Issues

• Export BenchmarkResult / UnifiedQuestion as EvalPort TestCase + ResultSet (portable eval format)

  supermemoryai/memorybench#89 · Filter by author adhabnr-ux opened on Aug 25, 2026

• Aggregated retrieval k is taken from the last question rather than the run, so reports can print "RETRIEVAL QUALITY (K=0)"

  supermemoryai/memorybench#78 · Filter by author rajarshidattapy opened on Aug 13, 2026
  1

• README quickstart fails at step 2: .env.example does not exist in the repo

  supermemoryai/memorybench#77 · Filter by author rajarshidattapy opened on Aug 13, 2026
  1

• The WebSocket server is dead code — the UI never connects, and no phase ever emits progress, contrary to the "real-time" claim

  supermemoryai/memorybench#76 · Filter by author rajarshidattapy opened on Aug 13, 2026

• Question order comes from readdirSync, so --limit N selects a filesystem-dependent subset and runs are not reproducible across machines

  supermemoryai/memorybench#75 · Filter by author rajarshidattapy opened on Aug 13, 2026

• BM25 index rebuilds its average document length on every chunk (O(n²) ingest) and double-counts on re-ingest

  supermemoryai/memorybench#74 · Filter by author rajarshidattapy opened on Aug 13, 2026

• A misspelled or unrecognised judge model silently becomes an OpenAI judge instead of failing

  supermemoryai/memorybench#73 · Filter by author rajarshidattapy opened on Aug 13, 2026

• Random sampling uses sort(() => Math.random() - 0.5), which is a biased shuffle — sampled runs are not representative

  supermemoryai/memorybench#72 · Filter by author rajarshidattapy opened on Aug 13, 2026

• CheckpointManager.save() is fire-and-forget — a failed write becomes an unhandled promise rejection and is never reported

  supermemoryai/memorybench#71 · Filter by author rajarshidattapy opened on Aug 13, 2026

• SupermemoryProvider.awaitIndexing polls forever with no timeout and treats API errors as "still pending"

  supermemoryai/memorybench#70 · Filter by author rajarshidattapy opened on Aug 13, 2026
  1

• chunkText in the RAG provider can loop forever and exhaust memory on text without whitespace

  supermemoryai/memorybench#69 · Filter by author rajarshidattapy opened on Aug 13, 2026

• Ingest loses the document IDs of already-ingested sessions when a later session in the same question fails

  supermemoryai/memorybench#68 · Filter by author rajarshidattapy opened on Aug 13, 2026

• Reported Recall@K, F1@K and NDCG are mathematically degenerate — they measure nothing beyond Hit@K

  supermemoryai/memorybench#67 · Filter by author rajarshidattapy opened on Aug 13, 2026

• RAG provider stores everything in process memory, so any resumed run silently searches an empty index and scores ~0%

  supermemoryai/memorybench#66 · Filter by author rajarshidattapy opened on Aug 13, 2026

• Supermemory provider ignores options.limit and returns 30 results while every other provider returns 10 — cross-provider comparisons are not like-for-like

  supermemoryai/memorybench#64 · Filter by author rajarshidattapy opened on Aug 13, 2026

• maxTokens is silently ignored on AI SDK v5 — every judge and answer call runs uncapped

  supermemoryai/memorybench#62 · Filter by author rajarshidattapy opened on Aug 13, 2026

• Path traversal in DELETE /api/runs/:runId and DELETE /api/compare/:compareId allows arbitrary directory deletion from any website

  supermemoryai/memorybench#60 · Filter by author rajarshidattapy opened on Aug 13, 2026

• fix: preserve LongMemEval question_date through checkpoint initialization

  supermemoryai/memorybench#51 · Filter by author 100yenadmin opened on Jul 19, 2026

• ## Automated Quality Checks

  supermemoryai/memorybench#31 · Filter by author x9x88pg2vd-prog opened on Mar 10, 2026

• > -

  supermemoryai/memorybench#27 · Filter by author x9x88pg2vd-prog opened on Feb 19, 2026

• > In scp, there might be a solution using the -o ProxyCommand option from SSH but I could not create a working example within some minutes.

  supermemoryai/memorybench#24 · Filter by author x9x88pg2vd-prog opened on Feb 14, 2026
  1

• Unable to login with oAuth

  supermemoryai/memorybench#22 · Filter by author PatelPratikkumar opened on Feb 5, 2026
  3