Skip to content

Add seeded Text LLM sampling - #9451

Open
JPPhoto wants to merge 1 commit into
invoke-ai:mainfrom
JPPhoto:add-seed-to-text-llm-nodes
Open

Add seeded Text LLM sampling#9451
JPPhoto wants to merge 1 commit into
invoke-ai:mainfrom
JPPhoto:add-seed-to-text-llm-nodes

Conversation

@JPPhoto

@JPPhoto JPPhoto commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds controlled seeded sampling to the Text LLM and Text LLM with System Prompt Preset nodes.

  • Exposes a validated seed input and bumps both nodes to version 1.1.0.
  • Uses invocation-local generators to prevent concurrent requests from interfering with each other's RNG state.
  • Preserves varied Expand Prompt results by selecting a fresh seed per request.
  • Documents seed behavior and reproducibility boundaries.

Related Issues / Discussions

Discord.

QA Instructions

  1. Run:
    pytest -n auto --no-cov tests/backend/text_llm tests/app/invocations/test_text_llm_with_preset.py
  2. Confirm all tests pass.
  3. Run the same Text LLM node twice with identical inputs and seed; confirm identical output on the same platform.
  4. Change the seed; confirm sampling can produce different output.
  5. Run concurrent seeded CPU sampling; confirm each request matches its isolated sequence.

Merge Plan

Checklist

  • The PR has a short but descriptive title, suitable for a changelog
  • Tests added / updated (if applicable)
  • ❗Changes to a redux slice have a corresponding migration
  • Documentation added / updated (if applicable)
  • Updated What's New copy (if doing a release after this PR)

@github-actions github-actions Bot added api python PRs that change python files invocations PRs that change invocations backend PRs that change backend files frontend PRs that change frontend files python-tests PRs that change python tests docs PRs that change docs labels Aug 3, 2026
@JPPhoto
JPPhoto force-pushed the add-seed-to-text-llm-nodes branch 3 times, most recently from 98a5a14 to 380663b Compare August 3, 2026 17:52
@JPPhoto JPPhoto moved this to 6.14.1: Bug fixes to 6.14.0 in Invoke - Community Roadmap Aug 3, 2026
@JPPhoto JPPhoto added 6.14.0 and removed 6.14.1 labels Aug 4, 2026
@JPPhoto JPPhoto moved this from 6.14.1: Bug fixes to 6.14.0 to 6.14.x Theme: USER EXPERIENCE in Invoke - Community Roadmap Aug 4, 2026
@JPPhoto
JPPhoto force-pushed the add-seed-to-text-llm-nodes branch 2 times, most recently from c46eb2b to 07a5af3 Compare August 5, 2026 16:20
@JPPhoto
JPPhoto force-pushed the add-seed-to-text-llm-nodes branch from 07a5af3 to b294a7f Compare August 5, 2026 17:48
@Pfannkuchensack

Copy link
Copy Markdown
Collaborator

PR #9451 — Add seeded Text LLM sampling

  • The tests do not prove the feature. Two independent one-line mutations destroy seeded sampling and leave all 30 tests green:
    • removing the self._generators cache (invokeai/backend/text_llm_pipeline.py:40-44) so the RNG never advances — measured effect: torch.multinomial returns the same value 10/10 times, i.e. a real model emits one repeated token;
    • moving with _SeededMultinomialMode(seed): (:127) out of the worker closure — TorchFunctionMode is thread-local, so the mode never covers generate and the seed does nothing.
      Cause: the RNG tests call torch.multinomial by hand inside the mode and never go through run(); everything that does go through run() uses a MagicMock model that never samples. An end-to-end test with a tiny randomly-initialised causal LM costs 0.18 s for three CPU runs and kills both mutations.
  • Runtime cost: +19.4% on every text-LLM invocation. Measured on real Qwen2.5-1.5B-Instruct weights (RTX 4090, bf16, 128 tokens, median of 5, non-overlapping ranges): 19.81 → 23.66 ms/token, +3.85 ms/token, from 1931 intercepted torch calls per token. That is ~+1.16 s at the node default max_tokens=300 and ~+6 s at the 2048 cap, on a path where the user watches tokens stream. The mode wraps the whole forward pass, not just sampling — narrowing it to the sampling step (a custom LogitsProcessor with explicit Temperature/TopP warpers) would be O(1) per token. If the cost is accepted, put it in the PR description.
  • Saved workflows silently become deterministic. The 1.0.0 → 1.1.0 bump makes validateWorkflow auto-run updateNode, whose defaultsDeep fills the new seed with the template default 0. A workflow that previously varied on each run now returns byte-identical text forever, with no signal to the user and no way back except manually wiring a rand_int. (Masked inside one server process by the node cache; visible across restarts.) Add a vitest case pinning the updateNode outcome for a stored text_llm node.
  • func is torch.multinomial misses the method form. p.multinomial(...) bypasses seeding entirely (verified: reproducible via the function form, not via the method form). Unreachable today — all four transformers sampling paths were checked and ruled out — but the method form already exists elsewhere in the same transformers version, so an upstream refactor would ship broken reproducibility with a fully green suite.
  • Expand Prompt cannot reproduce anything. invokeai/app/api/routers/utilities.py:170 draws a fresh seed that is never returned, logged, or accepted back; ExpandPromptRequest/ExpandPromptResponse have no seed field. The route already accepts max_tokens and system_prompt from the client, so accepting an optional seed is zero-risk. Decide explicitly.
  • MPS path unverified (no Apple Silicon available). Static review of the .cpu() / .to(device) rewrite is clean and num_samples/replacement survive, but no test covers the branch and the per-token .cpu() hop forces a device sync. Someone on an M-series machine should run one expansion twice with the same seed before merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.14 Nice-to-Have 6.14.0 api backend PRs that change backend files docs PRs that change docs frontend PRs that change frontend files invocations PRs that change invocations python PRs that change python files python-tests PRs that change python tests

Projects

Status: 6.14.x Theme: USER EXPERIENCE

Development

Successfully merging this pull request may close these issues.

2 participants