CLR: Claim-Level Reliability Assessment for Efficient Test-Time Reasoning

📄 Paper  |  🚀 Quick Start

Constructing a correct solution requires an entire reasoning path to hold. Refuting an incorrect one may require finding only a single decisive flaw.

CLR (Claim-Level Reliability Assessment) is a training-free test-time scaling framework used in our VibeThinker-3B technical report, that reallocates compute from sampling more solutions to falsifying decision-critical claims. It extracts key claims from reasoning traces, tests them for potential flaws, and uses the resulting reliability signals to weight solution voting.

By suppressing unreliable traces, CLR can let a reliable minority overturn an incorrect majority, improving reasoning accuracy and token efficiency without additional training or external verifiers.

Quick Start

pip install -r requirements.txt

Basic Usage

CLR supports any OpenAI-compatible API. Launch a vLLM server with --reasoning-parser so reasoning is split from the final answer, then run demo.py:

Step 1. Launch a vLLM server (example: gpt-oss):

vllm serve /path/to/gpt-oss-20b --port 8000 --reasoning-parser openai_gptoss

Step 2. Edit BASE_URL and MODEL in demo.py, then run:

python demo.py

run() Function

run() is the central interface for CLR. It takes a list of questions + ground truths and returns metrics.

Inputs

• questions / ground_truths: aligned lists of problem texts and answers
• base_url / model / api_key: the running OpenAI-compatible server
• workers: max concurrent requests (thread pool size)
• K: number of solution traces per flow (inner rollout)
• N: number of independent flows per question (outer rollout)
• num_claims: number of verification claims extracted per trace
• beta: exponent for weight (valid / num_claims) ** beta (defaults to num_claims)
• sampling_params: forwarded verbatim to the server — put any model-specific fields here
• output_path: if set, dump per-question details + metrics to JSON

sampling_params supports standard fields (temperature, top_p, top_k, max_tokens) plus model-specific extensions via chat_template_kwargs (e.g. reasoning_effort, enable_thinking).

Output Format

run() returns a dict with:

• Primary metrics
  • pass@1: mean correctness over N flow answers per question
  • pass@N: fraction of questions with ≥ 1 correct flow answer
  • cons@N: majority-vote accuracy over N flow answers (ties averaged)
• Stage-1 (raw trace) baselines
  • s1_pass@1: mean fraction of K traces that are correct
  • s1_pass@K: fraction of flows where ≥ 1 of K traces is correct
  • s1_cons@K: simple majority-vote accuracy over K traces
• Metadata
  • K, N, num_claims: run configuration
  • CLR_per_flow_avg_token: average output tokens per flow

Answer equivalence is decided by math_verify with an exact-string fast path.

Project Structure

.
├── demo.py              # Minimal runnable example
├── README.md
└── clr/
    ├── __init__.py      # Public API
    ├── pipeline.py      # Core two-stage pipeline + run() entry point
    ├── llm.py           # OpenAI-compatible chat client (worker-bounded concurrency)
    ├── prompt.py        # Stage-1/Stage-2 prompt assembly + response parsing
    └── evaluation.py    # math_verify equivalence, weighted/plain majority vote, metrics

Citation

@article{xu2026clr,
  title={Claim-Level Reliability Assessment for Efficient Test-Time Reasoning},
  author={Xu, Sen and Wang, Wei and Liu, Shixi and Min, Jixin and Dai, Yingwei and Yin, Zhibin and Chen, Yirong and Zhang, Junlin},
  journal={arXiv preprint arXiv:2608.11994},
  year={2026}
}