Observability for speculative decoding.
Sample sequences and inspect draft models.
View the accept/reject statistics for your draft model, and actually see the drafts. In my opinion this will give you really strong intuition for how good the model is.
pip install specspecs
To get started, try running:
python examples/qwen3_dflash.py "What is speculative decoding?"
Which will populate ./logs with an observed sequence and all its drafts.
To start the server:
specspecs-serve
This will default to monitoring ./logs as the parquet directory.
There are two calls you instrument your generation loop with: register_sequence (once at the beginning of each sequence) and update_sequence (once for each draft).
import specspecs
spec_logger = specspecs.SpecLogger("logs/", "Run Name", tokenizer)Register Sequence:
spec_logger.register_sequence(prompt_token_ids, sequence_id)If you don't pass a sequence_id it creates one and returns it.
Update Sequence:
Pass all of the relevant per-draft info here.
spec_logger.update_sequence(sequence_id, draft_tokens, acceptance_length, emitted_token, k, step)It writes sequences and drafts to parquet files, then analyzes them with DuckDB.
I'm working on turning it into a vLLM plugin, and moving the parquet writer to a separate thread. This should be enough to run it on production serving infra (sampling traffic, worst case).