How to get the number of tokens of the prompt?

[enzomich]

Hello everybody,
I'm new to llama-cpp-python, and I'm trying to understand how to get some statistics such as speed in tokens/second (prompt analysis and generation) after a call to llm.create_completion(...). I could measure the times by myself with time.perf_counter(), and at least in streaming mode get the number of generated tokens by counting the chunks yield()'d by create_completion(), but I don't know how to get the number of tokens of the prompt.
Inspecting the source code of Llama.py, I've found that in one case (at line 1718) _create_completion() yields a dict containing an item with key "usage" that is a dictionary containing the lengths of prompt_tokens[], completion_tokens[] and the sum of the two, but it's not clear to me how to have that yield() used and why usage is not present in the other calls to yield() (at least for the last chunk yielded: that's what the ollama-python library does, so it shouldn't be difficult).
Any idea of any other way of knowing the number of tokens of the prompt?

[gyanu2507]

Llama.tokenize is the straightforward count:

prompt_tokens = llm.tokenize(prompt.encode("utf-8"), add_bos=True)
n_prompt = len(prompt_tokens)

create_completion does put usage on the last chunk when you stream, same idea as the OpenAI API. You only see it if you keep iterating until the generator is exhausted and read that dict. Intermediate chunks omit it, which is why a loop that breaks on first content never observes usage.

Non-streaming:

out = llm.create_completion(prompt, max_tokens=64)
print(out["usage"]["prompt_tokens"], out["usage"]["completion_tokens"])

Speed: time around the call (or around the stream) and divide completion_tokens by the decode span. Prompt eval vs decode is not split out as two official counters in the Python binding. n_prompt from tokenize plus wall time until the first generated token is the usual stand-in for "prompt analysis."