Generate Text with Embedding as Input

[Sappique]

Is there a way to use an embedding as input to generate text?

I'm working on a research project for which I would like to obtain the embedding of a prompt using Llama.embed(), modify it in some way, and then use this modified embedding to generate further text.

As I understand it, the embedding is just the state of the last hidden layer of the network. So in theory this should be possible to do.

I'm currently using mistral-7b-openorca.Q4_K_M.

[gyanu2507]

Not through Llama.embed(). That API is "text in, pooled hidden out" for retrieval. It does not give you a hook to write a vector back into the first decode step.

Generation in llama.cpp starts from token ids. The model looks up token_embd, then runs transformer blocks. There is no first-class create_completion(inputs_embeds=...) in llama-cpp-python like Hugging Face.

What you can do for research:

1. Stay in Hugging Face (or MLX) with inputs_embeds= if you need to perturb the embedding and decode. GGUF Q4_K_M is a different stack.
2. In raw llama.cpp, people who inject custom embeddings do it by writing into the KV/input tensor before llama_decode. That is C API work. The Python Llama class does not wrap it.
3. mistral-7b-openorca.Q4_K_M embeddings are also in the quantized residual stream, so a vector you got from embed() is not a drop-in replacement for the fp16 embedding table row anyway.

If the project is "perturb the prompt embedding and generate," use a non-GGUF path. If it is "GGUF only," you are looking at a small C extension, not a flag on create_completion.