Skip to content
Merged

Dev #60

Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,9 @@ OLI_LOGS_DIR=""
OLI_TRUNCATION_SMALL=4000
OLI_TRUNCATION_LARGE=100000

# Max characters accepted by a single write_file call
OLI_MAX_WRITE_CHARS=100000

# Dry-run: preview destructive actions without executing
OLI_DRY_RUN=false

Expand Down
15 changes: 8 additions & 7 deletions AGENTS.md

Large diffs are not rendered by default.

6 changes: 5 additions & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,8 @@ ENV PYTHONUNBUFFERED=1 \

# Tell HuggingFace where to cache models inside the image
ENV TRANSFORMERS_CACHE=/app/.cache/huggingface \
HF_HOME=/app/.cache/huggingface
HF_HOME=/app/.cache/huggingface \
TIKTOKEN_CACHE_DIR=/app/.cache/tiktoken

# Install system dependencies (single RUN layer to keep image smaller)
RUN apt-get update && apt-get install -y \
Expand Down Expand Up @@ -43,6 +44,9 @@ RUN pip install --no-cache-dir transformers>=4.40 accelerate>=0.2
COPY . .
RUN pip install --no-cache-dir .

# Pre-cache tiktoken BPE files so token counting works without runtime network.
RUN python -c "import tiktoken; [tiktoken.get_encoding(n) for n in ('cl100k_base', 'o200k_base')]"

EXPOSE 9734

# Launch API Server
Expand Down
23 changes: 12 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,10 +29,11 @@ A lightweight, terminal-based AI agent harness for local-first, vendor-agnostic

- **Declarative sub-agent pooling (optional)** — with `--use-pool`, the root agent can fan tasks out concurrently to vendor-agnostic sub-agents defined in an optional [agents.yaml](agents.yaml) file via a `dispatch` tool. Each pool entry binds a model _and_ a backend, so dispatch decisions are also compute-location decisions — a frontier model can plan while sensitive work stays on a local model, or a local root can fan out to faster remote SLMs for latency-sensitive tool calls.
- **Agent profiles** — drop-in system prompts with permission manifests, base-profile inheritance, and auto-generated profiles via `/profile create`. Bundled profiles: `default`, `coder`, `reviewer`, `editor`, `writer`, `planner`, `researcher`, `analyst`.
- **Rich built-in tool set** — file ops, shell access (including PHP and read-only Docker inspection), web search/fetch, Wikipedia/GitHub/arXiv search, task tracking, reasoning scratchpad, notebook, interactive user-question modals, and more. Sandbox-locked with shell allowlists, SSRF protection, and sensitive-file gating.
- **Rich built-in tool set** — file ops, shell access, web search/fetch, Wikipedia/GitHub/arXiv search, task tracking, reasoning scratchpad, notebook, interactive user-question modals, and more. Sandbox-locked with shell allowlists, SSRF protection, and sensitive-file gating.
- **Permission system** — write operations and sensitive reads require user approval. Session grants, workspace scoping, and profile-level allow/deny lists.
- **OpenAI-compatible API server** — run the same agent harness behind `/v1/models` and `/v1/chat/completions` (streaming + non-streaming) so any workflow that speaks the OpenAI wire protocol (the `openai` Python SDK, curl, or plain REST) can drive the agent.
- **Voice mode (optional, experimental)** — `/voice` toggles a hands-free mic → STT → LLM → TTS loop (faster-whisper, Piper TTS, WebRTC VAD) for the TUI. Fully local; requires the `voice` extras and a downloaded Piper model.
- **Live context counter** — the TUI status bar shows a tiktoken-backed context-size snapshot (`ctx N tok`) that updates on submit, after thinking blocks, after each tool result, and at the end of every turn, so you can see how close the conversation is to the model's window before the next request.
- **Multi-backend support** — Ollama, OpenAI, HuggingFace (remote or local), and Transformers (local GPU/CPU). Switch at runtime.
- **MCP integration** — add stdio or HTTP MCP servers at runtime for custom tools. HTTP servers can authenticate with a static API token or OAuth 2.1 (set `OLI_USE_OAUTH=true`, or toggle it under `/config`).

Expand Down Expand Up @@ -153,16 +154,16 @@ Type `/voice` again (or `Ctrl+Q` to quit the app) to exit voice mode — the mic

## Documentation

| Document | Contents |
| ---------------------------------------------- | ------------------------------------------------------------------------------- |
| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | Architectural overview of the application, permission, configurations, and more |
| [docs/API_SERVER.md](docs/API_SERVER.md) | Running the `oli-server` REST/WebSocket API, endpoints, config, and examples |
| [docs/CONFIGURE.md](docs/CONFIGURE.md) | Full configuration reference, settings precedence, sessions, workspace |
| [docs/TOOLS.md](docs/TOOLS.md) | Built-in tools, permission system, security, dry-run/offline modes, truncation |
| [docs/BACKENDS.md](docs/BACKENDS.md) | Backend setup (Ollama, OpenAI, HuggingFace, Transformers), model tier switching |
| [docs/PROFILES.md](docs/PROFILES.md) | Profile structure, manifests, built-in profiles, creating and loading profiles |
| [docs/AGENT_POOLING.md](docs/AGENT_POOLING.md) | Agent pooling configuration, parsing, and usage |
| [docs/SECURITY.md](docs/SECURITY.md) | Security precedence and settings |
| Document | Contents |
| ---------------------------------------------- | ----------------------------------------------------------------------------------------- |
| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | Architectural overview of the application, permission, configurations, and more |
| [docs/API_SERVER.md](docs/API_SERVER.md) | Running the `oli-server` REST/WebSocket API, endpoints, config, and examples |
| [docs/CONFIGURE.md](docs/CONFIGURE.md) | Full configuration reference, settings precedence, sessions, workspace |
| [docs/TOOLS.md](docs/TOOLS.md) | Built-in tools, permission system, security, dry-run/offline modes, truncation, write cap |
| [docs/BACKENDS.md](docs/BACKENDS.md) | Backend setup (Ollama, OpenAI, HuggingFace, Transformers), model tier switching |
| [docs/PROFILES.md](docs/PROFILES.md) | Profile structure, manifests, built-in profiles, creating and loading profiles |
| [docs/AGENT_POOLING.md](docs/AGENT_POOLING.md) | Agent pooling configuration, parsing, and usage |
| [docs/SECURITY.md](docs/SECURITY.md) | Security precedence and settings |

## Docker

Expand Down
10 changes: 6 additions & 4 deletions docs/API_SERVER.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,15 +166,16 @@ Non-streaming response:
}
],
"usage": {
"prompt_tokens": 0,
"prompt_tokens": 18,
"completion_tokens": 42,
"total_tokens": 42
"total_tokens": 60
}
}
```

`usage` is an approximation (the agent tool loop does not surface exact prompt
counts); `completion_tokens` is a `~chars/4` estimate.
`usage` is the summed per-call usage reported by the backend for the whole
agent run (prompt + completion across every tool-loop round). When the backend
reports no counts, it falls back to a completion-only `~chars/4` estimate.

**Errors** are returned in OpenAI format with HTTP 500:

Expand Down Expand Up @@ -261,6 +262,7 @@ inbound message:
| `tool_call_executing` | `{"name": "...", "parameters": {...}}` |
| `tool_call_result` | `{"name": "...", "result": "..."}` |
| `assistant_response` | `{"content": "..."}` — final assembled assistant text |
| `token_count` | `{"prompt_tokens", "completion_tokens", "total_tokens", "estimated", "phase"}` — live context snapshot (`phase` is `prompt`/`thinking`/`round`/`tool_result`/`final`) |
| `usage` | `{"prompt_tokens": ..., "completion_tokens": ..., ...}` |
| `error` | `{"message": "..."}` |
| `done` | `{"full_text": "..."}` — run finished |
Expand Down
39 changes: 39 additions & 0 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,45 @@ flowchart TD

---

## Live Token Accounting

The agent emits a `TokenCount` snapshot (not a delta) at every point where the
request context changes, so a UI can show the current context size without
waiting for the run to finish. `oli_bot/tokens.py` uses `tiktoken` when it can
resolve an encoding for the active model, falls back to a generic BPE encoding
for unknown/local model names, and finally to the `~chars/4` heuristic; the
`estimated` flag drives the `~` prefix in the TUI.

```mermaid
sequenceDiagram
participant U as User
participant T as TUI
participant A as Agent
participant B as Backend
U->>T: submit message
T->>A: process(messages)
A-->>T: TokenCount(phase="prompt")
A->>B: stream_generate()
B-->>A: ThinkingChunk*
B-->>A: TextChunk*
A-->>T: TokenCount(phase="thinking")
B-->>A: ToolCallChunk*
A-->>T: ToolCallExecuting / ToolCallResult
A-->>T: TokenCount(phase="tool_result")
B-->>A: UsageChunk (exact round usage)
A-->>T: TokenCount(phase="round")
A->>B: next round / final stream
A-->>T: TokenCount(phase="final")
A-->>T: UsageEvent (cumulative run) + Done
```

`UsageEvent` remains the cumulative billed total for the run (persisted in
session JSON and returned by the REST API); `TokenCount` is the live context
size (rendered in the TUI status bar and relayed as `token_count` WebSocket
frames).

---

## Voice Mode Loop

`/voice` toggles a background `@work(exclusive=False)` worker (`_run_voice_loop`)
Expand Down
6 changes: 6 additions & 0 deletions docs/BACKENDS.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,12 @@ oli supports four model backends, selected via the `backend` setting (`OLI_BACKE
| `huggingface` | HuggingFace Inference API (remote) or TGI/vLLM (local) |
| `transformers` | Local model inference via HuggingFace `transformers` |

## Output-token cap

Every backend's `generate()` and `stream_generate()` accept a `max_tokens` cap. `Agent` passes the live `config.max_tokens` (set via `OLI_MAX_TOKENS`, `settings.json`, or `/config`) on each streaming round, so the value actually bounds a single generation — important for large tool calls like `write_file`, whose `content` argument is generated as output tokens. When `max_tokens` is omitted (`None`), Ollama/Transformers fall back to the import-time `MAX_TOKENS` default and OpenAI-compatible APIs use the provider default.

If generation is cut off mid-tool-call, the `arguments` JSON is truncated. Backends parse it leniently (`tool_call_from_raw`) and set `ToolCall.parse_error`; the agent returns that error as the tool result so the model can retry with a smaller payload instead of the run failing.

## Ollama

Connects to a local Ollama server. Default URL: `http://localhost:11434`.
Expand Down
3 changes: 2 additions & 1 deletion docs/CONFIGURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ If `~/.config/oli/settings.json` does not exist, it is auto-created on first loa
| `use_agent_pool` | `false` | `OLI_USE_AGENT_POOL` | Enable agent pooling (`--use-pool`); root agent can dispatch tasks to agents.yaml sub-agents |
| `agent_pool_size` | `5` | `OLI_AGENT_POOL_SIZE` | Max agents allowed per pool in agents.yaml |
| `agents_yaml` | `""` | `OLI_AGENTS_YAML` | Explicit path to agents.yaml for pooling (empty = auto-locate: package dir, repo root, `~/.config/oli`) |
| `max_tokens` | `2048` | `OLI_MAX_TOKENS` | Max tokens per generation |
| `max_tokens` | `2048` | `OLI_MAX_TOKENS` | Max tokens per generation, applied to streaming and non-streaming calls alike |
| `temperature` | `0.7` | `OLI_TEMPERATURE` | Model temperature |
| `max_retries` | `3` | `OLI_MAX_RETRIES` | Max retries on API failure |
| `retry_delay` | `1.0` | `OLI_RETRY_DELAY` | Delay between retries (seconds) |
Expand All @@ -52,6 +52,7 @@ If `~/.config/oli/settings.json` does not exist, it is auto-created on first loa
| `model_filters` | `""` | `OLI_MODEL_FILTERS` | Comma-separated model name exclusion filters |
| `truncation_max_chars_small` | `4000` | `OLI_TRUNCATION_SMALL` | Max chars for tool results under the small model tier |
| `truncation_max_chars_large` | `100000` | `OLI_TRUNCATION_LARGE` | Max chars for tool results under the large model tier |
| `max_write_chars` | `100000` | `OLI_MAX_WRITE_CHARS` | Max characters accepted by the `write_file` tool (also advertised in its description) |
| `dry_run` | `false` | `OLI_DRY_RUN` | Preview destructive actions without executing |
| `offline_mode` | `true` | `OLI_OFFLINE_MODE` | Block network access for web tools and MCP servers |
| `use_oauth` | `false` | `OLI_USE_OAUTH` | Use OAuth 2.1 (PKCE + dynamic client registration) for HTTP MCP servers added without an API token |
Expand Down
6 changes: 6 additions & 0 deletions docs/TOOLS.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,3 +126,9 @@ Tool results are automatically truncated to conserve context window:
- **Large tier** -- 100,000 characters (configurable via `OLI_TRUNCATION_LARGE`)

Truncation preserves sentence and paragraph boundaries when possible, appending a `[... truncated: N characters remaining]` notice.

## Write cap

`write_file` rejects a single write larger than `max_write_chars` (default 100,000 characters; configurable via `OLI_MAX_WRITE_CHARS` or `/config`). The limit is also advertised in the tool's description so the model can self-limit.

Note this is a *post-generation* guardrail: it fires only after the model has emitted the full `content` argument. The limit that actually truncates large writes is the per-round output-token budget (`max_tokens`). When generation hits that ceiling mid-call, the tool-call JSON is unterminated; the backend parses it leniently and the agent feeds back a retryable `parse_error` result ("...truncated JSON... split large content across multiple writes...") instead of aborting the run.
Loading
Loading