diff --git a/AGENTS.md b/AGENTS.md index 7a6be7c..f1d0355 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -15,6 +15,11 @@ the matching provider, and passes everything else through to GitHub unchanged. T interception. It never modifies the user's shell profile, and it writes credentials to disk only through the opt-in key store (`copilot-byok keys set` or `/byok`), never on its own initiative. +`--native-byok` publishes the same models through the CLI's supported `providers.json` registry +instead: discovery runs as usual, a private temporary registry is written, `COPILOT_PROVIDERS_CONFIG` +points at it for the child process, and it is deleted when the session ends. That file contains the +provider credentials for the session, which is why the mode stays opt-in. + `COPILOT_API_URL` is an internal Copilot CLI variable, absent from `copilot help environment`. Treat it as the project's main external risk: if a CLI release changes it, the router breaks and `--native` / `--legacy` are the fallbacks. Do not assume other undocumented variables exist — @@ -43,19 +48,27 @@ There is no VS Code extension, no bundler, no compiled output, and no hosted dep ```text bin/copilot-byok.mjs Executable entry point; only wires main() to process.exitCode src/args.mjs Pure argument parser; no I/O -src/cli.mjs Orchestration: router mode (default), legacy mode, native mode +src/argv-config.mjs Pure --config extraction for subcommands +src/cli.mjs Orchestration: router mode (default), native registry mode, legacy mode src/router.mjs The local proxy: model injection, provider routing, passthrough src/catalog.mjs Turns providers into the model entries Copilot expects from GET /models src/launcher.mjs Starts the router on an ephemeral port and spawns Copilot against it -src/upstream.mjs Resolves the Copilot API tier (individual / business / enterprise) +src/upstream.mjs Resolves the Copilot API tier (individual / business / enterprise, data residency) src/discovery.mjs Asks each provider for its live model list; falls back to the remembered list +src/model-cache.mjs Remembers the last model list each provider returned +src/models-auth.mjs Scopes provider credentials to the catalog origin (pure) src/config.mjs Provider config loading and validation; loads providers.default.json src/providers.default.json Generated catalog — never edit by hand, run npm run catalog:update +src/native-registry.mjs Pure translation of providers into the CLI's providers.json registry +src/registry-file.mjs Reads/writes the native registry; private temp file for one session src/keystore.mjs Optional on-disk key store (0600 / owner-only ACL) src/keys-command.mjs `copilot-byok keys ...` +src/doctor.mjs `copilot-byok doctor`; installation diagnostics +src/models-command.mjs `copilot-byok models`; discovery output without a session src/extension-install.mjs `copilot-byok extension ...`; rewrites the template's package root src/shim-install.mjs `copilot-byok shim ...`; writes the per-platform `copilot` shim src/session-log.mjs Router diagnostics to a file, never over Copilot's TUI +src/progress.mjs Startup activity line (TTY only) src/copilot-bin.mjs Resolves the Copilot executable, skipping the stale VS Code shim src/model-ranking.mjs Pure ranking of provider model catalogs (legacy mode) src/process-env.mjs Strips stale/secret variables from the child environment @@ -68,8 +81,8 @@ schemas/providers.schema.json Published JSON Schema for providers.json examples/providers.example.json Documented example configuration docs/provider-verification.md Evidence matrix for provider claims docs/providers.md Provider table, keys, model discovery, the /byok command -docs/configuration.md Custom providers.json, commands, environment variables, shim, reasoning effort -docs/how-it-works.md Router design, COPILOT_API_URL dependency, picker behavior +docs/configuration.md Custom providers.json, native registry mode, commands, environment variables, shim, reasoning effort +docs/how-it-works.md Router design, native registry mode, COPILOT_API_URL dependency, picker behavior docs/troubleshooting.md Failure modes and diagnostics ``` @@ -96,8 +109,9 @@ There is no `dev`, `build`, `format`, or `type-check` script. Do not invent one - Two-space indentation, single quotes, semicolons, trailing commas in multiline literals. - Named exports only; no default exports outside `bin/`. -- Modules stay small and single-purpose. `args.mjs`, `model-ranking.mjs`, `provider-env.mjs`, and - `process-env.mjs` must remain **pure** (no file, network, or process access) so they stay trivially testable. +- Modules stay small and single-purpose. `args.mjs`, `argv-config.mjs`, `catalog.mjs`, + `model-ranking.mjs`, `models-auth.mjs`, `native-registry.mjs`, `process-env.mjs`, and + `provider-env.mjs` must remain **pure** (no file, network, or process access) so they stay trivially testable. - All I/O is injected through the `io` object (`{ stdin, stdout, stderr, env }`) passed to `main()`. Never read `process.env` or write to `process.stdout` directly from `src/` outside the documented defaults. - Errors are thrown as `Error` with actionable messages; `bin/copilot-byok.mjs` prints them prefixed with @@ -130,6 +144,12 @@ This project handles third-party provider API keys. Every change must preserve t - Catalog requests must keep a bounded timeout (`AbortSignal.timeout`, default 10 s, configurable 10–300000 ms). - No `.env` file is used or expected; there is no `.env.example`. Do not add one. - Never commit real keys, tokens, or a `providers.json` containing credentials. +- `--native-byok` writes provider credentials to a temporary registry because the CLI's supported + path requires them inline. That file must keep `0600` (owner-only ACL on Windows), live in the + system temp directory, and be removed on exit, including on failure. It must never be written to + the repository or to the user's Copilot config directory. +- `--legacy` must keep isolating itself from a user `providers.json` when one is present: that file + takes precedence over `COPILOT_PROVIDER_*`, and silently using it would break the mode. ## Adding or changing a built-in provider @@ -173,7 +193,8 @@ that had already been verified working. ## Compatibility and anti-breaking-change rules - The CLI contract is public: existing flags, aliases, provider ids, and the `--dry-run` JSON shape must keep - working. Add options; do not rename or remove them. + working. Add options; do not rename or remove them. `--native-byok`, `doctor`, and `models` are additive; + the existing `--dry-run` shapes keep their keys, and the new native mode reports `"mode": "native-byok"`. - Unrecognized arguments and everything after `--` are forwarded verbatim to Copilot CLI. Do not start consuming new argument names without documenting the change in the README, the relevant `docs/` page, and `CHANGELOG.md`. diff --git a/CHANGELOG.md b/CHANGELOG.md index a8342ae..db638e8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,41 @@ All notable changes to this project are documented in this file. +## [Unreleased] + +### Added + +- **`--native-byok`** publishes the configured models through the Copilot CLI's supported + `providers.json` registry instead of the local router. Discovery runs as usual, a private + temporary registry is written and passed via `COPILOT_PROVIDERS_CONFIG`, and the file is removed + when the session ends. The mode needs a CLI that reads `providers.json` (verified on 1.0.105). + Model ids in `/model` are `provider/model`; a `--model byok-...` value is translated. +- **`copilot-byok doctor`** diagnoses the installation in one pass: Node, the Copilot binary and + its version, the provider config, credentials, key store, native registry, cache and log paths, + shim and extension. `--json` for scripts, `--online` to query keyed providers. +- **`copilot-byok models [provider]`** lists what each provider currently serves without starting + a session. `--json` and `--offline` are supported. +- Data residency: when `COPILOT_GH_HOST` or `GH_HOST` names a `*.ghe.com` tenant, the router derives + `copilot-api..ghe.com` instead of the public Copilot hosts. +- Legacy mode now passes `COPILOT_PROVIDER_HEADERS` (from `headers`) and + `COPILOT_PROVIDER_API_KEY_COMMAND` (from `apiKeyCommand`), and sets `COPILOT_PROVIDER_MODEL_ID` + when `catalogModelId` is configured. + +### Changed + +- The reasoning-effort selector is declared for models that report reasoning support, instead of + being opt-in: current CLIs forward the level to BYOK providers. It can still be hidden with + `reasoningEffortPicker: false`. Docs and verification evidence were updated for CLI 1.0.105. +- Legacy mode detects a user `providers.json` that would take precedence over `COPILOT_PROVIDER_*`, + points the child at an empty temporary registry, and says so on stderr. + +### Fixed + +- A reasoning-effort level sent by the CLI and rejected by the provider no longer loops: the retry + now carries the stepped-down level instead of replaying the original request. The close listener + is also registered once per client request instead of once per retry. +- `docs/troubleshooting.md` had a broken router log path. + ## [2.0.1] - 2026-10-21 ### Security diff --git a/README.md b/README.md index 7dfcd68..7886034 100644 --- a/README.md +++ b/README.md @@ -33,6 +33,9 @@ copilot-byok --list-providers # 3. Start Copilot copilot-byok + +# 4. Or publish the same models through the CLI's own BYOK registry +copilot-byok --native-byok ``` Open `/model` inside the session and pick one of your models. Arguments after `--` go to Copilot @@ -49,11 +52,15 @@ copilot-byok -- -p "fix the failing test" --allow-all-tools - Use as many providers per session as you configure. - GitHub models stay available while BYOK is active, and models can be switched with `/model` without restarting. +- Two ways to publish them: the default local router, or `--native-byok`, which feeds the CLI's + own supported `providers.json` registry from the same discovery. The native path has no local + proxy; it writes a private temporary registry for the session and deletes it on exit. - Live discovery: each configured provider is asked what it serves at startup, with a remembered list as the fallback when a request fails. - Zero AI credits for BYOK turns; you pay your provider directly. -- Credentials come from environment variables or an opt-in local key store, and never reach the - Copilot process in router mode. +- Credentials come from environment variables or an opt-in local key store. In router mode they + never reach the Copilot process; in native mode they are written only to a temporary file with + owner-only permissions and removed when the session ends. - No TLS interception: the router binds to loopback and exits with the session. - Windows, macOS and Linux are supported with Node.js only, with no build step. @@ -61,7 +68,8 @@ copilot-byok -- -p "fix the failing test" --allow-all-tools - Node.js 22.13 or newer. - GitHub Copilot CLI 1.0.79 or newer, available as `copilot` on `PATH` (set `COPILOT_BIN` to - override). + override). The current release line, 1.0.105, is the version this project is verified against; + `--native-byok` needs a CLI that reads `providers.json`, which 1.0.105 does. - A GitHub Copilot account, on any plan including free. ## Configuration @@ -78,6 +86,7 @@ the custom `providers.json` format, the command list and single-provider mode ar | `COPILOT_BYOK_PROGRESS=off` | Turns off the startup progress line | Progress line on | | `COPILOT_BYOK_DEBUG=1` | Streams router events live instead of only to the log | Log only | | `COPILOT_BYOK_PROVIDER_TIMEOUT_MS` | Idle timeout on a provider request | 10 minutes | +| `COPILOT_GH_HOST`, `GH_HOST` | Data residency tenant host; the router derives `copilot-api..ghe.com` from it | Public GitHub hosts | All 28 built-in providers, the environment variable for each key, and the model-discovery behavior are documented in the [providers guide](docs/providers.md). @@ -85,18 +94,21 @@ behavior are documented in the [providers guide](docs/providers.md). ## Privacy and security No TLS is intercepted, no certificate authority is installed, the router listens on loopback only -and exits with the session, provider keys never reach Copilot or GitHub, and your GitHub token -passes through the router unlogged because it has to. Provider keys come from environment -variables or the opt-in key store; keys in the store are plain text on disk protected by file -permissions, so environment variables remain the safer default. `SECURITY.md` covers the full -model, including what the undocumented `COPILOT_API_URL` dependency means for you. +and exits with the session, provider keys never reach GitHub, and your GitHub token passes through +the router unlogged because it has to. In router mode the keys also stay out of the Copilot +process; `--native-byok` writes them to a temporary registry with owner-only permissions for the +duration of the session because that is how the CLI's supported BYOK registry works, and removes it +on exit. Provider keys come from environment variables or the opt-in key store; keys in the store +are plain text on disk protected by file permissions, so environment variables remain the safer +default. `SECURITY.md` covers the full model, including what the undocumented `COPILOT_API_URL` +dependency means for you. ## Troubleshooting ### No BYOK models appear in `/model` No provider is active. Run `copilot-byok --list-providers` to see which provider is unlocked by -which environment variable or stored key. +which environment variable or stored key, and `copilot-byok doctor` for a full diagnosis. ### `Unknown BYOK model` @@ -109,7 +121,9 @@ The key for that provider is wrong or expired. This is not a GitHub authenticati ### Everything is a GitHub model again `COPILOT_API_URL` may no longer work on your CLI version. Run `copilot-byok --native` for a stock -session and check the router notes in [how it works](docs/how-it-works.md). +session, `copilot-byok --native-byok` to use the CLI's supported registry instead of the router, and +check the router notes in [how it works](docs/how-it-works.md). `copilot-byok doctor` reports the CLI +version it found. ### Startup pauses before the picker opens diff --git a/SECURITY.md b/SECURITY.md index 292dfe2..0a8e212 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -29,7 +29,7 @@ Out of scope: ## How credentials are handled -**Provider keys.** They are read from environment variables or, if you opt in, from the local key store. In router mode the keys stay in the router's own process: they are attached to that provider's outgoing request and stripped from the environment handed to Copilot CLI, so the agent process never sees them. In `--legacy` mode the selected credential is passed to Copilot itself, because that mode uses Copilot's built-in BYOK variables. +**Provider keys.** They are read from environment variables or, if you opt in, from the local key store. In router mode the keys stay in the router's own process: they are attached to that provider's outgoing request and stripped from the environment handed to Copilot CLI, so the agent process never sees them. In `--native-byok` mode the CLI reads them from a temporary `providers.json` written by this tool: the file is created in the system temp directory with `0600` and an owner-only ACL on Windows, passed to the CLI through `COPILOT_PROVIDERS_CONFIG`, and deleted when the session ends. In `--legacy` mode the selected credential is passed to Copilot itself, because that mode uses Copilot's built-in BYOK variables. **The local key store.** `copilot-byok keys set` and the `/byok` command write to `~/.config/copilot-byok/keys.json` (`%APPDATA%\copilot-byok\keys.json` on Windows), created with mode `0600` and an owner-only ACL on Windows. **Keys stored there are plain text on disk**, protected by file permissions alone; if that does not fit your threat model, use environment variables, which always take precedence and are the default. The store is never created unless you use it. Keys are only accepted from an interactive prompt, never from command-line arguments or pipes, so they do not reach shell history or process listings. @@ -45,7 +45,7 @@ Router mode relies on `COPILOT_API_URL`, an environment variable that Copilot CL - Nothing is intercepted at the TLS level, no certificate authority is installed, and no system trust store is modified. Copilot talks in the clear to loopback, and the router opens its own ordinary HTTPS connection to GitHub. - Requests that are not for a BYOK model are forwarded to GitHub unchanged. -- Because the variable is undocumented, a Copilot CLI update may change or remove it. If that happens the router stops receiving traffic; `--native` and `--legacy` remain available. +- Because the variable is undocumented, a Copilot CLI update may change or remove it. If that happens the router stops receiving traffic; `--native-byok` uses the CLI's supported `providers.json` registry instead, and `--native` and `--legacy` also remain available. ## The shim diff --git a/docs/configuration.md b/docs/configuration.md index 5ad879f..c94eddb 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -32,11 +32,19 @@ use `--list-models` to discover what a provider offers. **API keys are never written to the config** — only the names of the environment variables that hold them. Inline `apiKey` fields are rejected. +`apiKeyCommand` is the exception to environment variables: it names a command the Copilot CLI runs +itself to obtain a rotating credential, passed through as `COPILOT_PROVIDER_API_KEY_COMMAND` in +legacy mode. `headers` adds gateway headers to the provider requests, and to the native registry in +`--native-byok` mode. + ## Commands ```sh copilot-byok # Copilot with your models in the picker copilot-byok --list-providers # which providers are active, and what unlocks the rest +copilot-byok doctor # diagnose the installation without starting Copilot +copilot-byok models [provider] # what each provider serves right now +copilot-byok --native-byok # supported BYOK registry instead of the local router copilot-byok --native # stock Copilot, no router copilot-byok --dry-run # show what would start, then exit copilot-byok -- # pass anything through to Copilot @@ -55,6 +63,29 @@ copilot-byok shim status # which mechanism is actually in effect copilot-byok shim uninstall ``` +`doctor` exits non-zero only when a check fails; `models` accepts `--json` and `--offline` and +prints what discovery found without opening a session. + +## Native registry mode + +`--native-byok` publishes models through the CLI's own `providers.json` registry instead of the +local router. The session starts with `COPILOT_PROVIDERS_CONFIG` pointing at a temporary file +generated from the same live discovery; the file is removed when Copilot exits. + +Use it when you want the supported BYOK path, or when `COPILOT_API_URL` stops working. Model ids in +`/model` are `provider/model`, for example `openai/gpt-5.5`; passing a picker id from this project +(`--model byok-openai-gpt-5-5`) still works because it is translated before launch. + +Requirements and trade-offs: + +- a Copilot CLI that reads `providers.json` (verified on 1.0.105); +- provider keys are written to that temporary file for the session, unlike router mode where they + never reach the Copilot process; +- the router's reasoning-effort step-down does not apply because the CLI talks to the provider + directly; the levels a model advertises are what the picker offers; +- a `~/.copilot/providers.json` you maintain yourself is ignored for that session, since the + generated file replaces it. + ## Environment variables Everything has a working default; these are for when you need to change one. @@ -71,6 +102,7 @@ Everything has a working default; these are for when you need to change one. | `COPILOT_BYOK_ASCII=1` | ASCII spinner, for consoles that cannot render Braille | Unicode progress | | `COPILOT_BYOK_DEBUG=1` | Stream router events live instead of only to the log | Events go to the log only | | `COPILOT_BYOK_PROVIDER_TIMEOUT_MS` | Idle timeout on a provider request | 10 minutes | +| `COPILOT_GH_HOST`, `GH_HOST` | Data residency tenant host; the router derives `copilot-api..ghe.com` from it | Public GitHub hosts | ## Keeping the catalog honest @@ -93,6 +125,10 @@ mode, which is exactly what you want for `--offline` work against a local model: copilot-byok --legacy --provider ollama --model qwen3-coder --offline ``` +If `~/.copilot/providers.json` (or the file named by `COPILOT_PROVIDERS_CONFIG`) contains providers +or models, the CLI ignores `COPILOT_PROVIDER_*`. Legacy mode detects that case, points the child at +an empty temporary registry, and says so on stderr, so the mode keeps its documented meaning. + ## The shim `copilot-byok shim install` makes plain `copilot` use the router. It does two things, because on @@ -117,40 +153,32 @@ both are handled. `copilot-byok shim status` says which mechanism is actually in ## Reasoning effort -Copilot's own `--effort` flag and its effort selector do not reach BYOK providers. Verified against -a provider that records what it receives: the payloads produced with `low` and with `high` are -byte-identical, in both wire formats. This is a known CLI bug rather than a limitation of BYOK -itself — see [copilot-cli#4012](https://github.com/github/copilot-cli/issues/4012) (still open, -same symptom on GLM-5.2), [#3119](https://github.com/github/copilot-cli/issues/3119) and -[#3135](https://github.com/github/copilot-cli/issues/3135), which records that the level *was* -being sent in 1.0.41. +Current Copilot CLIs forward the level you pick to BYOK providers. Verified on 1.0.105, both through +the router and with the CLI's own provider variables. This was not true on 1.0.79, where the payloads +with `low` and with `high` were byte-identical: that was the CLI bug tracked in +[copilot-cli#4012](https://github.com/github/copilot-cli/issues/4012), resolved upstream. + +The effort selector is therefore declared for models that report reasoning support, and can be +hidden per model or per provider with `reasoningEffortPicker: false`. -Until it is fixed, set the level per model and the router applies it: +You can still force a level from the config, useful on older CLIs or when a model should always run +at a fixed effort: ```json { "model": "deepseek-reasoner", "reasoningEffort": "high" } ``` `reasoningEffort` also works at provider level as a default. Values are the ones Copilot uses: -`none`, `minimal`, `low`, `medium`, `high`, `xhigh`, `max`. +`none`, `minimal`, `low`, `medium`, `high`, `xhigh`, `max`. `reasoningEffortLevels` narrows what +the picker offers for a model. **Levels differ per model and providers do not fall back.** GLM-5.2 accepts `max` but answers `400` -to `xhigh`; an unsupported level would otherwise break every request to that model. The router -steps down one level at a time until one is accepted, records it, and remembers it for the session: +to `xhigh`. In router mode the router steps down one level at a time, applies the same ladder to a +level the CLI sent, records the rejection for the session, and logs the adjustment: ```text notice: chutes: zai-org/GLM-5.2-TEE rejected reasoning effort "xhigh", retrying with "high" ``` -### Ready for the fix - -Nothing here needs rewriting when the CLI starts forwarding the level again: the router already -prefers an incoming `reasoning_effort` over the configured one, so your pick in the picker will -simply take effect. The only step will be to turn the selector on: - -```json -{ "model": "deepseek-reasoner", "reasoningEffortPicker": true } -``` - -It ships off, because a selector that accepts a choice and changes nothing is worse than no -selector. `reasoningEffortLevels` narrows what it offers, for models that do not take every level. +In `--native-byok` mode there is no router in the path: the CLI's request reaches the provider +unchanged, so advertise only levels the model accepts, with `reasoningEffortLevels`. diff --git a/docs/how-it-works.md b/docs/how-it-works.md index ce34c98..f1ab270 100644 --- a/docs/how-it-works.md +++ b/docs/how-it-works.md @@ -31,21 +31,31 @@ Model changed to: byok-anthropic-claude-sonnet-4-6 for this session ## Why this exists -Copilot CLI does support BYOK out of the box, through `COPILOT_PROVIDER_BASE_URL`. That path has -three limits this project removes: - -| | Built-in BYOK | Copilot BYOK Switcher | -| --- | --- | --- | -| Providers per session | one | as many as you configure | -| `/model` picker | shows nothing to choose from | your models listed with GitHub's | -| GitHub models | unavailable while BYOK is on | still there, same session | -| Switching model | restart with new env vars | `/model` inside the session | - -Bringing your **own key** for OpenAI, Anthropic or Gemini is not possible natively on personal -accounts at all — GitHub reserves that for enterprise owners, who get custom models in the -picker. On a personal plan those vendors appear in the picker but are served by GitHub's own -subscription: they consume premium requests and follow your plan's restrictions. On the free plan -most of them are visible yet unusable. +Copilot CLI has two built-in ways to use your own models. + +The `COPILOT_PROVIDER_*` variables point the whole session at one provider. That path still has the +limits this project was built for: one provider per session, no provider list in `/model`, GitHub +models unavailable. + +The `providers.json` registry is the better supported path: multiple providers and models, listed +in `/model` next to GitHub's, with the CLI talking to each provider itself. It still asks you to +write every endpoint and model by hand, stores the keys as plain text in that file, and needs a +restart after edits. Ollama is the exception: the CLI discovers local Ollama models on its own. + +This project sits on top of both: it asks each provider what it serves now, ships endpoints for 28 +of them, keeps credentials in environment variables or the opt-in key store, and can feed the +native registry from the same discovery. The default mode remains the local router, which was +verified to keep working on CLI 1.0.105 and supports older CLIs as well. + +| | `COPILOT_PROVIDER_*` | `providers.json` | This project | +| --- | --- | --- | --- | +| Providers per session | one | as many as configured | as many as configured | +| `/model` picker | one fixed model | your models next to GitHub's | your models next to GitHub's | +| GitHub models | unavailable | available | available | +| Model lists | you write them | you write them (Ollama is discovered) | discovered from each provider | +| Credentials | environment | plain text in the file | environment or key store | +| Local proxy | no | no | router mode only | +| Works on older CLIs | yes | newer releases only | yes, via the router | ## The router @@ -70,6 +80,19 @@ loopback and shuts down when Copilot exits. > [!IMPORTANT] > `COPILOT_API_URL` is an internal variable — it is not listed in `copilot help environment`. It -> works today (verified on CLI 1.0.79) but GitHub may rename or change it. If your models ever -> vanish from the picker, that is the first thing to check, and `copilot-byok --native` always -> gets you back to a stock session. +> worked on CLI 1.0.79 and still worked on 1.0.105 when this project was re-verified on 2026-11-01, +> but GitHub may rename or change it. If your models ever vanish from the picker, that is the first +> thing to check, `copilot-byok doctor` reports the CLI version it found, and `copilot-byok +> --native` always gets you back to a stock session. + +## Native registry mode + +`copilot-byok --native-byok` publishes the same models through the CLI's supported BYOK registry +instead of the router. It runs the usual live discovery, writes a temporary `providers.json`, points +`COPILOT_PROVIDERS_CONFIG` at it, starts Copilot, and deletes the file when the session ends. There +is no local proxy and no dependency on `COPILOT_API_URL`. + +The trade-off is the one the registry itself imposes: provider keys are in that temporary file while +the session runs. The file is created in the system temp directory with owner-only permissions and +removed on exit, including when the session fails to start. Model ids are `provider/model`; a +`--model byok-...` value is translated to the matching native id. diff --git a/docs/provider-verification.md b/docs/provider-verification.md index ae75c13..3cefc0f 100644 --- a/docs/provider-verification.md +++ b/docs/provider-verification.md @@ -1,6 +1,7 @@ # Provider verification -Last verified: 2026-10-14 for endpoints and model lists; end-to-end runs below are from 2026-08-12. +Last verified: 2026-11-01 for Copilot CLI compatibility; provider endpoints and model lists +2026-10-14; end-to-end runs below are from 2026-08-12 and 2026-11-01. This document separates three different levels of evidence, because they are not interchangeable: @@ -21,6 +22,23 @@ interchangeable: 25-second timeout. No credentials were printed. - Endpoint and model-list tables were re-probed on 2026-10-14 (Windows, Node.js 24.21, invalid-key probes, no credentials). +- CLI compatibility was re-verified on 2026-11-01 (Windows 11, Node.js 24.21, + Copilot CLI 1.0.105 and 1.0.79) with an isolated `COPILOT_HOME`, mock providers + on loopback and no real GitHub credentials. + +## Copilot CLI compatibility + +Re-verified on 2026-11-01 against 1.0.105, with 1.0.79 as the baseline. + +| Area | 1.0.79 | 1.0.105 | Evidence | +|---|---|---|---| +| `COPILOT_API_URL` router | honoured | honoured | `GET /models` reached a local server on both; the full session completed on 1.0.105, including `POST /chat/completions` and `POST /mcp/readonly` | +| Reasoning effort to BYOK | not sent | sent | `reasoning_effort: "high"` on the wire with native provider variables, and through the router when the model entry advertises levels | +| Native `providers.json` registry | not loaded | loaded | two providers in one file; `--model p2/m2` reached the second base URL with its `wireModel` | +| Extension SDK | unchanged | unchanged | the 1.0.28 SDK still exports `joinSession`, `commands`, `ui.select` and `session.log`; extensions remain experimental and were not run interactively in this audit | + +`COPILOT_API_URL` remains undocumented: it is not listed in `copilot help environment` on 1.0.105. +The native registry is the supported alternative, which is why `--native-byok` exists. ## Endpoint reachability @@ -112,14 +130,20 @@ Measured wall-clock for the three configured providers, queried in parallel: 1.2 ## Reasoning effort -Copilot's own effort setting never reaches a BYOK provider. Verified by pointing -the router at a provider that records what it receives: the payloads produced -with `--effort low` and `--effort high` are byte-identical, and contain no -reasoning field at all. Declaring `reasoning_effort` in a model entry changes only -whether the CLI accepts the flag — not what it sends. +On CLI 1.0.79 the effort level never reached a BYOK provider: payloads produced with +`--effort low` and `--effort high` were byte-identical and contained no reasoning +field, in both wire formats. That was upstream +[copilot-cli#4012](https://github.com/github/copilot-cli/issues/4012), resolved later +in the 1.0.9x line. -Providers do accept the parameter when the router adds it, and they do **not** fall -back on their own. Measured against `zai-org/GLM-5.2-TEE` on Chutes: +Re-verified on 1.0.105 (2026-11-01): `--reasoning-effort high` produces +`"reasoning_effort":"high"` in the request, both with the native BYOK variables and +through the router when the model entry declares +`capabilities.supports.reasoning_effort`. Without that declaration the CLI rejects +the flag before sending anything. + +Providers accept the parameter when it arrives, and they do **not** fall back on +their own. Measured against `zai-org/GLM-5.2-TEE` on Chutes: | `reasoning_effort` | Result | |---|---| @@ -128,16 +152,8 @@ back on their own. Measured against `zai-org/GLM-5.2-TEE` on Chutes: | `banana` (control) | 400 validation error | So a level that a model does not accept breaks every request to it, which is why -the router steps the level down until one is accepted and remembers the outcome. - -The level was not delivered in any configuration tried on CLI 1.0.79: router mode -with the capability declared, the official BYOK variables over chat-completions, -the same over the responses wire format, model ids `gpt-5.4`, `gpt-5.5`, -`gpt-5.6-sol`, `claude-sonnet-4.6` and `gemini-3.1-pro-preview`, and -`COPILOT_OFFLINE=true` (the workaround reported in copilot-cli#3119). Declaring -`supported_endpoints: ['/responses']` does make the CLI switch to that format — -observed request keys: `include, input, instructions, model, parallel_tool_calls, -store, text, tools` — but still with no reasoning field. +the router steps the level down until one is accepted and remembers the outcome, +including when the level came from the CLI rather than from the config. ## End-to-end @@ -163,11 +179,23 @@ works — the errors came back from the provider, through the router, unchanged: | OpenCode Zen / Go | `Insufficient balance` | | Alibaba Token Plan | `Your token-plan 1-week quota has been exhausted` | +### Native registry path + +On 2026-11-01, `copilot-byok --native-byok` ran on CLI 1.0.105 with an isolated +`COPILOT_HOME` and no GitHub credentials, against a local OpenAI-compatible mock. +The generated `providers.json` was accepted, Copilot sent +`POST /v1/chat/completions` with the pinned model name, used the reply, and the +temporary registry was gone after exit. This verifies the registry shape and the +cleanup path, not a real provider's inference. + ## Reproducing ```sh +copilot-byok doctor # installation and CLI version copilot-byok --list-providers # which providers your keys activate +copilot-byok models --offline # what discovery would publish, no network copilot-byok --dry-run # router URL and published model ids +copilot-byok --native-byok --dry-run # native registry: providers and selection ids copilot-byok -- -p "read README.md and summarise it" --allow-all-tools ``` diff --git a/docs/providers.md b/docs/providers.md index b48a754..17bebe9 100644 --- a/docs/providers.md +++ b/docs/providers.md @@ -2,7 +2,8 @@ Every provider below is reached through its OpenAI-compatible endpoint, verified reachable at release time. A provider activates only when its key is present, so the `/model` picker stays as -short as your setup. +short as your setup. The same list feeds both modes: the router, and `--native-byok`, which publishes +them through the CLI's own `providers.json` registry. | Provider | Env var | Provider | Env var | | --- | --- | --- | --- | @@ -46,11 +47,16 @@ No entry ships a model list: the picker is filled by live discovery. To pin exac — for an offline setup or a fixed line-up — add them in your own configuration; see [configuration](configuration.md). +Ollama and LM Studio ship disabled because they have no key to detect, and current Copilot CLIs +discover local Ollama models themselves. Configure the local server explicitly to use it through +this tool, or add the model from the CLI's own picker. + ## Where the model list comes from **Providers are asked directly.** At startup each configured provider is queried for the models it serves right now, so the picker shows today's line-up rather than whatever was pinned at release -time — a model released this morning is there this morning. All 28 providers expose an +time — a model released this morning is there this morning. The same discovery fills the native +registry in `--native-byok` mode. All 28 providers expose an OpenAI-style `GET /models` (verified in [provider verification](provider-verification.md)); the ones that also publish capabilities (Chutes, OpenRouter, NIM and others) supply the real context window, output limit and tool-calling support along with it. @@ -72,6 +78,9 @@ Anything the provider does not report falls back to conservative defaults (128k output); image input and the reasoning selector appear only when the provider's own metadata declares them. +To see the same result without opening a session, run `copilot-byok models [provider]`; add +`--json` for scripts or `--offline` to print pinned and remembered lists without a network request. + If a request fails — a rate limit, a blip, an expired key — the list that key returned last time is used instead, kept in `~/.cache/copilot-byok/models.json` (`%LOCALAPPDATA%` on Windows). That matters: a provider's catalog belongs to the plan behind your key, so only what your key actually diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index bcf9538..1aebf42 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -1,5 +1,12 @@ # Troubleshooting +## Run `doctor` first + +`copilot-byok doctor` checks Node, the Copilot binary and its version, the provider config, +credentials, the key store, the native registry, the log paths, the shim and the extension in one +pass. Add `--json` for scripts and `--online` to query the providers that have a key. It never +starts Copilot and never prints a credential. + ## `copilot` starts, but without the BYOK models Plain `copilot` is GitHub's CLI and knows nothing about your providers. Either run @@ -60,6 +67,32 @@ Please open an issue with that version number. Meanwhile `copilot-byok --native` runs a stock session, and `--legacy` still gives you one BYOK provider through the officially documented `COPILOT_PROVIDER_*` variables. +## `--legacy` used a different provider than the one I picked + +`~/.copilot/providers.json` (or the file named by `COPILOT_PROVIDERS_CONFIG`) takes precedence over +the `COPILOT_PROVIDER_*` variables. Legacy mode notices that and hands the child an empty temporary +registry, printing: + +```text +copilot-byok: ... would take precedence over COPILOT_PROVIDER_*; using an empty registry for this session. +``` + +If the line is missing and the wrong provider answered, check the path named by +`COPILOT_PROVIDERS_CONFIG` in your shell. + +## `--native-byok` fails or lists nothing + +Native mode writes a temporary `providers.json` and starts Copilot with `COPILOT_PROVIDERS_CONFIG` +pointing at it. Requirements: + +- a Copilot CLI release that reads `providers.json` (verified on 1.0.105); +- at least one active provider after discovery, or the mode falls back to a stock session; +- model ids in `/model` are `provider/model`; `--model byok-...` is translated for you. + +If the session fails immediately, run `copilot-byok --native-byok --dry-run` to see the providers +and selection ids that would be written. The temporary file is removed when Copilot exits, and a +`providers.json` you maintain yourself is ignored for that session. + ## `Unknown BYOK model` The catalog changed while a session was running — for example after editing your @@ -89,7 +122,8 @@ The list comes from the provider itself. Three things can hide a model: of the picker. The log says which and why. Results are cached for ten minutes, so a model added upstream may take that long -to appear — restart to pick it up immediately. +to appear — restart to pick it up immediately. `--native-byok` uses the same discovery and the same +filters; `copilot-byok models` prints exactly what would be published. ## A provider returns 404 on a model @@ -145,6 +179,11 @@ or set `COPILOT_BYOK_UPSTREAM`. Note that Copilot Enterprise owners can add cust models natively from enterprise settings, which shows them in the picker without this tool. +On **GitHub Enterprise Cloud with data residency** the Copilot API lives at +`copilot-api..ghe.com`, not on the public hosts. The router derives it from +`COPILOT_GH_HOST` or `GH_HOST` when the host ends in `.ghe.com`; if your setup differs, +pass `--upstream https://copilot-api..ghe.com` explicitly. + ## Ollama or LM Studio returns nothing Both ship disabled in the catalog because they have no keys to detect. Add them in @@ -158,20 +197,21 @@ curl http://127.0.0.1:1234/v1/models # LM Studio ## Setting the reasoning effort has no effect -Copilot's `--effort` flag and its effort selector do not reach BYOK providers — -the level never leaves the CLI. Set it per model instead: +Current Copilot CLIs forward the level to BYOK providers; this was not true on +1.0.79, where the payload was the same for `low` and `high` +([copilot-cli#4012](https://github.com/github/copilot-cli/issues/4012), fixed later). +If the selector is missing, check that the model reports reasoning support and that +`reasoningEffortPicker` is not set to `false`. + +If the provider rejects the level, the router steps down automatically for a level from the picker +as well as one from the configuration, and says so in the log. To force a level, set it per model: ```json { "model": "deepseek-reasoner", "reasoningEffort": "high" } ``` -If a model rejects that level the router steps down automatically and says so in -the log, so a wrong value degrades rather than breaking every request. - -The selector is missing from `/model` on purpose: on CLI 1.0.79 the level it -returns never reaches the provider (github/copilot-cli#4012). Enable it with -`reasoningEffortPicker: true` once that is fixed — the router already honours an -incoming level, so nothing else needs changing. +In `--native-byok` mode the CLI talks to the provider directly, so there is no +step-down: set `reasoningEffortLevels` to the levels the model really accepts. ## A request hangs @@ -193,8 +233,7 @@ copilot-byok: 3 router errors during this session — see ~/.config/copilot-byok The log records forwarded requests, model injections and errors, with a timestamp and the process id. Credentials are never written to it. On Windows it lives in -`%APPDATA%\copilot-byok -outer.log`; `COPILOT_BYOK_LOG` overrides the path, and +`%APPDATA%\copilot-byok\router.log`; `COPILOT_BYOK_LOG` overrides the path, and the file is truncated once it passes 2 MB. To watch events live instead — accepting that the display will be disturbed: diff --git a/echo-config.json b/echo-config.json deleted file mode 100644 index 3c555cc..0000000 --- a/echo-config.json +++ /dev/null @@ -1 +0,0 @@ -{"providers":[{"id":"echo","name":"Echo","baseUrl":"http://127.0.0.1:8901/v1","modelsUrl":"http://127.0.0.1:8901/v1/models","authRequired":false,"models":[]}]} diff --git a/schemas/providers.schema.json b/schemas/providers.schema.json index cbbfb4d..2f48300 100644 --- a/schemas/providers.schema.json +++ b/schemas/providers.schema.json @@ -88,6 +88,11 @@ "bearerTokenEnv": { "$ref": "#/$defs/envNames" }, + "apiKeyCommand": { + "type": "string", + "minLength": 1, + "description": "Command the Copilot CLI runs to obtain a fresh credential (legacy mode, COPILOT_PROVIDER_API_KEY_COMMAND). It is passed through, never executed by this tool." + }, "modelsUrl": { "type": "string", "format": "uri", diff --git a/src/args.mjs b/src/args.mjs index 054d786..674f3f8 100644 --- a/src/args.mjs +++ b/src/args.mjs @@ -11,6 +11,7 @@ export function parseArgs(argv) { help: false, version: false, legacy: false, + nativeByok: false, listProviders: false, noDiscovery: false, upstream: null, @@ -40,6 +41,12 @@ export function parseArgs(argv) { continue; } + // Uses the CLI's own BYOK registry instead of the local router. + if (arg === '--native-byok') { + result.nativeByok = true; + continue; + } + // Classic mode: one provider per session through COPILOT_PROVIDER_*, with no // router and no GitHub models in the same list. if (arg === '--legacy' || arg === '--single-provider') { @@ -153,6 +160,18 @@ export function parseArgs(argv) { throw new Error('--offline and --wire-api require a BYOK provider'); } + if (result.nativeByok && result.legacy) { + throw new Error('--native-byok cannot be combined with --legacy'); + } + + if (result.nativeByok && result.providerName === 'native') { + throw new Error('--native-byok cannot be combined with --native'); + } + + if (result.nativeByok && result.upstream) { + throw new Error('--upstream applies only to router mode; --native-byok talks to the providers directly'); + } + // These only mean something to the single-provider flow. Accepting them in // router mode and ignoring them silently produced surprising results, such as // --list-models launching a full session instead of printing a list. diff --git a/src/argv-config.mjs b/src/argv-config.mjs new file mode 100644 index 0000000..bdee8af --- /dev/null +++ b/src/argv-config.mjs @@ -0,0 +1,19 @@ +/** Removes --config/-c from subcommand arguments and returns its value. */ +export function extractConfigPath(argv) { + for (let index = 0; index < argv.length; index += 1) { + const arg = argv[index]; + if (arg === '--config' || arg === '-c') { + const value = argv[index + 1]; + if (!value || value.startsWith('-')) throw new Error(`${arg} requires a file path`); + argv.splice(index, 2); + return value; + } + if (arg.startsWith('--config=')) { + const value = arg.slice('--config='.length); + if (!value) throw new Error('--config requires a file path'); + argv.splice(index, 1); + return value; + } + } + return null; +} diff --git a/src/catalog.mjs b/src/catalog.mjs index 72011ea..9e33459 100644 --- a/src/catalog.mjs +++ b/src/catalog.mjs @@ -7,7 +7,7 @@ const ID_PREFIX = 'byok'; // The levels Copilot offers. Overridable per model, since they differ: Claude // Sonnet 4.6 has no `xhigh`, Gemini stops at `high`. -const DEFAULT_EFFORT_LEVELS = ['low', 'medium', 'high']; +export const DEFAULT_EFFORT_LEVELS = ['low', 'medium', 'high']; /** * @param {object[]} providers Already-resolved providers (with apiKey where needed). @@ -95,16 +95,12 @@ function buildEntry({ id, provider, model }) { parallel_tool_calls: true, structured_outputs: true, ...(model.vision ? { vision: true } : {}), - // Off by default, and deliberately so. Declaring this makes Copilot show - // its effort selector, but on CLI 1.0.79 the chosen level never reaches a - // BYOK provider: payloads sent with low and with high are byte-identical, - // in both wire formats (github/copilot-cli#4012, #3119, #3135). A selector - // that changes nothing is worse than no selector. - // - // Turn it on with `reasoningEffortPicker: true` once the CLI forwards the - // level again — the router already prefers an incoming reasoning_effort - // over the configured one, so it will start working with no other change. - ...(model.reasoning && model.reasoningEffortPicker + // Declared for reasoning models by default: current CLIs forward the + // level to BYOK providers, so the selector is meaningful. Older CLIs + // accepted the choice without sending it (github/copilot-cli#4012, + // fixed in 1.0.91); there the selector is inert, and + // `reasoningEffortPicker: false` hides it. + ...(model.reasoning && model.reasoningEffortPicker !== false ? { reasoning_effort: model.reasoningEffortLevels || DEFAULT_EFFORT_LEVELS } : {}), }, @@ -119,7 +115,7 @@ function buildEntry({ id, provider, model }) { * prompt; in that case output is halved. Otherwise output is kept as published * and the prompt takes what is left. */ -function splitBudget(context, output) { +export function splitBudget(context, output) { const outputBudget = output >= context ? Math.floor(context / 2) : output; return { outputBudget, promptBudget: Math.max(context - outputBudget, 1) }; } diff --git a/src/cli.mjs b/src/cli.mjs index 35d46b6..b8d6d3f 100644 --- a/src/cli.mjs +++ b/src/cli.mjs @@ -1,9 +1,12 @@ +import { rmSync } from 'node:fs'; import { readFile } from 'node:fs/promises'; import readline from 'node:readline/promises'; import spawn from 'cross-spawn'; import { parseArgs } from './args.mjs'; +import { extractConfigPath } from './argv-config.mjs'; +import { buildCatalog, isByokModelId } from './catalog.mjs'; import { resolveCopilotBin } from './copilot-bin.mjs'; import { findProvider, loadConfig } from './config.mjs'; import { rankModels } from './model-ranking.mjs'; @@ -11,8 +14,13 @@ import { providerModelHeaders } from './models-auth.mjs'; import { buildProviderEnvironment } from './provider-env.mjs'; import { sanitizeCopilotEnvironment } from './process-env.mjs'; import { isQueryable, resolveLiveModels, resolveOfflineModels, runCopilotWithRouter, selectActiveProviders, startRouter } from './launcher.mjs'; +import { dataResidencyOrigin } from './upstream.mjs'; import { readCache, recallModels } from './model-cache.mjs'; import { runKeysCommand } from './keys-command.mjs'; +import { runDoctorCommand } from './doctor.mjs'; +import { runModelsCommand } from './models-command.mjs'; +import { buildNativeRegistry, nativeSelectionId } from './native-registry.mjs'; +import { createRegistryFile, inspectNativeRegistry } from './registry-file.mjs'; import { loadKeys } from './keystore.mjs'; import { runExtensionCommand } from './extension-install.mjs'; import { runShimCommand } from './shim-install.mjs'; @@ -41,6 +49,16 @@ export async function main(argv = process.argv.slice(2), io = defaultIo()) { return runShimCommand({ argv: argv.slice(1), io }); } + // `doctor` reports the state of the installation without starting Copilot. + if (argv[0] === 'doctor') { + return runDoctorCommand({ argv: argv.slice(1), io }); + } + + // `models` prints what the providers currently serve, without a session. + if (argv[0] === 'models') { + return runModelsCommand({ argv: argv.slice(1), io }); + } + const args = parseArgs(argv); if (args.help) { @@ -67,6 +85,11 @@ export async function main(argv = process.argv.slice(2), io = defaultIo()) { return await listProviders({ config, io }); } + if (args.nativeByok) { + const config = await loadConfig({ configPath: args.configPath, env: io.env, secrets }); + return runNativeByokMode({ args, config, io }); + } + if (!args.legacy) { const config = await loadConfig({ configPath: args.configPath, env: io.env, secrets }); return runRouterMode({ args, config, io }); @@ -96,7 +119,7 @@ async function runRouterMode({ args, config, io }) { const router = await startRouter({ providers, - upstreamOrigin: args.upstream || io.env.COPILOT_BYOK_UPSTREAM, + upstreamOrigin: args.upstream || io.env.COPILOT_BYOK_UPSTREAM || dataResidencyOrigin(io.env), onEvent: sessionLog.onEvent, // Re-read config and key store so a key added mid-session (via /byok) // shows up in the picker without restarting. @@ -177,6 +200,99 @@ async function listProviders({ config, io }) { return 0; } +/** Native BYOK: the CLI reads a generated providers.json and talks to the providers itself. */ +async function runNativeByokMode({ args, config, io }) { + const sessionLog = createSessionLog(io); + + const providers = args.noDiscovery + ? await resolveOfflineModels(config.providers, { env: io.env }) + : await discoverWithProgress({ config, io, sessionLog }); + const active = selectActiveProviders(providers); + + if (active.length === 0) { + io.stderr.write( + 'No BYOK provider configured: starting Copilot with GitHub models only.\n' + + 'Run "copilot-byok --list-providers" to see which environment variables to set.\n\n' + ); + return runCopilot({ copilotArgs: nativeCopilotArgs(args, []), env: {}, io, dryRun: args.dryRun, native: true, config }); + } + + const registry = buildNativeRegistry(active); + const registryFile = await createRegistryFile(registry); + // A hard kill still leaves the file behind if only `finally` cleaned up. + const cleanupOnExit = () => rmSync(registryFile.dir, { recursive: true, force: true }); + process.once('exit', cleanupOnExit); + + const selectionIds = registry.models.map((model) => nativeSelectionId(model.provider, model.id)); + const copilotArgs = nativeCopilotArgs(args, active); + + if (args.dryRun) { + process.removeListener('exit', cleanupOnExit); + await registryFile.cleanup(); + io.stdout.write(`${JSON.stringify({ + mode: 'native-byok', + providers: active.map((provider) => provider.id), + models: selectionIds, + args: copilotArgs, + }, null, 2)}\n`); + return 0; + } + + io.stderr.write( + `copilot-byok: added ${registry.models.length} models from ${active.length} providers through the native BYOK registry\n` + ); + + try { + return await runCopilot({ + copilotArgs, + env: { COPILOT_PROVIDERS_CONFIG: registryFile.path }, + io, + config, + }); + } finally { + process.removeListener('exit', cleanupOnExit); + await registryFile.cleanup(); + sessionLog.summarize(); + } +} + +/** + * Builds the Copilot arguments for native mode, translating a picker id from + * this project (`byok-...`) into the `provider/model` id the CLI expects. + */ +function nativeCopilotArgs(args, providers) { + const copilotArgs = [...args.copilotArgs]; + if (!args.explicitModel) return copilotArgs; + + let model = args.explicitModel; + if (isByokModelId(model) && providers.length > 0) { + const catalog = buildCatalog(providers.map((provider) => ({ + ...provider, + apiKey: provider.bearerToken || provider.apiKey, + }))); + const route = catalog.routes.get(model); + if (route) model = nativeSelectionId(route.provider.id, route.model.model); + } + + return ['--model', model, ...copilotArgs]; +} + +/** + * A providers.json with entries takes precedence over COPILOT_PROVIDER_*, so + * --legacy would silently talk to the wrong provider. When one is present, the + * child gets an empty registry instead: the mode keeps its documented meaning. + */ +async function isolateLegacyRegistry({ io }) { + const native = await inspectNativeRegistry(io.env); + if (!native.present || (native.providers === 0 && native.models === 0)) return null; + + const registryFile = await createRegistryFile({ providers: [], models: [] }); + io.stderr.write( + `copilot-byok: ${native.path} would take precedence over COPILOT_PROVIDER_*; using an empty registry for this session.\n` + ); + return registryFile; +} + /** Classic mode: one provider per session, without the GitHub models. */ async function runLegacyMode({ args, io }) { const config = await loadConfig({ configPath: args.configPath, env: io.env, secrets: await loadKeys(io.env) }); @@ -209,7 +325,14 @@ async function runLegacyMode({ args, io }) { const wireModel = args.explicitModel || await selectModel({ provider, models, noPrompt: args.noModelPrompt, io }); const effectiveProvider = args.wireApi ? { ...provider, wireApi: args.wireApi } : provider; const env = buildProviderEnvironment({ provider: effectiveProvider, wireModel, offline: args.offline }); - return runCopilot({ copilotArgs: args.copilotArgs, env, io, dryRun: args.dryRun, provider, wireModel, config }); + const isolation = await isolateLegacyRegistry({ io }); + if (isolation) env.COPILOT_PROVIDERS_CONFIG = isolation.path; + + try { + return await runCopilot({ copilotArgs: args.copilotArgs, env, io, dryRun: args.dryRun, provider, wireModel, config }); + } finally { + await isolation?.cleanup(); + } } function defaultIo() { @@ -221,30 +344,6 @@ function defaultIo() { }; } -/** - * Subcommands are dispatched before the argument parser runs, so they pick up - * --config here. Without this, anyone with a custom catalog could not store a key - * for a provider defined only in it. - */ -function extractConfigPath(argv) { - for (let index = 0; index < argv.length; index += 1) { - const arg = argv[index]; - if (arg === '--config' || arg === '-c') { - const value = argv[index + 1]; - if (!value || value.startsWith('-')) throw new Error(`${arg} requires a file path`); - argv.splice(index, 2); - return value; - } - if (arg.startsWith('--config=')) { - const value = arg.slice('--config='.length); - if (!value) throw new Error('--config requires a file path'); - argv.splice(index, 1); - return value; - } - } - return null; -} - async function resolveProvider({ config, requested, io }) { if (requested === 'native') return 'native'; @@ -473,6 +572,7 @@ GitHub, and /model lists the GitHub models together with your providers'. Options: --list-providers List providers and which ones are usable already --native Start Copilot without the router, GitHub models only + --native-byok Use the CLI's own providers.json registry instead of the router -m, --model Starting model (picker id, e.g. byok-openai-gpt-5-5) -c, --config Provider configuration file --upstream Force the Copilot API tier (individual/business/enterprise) @@ -481,6 +581,10 @@ Options: -h, --help Show this help -v, --version Print the copilot-byok version +Inspection: + copilot-byok doctor Diagnose the installation without starting Copilot + copilot-byok models [provider] List the models each provider currently serves + Credentials: copilot-byok keys list Show where each provider's key comes from copilot-byok keys set Store a key (prompted, never echoed) diff --git a/src/config.mjs b/src/config.mjs index f629d2b..1123626 100644 --- a/src/config.mjs +++ b/src/config.mjs @@ -99,6 +99,7 @@ function normalizeProvider(provider, env, index, secrets = {}) { : requiredString(provider.azureApiVersion, `Provider ${name} azureApiVersion must be a non-empty string`); const modelIncludePrefixes = normalizeStringList(provider.modelIncludePrefixes, name, 'modelIncludePrefixes'); const modelExcludePrefixes = normalizeStringList(provider.modelExcludePrefixes, name, 'modelExcludePrefixes'); + const apiKeyCommand = normalizeApiKeyCommand(provider.apiKeyCommand, name); if (Object.hasOwn(provider, 'apiKey')) { throw new Error(`Inline apiKey is not allowed for provider ${name}. Use apiKeyEnv instead.`); @@ -129,6 +130,7 @@ function normalizeProvider(provider, env, index, secrets = {}) { ...(azureApiVersion ? { azureApiVersion } : {}), ...(modelIncludePrefixes ? { modelIncludePrefixes } : {}), ...(modelExcludePrefixes ? { modelExcludePrefixes } : {}), + ...(apiKeyCommand ? { apiKeyCommand } : {}), // The environment wins over the optional key store, so a shell variable can // always override a stored key for one session. apiKey: readFirstEnv(env, apiKeyEnv) ?? secrets[id] ?? null, @@ -150,6 +152,9 @@ function normalizeModels(value, providerName) { const model = requiredString(entry.model, `${where} requires a model name`); if (entry.reasoningEffort != null) validateEffort(entry.reasoningEffort, where); + if (entry.reasoningEffortPicker != null && typeof entry.reasoningEffortPicker !== 'boolean') { + throw new Error(`${where} reasoningEffortPicker must be a boolean`); + } if (entry.reasoningEffortLevels != null) { if (!Array.isArray(entry.reasoningEffortLevels) || entry.reasoningEffortLevels.length === 0) { throw new Error(`${where} reasoningEffortLevels must be a non-empty array`); @@ -180,6 +185,9 @@ function validateEffort(value, where) { function validateProviderOptions(provider, name, type) { if (provider.reasoningEffort != null) validateEffort(provider.reasoningEffort, `Provider ${name}`); + if (provider.reasoningEffortPicker != null && typeof provider.reasoningEffortPicker !== 'boolean') { + throw new Error(`Provider ${name} reasoningEffortPicker must be a boolean`); + } for (const field of ['catalogModelId', 'defaultModel']) { if (provider[field] != null) requiredString(provider[field], `Provider ${name} ${field} must be a non-empty string`); } @@ -254,6 +262,12 @@ function rejectSecretModelHeaders(provider, name) { for (const { key, label } of fields) { for (const [headerName, value] of Object.entries(provider[key] || {})) { + // These reach an HTTP client or the newline-separated + // COPILOT_PROVIDER_HEADERS variable: a line break would smuggle an + // extra header into the request. + if (/[\r\n]/.test(headerName) || /[\r\n]/.test(value)) { + throw new Error(`Provider ${name} ${label} must not contain line breaks`); + } const combined = `${headerName}: ${value}`; if (/authorization|cookie|session|credential|password|api[-_]?key|token|secret|bearer|x-auth/i.test(combined)) { throw new Error(`Secret ${label} are not allowed for provider ${name}. Use apiKeyEnv or bearerTokenEnv instead.`); @@ -309,6 +323,16 @@ function normalizeStringList(value, providerName, field) { return [...new Set(normalized)]; } +/** + * A command the CLI runs to obtain a fresh credential for each request. It is + * passed through, never executed here: in legacy mode it becomes + * COPILOT_PROVIDER_API_KEY_COMMAND. It must stay a command, not a credential. + */ +function normalizeApiKeyCommand(value, providerName) { + if (value == null) return undefined; + return requiredString(value, `Provider ${providerName} apiKeyCommand must be a non-empty string`); +} + function assertUniqueProviderNames(providers) { const claimed = new Map(); for (const provider of providers) { diff --git a/src/discovery.mjs b/src/discovery.mjs index 7c1de84..9d3dae3 100644 --- a/src/discovery.mjs +++ b/src/discovery.mjs @@ -168,7 +168,6 @@ export function declaresToolSupport(entry) { return undefined; } -/** Reads the context window under any of the names providers actually use. */ /** * Whether the model reasons before answering. Copilot only shows its effort * selector for models that say so, and only then does it forward the level. @@ -220,6 +219,7 @@ export function declaresVision(entry) { return Boolean(supports?.vision); } +/** Reads the context window under any of the names providers actually use. */ export function readContextWindow(entry) { return firstPositive( entry?.context_length, diff --git a/src/doctor.mjs b/src/doctor.mjs new file mode 100644 index 0000000..a90a617 --- /dev/null +++ b/src/doctor.mjs @@ -0,0 +1,270 @@ +// `copilot-byok doctor` - one command that reports the state of the install. +// +// It never starts Copilot and never prints a credential: each check reports +// whether something is present, not its value. Exit code 0 when nothing failed, +// 1 when at least one check failed, so it can gate a script. + +import { readFile, readdir, stat } from 'node:fs/promises'; +import { join } from 'node:path'; + +import spawn from 'cross-spawn'; + +import { extractConfigPath } from './argv-config.mjs'; +import { resolveCopilotBin } from './copilot-bin.mjs'; +import { loadConfig } from './config.mjs'; +import { extensionDir } from './extension-install.mjs'; +import { isQueryable, resolveLiveModels } from './launcher.mjs'; +import { cachePath, readCache } from './model-cache.mjs'; +import { nativeRegistryPath, inspectNativeRegistry } from './registry-file.mjs'; +import { keystorePath, loadKeys } from './keystore.mjs'; +import { logPath } from './session-log.mjs'; +import { shimDir } from './shim-install.mjs'; +import { dataResidencyOrigin } from './upstream.mjs'; + +const MINIMUM_COPILOT_CLI = '1.0.79'; + +export function detectCopilotVersion(bin, env) { + return new Promise((resolve) => { + let child; + try { + child = spawn(bin, ['--version'], { + env, + shell: false, + stdio: ['ignore', 'pipe', 'pipe'], + windowsHide: true, + }); + } catch { + resolve(null); + return; + } + + let output = ''; + const timer = setTimeout(() => { + child.kill(); + resolve(null); + }, 15_000); + + child.stdout.on('data', (chunk) => { output += chunk; }); + child.stderr.on('data', (chunk) => { output += chunk; }); + child.on('error', () => { + clearTimeout(timer); + resolve(null); + }); + child.on('exit', (code) => { + clearTimeout(timer); + if (code !== 0) { + resolve(null); + return; + } + const match = /(\d+\.\d+\.\d+)/.exec(output); + resolve(match ? match[1] : null); + }); + }); +} + +export async function runDoctorCommand({ argv = [], io, probeVersion = detectCopilotVersion }) { + const args = { json: false, online: false, configPath: null, help: false }; + const rest = [...argv]; + args.configPath = extractConfigPath(rest); + + for (const arg of rest) { + if (arg === '--json') args.json = true; + else if (arg === '--online') args.online = true; + else if (arg === '--help' || arg === '-h') args.help = true; + else throw new Error(`Unknown doctor option: ${arg}`); + } + + if (args.help) { + io.stdout.write(doctorHelpText()); + return 0; + } + + const checks = []; + const add = (level, name, detail) => checks.push({ level, name, detail }); + + await checkNode(add); + await checkCopilot({ add, io, probeVersion }); + const { config, secrets } = await checkConfig({ add, io, configPath: args.configPath }); + + if (config) { + checkProviders({ add, config }); + if (args.online) await checkOnline({ add, config, io }); + } + + checkKeystore({ add, secrets, env: io.env }); + await checkNativeRegistry({ add, env: io.env }); + await checkPaths({ add, env: io.env }); + await checkShim({ add, env: io.env }); + await checkExtension({ add, env: io.env }); + checkDataResidency({ add, env: io.env }); + + const failed = checks.filter((check) => check.level === 'fail').length; + if (args.json) { + io.stdout.write(`${JSON.stringify({ ok: failed === 0, checks }, null, 2)}\n`); + } else { + for (const check of checks) io.stdout.write(`${formatCheck(check)}\n`); + io.stdout.write(`\n${failed === 0 ? 'No failures.' : `${failed} check(s) failed.`}\n`); + } + + return failed === 0 ? 0 : 1; +} + +function formatCheck({ level, name, detail }) { + const tag = level === 'fail' ? 'FAIL' : level === 'warn' ? 'WARN' : 'ok '; + return `${tag} ${name}: ${detail}`; +} + +async function checkNode(add) { + const minimum = await readEngineMinimum(); + const current = process.versions.node; + if (minimum && !versionAtLeast(current, minimum)) { + add('fail', 'node', `${current}, the package needs ${minimum} or newer`); + return; + } + add('ok', 'node', minimum ? `${current} (minimum ${minimum})` : current); +} + +async function checkCopilot({ add, io, probeVersion }) { + const bin = resolveCopilotBin({ env: io.env }); + const version = await probeVersion(bin, io.env); + if (!version) { + add('warn', 'copilot', `${bin} did not answer --version; set COPILOT_BIN if the path is wrong`); + return; + } + + if (!versionAtLeast(version, MINIMUM_COPILOT_CLI)) { + add('warn', 'copilot', `${version} at ${bin}; this project targets ${MINIMUM_COPILOT_CLI} or newer`); + return; + } + + add('ok', 'copilot', `${version} at ${bin}`); +} + +async function checkConfig({ add, io, configPath }) { + let secrets = {}; + try { + secrets = await loadKeys(io.env); + } catch (error) { + add('fail', 'key store', error.message); + } + + try { + const config = await loadConfig({ configPath, env: io.env, secrets }); + add('ok', 'config', `${config.providers.length} providers`); + return { config, secrets }; + } catch (error) { + add('fail', 'config', error.message); + return { config: null, secrets }; + } +} + +function checkProviders({ add, config }) { + const active = config.providers.filter(isQueryable); + const missing = config.providers + .filter((provider) => provider.authRequired !== false && !provider.apiKey && !provider.bearerToken) + .map((provider) => provider.id); + + add(active.length > 0 ? 'ok' : 'warn', 'providers', `${active.length} of ${config.providers.length} have a credential`); + if (missing.length > 0 && active.length === 0) { + add('warn', 'credentials', `set a key for one of: ${missing.slice(0, 6).join(', ')}${missing.length > 6 ? ', ...' : ''}`); + } +} + +async function checkOnline({ add, config, io }) { + try { + const providers = await resolveLiveModels(config.providers, { env: io.env }); + const withModels = providers.filter((provider) => provider.models?.length); + add('ok', 'discovery', `${withModels.length} providers returned models`); + } catch (error) { + add('warn', 'discovery', error.message); + } +} + +function checkKeystore({ add, secrets, env }) { + const stored = Object.keys(secrets).length; + add('ok', 'key store', `${keystorePath(env)}${stored > 0 ? ` (${stored} stored keys)` : ' (empty)'}`); +} + +async function checkNativeRegistry({ add, env }) { + const native = await inspectNativeRegistry(env); + if (native.error) { + add('warn', 'native registry', `${native.path}: ${native.error}`); + return; + } + if (!native.present) { + add('ok', 'native registry', `${nativeRegistryPath(env)} (not present)`); + return; + } + add( + native.providers + native.models > 0 ? 'warn' : 'ok', + 'native registry', + `${native.path}: ${native.providers} providers, ${native.models} models${native.providers + native.models > 0 ? ' (takes precedence over COPILOT_PROVIDER_*)' : ''}`, + ); +} + +async function checkPaths({ add, env }) { + const cache = await readCache(env).catch(() => ({})); + add('ok', 'model cache', `${nativeCount(cache)} providers remembered at ${cachePath(env)}`); + + const log = await stat(logPath(env)).catch(() => null); + add('ok', 'router log', log ? `${logPath(env)} (${log.size} bytes)` : `${logPath(env)} (not created yet)`); +} + +async function checkShim({ add, env }) { + const dir = shimDir(env); + const entries = await readdir(dir).catch(() => []); + const hasShim = entries.includes('copilot') || entries.includes('copilot.cmd'); + add(hasShim ? 'ok' : 'warn', 'shim', hasShim ? dir : `not installed (${dir})`); +} + +async function checkExtension({ add, env }) { + const dir = extensionDir(env); + try { + await readFile(join(dir, 'extension.mjs'), 'utf8'); + add('ok', 'extension', `installed at ${dir}`); + } catch { + add('warn', 'extension', 'not installed; /byok is unavailable'); + } +} + +function checkDataResidency({ add, env }) { + const origin = dataResidencyOrigin(env); + add('ok', 'upstream', origin ? `data residency: ${origin}` : 'public GitHub Copilot hosts'); +} + +function nativeCount(cache) { + return Object.keys(cache || {}).length; +} + +async function readEngineMinimum() { + try { + const manifest = JSON.parse(await readFile(new URL('../package.json', import.meta.url), 'utf8')); + const match = /(\d+\.\d+\.\d+)/.exec(manifest.engines?.node || ''); + return match ? match[1] : null; + } catch { + return null; + } +} + +function versionAtLeast(current, minimum) { + const [a, b, c] = current.split('.').map(Number); + const [x, y, z] = minimum.split('.').map(Number); + if ([a, b, c].some((value) => !Number.isInteger(value))) return false; + if (a !== x) return a > x; + if (b !== y) return b > y; + return c >= z; +} + +function doctorHelpText() { + return `copilot-byok doctor - report the state of the installation + +Usage: + copilot-byok doctor [--json] [--online] [--config ] + +Options: + --json Machine-readable output + --online Also query the providers that have a key + -c, --config Provider configuration file + -h, --help Show this help +`; +} diff --git a/src/models-command.mjs b/src/models-command.mjs new file mode 100644 index 0000000..0cea837 --- /dev/null +++ b/src/models-command.mjs @@ -0,0 +1,92 @@ +// `copilot-byok models` - print what the configured providers currently serve. +// +// Router mode discovers models at startup, but there was no way to inspect them +// without opening a session: --list-models only exists in legacy mode and only +// covers one provider. This command reuses the same discovery path. + +import { extractConfigPath } from './argv-config.mjs'; +import { findProvider, loadConfig } from './config.mjs'; +import { resolveLiveModels, resolveOfflineModels } from './launcher.mjs'; +import { loadKeys } from './keystore.mjs'; + +export async function runModelsCommand({ argv = [], io }) { + const args = { json: false, offline: false, help: false, configPath: null, filter: null }; + const rest = [...argv]; + args.configPath = extractConfigPath(rest); + + for (const arg of rest) { + if (arg === '--json') args.json = true; + else if (arg === '--offline') args.offline = true; + else if (arg === '--help' || arg === '-h') args.help = true; + else if (arg.startsWith('-')) throw new Error(`Unknown models option: ${arg}`); + else if (args.filter) throw new Error(`Only one provider can be selected, got "${args.filter}" and "${arg}"`); + else args.filter = arg; + } + + if (args.help) { + io.stdout.write(modelsHelpText()); + return 0; + } + + const config = await loadConfig({ + configPath: args.configPath, + env: io.env, + secrets: await loadKeys(io.env), + }); + + const requested = args.filter ? findProvider(config, args.filter) : null; + if (args.filter && !requested) { + throw new Error(`Unknown provider: ${args.filter}. Run "copilot-byok --list-providers" to see them all.`); + } + + const providers = args.offline + ? await resolveOfflineModels(config.providers, { env: io.env }) + : await resolveLiveModels(config.providers, { env: io.env }); + + const rows = providers + .filter((provider) => !requested || provider.id === requested.id) + .map((provider) => ({ + id: provider.id, + name: provider.name, + models: (provider.models || []).map((model) => ({ + id: model.model, + label: model.label || null, + contextWindow: model.contextWindow || null, + vision: Boolean(model.vision), + reasoning: Boolean(model.reasoning), + })), + })); + + if (args.json) { + io.stdout.write(`${JSON.stringify(rows, null, 2)}\n`); + return 0; + } + + for (const row of rows) { + io.stdout.write(`${row.id} (${row.models.length} models${args.offline ? ', offline lists' : ''})\n`); + for (const model of row.models) { + const details = [ + model.contextWindow ? `${Math.round(model.contextWindow / 1000)}k context` : null, + model.vision ? 'vision' : null, + model.reasoning ? 'reasoning' : null, + ].filter(Boolean).join(', '); + io.stdout.write(` ${model.id}${details ? ` (${details})` : ''}\n`); + } + } + + return 0; +} + +function modelsHelpText() { + return `copilot-byok models - list the models each provider currently serves + +Usage: + copilot-byok models [provider] [--json] [--offline] [--config ] + +Options: + --json Machine-readable output + --offline Skip discovery; use pinned or remembered lists + -c, --config Provider configuration file + -h, --help Show this help +`; +} diff --git a/src/native-registry.mjs b/src/native-registry.mjs new file mode 100644 index 0000000..6114391 --- /dev/null +++ b/src/native-registry.mjs @@ -0,0 +1,95 @@ +// Builds the registry the Copilot CLI reads from providers.json. +// +// The native registry is a supported surface: models listed there appear in the +// /model picker next to the GitHub ones, and the CLI talks to each provider +// itself, without a local router. This module only translates the project's +// resolved providers into that shape; reading and writing the file lives in +// registry-file.mjs and the launch flow in cli.mjs. + +import { DEFAULT_EFFORT_LEVELS, splitBudget } from './catalog.mjs'; + +const DEFAULT_CONTEXT_WINDOW = 128_000; +const DEFAULT_MAX_OUTPUT_TOKENS = 32_000; + +/** The id the CLI accepts for `--model`: PROVIDER-NAME/MODEL-ID. */ +export function nativeSelectionId(providerId, modelName) { + return `${providerId}/${modelName}`; +} + +/** + * @param {object[]} providers Active providers (models + credential resolved). + * @returns {{providers: object[], models: object[]}} + */ +export function buildNativeRegistry(providers) { + const nativeProviders = []; + const nativeModels = []; + + for (const provider of providers) { + nativeProviders.push(buildNativeProvider(provider)); + for (const model of provider.models || []) { + nativeModels.push(buildNativeModel(provider, model)); + } + } + + return { providers: nativeProviders, models: nativeModels }; +} + +/** Maps a native selection id back to the provider and model it describes. */ +export function nativeSelectionMap(providers) { + const map = new Map(); + for (const provider of providers) { + for (const model of provider.models || []) { + map.set(nativeSelectionId(provider.id, model.model), { provider, model }); + } + } + return map; +} + +function buildNativeProvider(provider) { + return { + name: provider.id, + type: provider.type ?? 'openai', + baseUrl: provider.baseUrl, + ...(provider.bearerToken + ? { bearerToken: provider.bearerToken } + : provider.apiKey + ? { apiKey: provider.apiKey } + : {}), + ...(provider.wireApi ? { wireApi: provider.wireApi } : {}), + ...(provider.transport ? { transport: provider.transport } : {}), + ...(provider.azureApiVersion ? { azure: { apiVersion: provider.azureApiVersion } } : {}), + ...(provider.headers && Object.keys(provider.headers).length > 0 + ? { headers: { ...provider.headers } } + : {}), + }; +} + +function buildNativeModel(provider, model) { + const context = model.contextWindow || DEFAULT_CONTEXT_WINDOW; + const output = model.maxOutputTokens || DEFAULT_MAX_OUTPUT_TOKENS; + const { promptBudget, outputBudget } = splitBudget(context, output); + const picker = model.reasoningEffortPicker ?? provider.reasoningEffortPicker; + const reasoningLevels = picker === false ? null : (model.reasoningEffortLevels || DEFAULT_EFFORT_LEVELS); + + return { + id: model.model, + provider: provider.id, + name: model.label || `${model.model} (${provider.name})`, + maxContextWindowTokens: context, + maxPromptTokens: promptBudget, + maxOutputTokens: outputBudget, + capabilities: { + supports: { + toolCalls: true, + streaming: true, + ...(model.vision ? { vision: true } : {}), + ...(model.reasoning && reasoningLevels ? { reasoning_effort: reasoningLevels } : {}), + }, + limits: { + max_context_window_tokens: context, + max_prompt_tokens: promptBudget, + max_output_tokens: outputBudget, + }, + }, + }; +} diff --git a/src/provider-env.mjs b/src/provider-env.mjs index a2c46c4..bca9640 100644 --- a/src/provider-env.mjs +++ b/src/provider-env.mjs @@ -10,6 +10,24 @@ export function buildProviderEnvironment({ provider, wireModel, offline = false COPILOT_PROVIDER_WIRE_MODEL: wireModel, }; + if (provider.catalogModelId) { + env.COPILOT_PROVIDER_MODEL_ID = provider.catalogModelId; + } + + // The CLI reads this as newline-separated "Name: Value" pairs. Custom + // `headers` are already validated against credential-bearing names. + if (provider.headers && Object.keys(provider.headers).length > 0) { + env.COPILOT_PROVIDER_HEADERS = Object.entries(provider.headers) + .map(([headerName, value]) => `${headerName}: ${value}`) + .join('\n'); + } + + // A rotating credential: the CLI runs the command itself, so it takes + // precedence over a static key and is never resolved by this process. + if (provider.apiKeyCommand) { + env.COPILOT_PROVIDER_API_KEY_COMMAND = provider.apiKeyCommand; + } + if (provider.bearerToken) { env.COPILOT_PROVIDER_BEARER_TOKEN = provider.bearerToken; } else if (provider.apiKey) { diff --git a/src/registry-file.mjs b/src/registry-file.mjs new file mode 100644 index 0000000..8f152be --- /dev/null +++ b/src/registry-file.mjs @@ -0,0 +1,66 @@ +// File handling for the Copilot CLI's native BYOK registry (providers.json). +// +// Native mode writes a temporary registry for the session and deletes it on +// exit. Legacy mode uses the same writer to hand the CLI an empty registry, so +// a providers.json in the user's config directory cannot silently override the +// COPILOT_PROVIDER_* variables that mode is built on. + +import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'; +import { homedir, tmpdir } from 'node:os'; +import { join } from 'node:path'; + +import { restrictPermissions } from './keystore.mjs'; + +/** Where the CLI looks for the registry, honouring its own overrides. */ +export function nativeRegistryPath(env = process.env) { + if (env.COPILOT_PROVIDERS_CONFIG) return env.COPILOT_PROVIDERS_CONFIG; + const home = env.COPILOT_HOME || join(homedir(), '.copilot'); + return join(home, 'providers.json'); +} + +/** + * Reads the registry the CLI would load. Never returns credentials. + * @returns {Promise<{path: string, present: boolean, providers: number, models: number, error?: string}>} + */ +export async function inspectNativeRegistry(env = process.env) { + const path = nativeRegistryPath(env); + let contents; + try { + contents = await readFile(path, 'utf8'); + } catch (error) { + if (error?.code === 'ENOENT') return { path, present: false, providers: 0, models: 0 }; + return { path, present: false, providers: 0, models: 0, error: error.message }; + } + + try { + const parsed = JSON.parse(contents); + return { + path, + present: true, + providers: Array.isArray(parsed?.providers) ? parsed.providers.length : 0, + models: Array.isArray(parsed?.models) ? parsed.models.length : 0, + }; + } catch (error) { + return { path, present: true, providers: 0, models: 0, error: `invalid JSON: ${error.message}` }; + } +} + +/** + * Writes a registry to a private temporary directory. + * @returns {Promise<{path: string, dir: string, cleanup: function(): Promise}>} + */ +export async function createRegistryFile(registry) { + const dir = await mkdtemp(join(tmpdir(), 'copilot-byok-')); + const path = join(dir, 'providers.json'); + + await writeFile(path, `${JSON.stringify(registry, null, 2)}\n`, { encoding: 'utf8', mode: 0o600 }); + await restrictPermissions(path); + + return { + path, + dir, + async cleanup() { + await rm(dir, { recursive: true, force: true }).catch(() => {}); + }, + }; +} diff --git a/src/router.mjs b/src/router.mjs index 3617c64..5a3f4f7 100644 --- a/src/router.mjs +++ b/src/router.mjs @@ -141,6 +141,7 @@ export function mergeModels(payload, entries) { // is stepped down until one is accepted, and the outcome is remembered so the // same round trip is not repeated for the rest of the session. const EFFORT_LADDER = ['max', 'xhigh', 'high', 'medium', 'low', 'minimal', 'none']; +const MAX_EFFORT_ATTEMPTS = EFFORT_LADDER.length + 1; const rejectedEfforts = new Map(); function nextEffortDown(effort) { @@ -148,22 +149,30 @@ function nextEffortDown(effort) { return index === -1 || index === EFFORT_LADDER.length - 1 ? null : EFFORT_LADDER[index + 1]; } -function forwardToProvider({ req, res, payload, route, onEvent }) { +function forwardToProvider({ req, res, payload, route, onEvent, effortOverride, attempt = 0, state = { providerReq: null } }) { const { provider, model } = route; const target = new URL(`${provider.baseUrl.replace(/\/$/, '')}/chat/completions`); const client = target.protocol === 'https:' ? https : http; const outbound = { ...payload, model: model.model }; - // Copilot accepts a reasoning effort but never forwards it to a BYOK provider — - // requests sent with low and high are identical on the wire. Applying the - // configured level here is what makes the setting take effect at all. - const configured = model.reasoningEffort || provider.reasoningEffort; - if (configured && outbound.reasoning_effort === undefined) { - const rejected = rejectedEfforts.get(`${provider.id}:${model.model}`) || new Set(); - let effort = configured; - while (effort && rejected.has(effort)) effort = nextEffortDown(effort); - if (effort) outbound.reasoning_effort = effort; + // The CLI now forwards the level the user picked; keep it, unless this + // provider already rejected that level earlier in the session. A configured + // level applies only when the request carries none. `effortOverride` is what + // the retry below uses to replace a rejected value instead of resending it. + const effortKey = `${provider.id}:${model.model}`; + if (effortOverride !== undefined) { + if (effortOverride === null) delete outbound.reasoning_effort; + else outbound.reasoning_effort = effortOverride; + } else { + const configured = model.reasoningEffort || provider.reasoningEffort; + let effort = outbound.reasoning_effort ?? configured; + if (effort) { + const rejected = rejectedEfforts.get(effortKey) || new Set(); + while (effort && rejected.has(effort)) effort = nextEffortDown(effort); + if (effort) outbound.reasoning_effort = effort; + else delete outbound.reasoning_effort; + } } const body = Buffer.from(JSON.stringify(outbound), 'utf8'); @@ -190,12 +199,14 @@ function forwardToProvider({ req, res, payload, route, onEvent }) { }, (providerRes) => { // A 400 caused by the effort is recoverable: drop a level and try again, - // before a single byte has reached the client. - if (providerRes.statusCode === 400 && outbound.reasoning_effort) { - const key = `${provider.id}:${model.model}`; - const rejected = rejectedEfforts.get(key) || new Set(); + // before a single byte has reached the client. The rejected level is + // remembered for the session, and the retry must carry the replacement: + // the CLI sends its own level now, so replaying the original payload + // would resend the same value and loop forever. + if (providerRes.statusCode === 400 && outbound.reasoning_effort && attempt < MAX_EFFORT_ATTEMPTS) { + const rejected = rejectedEfforts.get(effortKey) || new Set(); rejected.add(outbound.reasoning_effort); - rejectedEfforts.set(key, rejected); + rejectedEfforts.set(effortKey, rejected); const fallback = nextEffortDown(outbound.reasoning_effort); onEvent({ @@ -206,7 +217,16 @@ function forwardToProvider({ req, res, payload, route, onEvent }) { }); providerRes.resume(); // discard the error body - return forwardToProvider({ req, res, payload, route, onEvent }); + return forwardToProvider({ + req, + res, + payload, + route, + onEvent, + effortOverride: fallback, + attempt: attempt + 1, + state, + }); } res.writeHead(providerRes.statusCode, providerRes.headers); @@ -220,6 +240,8 @@ function forwardToProvider({ req, res, payload, route, onEvent }) { } ); + state.providerReq = providerReq; + providerReq.setTimeout(PROVIDER_IDLE_TIMEOUT_MS, () => { providerReq.destroy(new Error(`no data for ${Math.round(PROVIDER_IDLE_TIMEOUT_MS / 1000)}s`)); }); @@ -233,9 +255,13 @@ function forwardToProvider({ req, res, payload, route, onEvent }) { // A cancelled generation must stop the provider request too: otherwise it keeps // streaming, and paid providers keep billing, for an answer nobody will read. - res.on('close', () => { - if (!res.writableEnded) providerReq.destroy(); - }); + // Attached once per client request: retries replace the active provider + // request, and a listener per attempt would pile up on the response. + if (attempt === 0) { + res.on('close', () => { + if (!res.writableEnded) state.providerReq?.destroy(); + }); + } providerReq.end(body); } diff --git a/src/upstream.mjs b/src/upstream.mjs index ea68e68..f6c92db 100644 --- a/src/upstream.mjs +++ b/src/upstream.mjs @@ -12,6 +12,29 @@ export const UPSTREAM_CANDIDATES = [ 'https://api.enterprise.githubcopilot.com', ]; +const GHE_HOST_PATTERN = /^[a-z0-9-]+(\.[a-z0-9-]+)*\.ghe\.com$/i; + +/** + * GitHub Enterprise Cloud with data residency serves the Copilot API from the + * tenant host, not from the public api.githubcopilot.com hosts. The CLI reads + * COPILOT_GH_HOST (or GH_HOST) for its own requests, so the router derives the + * matching origin from the same variables. + */ +export function dataResidencyOrigin(env = process.env) { + const raw = env.COPILOT_GH_HOST || env.GH_HOST; + if (!raw) return null; + + let host; + try { + host = new URL(/^[a-z][a-z0-9+.-]*:\/\//i.test(raw) ? raw : `https://${raw}`).hostname.toLowerCase(); + } catch { + return null; + } + + if (!GHE_HOST_PATTERN.test(host)) return null; + return `https://copilot-api.${host}`; +} + /** * @param {object} options * @param {string} [options.explicit] Origin set by hand: skips detection. diff --git a/test/args.test.mjs b/test/args.test.mjs index 4bd14df..fe04e1f 100644 --- a/test/args.test.mjs +++ b/test/args.test.mjs @@ -86,3 +86,15 @@ test('--upstream must be an http(s) URL and is reduced to its origin', () => { assert.throws(() => parseArgs(['--upstream', 'business']), /must be a URL/); assert.throws(() => parseArgs(['--upstream', 'ftp://example.com']), /must be an http\(s\) URL/); }); + +test('--native-byok is parsed and cannot be combined with the other modes', () => { + const parsed = parseArgs(['--native-byok', '--no-discovery', '--model', 'byok-echo-m1', '-p', 'hello']); + assert.equal(parsed.nativeByok, true); + assert.equal(parsed.noDiscovery, true); + assert.equal(parsed.explicitModel, 'byok-echo-m1'); + assert.deepEqual(parsed.copilotArgs, ['-p', 'hello']); + + assert.throws(() => parseArgs(['--native-byok', '--legacy']), /cannot be combined with --legacy/); + assert.throws(() => parseArgs(['--native-byok', '--native']), /cannot be combined with --native/); + assert.throws(() => parseArgs(['--native-byok', '--upstream', 'https://api.example.com']), /only to router mode/); +}); diff --git a/test/catalog.test.mjs b/test/catalog.test.mjs index 309824c..5fd57f9 100644 --- a/test/catalog.test.mjs +++ b/test/catalog.test.mjs @@ -120,31 +120,38 @@ test('models whose names collide are reported rather than dropped in silence', ( assert.match(events[0].message, /collide as byok-p-gpt-4-1/); }); -test('the effort selector is declared only when explicitly enabled', () => { +test('the effort selector is declared for reasoning models unless disabled', () => { const base = { id: 'p', name: 'P', baseUrl: 'https://x.test/v1', apiKey: 'k' }; - const off = buildCatalog([{ ...base, models: [{ model: 'm', reasoning: true }] }]); - assert.equal(off.entries[0].capabilities.supports.reasoning_effort, undefined, - 'a selector that changes nothing must not appear'); + const on = buildCatalog([{ ...base, models: [{ model: 'm', reasoning: true }] }]); + assert.deepEqual(on.entries[0].capabilities.supports.reasoning_effort, ['low', 'medium', 'high'], + 'current CLIs forward the level, so the selector is meaningful by default'); - const on = buildCatalog([{ ...base, models: [{ model: 'm', reasoning: true, reasoningEffortPicker: true }] }]); - assert.deepEqual(on.entries[0].capabilities.supports.reasoning_effort, ['low', 'medium', 'high']); + const off = buildCatalog([{ ...base, models: [{ model: 'm', reasoning: true, reasoningEffortPicker: false }] }]); + assert.equal(off.entries[0].capabilities.supports.reasoning_effort, undefined, + 'the selector can still be turned off explicitly'); const custom = buildCatalog([{ ...base, - models: [{ model: 'm', reasoning: true, reasoningEffortPicker: true, reasoningEffortLevels: ['low', 'high', 'max'] }], + models: [{ model: 'm', reasoning: true, reasoningEffortLevels: ['low', 'high', 'max'] }], }]); assert.deepEqual(custom.entries[0].capabilities.supports.reasoning_effort, ['low', 'high', 'max']); }); test('the provider-level switch applies to its models', () => { - const { entries } = buildCatalog([{ + const enabled = buildCatalog([{ id: 'p', name: 'P', baseUrl: 'https://x.test/v1', apiKey: 'k', - reasoningEffortPicker: true, models: [{ model: 'thinker', reasoning: true }, { model: 'plain' }], }]); - - assert.ok(entries.find((e) => e.id.endsWith('thinker')).capabilities.supports.reasoning_effort); - assert.equal(entries.find((e) => e.id.endsWith('plain')).capabilities.supports.reasoning_effort, undefined, + assert.ok(enabled.entries.find((e) => e.id.endsWith('thinker')).capabilities.supports.reasoning_effort); + assert.equal(enabled.entries.find((e) => e.id.endsWith('plain')).capabilities.supports.reasoning_effort, undefined, 'a model that does not reason never gets the selector'); + + const disabled = buildCatalog([{ + id: 'q', name: 'Q', baseUrl: 'https://x.test/v1', apiKey: 'k', + reasoningEffortPicker: false, + models: [{ model: 'thinker', reasoning: true }], + }]); + assert.equal(disabled.entries[0].capabilities.supports.reasoning_effort, undefined, + 'a provider-level false disables it for every model'); }); diff --git a/test/cli.test.mjs b/test/cli.test.mjs index fc5bf6f..ffaf84c 100644 --- a/test/cli.test.mjs +++ b/test/cli.test.mjs @@ -1,7 +1,7 @@ import assert from 'node:assert/strict'; -import { mkdtemp, readFile, writeFile } from 'node:fs/promises'; +import { mkdtemp, readFile, stat, writeFile } from 'node:fs/promises'; import { tmpdir } from 'node:os'; -import { join } from 'node:path'; +import { dirname, join } from 'node:path'; import { Readable, Writable } from 'node:stream'; import test from 'node:test'; @@ -650,6 +650,77 @@ test('explains how to install Copilot CLI when the binary is missing', async () ); }); +test('native BYOK dry-run publishes selection ids and translates picker ids', async () => { + const dir = await mkdtemp(join(tmpdir(), 'copilot-byok-native-byok-')); + const configPath = join(dir, 'providers.json'); + await writeFile(configPath, JSON.stringify({ + providers: [{ + id: 'echo', + name: 'Echo', + baseUrl: 'http://127.0.0.1:1/v1', + authRequired: false, + models: [{ model: 'm1', contextWindow: 128000, maxOutputTokens: 32000 }], + }], + })); + + const output = captureWritable(); + const exitCode = await main(['--native-byok', '--no-discovery', '--dry-run', '--model', 'byok-echo-m1', '--config', configPath], { + stdin: { isTTY: false }, + stdout: output, + stderr: captureWritable(), + env: { + COPILOT_BYOK_KEYSTORE: join(dir, 'keys.json'), + COPILOT_BYOK_MODEL_CACHE: join(dir, 'models.json'), + }, + }); + + assert.equal(exitCode, 0); + const plan = JSON.parse(output.text()); + assert.equal(plan.mode, 'native-byok'); + assert.deepEqual(plan.providers, ['echo']); + assert.deepEqual(plan.models, ['echo/m1']); + assert.deepEqual(plan.args, ['--model', 'echo/m1']); +}); + +test('legacy mode isolates a native registry that would take precedence', async () => { + const home = await mkdtemp(join(tmpdir(), 'copilot-byok-legacy-home-')); + const dir = await mkdtemp(join(tmpdir(), 'copilot-byok-legacy-')); + const configPath = join(dir, 'providers.json'); + await writeFile(configPath, JSON.stringify({ + providers: [{ + id: 'echo', + name: 'Echo', + baseUrl: 'http://127.0.0.1:1/v1', + authRequired: false, + }], + })); + await writeFile(join(home, 'providers.json'), JSON.stringify({ + providers: [{ name: 'native', baseUrl: 'https://api.example.com/v1' }], + models: [{ id: 'm', provider: 'native' }], + })); + + const output = captureWritable(); + const stderr = captureWritable(); + const exitCode = await main(['--legacy', '--provider', 'echo', '--model', 'm1', '--dry-run', '--config', configPath], { + stdin: { isTTY: false }, + stdout: output, + stderr, + env: { + COPILOT_HOME: home, + COPILOT_BYOK_KEYSTORE: join(dir, 'keys.json'), + COPILOT_BYOK_MODEL_CACHE: join(dir, 'models.json'), + }, + }); + + assert.equal(exitCode, 0); + assert.match(stderr.text(), /would take precedence over COPILOT_PROVIDER_\*/); + const plan = JSON.parse(output.text()); + assert.ok(plan.env.COPILOT_PROVIDERS_CONFIG, 'the child gets an empty registry'); + const registryDir = dirname(plan.env.COPILOT_PROVIDERS_CONFIG); + await assert.rejects(() => stat(plan.env.COPILOT_PROVIDERS_CONFIG)); + await assert.rejects(() => stat(registryDir)); +}); + function captureWritable({ isTTY = false } = {}) { let buffer = ''; const stream = new Writable({ diff --git a/test/config.test.mjs b/test/config.test.mjs index 7a36734..46877e1 100644 --- a/test/config.test.mjs +++ b/test/config.test.mjs @@ -308,6 +308,54 @@ test('rejects invalid additive provider option values', async () => { await assert.rejects(() => loadConfigFromPath(configPath, {}), /modelIncludePrefixes|transport|authRequired|modelsAuth/); }); +test('normalizes a rotating credential command', async () => { + const dir = await mkdtemp(join(tmpdir(), 'copilot-byok-')); + const configPath = join(dir, 'providers.json'); + await writeFile(configPath, JSON.stringify({ + providers: [{ + id: 'gateway', + name: 'Gateway', + baseUrl: 'https://gateway.example.com/v1', + apiKeyCommand: ' vault read -field=key secret/llm ', + }], + })); + + const provider = (await loadConfigFromPath(configPath, {})).providers[0]; + assert.equal(provider.apiKeyCommand, 'vault read -field=key secret/llm'); + assert.equal(provider.apiKey, null, 'the command is passed through, never executed here'); +}); + +test('rejects a non-boolean reasoningEffortPicker', async () => { + const dir = await mkdtemp(join(tmpdir(), 'copilot-byok-')); + const configPath = join(dir, 'providers.json'); + await writeFile(configPath, JSON.stringify({ + providers: [{ + id: 'picker', + name: 'Picker', + baseUrl: 'https://api.example.com/v1', + reasoningEffortPicker: 'yes', + models: [{ model: 'm', reasoning: true, reasoningEffortPicker: 1 }], + }], + })); + + await assert.rejects(() => loadConfigFromPath(configPath, {}), /reasoningEffortPicker must be a boolean/); +}); + +test('rejects header values with line breaks', async () => { + const dir = await mkdtemp(join(tmpdir(), 'copilot-byok-')); + const configPath = join(dir, 'providers.json'); + await writeFile(configPath, JSON.stringify({ + providers: [{ + id: 'header-injection', + name: 'Header Injection', + baseUrl: 'https://api.example.com/v1', + headers: { 'X-Team': 'a\nX-Admin: true' }, + }], + })); + + await assert.rejects(() => loadConfigFromPath(configPath, {}), /must not contain line breaks/); +}); + test('rejects an OpenAI wire API override on Anthropic providers', async () => { const dir = await mkdtemp(join(tmpdir(), 'copilot-byok-')); const configPath = join(dir, 'providers.json'); diff --git a/test/doctor.test.mjs b/test/doctor.test.mjs new file mode 100644 index 0000000..2991b66 --- /dev/null +++ b/test/doctor.test.mjs @@ -0,0 +1,116 @@ +import assert from 'node:assert/strict'; +import { mkdtemp, writeFile } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { Writable } from 'node:stream'; +import test from 'node:test'; + +import { runDoctorCommand } from '../src/doctor.mjs'; + +test('reports a healthy installation as ok', async () => { + const { io, configPath } = await fixture(); + const stdout = captureWritable(); + + const exitCode = await runDoctorCommand({ + argv: ['--config', configPath], + io: { ...io, stdout }, + probeVersion: () => '1.0.105', + }); + + assert.equal(exitCode, 0); + const text = stdout.text(); + assert.match(text, /copilot: 1\.0\.105/); + assert.match(text, /config: 1 providers/); + assert.match(text, /providers: 1 of 1 have a credential/); + assert.match(text, /native registry: .*not present/); + assert.match(text, /No failures\./); +}); + +test('--json exposes the same checks', async () => { + const { io, configPath } = await fixture(); + const stdout = captureWritable(); + + const exitCode = await runDoctorCommand({ + argv: ['--json', '--config', configPath], + io: { ...io, stdout }, + probeVersion: () => '1.0.105', + }); + + assert.equal(exitCode, 0); + const report = JSON.parse(stdout.text()); + assert.equal(report.ok, true); + assert.ok(Array.isArray(report.checks)); + assert.ok(report.checks.some((check) => check.name === 'node' && check.level === 'ok')); +}); + +test('a broken configuration fails the doctor', async () => { + const { io } = await fixture(); + const dir = await mkdtemp(join(tmpdir(), 'copilot-byok-doctor-')); + const configPath = join(dir, 'broken.json'); + await writeFile(configPath, '{broken'); + + const stdout = captureWritable(); + const exitCode = await runDoctorCommand({ + argv: ['--config', configPath], + io: { ...io, stdout }, + probeVersion: () => '1.0.105', + }); + + assert.equal(exitCode, 1); + assert.match(stdout.text(), /FAIL config:/); + assert.match(stdout.text(), /1 check\(s\) failed/); +}); + +test('a missing Copilot binary is a warning, not a failure', async () => { + const { io, configPath } = await fixture(); + const stdout = captureWritable(); + + const exitCode = await runDoctorCommand({ + argv: ['--config', configPath], + io: { ...io, stdout }, + probeVersion: () => null, + }); + + assert.equal(exitCode, 0); + assert.match(stdout.text(), /WARN copilot:/); +}); + +async function fixture() { + const dir = await mkdtemp(join(tmpdir(), 'copilot-byok-doctor-')); + const configPath = join(dir, 'providers.json'); + await writeFile(configPath, JSON.stringify({ + providers: [{ + id: 'echo', + name: 'Echo', + baseUrl: 'http://127.0.0.1:1/v1', + authRequired: false, + models: [{ model: 'm1' }], + }], + })); + + const env = { + COPILOT_BIN: 'copilot', + COPILOT_HOME: join(dir, 'copilot-home'), + COPILOT_BYOK_KEYSTORE: join(dir, 'keys.json'), + COPILOT_BYOK_MODEL_CACHE: join(dir, 'models.json'), + COPILOT_BYOK_LOG: join(dir, 'router.log'), + COPILOT_BYOK_SHIM_DIR: join(dir, 'shim'), + }; + + return { + configPath, + io: { stdout: captureWritable(), stderr: captureWritable(), env, stdin: { isTTY: false } }, + }; +} + +function captureWritable() { + let buffer = ''; + const stream = new Writable({ + write(chunk, encoding, callback) { + buffer += chunk.toString(); + callback(); + }, + }); + stream.text = () => buffer; + return stream; +} diff --git a/test/models-command.test.mjs b/test/models-command.test.mjs new file mode 100644 index 0000000..c5cc8a3 --- /dev/null +++ b/test/models-command.test.mjs @@ -0,0 +1,102 @@ +import assert from 'node:assert/strict'; +import { mkdtemp, writeFile } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { Writable } from 'node:stream'; +import test from 'node:test'; + +import { runModelsCommand } from '../src/models-command.mjs'; + +test('lists pinned models without touching the network in offline mode', async () => { + const { configPath, env } = await fixture(); + const stdout = captureWritable(); + + const exitCode = await runModelsCommand({ + argv: ['--offline', '--config', configPath], + io: { stdout, stderr: captureWritable(), env }, + }); + + assert.equal(exitCode, 0); + const text = stdout.text(); + assert.match(text, /echo {2}\(1 models, offline lists\)/); + assert.match(text, /m1 {2}\(128k context, reasoning\)/); + assert.match(text, /keyed {2}\(1 models, offline lists\)/); +}); + +test('--json is machine-readable and can be filtered by provider', async () => { + const { configPath, env } = await fixture(); + const stdout = captureWritable(); + + const exitCode = await runModelsCommand({ + argv: ['--offline', '--json', 'echo', '--config', configPath], + io: { stdout, stderr: captureWritable(), env }, + }); + + assert.equal(exitCode, 0); + const rows = JSON.parse(stdout.text()); + assert.equal(rows.length, 1); + assert.equal(rows[0].id, 'echo'); + assert.deepEqual(rows[0].models, [{ + id: 'm1', + label: null, + contextWindow: 128000, + vision: false, + reasoning: true, + }]); +}); + +test('an unknown provider is rejected', async () => { + const { configPath, env } = await fixture(); + + await assert.rejects( + () => runModelsCommand({ + argv: ['--offline', 'missing', '--config', configPath], + io: { stdout: captureWritable(), stderr: captureWritable(), env }, + }), + /Unknown provider: missing/, + ); +}); + +async function fixture() { + const dir = await mkdtemp(join(tmpdir(), 'copilot-byok-models-')); + const configPath = join(dir, 'providers.json'); + await writeFile(configPath, JSON.stringify({ + providers: [ + { + id: 'echo', + name: 'Echo', + baseUrl: 'http://127.0.0.1:1/v1', + authRequired: false, + models: [{ model: 'm1', contextWindow: 128000, reasoning: true }], + }, + { + id: 'keyed', + name: 'Keyed', + baseUrl: 'https://api.example.com/v1', + apiKeyEnv: 'TEST_PROVIDER_KEY', + models: [{ model: 'm2' }], + }, + ], + })); + + return { + configPath, + env: { + TEST_PROVIDER_KEY: 'secret', + COPILOT_BYOK_KEYSTORE: join(dir, 'keys.json'), + COPILOT_BYOK_MODEL_CACHE: join(dir, 'models.json'), + }, + }; +} + +function captureWritable() { + let buffer = ''; + const stream = new Writable({ + write(chunk, encoding, callback) { + buffer += chunk.toString(); + callback(); + }, + }); + stream.text = () => buffer; + return stream; +} diff --git a/test/native-registry.test.mjs b/test/native-registry.test.mjs new file mode 100644 index 0000000..3f4ef30 --- /dev/null +++ b/test/native-registry.test.mjs @@ -0,0 +1,90 @@ +import assert from 'node:assert/strict'; +import test from 'node:test'; + +import { buildNativeRegistry, nativeSelectionId, nativeSelectionMap } from '../src/native-registry.mjs'; + +const providers = [{ + id: 'acme', + name: 'Acme', + type: 'openai', + baseUrl: 'https://api.acme.test/v1', + apiKey: 'secret', + wireApi: 'responses', + headers: { 'X-Tenant': 'team-a' }, + models: [{ + model: 'reasoner', + label: 'Acme Reasoner', + contextWindow: 200_000, + maxOutputTokens: 50_000, + reasoning: true, + vision: true, + }], +}]; + +test('builds a providers.json registry from resolved providers', () => { + const registry = buildNativeRegistry(providers); + + assert.deepEqual(registry.providers, [{ + name: 'acme', + type: 'openai', + baseUrl: 'https://api.acme.test/v1', + apiKey: 'secret', + wireApi: 'responses', + headers: { 'X-Tenant': 'team-a' }, + }]); + + const [model] = registry.models; + assert.equal(model.id, 'reasoner'); + assert.equal(model.provider, 'acme'); + assert.equal(model.name, 'Acme Reasoner'); + assert.equal(model.maxContextWindowTokens, 200_000); + assert.equal(model.maxOutputTokens, 50_000); + assert.equal(model.maxPromptTokens, 150_000); + assert.equal(model.capabilities.supports.toolCalls, true); + assert.equal(model.capabilities.supports.streaming, true); + assert.equal(model.capabilities.supports.vision, true); + assert.deepEqual(model.capabilities.supports.reasoning_effort, ['low', 'medium', 'high']); +}); + +test('the model id can be disabled explicitly and bearer tokens win', () => { + const registry = buildNativeRegistry([{ + id: 'gateway', + name: 'Gateway', + baseUrl: 'https://gateway.test/v1', + bearerToken: 'bearer', + apiKey: 'ignored', + azureApiVersion: '2024-10-21', + models: [{ + model: 'plain', + reasoning: true, + reasoningEffortPicker: false, + reasoningEffortLevels: ['low', 'high'], + }], + }]); + + assert.equal(registry.providers[0].bearerToken, 'bearer'); + assert.equal(Object.hasOwn(registry.providers[0], 'apiKey'), false); + assert.deepEqual(registry.providers[0].azure, { apiVersion: '2024-10-21' }); + assert.equal(registry.models[0].capabilities.supports.reasoning_effort, undefined); +}); + +test('authless providers carry no credential field', () => { + const registry = buildNativeRegistry([{ + id: 'local', + name: 'Local', + baseUrl: 'http://127.0.0.1:11434/v1', + authRequired: false, + models: [{ model: 'm' }], + }]); + + assert.equal(Object.hasOwn(registry.providers[0], 'apiKey'), false); + assert.equal(Object.hasOwn(registry.providers[0], 'bearerToken'), false); +}); + +test('selection ids follow PROVIDER/MODEL and map back to the source entries', () => { + const map = nativeSelectionMap(providers); + assert.equal(nativeSelectionId('acme', 'reasoner'), 'acme/reasoner'); + assert.ok(map.has('acme/reasoner')); + assert.equal(map.get('acme/reasoner').provider.id, 'acme'); + assert.equal(map.get('acme/reasoner').model.model, 'reasoner'); +}); diff --git a/test/provider-env.test.mjs b/test/provider-env.test.mjs index 9c5b5ca..7d1abfa 100644 --- a/test/provider-env.test.mjs +++ b/test/provider-env.test.mjs @@ -18,8 +18,8 @@ test('separates Copilot catalog model from provider wire model', () => { assert.equal(env.COPILOT_PROVIDER_BASE_URL, 'https://llm.chutes.ai/v1'); assert.equal(env.COPILOT_PROVIDER_API_KEY, 'secret'); assert.equal(env.COPILOT_MODEL, 'gpt-4.1'); + assert.equal(env.COPILOT_PROVIDER_MODEL_ID, 'gpt-4.1'); assert.equal(env.COPILOT_PROVIDER_WIRE_MODEL, 'moonshotai/Kimi-K2.6-TEE'); - assert.equal(Object.hasOwn(env, 'COPILOT_PROVIDER_MODEL_ID'), false); }); test('passes explicit token limits when configured', () => { @@ -73,3 +73,19 @@ test('prefers bearer authentication and passes transport-specific settings', () assert.equal(env.COPILOT_PROVIDER_TRANSPORT, 'websockets'); assert.equal(env.COPILOT_PROVIDER_AZURE_API_VERSION, '2025-04-01-preview'); }); + +test('passes gateway headers and a rotating credential command', () => { + const env = buildProviderEnvironment({ + provider: { + type: 'openai', + baseUrl: 'https://gateway.example.com/v1', + apiKey: 'secret', + apiKeyCommand: 'vault read -field=key secret/llm', + headers: { 'X-Tenant': 'team-a', 'X-Project': 'copilot' }, + }, + wireModel: 'custom-model', + }); + + assert.equal(env.COPILOT_PROVIDER_API_KEY_COMMAND, 'vault read -field=key secret/llm'); + assert.equal(env.COPILOT_PROVIDER_HEADERS, 'X-Tenant: team-a\nX-Project: copilot'); +}); diff --git a/test/registry-file.test.mjs b/test/registry-file.test.mjs new file mode 100644 index 0000000..7babc15 --- /dev/null +++ b/test/registry-file.test.mjs @@ -0,0 +1,71 @@ +import assert from 'node:assert/strict'; +import { mkdtemp, readFile, writeFile } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import test from 'node:test'; + +import { + createRegistryFile, + inspectNativeRegistry, + nativeRegistryPath, +} from '../src/registry-file.mjs'; + +test('writes a registry to a private temp directory and removes it on cleanup', async () => { + const registry = { + providers: [{ name: 'acme', baseUrl: 'https://api.acme.test/v1', apiKey: 'secret' }], + models: [{ id: 'm', provider: 'acme' }], + }; + + const file = await createRegistryFile(registry); + try { + assert.deepEqual(JSON.parse(await readFile(file.path, 'utf8')), registry); + } finally { + await file.cleanup(); + } + + await assert.rejects(() => readFile(file.path, 'utf8')); +}); + +test('honours COPILOT_PROVIDERS_CONFIG and counts the registry entries', async () => { + const dir = await mkdtemp(join(tmpdir(), 'copilot-byok-registry-')); + const path = join(dir, 'custom.json'); + await writeFile(path, JSON.stringify({ + providers: [{ name: 'acme', baseUrl: 'https://api.acme.test/v1' }], + models: [{ id: 'm', provider: 'acme' }, { id: 'm2', provider: 'acme' }], + })); + + const inspected = await inspectNativeRegistry({ COPILOT_PROVIDERS_CONFIG: path }); + assert.equal(inspected.path, path); + assert.equal(inspected.present, true); + assert.equal(inspected.providers, 1); + assert.equal(inspected.models, 2); +}); + +test('an absent registry is reported as absent', async () => { + const dir = await mkdtemp(join(tmpdir(), 'copilot-byok-registry-')); + const inspected = await inspectNativeRegistry({ COPILOT_PROVIDERS_CONFIG: join(dir, 'missing.json') }); + assert.equal(inspected.present, false); + assert.equal(inspected.providers, 0); + assert.equal(inspected.models, 0); +}); + +test('invalid JSON is reported instead of thrown', async () => { + const dir = await mkdtemp(join(tmpdir(), 'copilot-byok-registry-')); + const path = join(dir, 'broken.json'); + await writeFile(path, '{broken'); + + const inspected = await inspectNativeRegistry({ COPILOT_PROVIDERS_CONFIG: path }); + assert.equal(inspected.present, true); + assert.match(inspected.error, /invalid JSON/); +}); + +test('the default path follows COPILOT_HOME', () => { + assert.equal( + nativeRegistryPath({ COPILOT_HOME: 'C:\\copilot-home' }), + join('C:\\copilot-home', 'providers.json'), + ); + assert.equal( + nativeRegistryPath({ COPILOT_PROVIDERS_CONFIG: '/tmp/custom.json' }), + '/tmp/custom.json', + ); +}); diff --git a/test/router.test.mjs b/test/router.test.mjs index 8d583c2..61e3250 100644 --- a/test/router.test.mjs +++ b/test/router.test.mjs @@ -406,14 +406,14 @@ test('when the CLI starts sending an effort, the user choice wins over the confi const url = `http://127.0.0.1:${router.address().port}/chat/completions`; try { - // Today: the CLI sends nothing, so the configured level is applied. + // The CLI used to send nothing, so the configured level is applied. await fetch(url, { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ model: 'byok-acme-thinker', messages: [] }), }); - // After the CLI is fixed: what the user picked must not be overridden. + // The CLI forwards what the user picked: it must not be overridden. await fetch(url, { method: 'POST', headers: { 'content-type': 'application/json' }, @@ -427,3 +427,102 @@ test('when the CLI starts sending an effort, the user choice wins over the confi await new Promise((resolve) => provider.close(resolve)); } }); + +test('a caller-sent effort the provider rejects steps down instead of looping', async () => { + const attempts = []; + const provider = http.createServer((req, res) => { + let body = ''; + req.on('data', (chunk) => { body += chunk; }); + req.on('end', () => { + const parsed = JSON.parse(body); + attempts.push(parsed.reasoning_effort); + if (parsed.reasoning_effort === 'xhigh') { + res.writeHead(400, { 'content-type': 'application/json' }); + return res.end(JSON.stringify({ detail: 'Invalid request: reasoning_effort' })); + } + res.writeHead(200, { 'content-type': 'application/json' }); + res.end(JSON.stringify({ ok: true })); + }); + }); + await new Promise((resolve) => provider.listen(0, '127.0.0.1', resolve)); + + const events = []; + const catalog = buildCatalog([{ + id: 'beta', name: 'Beta', baseUrl: `http://127.0.0.1:${provider.address().port}/v1`, + apiKey: 'k', models: [{ model: 'cli-effort' }], + }]); + + const router = createRouter({ + catalog, + upstream: createUpstreamResolver({ explicit: 'https://upstream.invalid' }), + onEvent: (event) => events.push(event), + }); + await new Promise((resolve) => router.listen(0, '127.0.0.1', resolve)); + const url = `http://127.0.0.1:${router.address().port}/chat/completions`; + + try { + const response = await fetch(url, { + method: 'POST', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ model: 'byok-beta-cli-effort', messages: [], reasoning_effort: 'xhigh' }), + }); + + assert.equal(response.status, 200); + assert.deepEqual(attempts, ['xhigh', 'high'], 'the retry carries the fallback level'); + + // Remembered: the same pick goes straight to the level that worked. + attempts.length = 0; + await fetch(url, { + method: 'POST', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ model: 'byok-beta-cli-effort', messages: [], reasoning_effort: 'xhigh' }), + }); + assert.deepEqual(attempts, ['high'], 'no repeated round trip for a rejected level'); + } finally { + router.closeAllConnections(); + await new Promise((resolve) => router.close(resolve)); + await new Promise((resolve) => provider.close(resolve)); + } +}); + +test('an effort every level rejects ends with the provider error', async () => { + const attempts = []; + const provider = http.createServer((req, res) => { + let body = ''; + req.on('data', (chunk) => { body += chunk; }); + req.on('end', () => { + const parsed = JSON.parse(body); + attempts.push(parsed.reasoning_effort); + res.writeHead(400, { 'content-type': 'application/json' }); + res.end(JSON.stringify({ detail: 'Invalid request: reasoning_effort' })); + }); + }); + await new Promise((resolve) => provider.listen(0, '127.0.0.1', resolve)); + + const catalog = buildCatalog([{ + id: 'gamma', name: 'Gamma', baseUrl: `http://127.0.0.1:${provider.address().port}/v1`, + apiKey: 'k', models: [{ model: 'picky' }], + }]); + + const router = createRouter({ + catalog, + upstream: createUpstreamResolver({ explicit: 'https://upstream.invalid' }), + }); + await new Promise((resolve) => router.listen(0, '127.0.0.1', resolve)); + const url = `http://127.0.0.1:${router.address().port}/chat/completions`; + + try { + const response = await fetch(url, { + method: 'POST', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ model: 'byok-gamma-picky', messages: [], reasoning_effort: 'xhigh' }), + }); + + assert.equal(response.status, 400, 'the provider error reaches the client'); + assert.deepEqual(attempts, ['xhigh', 'high', 'medium', 'low', 'minimal', 'none', undefined]); + } finally { + router.closeAllConnections(); + await new Promise((resolve) => router.close(resolve)); + await new Promise((resolve) => provider.close(resolve)); + } +}); diff --git a/test/upstream.test.mjs b/test/upstream.test.mjs index bac538e..b4d5add 100644 --- a/test/upstream.test.mjs +++ b/test/upstream.test.mjs @@ -1,7 +1,7 @@ import assert from 'node:assert/strict'; import test from 'node:test'; -import { createUpstreamResolver, UPSTREAM_CANDIDATES } from '../src/upstream.mjs'; +import { createUpstreamResolver, dataResidencyOrigin, UPSTREAM_CANDIDATES } from '../src/upstream.mjs'; test('an explicit origin skips detection entirely', async () => { let calls = 0; @@ -85,3 +85,26 @@ test('a definitive rejection from every candidate is cached', async () => { assert.equal(calls, UPSTREAM_CANDIDATES.length, 'the servers answered, so there is nothing to retry'); }); + +test('a data residency tenant derives its Copilot API origin', () => { + assert.equal( + dataResidencyOrigin({ COPILOT_GH_HOST: 'acme.ghe.com' }), + 'https://copilot-api.acme.ghe.com', + ); + assert.equal( + dataResidencyOrigin({ GH_HOST: 'https://acme.ghe.com/' }), + 'https://copilot-api.acme.ghe.com', + ); + assert.equal( + dataResidencyOrigin({ COPILOT_GH_HOST: 'acme.ghe.com', GH_HOST: 'github.com' }), + 'https://copilot-api.acme.ghe.com', + 'COPILOT_GH_HOST wins over GH_HOST', + ); +}); + +test('ordinary GitHub hosts do not trigger data residency detection', () => { + assert.equal(dataResidencyOrigin({ GH_HOST: 'github.com' }), null); + assert.equal(dataResidencyOrigin({ GH_HOST: 'ghe.com' }), null); + assert.equal(dataResidencyOrigin({ GH_HOST: 'acme.ghe.com.example.com' }), null); + assert.equal(dataResidencyOrigin({}), null); +});