Add tool calling to the Core ML provider - #11
Open
ActuallyTaylor wants to merge 2 commits into
Open
Conversation
ActuallyTaylor
force-pushed
the
feature/central-tool-resolution
branch
from
August 3, 2026 19:35
3f72c2c to
6925e25
Compare
ActuallyTaylor
force-pushed
the
feature/coreml-streaming-tool-calling
branch
from
August 3, 2026 19:36
6b93b8b to
574b284
Compare
ActuallyTaylor
force-pushed
the
feature/central-tool-resolution
branch
from
August 3, 2026 19:39
6925e25 to
28e5c10
Compare
ActuallyTaylor
force-pushed
the
feature/coreml-streaming-tool-calling
branch
2 times, most recently
from
August 3, 2026 19:47
21f6bcf to
fef1463
Compare
ActuallyTaylor
changed the base branch from
feature/central-tool-resolution
to
feature/llama-streaming-tool-calling
August 3, 2026 19:48
ActuallyTaylor
force-pushed
the
feature/llama-streaming-tool-calling
branch
from
August 3, 2026 20:03
f0d25d1 to
df732ac
Compare
ActuallyTaylor
force-pushed
the
feature/coreml-streaming-tool-calling
branch
from
August 3, 2026 20:03
fef1463 to
92eebd2
Compare
ActuallyTaylor
force-pushed
the
feature/llama-streaming-tool-calling
branch
from
August 3, 2026 20:07
df732ac to
6cbb803
Compare
ActuallyTaylor
force-pushed
the
feature/coreml-streaming-tool-calling
branch
from
August 3, 2026 20:07
92eebd2 to
a72e53f
Compare
ActuallyTaylor
force-pushed
the
feature/llama-streaming-tool-calling
branch
from
August 3, 2026 20:09
6cbb803 to
7d02c02
Compare
ActuallyTaylor
force-pushed
the
feature/coreml-streaming-tool-calling
branch
2 times, most recently
from
August 3, 2026 20:17
78cbcab to
ad2e937
Compare
ActuallyTaylor
force-pushed
the
feature/llama-streaming-tool-calling
branch
2 times, most recently
from
August 3, 2026 20:18
2cc5490 to
3179cd5
Compare
ActuallyTaylor
force-pushed
the
feature/coreml-streaming-tool-calling
branch
2 times, most recently
from
August 3, 2026 20:40
859a64d to
c61dff5
Compare
ActuallyTaylor
force-pushed
the
feature/llama-streaming-tool-calling
branch
2 times, most recently
from
August 3, 2026 20:44
84269fc to
1c578c9
Compare
ActuallyTaylor
force-pushed
the
feature/coreml-streaming-tool-calling
branch
from
August 3, 2026 20:44
c61dff5 to
360efcb
Compare
ActuallyTaylor
force-pushed
the
feature/llama-streaming-tool-calling
branch
from
August 3, 2026 22:23
1c578c9 to
fe249eb
Compare
ActuallyTaylor
force-pushed
the
feature/coreml-streaming-tool-calling
branch
from
August 3, 2026 22:23
360efcb to
97c09d6
Compare
The Core ML provider forwarded tool specs into the chat template and then ignored the result: generated output was never inspected for tool calls, nothing was executed, and nothing reached the transcript or the tool execution delegate. Both `respond` and `streamResponse` now detect tool calls, execute them, feed the results back as another turn, and loop until the model answers without requesting tools. Core ML models have no structured tool-call channel. swift-transformers renders tool specs into the prompt through the tokenizer's Jinja chat template and stops there — it ships no tool-call parser — so the model's request comes back as ordinary generated text in whatever format its chat template taught it. `CoreMLToolCallParser` therefore supports an explicit, closed set of formats: Hermes/Qwen `<tool_call>` tags, Mistral `[TOOL_CALLS]`, Llama 3.x `<|python_tag|>`, fenced JSON blocks, and bare JSON. The last two are ambiguous with a model simply answering in JSON, so they only count as a tool call when the name matches a tool the session actually has. Observable behavior matches the Anthropic baseline. Tool calls go through the session's `toolExecutionDelegate`, including `.stop` and `.provideOutput`. During streaming, tool activity lands in the transcript as it happens, with `.toolCalls` always appended before the corresponding `.toolOutput`, and a `.stop` decision appends the tool calls, finishes the stream, and executes nothing. Text growth goes through `growStreamingTranscript`, and tool-call markup is held back mid-stream so it never reaches the transcript or the caller. Repeated-tool-call-loop protection and an iteration cap mirror the MLX provider. A caller-supplied `toolsHandler` still wins and is still called exactly as before. When no handler is supplied, the session's tools are now converted with the conventional OpenAI-style function schema that Hugging Face chat templates expect, instead of being silently dropped. This also fixes the free-form generation config, which tool calling depends on. `GenerationConfig.maxLength` defaults to 20 and `generate` stops as soon as the total sequence reaches it, so with the prompt unaccounted for it generated nothing at all for any realistic prompt. `eosTokenId` likewise had no default, so generation never stopped early and decoded past the end of the turn. The constrained-decoding path in this file already set both; the plain path now does the same. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds `withTools` and `streamWithTools` to the existing Core ML suite, modeled on the Anthropic equivalents. `streamWithTools` additionally asserts that tool-call markup never appears in the streamed assistant text. Both follow the suite's existing gating and so need the downloaded Core ML model. Because that model is not available in most environments, this also adds an ungated `CoreMLToolCallParsing` suite covering the tool-call text formats the provider claims to support: Hermes/Qwen tags, Mistral `[TOOL_CALLS]`, Llama `<|python_tag|>`, arguments encoded as a JSON string, the OpenAI function envelope, bare JSON gated on known tool names, the mid-stream hold-back heuristic, loop-detection signatures, and the nine-character alphanumeric tool call ids that Mistral's chat template requires. These need no model and run wherever the CoreML trait is enabled. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ActuallyTaylor
changed the base branch from
feature/llama-streaming-tool-calling
to
main
August 4, 2026 00:10
ActuallyTaylor
force-pushed
the
feature/coreml-streaming-tool-calling
branch
from
August 4, 2026 00:10
97c09d6 to
59b415d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The provider forwarded tool specs into the chat template but never inspected the generated output for tool calls, so nothing was ever executed and nothing reached the transcript or the delegate.
swift-transformers has no tool-call parser —
applyChatTemplate(messages:tools:)only injects tools into the Jinja context, and generation returns token ids. Detection and parsing are entirely on the provider and are necessarily model-format-specific. Because messages are untyped dictionaries, feeding results back works cleanly on Hermes/Qwen, Llama 3.x and Mistral templates.Both
respondandstreamResponsenow run a tool loop (8-iteration cap, repeated-signature guard) with full delegate coverage and transcript ordering. Supported formats are a closed, documented set: Hermes/Qwen<tool_call>, Mistral[TOOL_CALLS], Llama 3.x<|python_tag|>, fenced JSON, and bare JSON — the last two only when the name matches a registered tool. DeepSeek markers and XML-argument formats are explicitly unsupported. Tool call ids are 9-character alphanumeric because Mistral's template rejects anything else.toolsHandlerstill wins and behaves exactly as before; when no handler is supplied, tools are now converted with a standard function schema instead of being silently dropped.Also fixes a pre-existing bug the feature depends on:
GenerationConfig.maxLengthdefaults to 20 and generation stops once the sequence reaches it, so the plain (non-constrained) path produced nothing at all for any prompt longer than 20 tokens.eosTokenIdwas also unset, so generation ran past end of turn. The constrained path already set both.12 ungated parser unit tests. Verified live against the real
apple/mistral-coreml7B Int4 model: 29/29 pass includingwithToolsandstreamWithTools.Based on #4; merge that first.
🤖 Generated with Claude Code