feat(media): convert HEIC images to JPEG with sips on macOS - #3647
feat(media): convert HEIC images to JPEG with sips on macOS#3647RealKai42 wants to merge 1 commit into
Conversation
HEIC/HEIF is the default photo and screenshot format on Apple devices, but model providers only accept PNG, JPEG, GIF, and WebP, so every ingestion point refused it with a "convert it first" message. On macOS the system `sips` tool decodes HEIC with Apple's codecs, so the engine now converts HEIC to JPEG in-process before the existing format gate, and the converted image flows through the unchanged compression pipeline (resize, byte budget, region crops, originals): - a shared transcoder that only activates on macOS with a host process service, runs sips against a scratch directory, and returns null on any failure so callers keep their previous behavior - ReadMediaFile hands the file path straight to sips on the runtime that owns the file and notes the conversion for the model - the daemon-file image resolver (@file references and pasted uploads) converts and compresses before inlining - kap-server prompt attachments (inline base64, path, uploaded file) accept an image transcoder; the prompt and skill routes wire the macOS one in, keeping the HEIC original next to the compression caption - the TUI clipboard reader converts a copied HEIC file through sips on macOS instead of silently ignoring it Other platforms keep the existing conversion guidance.
🦋 Changeset detectedLatest commit: 5745c0d The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
commit: |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5745c0db9a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const result = run( | ||
| 'sips', | ||
| ['-s', 'format', 'jpeg', '-s', 'formatOptions', '90', path, '--out', target], | ||
| { timeoutMs: SIPS_TIMEOUT_MS }, |
There was a problem hiding this comment.
When sips is slow or wedged, this calls the default synchronous runCommand implementation, which uses spawnSync, from the explicit paste handler before the attachment placeholder is inserted. The entire Node event loop therefore stops processing input, rendering, and cancellation for as long as the 15-second timeout; run the conversion asynchronously or move it into the existing background image-ingestion work.
Useful? React with 👍 / 👎.
| const heicTranscodable = | ||
| fileType.kind === 'image' && | ||
| isHeicMime(fileType.mimeType) && | ||
| canTranscodeHeic({ osKind: env.osKind, process }); |
There was a problem hiding this comment.
hevx brand before gating transcoding
For a valid HEIF/HEIC file whose ftyp major brand is hevx, detectFileType(..., 'media') returns unknown because the core FTYP_IMAGE_BRANDS table omits that brand, so this condition never reaches the new transcoder and ReadMediaFile rejects the same file. The new clipboard detector explicitly recognizes hevx, making behavior inconsistent across paste, upload, and read paths; add it to the shared core sniffer, ideally using one common brand definition.
Useful? React with 👍 / 👎.
| const transcoded = heicTranscodable | ||
| ? await transcodeHeicToJpeg({ path: safePath }, fileType.mimeType, { | ||
| osKind: env.osKind, | ||
| process, | ||
| telemetry: this.telemetry, | ||
| telemetrySource: 'read_media', | ||
| }) |
There was a problem hiding this comment.
When ReadMediaFile is backed by a runtime whose filesystem is not shared with its process host, this passes the runtime path directly to sips even though the file was read and validated through IHostFileSystem. For example, the ACP runtime delegates filesystem reads over ACP but falls back to the server-local process service for non-shell commands such as sips, so a HEIC visible to the ACP client can fail conversion because the same path does not exist on the server. Read the bytes through fs and use the transcoder's byte-input form instead of assuming both abstractions share paths.
Useful? React with 👍 / 👎.
| let mediaType = file.meta.media_type; | ||
| mediaType = resolveEffectiveImageMime(mediaType, data); | ||
| if (!isModelAcceptedImageMime(mediaType)) { | ||
| const transcoded = await options.transcodeImage?.(data, mediaType); |
There was a problem hiding this comment.
When an uploaded HEIC exceeds MAX_IMAGE_DECODE_BYTES, this branch still buffers the complete upload and invokes sips, even though the server-local path branch rejects images over that limit and /api/v1/files permits files up to Number.MAX_SAFE_INTEGER. Previously an unsupported HEIC was only retained as an attachment; the new path now asks an external decoder to expand it and then buffers the JPEG too, allowing a large upload to consume substantial child-process and server memory. Skip conversion and retain the attachment once file.meta.size exceeds the decode limit.
Useful? React with 👍 / 👎.
1 participant
Problem
HEIC/HEIF is the default photo and screenshot format on iPhone and macOS, but model providers only accept PNG, JPEG, GIF, and WebP. Every ingestion point therefore refused HEIC:
ReadMediaFileanswered with a "convert it to JPEG first" message, the agent had to runsipsitself through Bash (leaving a stray.jpgnext to the user's file),@filereferences and prompt attachments degraded to a text notice, and a HEIC file pasted into the TUI was silently ignored.What changed
macOS ships
sips, which decodes HEIC with Apple's system codecs, so on macOS the conversion now happens in-process before the existing format gate. The converted JPEG then flows through the unchanged compression pipeline (edge cap, byte budget,regioncrops, persisted originals). Other platforms are untouched and still receive the platform-matched conversion guidance; Windows/Linux support is deliberately out of scope because the only cross-platform decoders are libheif/libde265 wasm builds, which raise LGPL and HEVC patent questions that are being evaluated separately.heic-transcode.ts(agent-core-v2): a shared transcoder that activates only whenosKind === 'macOS'and a host process service is available. It runssips -s format jpeg -s formatOptions 90 <in> --out <tmp>.jpgin amkdtempscratch directory with a 15 s timeout, removes the directory on every exit path, and returnsnullon any failure so each caller keeps its previous behavior.createImageTranscoderbinds the host for callers outside DI. Emits animage_transcodetelemetry event.ReadMediaFile: HEIC passes the gate on a transcode-capable runtime; the file path is handed straight tosipsthrough the runtime lease that owns the file (processis optional on the lease, so runtimes without it fall back to the guidance). The<system>note keeps the original mime and size and states the conversion. A failed conversion returns the original guidance.AgentMediaResolverService(kimi-file://references from@fileand pasted uploads): injectsIHostEnvironment+IHostProcessService, converts HEIC and runs it throughcompressImageForModelbefore inlining; the result is memoized per file id like any other image.resolvePromptMediaFiles: new optionaltranscodeImagehook used by the inline-base64, path, and uploaded-file branches. A converted image keeps its HEIC original (persisted to media-originals, or the source path) and gets the standard compression caption, soReadMediaFileregion readback keeps working. Thepromptsandskillsroutes wire in the macOS transcoder fromcore.accessor.apps/kimi-code):isHeicImageftyp sniffing; on darwin a copied HEIC file is converted throughsips(the same synchronousrunCommandthe module already uses forosascript) before the usualparseImageMetapath. Elsewhere the paste is still declined.image-originals:.heic/.heifextensions for persisted originals.Tests: unit tests drive a fake
IHostProcessServicethat writes the--outfile; real-sipsintegration tests are gated withskipIf(process.platform !== 'darwin')and generate their HEIC fixture at test time viasips -s format heicfrom a PNG, so no binary fixture is committed. Full suites of agent-core-v2, kap-server, and the CLI pass; lint and typecheck are clean.Changeset:
minorfor@moonshot-ai/kimi-code.