Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 33 additions & 1 deletion docs/packages/cli.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -186,10 +186,41 @@ npx hyperframes capture https://example.com --json

Screenshots: 12
Assets: 45
Dropped: 9 (6 size-floor, 3 cap-reached)
Sections: 15
Fonts: sohne-var
```

`Dropped` is how many assets the page referenced that are **not** in the
folder, and why. It is printed only when it is non-zero, and `--json` always
carries it as a `dropped` object. Without it, a capture of a spare page and a
capture that a limit truncated are the same three-line summary, and the only
way to tell them apart is to open the page yourself.

| Reason | What it means |
| ------------------ | ------------------------------------------------------------------------------------------------------ |
| `size-floor` | Fetched, then judged too small to be a real asset rather than a spacer or tracking pixel. |
| `budget-exhausted` | `--capture-budget` ran out before this one was reached. Raise it, or pass `--skip-vision` to buy time. |
| `cap-reached` | A per-run or per-family limit was already met: 30 inline SVGs, 30 fonts, 6 faces per family. |
| `unavailable` | The request or the write failed: network error, timeout, refused address, bad status, disk. |

Three of those are decisions the capture made and one is a failure it hit, so a
run that is thin with an all-zero `dropped` is a thin page, and a run that is
thin with counts on it was cut short.

A page that declares no `<link rel="icon">` still gets a favicon: capture falls
back to `/favicon.ico` and then `/apple-touch-icon.png` at the site root, the
same paths a browser requests on its own, and keeps the first that answers with
an image. A declared icon link always wins, and a fallback that answers with
nothing usable is counted under `unavailable`.

```json
{
"assets": 45,
"dropped": { "size-floor": 6, "budget-exhausted": 0, "cap-reached": 3, "unavailable": 0 }
}
```

| Flag | Description |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--output, -o` | Output directory. Default `./capture`, then `./capture-2/`, `./capture-3/`, … if that name is taken. |
Expand All @@ -207,7 +238,8 @@ metadata, and contact sheets — plus whatever Lottie, video, and WebGL context
the page exposed. It is raw material for an agent, not a finished composition;
the `/product-launch-video` workflow uses it when a real product has to appear
on screen. Dynamic sites, protected pages, and unusual media loaders produce
partial results, so read the warnings and contact sheets before you build.
partial results, so read `dropped`, the warnings, and the contact sheets before
you build.

For AI image descriptions, set `GEMINI_API_KEY` in a `.env` file
(~$0.001/image), or `OPENROUTER_API_KEY` to route any vision model through
Expand Down
223 changes: 223 additions & 0 deletions packages/cli/src/capture/assetDownloader.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,12 @@ import { tmpdir } from "node:os";
import { join } from "node:path";
import {
downloadAndRewriteFonts,
downloadAssets,
isPrivateUrl,
safeFetch,
toStandaloneSvg,
} from "./assetDownloader.js";
import type { DesignTokens } from "./types.js";

describe("isPrivateUrl — SSRF denylist (security: F-003)", () => {
it("blocks loopback, private, and metadata IPv4", () => {
Expand Down Expand Up @@ -184,3 +186,224 @@ describe("downloadAndRewriteFonts — attempt caps", () => {
}
});
});

describe("drop counts — why a referenced asset is not in the capture", () => {
afterEach(() => vi.unstubAllGlobals());

/** `n` @font-face rules, each naming a DIFFERENT family, so only the global cap can bite. */
function fontCss(n: number): string {
return Array.from(
{ length: n },
(_, i) =>
`@font-face { font-family: Family${i}; src: url(https://fonts${i}.example/font-${i}.woff2); }`,
).join("\n");
}

function withTempDir<T>(run: (dir: string) => Promise<T>): Promise<T> {
const dir = mkdtempSync(join(tmpdir(), "hf-drops-"));
return run(dir).finally(() => rmSync(dir, { recursive: true, force: true }));
}

it("counts every face the budget never let it reach, not just the one it stopped on", async () => {
// Four declared, zero budget: the honest number is four, and a warning string could only
// ever have said "some".
await withTempDir(async (dir) => {
const fetchMock = vi.fn();
vi.stubGlobal("fetch", fetchMock);
const { drops } = await downloadAndRewriteFonts(fontCss(4), dir, { remainingMs: () => 0 });
expect(fetchMock).not.toHaveBeenCalled();
expect(drops["budget-exhausted"]).toBe(4);
expect(drops["cap-reached"]).toBe(0);
expect(drops.unavailable).toBe(0);
});
});

it("separates the faces the global cap refused from the ones that failed", async () => {
// 35 declared, cap 30: 30 are attempted and every attempt 503s, 5 are never reached.
await withTempDir(async (dir) => {
vi.stubGlobal(
"fetch",
vi.fn(async () => new Response("no", { status: 503 })),
);
const { drops } = await downloadAndRewriteFonts(fontCss(35), dir);
expect(drops["cap-reached"]).toBe(5);
expect(drops.unavailable).toBe(30);
expect(drops["budget-exhausted"]).toBe(0);
});
});

it("counts the faces the per-family cap refused", async () => {
// 10 rules, all one family, per-family cap 6: 6 attempted, 4 refused.
await withTempDir(async (dir) => {
vi.stubGlobal(
"fetch",
vi.fn(async () => new Response("no", { status: 503 })),
);
const css = Array.from(
{ length: 10 },
(_, i) =>
`@font-face { font-family: Shared; src: url(https://fonts.example/f-${i}.woff2); }`,
).join("\n");
const { drops } = await downloadAndRewriteFonts(css, dir);
expect(drops["cap-reached"]).toBe(4);
expect(drops.unavailable).toBe(6);
});
});

it("reports all zeroes when nothing was refused, which is what makes thin readable", async () => {
// The whole point of the tally: this page declared one face and we have it. A reader can now
// tell this apart from a page that declared thirty and got truncated to one.
await withTempDir(async (dir) => {
vi.stubGlobal(
"fetch",
vi.fn(async () => new Response(new Uint8Array(2048), { status: 200 })),
);
const { css, drops } = await downloadAndRewriteFonts(fontCss(1), dir);
expect(css).toContain("assets/fonts/font-0.woff2");
expect(drops).toEqual({
"size-floor": 0,
"budget-exhausted": 0,
"cap-reached": 0,
unavailable: 0,
});
});
});

/** The two fields `downloadAssets` reads off the token bundle. */
function tokensWithNoSvgs(): DesignTokens {
return { svgs: [], sections: [], ogImage: "" } as unknown as DesignTokens;
}

it("counts an image dropped for being under the raster floor", async () => {
// 9 KB is under the 10 KB floor. Nothing lands, and the reason is now on the record.
await withTempDir(async (dir) => {
vi.stubGlobal(
"fetch",
vi.fn(async () => new Response(new Uint8Array(9000), { status: 200 })),
);
const { assets, drops } = await downloadAssets(tokensWithNoSvgs(), dir, [
{ type: "Image", url: "https://cdn.example/hero.png", contexts: ["img[src]"] },
] as never);
expect(assets).toEqual([]);
expect(drops["size-floor"]).toBe(1);
expect(drops.unavailable).toBe(0);
});
});

it("counts every catalogued image the budget never let it reach", async () => {
await withTempDir(async (dir) => {
const fetchMock = vi.fn();
vi.stubGlobal("fetch", fetchMock);
const catalog = Array.from({ length: 7 }, (_, i) => ({
type: "Image",
url: `https://cdn.example/img-${i}.png`,
contexts: ["img[src]"],
}));
const { assets, drops } = await downloadAssets(
tokensWithNoSvgs(),
dir,
catalog as never,
[],
{ remainingMs: () => 0 },
);
expect(fetchMock).not.toHaveBeenCalled();
expect(assets).toEqual([]);
expect(drops["budget-exhausted"]).toBe(7);
});
});

it("falls back to the site root's well-known icon paths when the page declares none", async () => {
// The case that used to produce no favicon AND no drop reason: a page whose head declares
// no icon link, on a site that serves `/favicon.ico` perfectly well.
await withTempDir(async (dir) => {
const fetchMock = vi.fn(
async (url: string) =>
new Response(new Uint8Array([0, 0, 1, 0]), {
status: url.endsWith("/favicon.ico") ? 200 : 404,
headers: { "content-type": "image/x-icon" },
}),
);
vi.stubGlobal("fetch", fetchMock);
const { assets, drops } = await downloadAssets(tokensWithNoSvgs(), dir, [], [], {
pageUrl: "https://brand.example/pricing?ref=x",
});
expect(assets).toEqual([
{
url: "https://brand.example/favicon.ico",
localPath: "assets/favicon.ico",
type: "favicon",
},
]);
expect(drops.unavailable).toBe(0);
});
});

it("counts a well-known icon path that answers with a page, never drops it silently", async () => {
// A site with no icon answers `/favicon.ico` with its 200 HTML shell. Nothing is kept and
// both guesses are on the record.
await withTempDir(async (dir) => {
vi.stubGlobal(
"fetch",
vi.fn(
async () =>
new Response("<!doctype html><html></html>", {
status: 200,
headers: { "content-type": "text/html" },
}),
),
);
const { assets, drops } = await downloadAssets(tokensWithNoSvgs(), dir, [], [], {
pageUrl: "https://brand.example/",
});
expect(assets).toEqual([]);
expect(drops.unavailable).toBe(2);
});
});

it("keeps a declared icon link ahead of the well-known paths", async () => {
// The control: when the page vouches for an icon, the guess must not be attempted at all.
await withTempDir(async (dir) => {
const fetchMock = vi.fn(
async (_url: string) =>
new Response(new Uint8Array([137, 80, 78, 71]), {
status: 200,
headers: { "content-type": "image/png" },
}),
);
vi.stubGlobal("fetch", fetchMock);
const { assets } = await downloadAssets(
tokensWithNoSvgs(),
dir,
[],
[{ rel: "icon", href: "https://cdn.example/brand/icon-512.png" }],
{ pageUrl: "https://brand.example/" },
);
expect(assets).toEqual([
{
url: "https://cdn.example/brand/icon-512.png",
localPath: "assets/favicon.png",
type: "favicon",
},
]);
expect(fetchMock.mock.calls.map((c) => c[0])).toEqual([
"https://cdn.example/brand/icon-512.png",
]);
});
});

it("counts the inline SVGs the 30-per-run cap refused", async () => {
// 34 inline SVGs on the page, 30 kept: the four the cap dropped are now countable.
await withTempDir(async (dir) => {
const svgs = Array.from({ length: 34 }, (_, i) => ({
outerHTML: `<svg viewBox="0 0 ${i} 10"><rect width="10" height="10" fill="#abc"/></svg>`,
isLogo: false,
}));
const { assets, drops } = await downloadAssets(
{ svgs, sections: [], ogImage: "" } as unknown as DesignTokens,
dir,
);
expect(assets).toHaveLength(30);
expect(drops["cap-reached"]).toBe(4);
});
});
});
Loading
Loading