Skip to content

feat(bench): add implement-hex-color benchmark task fixture - #60

Open
snowyukitty wants to merge 1 commit into
KlaatAI:mainfrom
snowyukitty:feat/bench-fixture-hex-to-rgb
Open

feat(bench): add implement-hex-color benchmark task fixture#60
snowyukitty wants to merge 1 commit into
KlaatAI:mainfrom
snowyukitty:feat/bench-fixture-hex-to-rgb

Conversation

@snowyukitty

Copy link
Copy Markdown

Summary

Addresses #7 by adding the \implement-hex-color\ benchmark task fixture to \�ench/tasks/\ with its reference solution in \�ench/solutions/\ and registration in \�ench/suite.json.

Changes

  • Added \�ench/tasks/implement-hex-color/src/color.ts\ with problem specification comments and stub function.
  • Added \�ench/tasks/implement-hex-color/src/color.test.ts\ covering 3, 4, 6, 8 digit hex color inputs with/without #, shorthand component expansion, alpha channel scaling/rounding, and invalid input validation.
  • Added \�ench/solutions/implement-hex-color/src/color.ts\ providing the reference implementation.
  • Registered task \implement-hex-color\ in \�ench/suite.json\ and updated task counts in \�ench/README.md.

Verification

  • \�un bench/selfcheck.ts\ (34/34 fixtures verified: fail as shipped, pass with reference solution)
  • \�un run typecheck\ (passed)
  • \�un test\ (445/445 unit tests passed)
  • \�un run build\ (passed, bundled 106 modules)

@shivaylamba shivaylamba left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, thanks for the contribution. Can you also share what were the benchmark results that you got?

@snowyukitty

Copy link
Copy Markdown
Author

Thanks for looking at it — and to be straight with you, I don't have agent numbers for this task.

What I ran was the suite-integrity path the "Add a task" section asks for, not the benchmark itself:

  • bun bench/selfcheck.ts — 34/34, i.e. the new fixture fails as shipped and passes with the reference solution overlaid, and no existing task regressed
  • bun run typecheck — pass
  • bun test — 445/445
  • bun run build — pass

I did not run bun run bench, because that drives the headless agent and needs klaatai login or a KLAATAI_API_KEY, which I don't have. So I have no solved/cost/tokens/turns row for implement-hex-color, and I'd rather say that than hand you a number from a different harness and let it get compared against your own.

Happy to close that gap either way you prefer: if you want to run bun run bench -- --only implement-hex-color yourself, or point me at a way to authenticate, I'll fold the row into the PR.

On the calibration in the meantime — I marked it easy / implement on the same basis as the existing 13 implement tasks, and the two things it actually discriminates on are:

  1. 4-digit shorthand alpha. #f008 has to expand the nibble (80x88), scale by /255, and round to 3 dp, giving a: 0.533. A naive parseInt(hex[3], 16) / 15 or an unrounded 0.5333333333333333 both fail.
  2. Rejecting rather than coercing. Non-hex characters and lengths other than 3/4/6/8 must return null; parseInt is happy to partially parse "#gg0000" or "#12345", so a stub that leans on it passes the happy path and fails validation.

The rest (6-digit with/without #, case-insensitivity, a: 1 default) is happy-path coverage, which is why easy rather than medium.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants