From a6d2ebe9d3441c8a7ae955ca7519a0abb0d1029a Mon Sep 17 00:00:00 2001 From: Simon Corry Date: Fri, 4 Sep 2026 16:46:24 -0400 Subject: [PATCH] Add candidate assessment skill --- README.md | 4 + skills/candidate-assessment/SKILL.md | 64 ++++++++++++++ .../candidate-assessment/agents/openai.yaml | 4 + .../references/ashby-setup.md | 64 ++++++++++++++ .../references/design-exercise.md | 63 +++++++++++++ .../references/final-synthesis.md | 36 ++++++++ .../references/hiring-manager.md | 58 ++++++++++++ .../references/portfolio-review.md | 62 +++++++++++++ .../references/research-basis.md | 28 ++++++ .../references/scoring-and-output.md | 88 +++++++++++++++++++ 10 files changed, 471 insertions(+) create mode 100644 skills/candidate-assessment/SKILL.md create mode 100644 skills/candidate-assessment/agents/openai.yaml create mode 100644 skills/candidate-assessment/references/ashby-setup.md create mode 100644 skills/candidate-assessment/references/design-exercise.md create mode 100644 skills/candidate-assessment/references/final-synthesis.md create mode 100644 skills/candidate-assessment/references/hiring-manager.md create mode 100644 skills/candidate-assessment/references/portfolio-review.md create mode 100644 skills/candidate-assessment/references/research-basis.md create mode 100644 skills/candidate-assessment/references/scoring-and-output.md diff --git a/README.md b/README.md index b7094aa..8fdea23 100644 --- a/README.md +++ b/README.md @@ -56,6 +56,10 @@ Two folders keep the chain from starting every session at zero. `docs/sessions/` The wiki ships stocked. Three reference shelves come with the repo: engineering fundamentals (Brooks, Parnas, Naur, and the essential-vs-accidental and wrong-abstraction lenses the challenge rounds cite), design fundamentals (the Norman-to-Rams canon distilled for hierarchy, grouping, type, and color arguments), and motion fundamentals (easing, springs, gesture feel, and motion cost, adapted with credit from Emil Kowalski's and Meng To's MIT-licensed work). Start at [the index](docs/wiki/INDEX.md). +## Optional skills + +Foundry also keeps a small set of standalone skills that are useful outside the build chain. [Candidate Assessment](skills/candidate-assessment/SKILL.md) turns rough Product Design interview notes into direct, evidence-led Ashby feedback for hiring manager interviews, portfolio reviews, design exercises, and final decisions. It is deliberately opinionated about early-stage designers: problem finding, a useful spike, customer-led velocity, visible craft judgment, and real AI build practice. Install it by asking Codex to install `skills/candidate-assessment` from this repository. + ## What it costs The full chain is thorough and token-heavy. On a mid-size feature it plausibly lands in the low hundreds of thousands of tokens end to end, which is real money on metered plans. The review rounds are not automatically the expensive part: in the source project's measured runs, build volume, repeated context-heavy reads, duplicate checks, and late mechanical work were often larger contributors. The rounds still found defects late in the chain. diff --git a/skills/candidate-assessment/SKILL.md b/skills/candidate-assessment/SKILL.md new file mode 100644 index 0000000..742efd9 --- /dev/null +++ b/skills/candidate-assessment/SKILL.md @@ -0,0 +1,64 @@ +--- +name: candidate-assessment +description: Turn rough Product Design interview notes into concise, evidence-led Ashby feedback using Simon Corry's startup hiring bar. Use when Simon asks to run candidate assessment, assess a hiring manager interview, review a portfolio interview, score a design exercise, or synthesize a final hiring decision. Apply only to Product Design candidates Simon is assessing. +--- + +# Candidate Assessment + +Assess Product Design candidates for a small, high-velocity startup. Look for the person who makes the role and the team more ambitious, not the person who most cleanly resembles a conventional designer. + +This is a demanding bar. It is not a vibe check and it is not an HR compliance exercise. Strong evidence can be unconventional. Weak evidence does not become strong because the candidate is polished, likable, prestigious, or fluent in design language. + +## Inputs + +Require: + +- Interview type: `hiring manager`, `portfolio review`, `design exercise`, or `final synthesis` +- Target level or role +- Simon's rough notes, transcript excerpts, or supporting paragraphs + +Run one interview type at a time. Use `final synthesis` only when evidence from multiple interviews is available. + +If the interview type is missing, ask one short question. If the level is missing but obvious from the role title, state the assumption and continue. Do not invent missing observations. + +## Route the Assessment + +Always read [scoring-and-output.md](references/scoring-and-output.md). Then read only the reference for the requested interview: + +- [hiring-manager.md](references/hiring-manager.md) +- [portfolio-review.md](references/portfolio-review.md) +- [design-exercise.md](references/design-exercise.md) +- [final-synthesis.md](references/final-synthesis.md) + +Read [research-basis.md](references/research-basis.md) only when revising the rubric or explaining why it works. Do not add research citations to candidate feedback. + +Read [ashby-setup.md](references/ashby-setup.md) when creating or changing the actual Ashby interview plans. + +## Assessment Method + +1. Pull out observed behavior, candidate claims, outcomes, contradictions, and missing evidence. +2. Keep claims separate from proof. Use phrases such as `said they...` when the candidate described something but the interview did not substantiate it. +3. Score each criterion independently against its behavioral anchors and the target level. +4. Apply the core gates for that interview. Do not average scores. +5. Write the Ashby fields in the exact order and format in the scoring reference. + +## Simon's Bar + +- Prefer a sharp, useful spike over smooth general competence. Name the spike and its likely cost. +- Reward people who find consequential problems and act without waiting for a task list. +- Treat ambiguity as the work. The candidate should create a point of view, not ask the interviewer to finish the brief. +- Pair velocity with learning. Shipping quickly matters when it produces customer evidence and the candidate owns the misses. +- Reward constructive challenge. Agreement, deference, and easy chemistry are not collaboration signals. +- Treat craft as judgment made visible. Look for intent, behavior, hierarchy, detail, and a human pulse. Portfolio polish alone proves very little. +- AI fluency is mandatory. Chat use and one-shot generation do not meet the bar. Look for durable workflows, working builds, model understanding, evaluation, and explicit human judgment. +- Background, pedigree, role labels, accent, charisma, hours worked, and generic culture fit are not evidence. + +## Writing Rules + +- Sound like a thoughtful design leader recording a decision, not a recruiting department. +- Be candid, specific, and restrained. No inflated praise. No HR euphemisms. +- Use short bullets with a clear reason. Name the evidence and what it means. +- Avoid generic phrases such as `strong communicator`, `great culture fit`, `passionate`, or `impressive background` unless the following words make the claim concrete. +- Do not soften a No into ambiguity. Do not manufacture certainty when evidence is missing. +- Keep the full response between roughly 175 and 300 words unless the evidence genuinely needs less. +- Never mention this skill, AI authorship, or the research in the Ashby-ready output. diff --git a/skills/candidate-assessment/agents/openai.yaml b/skills/candidate-assessment/agents/openai.yaml new file mode 100644 index 0000000..3415792 --- /dev/null +++ b/skills/candidate-assessment/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "Candidate Assessment" + short_description: "Turn interview notes into sharp Ashby feedback" + default_prompt: "Use $candidate-assessment to assess this Product Design interview for Ashby." diff --git a/skills/candidate-assessment/references/ashby-setup.md b/skills/candidate-assessment/references/ashby-setup.md new file mode 100644 index 0000000..4cd6654 --- /dev/null +++ b/skills/candidate-assessment/references/ashby-setup.md @@ -0,0 +1,64 @@ +# Ashby Setup + +Create three Product Design interview plans. Keep the shared fields, replace the generic company traits with the interview-specific criteria below, and use the same 1 to 4 labels everywhere. + +## Shared Fields + +### Overall Recommendation + +- `4`: Strong Yes +- `3`: Yes +- `2`: No +- `1`: Strong No + +### Reason for Recommendation + +Prompt: `What is the decisive evidence for or against this candidate? Use three to five short bullets.` + +### Next Steps + +Prompt: `Advance, decline, or name the precise condition for continuing.` + +### Areas to Probe + +Prompt: `What important question remains unresolved? Maximum three.` + +### What Candidate Cares About + +Prompt: `What did the candidate explicitly say they want to build, learn, change, or avoid?` + +Remove the duplicate general Comments/Notes field from these interview plans. Keep recruiter-only fields outside the design scorecard. Granola Enhanced Notes can remain as raw source material but should not replace interviewer judgment. + +## Hiring Manager Scorecard + +- `Distinctive Edge and Generative Curiosity`: Do they bring a useful spike, original product perspective, and questions that make the conversation better? +- `Problem Finding, Autonomy, and Impact`: Do they notice consequential problems, create momentum without a task list, and produce a real outcome? +- `Velocity, Learning, and Product Judgment`: Do they ship the smallest useful move, learn from customers, make explicit tradeoffs, and own mistakes? +- `Constructive Challenge, Self-Awareness, and Trust`: Can they disagree plainly, absorb better evidence, know their limits, and preserve trust? +- `AI-Native Builder Practice`: Have they built durable AI-enabled products or workflows, and can they explain model behavior, evaluation, and the human judgment that remains? + +## Portfolio Review Scorecard + +- `Problem Choice and Framing`: Did they find the real problem beneath the brief and explain why it mattered? +- `Personal Ownership and Judgment`: What did they personally decide, change, and move across the organization? +- `Craft, Taste, and Human Intent`: Does the work show intentional behavior, hierarchy, language, detail, and a human pulse? +- `Shipping, Customer Contact, and Learning`: Did real use change the work, or is this a tidy retrospective story? +- `Distinctive Spike`: What can they do unusually well, and what does that strength cost? +- `AI-Enabled Build Practice`: Did AI materially change how they built, tested, or learned, beyond chat and one-shot generation? + +## Design Exercise Scorecard + +- `Question Quality and Discovery`: Do their questions expose consequential uncertainty and alter what they do next? +- `Problem Framing and Perspective`: Can they turn a vague prompt into a specific, defensible point of view? +- `Exploration and Distinctive Thinking`: Do they consider meaningfully different directions rather than decorate the first idea? +- `Prioritization, Tradeoffs, and Velocity`: Can they choose, explain what they are sacrificing, and use the hour well? +- `Collaboration and Adaptation`: Can they challenge, listen, and improve the frame when a new constraint arrives? +- `Direction, Craft, and Human Intent`: Does the rough direction cohere, and does it show care for actual human use? + +Do not add an AI score to the design exercise unless AI use is explicitly part of that exercise. The final decision still requires AI evidence from another stage. + +## Field-Level Scale Description + +Use this description on every scored criterion: + +`1 Strong No: evidence runs against a core requirement. 2 No: some signal, but below the independence or judgment required. 3 Yes: credible independent performance at level. 4 Strong Yes: rare, repeatable evidence that materially raises the team's bar. Do not average. When something was not observed, leave it unscored and say so in the comment.` diff --git a/skills/candidate-assessment/references/design-exercise.md b/skills/candidate-assessment/references/design-exercise.md new file mode 100644 index 0000000..cdef4d4 --- /dev/null +++ b/skills/candidate-assessment/references/design-exercise.md @@ -0,0 +1,63 @@ +# Design Exercise + +## What This Interview Is For + +Watch the candidate create clarity from a deliberately vague prompt. The point is not a finished screen. The point is how they question, frame, choose, explore, and adapt while another person is in the room. + +Use a fictional or adjacent problem, not free consulting work. Give a two-sentence prompt with genuine ambiguity. Around the middle of the session, introduce one meaningful constraint that tests the candidate's frame. Answer questions honestly, but do not rescue them with a hidden brief. + +The candidate may use paper, Figma, code, or AI after they have made the frame visible. Tool use earns no points on its own. + +## Criteria + +### Question Quality and Discovery + +Core gate. + +- `1`: Runs straight to a solution or asks a checklist of questions without using the answers. +- `2`: Covers obvious users and constraints, but does not find the uncertainty that matters. +- `3`: Asks economical questions, turns answers into hypotheses, and deliberately reduces the riskiest uncertainty. +- `4`: Finds the question beneath the prompt and changes what the room believes the exercise is about. + +### Problem Framing and Perspective + +Core gate. + +- `1`: Treats ambiguity as missing instructions and never establishes a point of view. +- `2`: Produces a plausible frame, but it is generic or disconnected from the evidence gathered. +- `3`: Defines a specific user, problem, outcome, and boundary. States assumptions and chooses a defensible perspective. +- `4`: Creates a sharp, surprising frame that opens a materially better product direction. + +### Exploration and Distinctive Thinking + +- `1`: Fixates on the first familiar pattern. +- `2`: Generates variations, but not meaningfully different approaches. +- `3`: Explores distinct possibilities, compares their consequences, and brings an original idea or useful spike. +- `4`: Reimagines the problem without losing practical contact with the user or business. + +### Prioritization, Tradeoffs, and Velocity + +- `1`: Cannot choose, or confuses speed with skipping thought. +- `2`: Chooses a direction but cannot explain what was sacrificed or learned. +- `3`: Makes explicit tradeoffs, selects the smallest useful move, and manages the hour well. +- `4`: Uses time and evidence exceptionally well, advancing the idea while keeping the most consequential uncertainty visible. + +### Collaboration and Adaptation + +- `1`: Defends the artifact, ignores input, or waits for permission. +- `2`: Accepts input but merely complies, or pushes a view without incorporating evidence. +- `3`: Thinks aloud, challenges assumptions respectfully, uses the interviewer as a collaborator, and adapts without losing coherence. +- `4`: The exchange improves both people's thinking. The candidate can absorb a disruptive constraint and produce a stronger frame. + +### Direction, Craft, and Human Intent + +- `1`: The direction is incoherent or careless about actual use. +- `2`: Understandable but generic, with important behaviors or edge cases ignored. +- `3`: The chosen direction is coherent and intentional. Key behavior, hierarchy, language, and human consequences are considered. +- `4`: Even in rough form, the direction shows exceptional taste and makes the intended experience palpable. + +## Interpretation + +Problem framing is the primary gate. Strong visual output cannot rescue a candidate who needed the interviewer to define the work. A rough outcome can still pass when the reasoning, perspective, and judgment are strong. + +Do not score AI unless the candidate uses it or the interview explicitly tests it. If used, note whether it extended thinking or merely generated surface output. Final synthesis still requires separate evidence that the AI bar is met. diff --git a/skills/candidate-assessment/references/final-synthesis.md b/skills/candidate-assessment/references/final-synthesis.md new file mode 100644 index 0000000..7e1ec39 --- /dev/null +++ b/skills/candidate-assessment/references/final-synthesis.md @@ -0,0 +1,36 @@ +# Final Synthesis + +## What This Assessment Is For + +Make the hiring decision across interviews. Reconcile evidence. Do not average scorecards and do not let the loudest interviewer substitute for a decision. + +## Required Evidence + +A final Yes or Strong Yes requires credible evidence of all four: + +- Problem finding and independent ownership at the target level +- Product craft, taste, and human judgment +- Velocity tied to shipping, customer contact, or a real learning loop +- AI-native build practice at 3 or higher + +For staff, lead, or manager roles, also require evidence that the person creates leverage beyond their own output. This can be growing others, improving the build system, changing the quality of decisions, or creating a capability the team did not have. + +## Synthesis Method + +1. Name the candidate's spike in one plain sentence. +2. Name the cost or risk that comes with it. +3. Separate repeated evidence from a single strong story. +4. Resolve contradictions by asking which interview observed behavior closest to the real work. Do not erase the contradiction. +5. Check all required evidence and level expectations. +6. Decide what the team would be able to do with this person that it cannot do now. + +## Overall Anchors + +- `1: Strong No`: Evidence contradicts a required capability, ownership is not credible, or behavior presents a serious trust risk. +- `2: No`: Interesting strengths exist, but at least one required capability remains below bar or unproved after the process. +- `3: Yes`: Clears every required capability at level, has a useful spike, and the associated risks are manageable and explicit. +- `4: Strong Yes`: Rare. Clears every requirement, brings a distinctive capability, expands the role or company's ambition, and is someone Simon would reorganize meaningful work to hire. + +Do not use team similarity as evidence. A candidate can create productive disruption and still be a Yes. The line is whether their challenge produces better work while preserving trust and follow-through. + +If an exceptional candidate describes a role that differs from the opening, say so directly in Next Steps. Shaping a role around a rare person is allowed. Pretending a mismatch does not exist is not. diff --git a/skills/candidate-assessment/references/hiring-manager.md b/skills/candidate-assessment/references/hiring-manager.md new file mode 100644 index 0000000..2de6650 --- /dev/null +++ b/skills/candidate-assessment/references/hiring-manager.md @@ -0,0 +1,58 @@ +# Hiring Manager Interview + +## What This Interview Is For + +Decide whether this person should work directly on Simon's team. Test whether broad trust will make them more useful or simply leave them waiting for direction. The conversation can roam, but the scoring cannot become a chemistry score. + +## Criteria + +### Distinctive Edge and Generative Curiosity + +Core gate. + +- `1`: Rehearsed, derivative answers. No visible curiosity beyond the role in front of them. +- `2`: Has interests and preferences, but cannot turn them into an original product point of view. +- `3`: Has a clear spike, asks questions that change the conversation, and makes grounded connections others may miss. +- `4`: Brings an unusual, deeply developed capability or perspective that makes the possible role bigger. + +Do not confuse eccentricity, confidence, or good banter with a spike. + +### Problem Finding, Autonomy, and Impact + +Core gate. + +- `1`: Waits for assignments or describes ownership that collapses under questioning. +- `2`: Can execute a framed problem but depends on others to choose the problem, remove ambiguity, or create momentum. +- `3`: Repeatedly notices consequential problems, frames them, recruits the right people, and gets useful work into the world. +- `4`: Creates opportunities nobody assigned, changes company direction or capability, and can show the resulting impact. + +### Velocity, Learning, and Product Judgment + +- `1`: Optimizes for output or perfection without a learning loop. Cannot name tradeoffs or mistakes. +- `2`: Moves quickly in familiar conditions, but customer evidence and decision quality are inconsistent. +- `3`: Chooses the smallest useful move, gets it in front of real people, changes course from evidence, and owns what went wrong. +- `4`: Consistently turns uncertain product bets into fast, high-quality learning without becoming reckless or precious. + +### Constructive Challenge, Self-Awareness, and Trust + +- `1`: Protects ego, blames others, or uses sharpness as permission to diminish people. +- `2`: Either defers too easily or pushes without listening and follow-through. +- `3`: Can disagree plainly, revise their view, name their limits, and preserve trust while moving the work. +- `4`: Makes the room more ambitious. They challenge the premise, absorb better evidence quickly, and help other people find their own best work. + +### AI-Native Builder Practice + +Core gate. Use the shared AI anchors. + +Ask for concrete artifacts and workflows. Two useful prompts are: + +- How has AI changed your actual design and build process? What is different now? +- How has your understanding of models changed what you believe products can become? + +A chatbot feature, prompt collection, or enthusiasm for Cursor and Claude is not enough. Look for working systems, agent orchestration, evaluation, customer contact, and a developing mental model. + +## Motivation and Anti-Sell + +Capture, but do not score, what the candidate wants to build and what their version of a perfect role looks like. Be honest about the environment: little task-list management, high autonomy, frequent shipping, real customer contact, and permission to challenge the premise. + +The decisive fit question is not `Do they want this job?` It is `Will this environment release their best work?` diff --git a/skills/candidate-assessment/references/portfolio-review.md b/skills/candidate-assessment/references/portfolio-review.md new file mode 100644 index 0000000..efcb8b5 --- /dev/null +++ b/skills/candidate-assessment/references/portfolio-review.md @@ -0,0 +1,62 @@ +# Portfolio Review + +## What This Interview Is For + +Assess the candidate's judgment and craft through work they actually shaped. A polished case study is optional. Products, prototypes, code, experiments, side projects, and public work are valid evidence. Deck quality is not a proxy for product quality. + +Ask what they did, what changed because of them, and what they would now do differently. Keep team outcomes separate from personal contribution. + +## Criteria + +### Problem Choice and Framing + +Core gate. + +- `1`: Starts from a supplied solution and cannot recover the real problem. +- `2`: Repeats the brief or process steps with limited evidence of reframing. +- `3`: Identifies the consequential user and business problem, tests assumptions, and explains why this problem deserved attention. +- `4`: Finds the problem beneath the brief and changes the team's understanding of what should be built. + +### Personal Ownership and Judgment + +Core gate. + +- `1`: Uses `we` to hide their contribution or claims decisions they cannot explain. +- `2`: Contribution is real but narrow, heavily directed, or unclear at the target level. +- `3`: Clearly owns important decisions, tradeoffs, and cross-functional movement. +- `4`: Created unusual leverage, changed the trajectory, and can defend the hard calls without rewriting history. + +### Craft, Taste, and Human Intent + +Core gate. + +- `1`: The work is careless, incoherent, or disconnected from human use. +- `2`: Competent surface execution, but generic patterns do most of the thinking. +- `3`: Strong interaction and visual judgment. Behavior, hierarchy, language, edge cases, and emotional intent work together. +- `4`: Exceptional taste with a reason behind it. The work has a human pulse and makes a hard product feel inevitable without becoming ornamental. + +Do not reward polish detached from use. Do not punish rough artifacts that reveal exceptional judgment. + +### Shipping, Customer Contact, and Learning + +- `1`: Cannot connect the work to real use, outcomes, or learning. +- `2`: Shipped or tested, but evidence did not meaningfully change decisions. +- `3`: Put work in front of customers, learned quickly, and can trace important iterations to evidence. +- `4`: Built a repeatable learning loop that improved both the product and how the team worked. + +### Distinctive Spike + +- `1`: No credible strength beyond baseline execution. +- `2`: Broadly capable, but the work could have come from many designers. +- `3`: One clear and useful spike is visible across the work. +- `4`: The spike is rare, generative, and gives the company a capability it would struggle to hire elsewhere. + +Name the cost of the spike. A strength that slows, alienates, or narrows the work is still a tradeoff worth recording. + +### AI-Enabled Build Practice + +Use the shared AI anchors. Ask to see where AI changed the work, not just where it made the same work faster. Strong evidence includes direct implementation, functional prototypes, durable agents or harnesses, explicit evaluation, and informed choices about model limitations. + +## Reading Metrics + +Metrics are useful but not magical. Check causal proximity. A designer should distinguish what the team measured, what they inferred, and what remains unknown. A tidy number with no credible decision behind it is weaker than an honest account of uncertainty. diff --git a/skills/candidate-assessment/references/research-basis.md b/skills/candidate-assessment/references/research-basis.md new file mode 100644 index 0000000..49bb05d --- /dev/null +++ b/skills/candidate-assessment/references/research-basis.md @@ -0,0 +1,28 @@ +# Research Basis + +This file explains the durable choices behind the rubric. It is not candidate evidence and should not be cited in Ashby output. + +## Simon's Published Hiring View + +- [Double Diamond NYC interview](https://doublediamondnyc.substack.com/p/simon-corry-senior-director-of-product): hire for velocity, autonomy, curiosity, ambiguity, and the ability to take prototypes to customers. Portfolio polish and familiar role labels are weak proxies. +- [Liftoff interview](https://liftoff.substack.com/p/simon-corry-on-how-to-hire-in-the): seek spiky people who challenge defaults, build fluently with AI, cross disciplines, and chart their own course. A designed chatbot is not meaningful AI evidence. +- [Dry Run interview](https://www.youtube.com/watch?v=W1bB4J7rz2I): the strongest designer can act like a spiritual co-founder, bring imagination and productive disruption, ship to learn, and preserve human taste while using AI. Durable agent workflows matter more than one-shot demos. +- [Due Diligence interview](https://www.youtube.com/watch?v=Iq6FK3y7x64): look for people who see problems and create without direction. Velocity matters because it creates customer learning, and owning the damage from moving too fast matters just as much. +- [The Cost of Context](https://simoncorry.com/blog/2026/08/06/the-cost-of-context), [Design Process, Meet Agent Process](https://simoncorry.com/blog/2026/07/15/design-process-meet-agent-process), and [A Researcher Out of Time](https://simoncorry.com/blog/2026/05/20/a-researcher-out-of-time): strong AI practice is an inspectable, long-lived process with explicit claims, tests, reversals, and human review. + +## External Evidence and Startup Practice + +- [First Round on the founding designer role](https://review.firstround.com/the-ultimate-guide-to-the-founding-designer-role/): early designers need autonomy, ambiguity tolerance, business prioritization, self-critique, and a lightweight process. Copying a large-company design process is a poor fit. +- [Emotive on hiring startup product designers](https://medium.com/emotive-design/how-we-hire-product-designers-at-emotive-12854dfc2b99): early-stage work rewards versatility, context switching, problem-solving, autonomy, and agility. +- [Monzo's product design hiring process](https://monzo.com/blog/2019/01/17/monzo-product-designer-jobs): consistent questions, realistic challenges, and attention to framing, collaboration, and tradeoffs produce more useful evidence than puzzle exercises. +- [Hartwell et al. on structured interviews](https://www.sciencedirect.com/science/article/pii/S0148296319301985): job-related behavioral and situational questions improve the usefulness of interview evidence. +- [Ashby score field guidance](https://www.ashbyhq.com/product-updates/form-sections-question-descriptions-and-scores): explicit criteria, descriptions, comments, and anchored scores support a usable scorecard. +- [Figma's AI Model Designer role](https://job-boards.greenhouse.io/figma/jobs/5711913004) and [Anthropic's product design workflow](https://www-cdn.anthropic.com/58284b19e702b49db9302d5b6f135ad8871e7658.pdf): current AI-native design includes model evaluation, technical understanding, direct implementation, functional prototypes, and attention to states and edge cases. + +## Deliberate Departures From Conventional Rubrics + +- `Smile` is excluded. Affect is not job evidence. +- `Fast-brained` is replaced with framing and judgment. Processing speed is not the same as clear thinking. +- `Care and intensity` is observed through ownership and follow-through, not performed enthusiasm or willingness to work long hours. +- `Culture fit` is replaced with constructive challenge, trust, and the conditions under which the person's best work emerges. +- Portfolio presentation is not a compliance test. The work and the candidate's decisions carry the signal. diff --git a/skills/candidate-assessment/references/scoring-and-output.md b/skills/candidate-assessment/references/scoring-and-output.md new file mode 100644 index 0000000..3544c08 --- /dev/null +++ b/skills/candidate-assessment/references/scoring-and-output.md @@ -0,0 +1,88 @@ +# Scoring and Ashby Output + +## The 1 to 4 Scale + +Use whole numbers only. + +### 1: Strong No + +- The evidence runs against a core requirement. +- The candidate cannot substantiate ownership, judgment, or claimed impact. +- Their behavior would create material risk at this level and is unlikely to be fixed with normal onboarding. + +### 2: No + +- There is some useful signal, but the candidate needs more framing, direction, or support than this role allows. +- Evidence is thin, inconsistent, derivative, or below the target level. +- This is a real No, not a polite maybe. + +### 3: Yes + +- The candidate gave credible, repeated evidence of independent performance at the target level. +- Tradeoffs, ownership, and outcomes are clear enough to trust. +- Some gaps can remain, but none undercut a core requirement. + +### 4: Strong Yes + +- Rare evidence. The candidate would materially raise the team's bar or add a capability the team does not have. +- Their judgment is original, repeatable, and grounded in shipped work or observed behavior. +- They meet every core gate. A merely solid interview is not a 4. + +## Level Calibration + +- Mid-level: owns a defined problem, makes sound decisions, and seeks help without surrendering the work. +- Senior: finds and frames ambiguous problems, moves across product and engineering boundaries, and ships without close direction. +- Staff or lead: changes how the company sees a problem, creates leverage beyond their own output, and makes other people better while remaining a credible builder. +- Design manager: builds the environment for strong work, develops individual spikes, makes hard talent decisions, and can still reason concretely about the product. + +Do not give extra credit for senior titles at well-known companies. + +## Decision Rules + +- Do not average the criterion scores. Decide whether the candidate cleared the essential bar. +- A core criterion below 3 normally caps the interview at 2. +- `Not observed` is not a score. It becomes an Area to Probe if the stage was not meant to test it. If the stage was meant to test it, missing evidence can justify a 2. +- One excellent artifact cannot cancel weak ownership. One charismatic conversation cannot cancel weak craft. +- A 4 needs both breadth across the core bar and one unmistakable spike. +- Separate productive friction from destructive behavior. Challenging defaults is valuable. Contempt, ego protection, or failure to follow through is not. +- Velocity without customer contact, evaluation, or ownership of mistakes is just speed. + +## AI Bar + +Use this wherever AI practice is assessed: + +- `1`: No practical fluency, misleading claims, or treats generated output as inherently correct. +- `2`: Uses chat or one-off generation. Cannot explain workflow design, model behavior, evaluation, or what remains human judgment. +- `3`: Has built real prototypes, products, agents, or repeatable workflows. Can explain context design, iteration, model limits, edge cases, evaluation, and how customer learning changed the work. +- `4`: Has built or materially extended a durable AI-enabled system. Chooses models and tools deliberately, maintains an evaluation loop, understands nondeterministic behavior, and has a clear view of where human taste and responsibility must remain. + +For a final hiring decision, AI must be at least 3. Do not infer a 3 from enthusiasm or vocabulary. + +## Ashby-Ready Format + +Return only the following blocks. Omit a bullet rather than filling it with generic language. + +```markdown +## Overall Recommendation +3 - Yes + +## Reason for Recommendation +- [Three to five evidence-led bullets. Lead with the decisive signal.] + +## Next Steps +[One direct sentence: advance, decline, or advance with a precise condition.] + +## Areas to Probe +- [Zero to three unresolved questions. Say `None` when there are none.] + +## What Candidate Cares About +- [Only priorities the candidate explicitly expressed. Say `Not discussed` when absent.] + +## Criterion Scores +- [Criterion]: 3/4 + [One or two sentences of evidence and meaning.] +``` + +Do not produce a separate Comments/Notes section. Do not fill recruiter-only fields. Treat Granola notes as source material, not an output field. + +The recommendation must be consistent with the criterion scores and core gates. Keep the reason bullets compact enough to scan during a debrief.