Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion skills/coderuler-new-rule/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: coderuler-new-rule
description: Create CodeRuler rules, including custom selection, filters and checks, inline or companion tests, and inspection.
description: Create CodeRuler rules, including selection, checks, tests, inspection, and suppressing accepted violations.
---

# Create a CodeRuler rule
Expand Down Expand Up @@ -136,6 +136,12 @@ Finally exercise normal evaluation:
coderuler check . --rules .coderuler/k8s-replica-floor.rule.ts --output-all
```

## 5. Record a valid exception

After the project check, suppress only a finding the rule correctly identifies but the project accepts. Find its exact rule name and location with `coderuler inspect <rule> . --json`, then add a narrow entry to `.coderuler/suppressions.yml` with a short reason. Prefer a scope ID; use inclusive lines when a scope is unavailable or too broad.

Rerun `coderuler check . --output-all` and confirm only the intended finding is suppressed. Fix a rule or fixture when its policy is wrong; never add a suppression just to pass a test. The engine applies suppressions between a valid check result and its report; this does not change applicability or skip checks and AI calls. See [violation suppressions](reference.md#violation-suppressions) for the YAML shape and root rules.

## Good and bad practices

| Prefer | Avoid |
Expand Down
22 changes: 21 additions & 1 deletion skills/coderuler-new-rule/reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ Default-export a constructible class extending `Rule<T>` (`T = JsonValue`). Extr
| `check: Evaluator<T>` | Required | Function-valued property, such as `check = regex(...)` or `check: Evaluator<T> = ...`, rather than a method. |


Execution: load and validate all rules → apply `check` tag filters → enabled → `when` → select → validate subjects → filter → group → check → report. `Awaitable<T>` means `T | Promise<T>`.
Execution: load and validate all rules → apply `check` tag filters → read root suppressions → enabled → `when` → select → validate subjects → filter → group → check → validate/normalize result → apply suppressions → report. Suppression leaves applicability and check execution unchanged, including AI calls. `Awaitable<T>` means `T | Promise<T>`.

`files` and `exclude` must be dense arrays of nonempty strings, including on disabled rules. `tags`, when present, must be dense arrays of nonempty strings.

Expand Down Expand Up @@ -344,6 +344,26 @@ Scalar AI classification supplies no offending-member attribution: scope AI can

Results validate; supplied messages survive reporting. Violations without a message report rule/location without generated text. Return real probability/model/threshold metadata only; supplied probability/threshold must be finite in `[0, 1]`. Throw for execution errors. External libraries/providers may be called directly; no backend registry is required.

## Violation suppressions

Rules return `CheckResult` with status `pass` or `violation`; only the engine creates `EvaluationResult` with status `suppressed`. The engine applies suppressions after validating and normalizing a check result, before reporting it. A partial match remains a violation with uncovered locations active; a full match has status `suppressed` and records each covered location and reason. Only active violations reach `Rule.comment`; fully suppressed results do not create PR comments. Suppression does not change applicability, skip checks, hide execution errors, or avoid AI costs. There is no bypass setting.

Each run reads `.coderuler/suppressions.yml` from its project root: the invocation root for `check`, the selected folder for `inspect`, and each isolated fixture root for rule tests. Rule-test fixtures do not inherit the invocation project's configuration. Use `coderuler inspect <rule> . --json` to find the exact rule name, scope IDs, and line ranges.

```yaml
suppressions:
field-naming:
- files: ["api/orders.proto"]
scope: "acme.Order.legacy_id"
reason: "Existing public contract"
no-console:
- files: ["src/legacy.js"]
lines: { start: 42, end: 45 }
reason: "Required diagnostic output"
```

`files` uses root-relative globs. `scope` is an exact selector ID; `lines` is an inclusive range that must contain the entire finding range. Use at most one selector; omitting both covers all findings in the matching files. Each entry needs a reason, and the first matching entry supplies it. Fully suppressed results and their reasons appear with `--output-all`; default check output omits them. Rules cannot return `suppressed`, and their applicability filters should not encode local exceptions.

### Context helpers


Expand Down
25 changes: 22 additions & 3 deletions src/adapter/cli/inspect-json.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
import type { RuleReport, RunEvent } from '../../app/ports.js';
import type { CheckResult, GroupSubject, JsonValue, ScopeSubject, Subject } from '../../app/rule.js';
import type { InspectRenderer } from './reports.js';
import type { EvaluationResult } from '../../app/suppressions.js';
import type { GroupSubject, JsonValue, ScopeSubject, Subject } from '../../app/rule.js';
import type { TextRedactor, TextWriter } from './text.js';
import type { InspectRenderer } from './reports.js';
import { createUsageTracker } from './inspect-usage.js';

interface InspectJsonOptions {
Expand Down Expand Up @@ -60,12 +61,30 @@ function redactSubject(subject: Subject, redact: TextRedactor): Subject {
return subject.kind === 'scope' ? redactScope(subject, redact) : redactGroup(subject, redact);
}

function redactResult(result: CheckResult, redact: TextRedactor): CheckResult {
function redactResult(result: EvaluationResult, redact: TextRedactor): EvaluationResult {
const suppressedLocations = result.suppressedLocations?.map(suppressed => ({
location: {
...suppressed.location,
filepath: redact(suppressed.location.filepath),
...(suppressed.location.scopeId === undefined ? {} : { scopeId: redact(suppressed.location.scopeId) }),
},
reason: redact(suppressed.reason),
}));
if (result.status === 'suppressed') {
return {
...result,
...(suppressedLocations === undefined ? {} : { suppressedLocations }),
...(result.message === undefined ? {} : { message: redact(result.message) }),
...(result.model === undefined ? {} : { model: redact(result.model) }),
...(result.usage === undefined ? {} : { usage: { ...result.usage } }),
};
}
return {
...result,
...(result.message === undefined ? {} : { message: redact(result.message) }),
...(result.model === undefined ? {} : { model: redact(result.model) }),
...(result.usage === undefined ? {} : { usage: { ...result.usage } }),
...(suppressedLocations === undefined ? {} : { suppressedLocations }),
};
}

Expand Down
58 changes: 44 additions & 14 deletions src/adapter/cli/reports.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import type { RuleTestResult } from '../../app/rule-tests.js';
import type { AiUsage, JsonValue, ScopeSubject, Subject } from '../../app/rule.js';
import { addUsage } from '../../app/usage.js';
import type { ViolationComment } from '../../app/violation-comments.js';
import type { SuppressedLocation } from '../../app/suppressions.js';
import { createUsageTracker } from './inspect-usage.js';
import { escapeText, type TextRedactor, type TextWriter } from './text.js';

Expand Down Expand Up @@ -432,14 +433,20 @@ export function createWalkthroughRenderer(options: WalkthroughOptions, inspectUs
paint(' body:', 'gray', color),
...walkthroughPayload(event.body, redact, raw).map(line => paint(` ${line}`, 'gray', color)),
]);
case 'result':
case 'result': {
const result = event.result;
return section(card, 'Result', [
` ${field('status', event.result.status, event.result.status === 'pass' ? 'green' : 'red')}`,
...(event.result.message === undefined ? [] : [` ${field('message', event.result.message)}`]),
...(event.result.probability === undefined ? [] : [` ${field('probability', String(event.result.probability))}`]),
...(event.result.threshold === undefined ? [] : [` ${field('threshold', String(event.result.threshold))}`]),
...(event.result.model === undefined ? [] : [` ${field('model', event.result.model)}`]),
` ${field('status', result.status, result.status === 'pass' ? 'green' : result.status === 'suppressed' ? 'yellow' : 'red')}`,
...(result.message === undefined ? [] : [` ${field('message', result.message)}`]),
...(result.probability === undefined ? [] : [` ${field('probability', String(result.probability))}`]),
...(result.threshold === undefined ? [] : [` ${field('threshold', String(result.threshold))}`]),
...(result.model === undefined ? [] : [` ${field('model', result.model)}`]),
...(result.suppressedLocations ?? []).flatMap(({ location, reason }) => [
` ${field('suppressed location', `${location.filepath}${location.lines === undefined ? '' : `:${location.lines.start}-${location.lines.end}`}${location.scopeId === undefined ? '' : ` ${location.scopeId}`}`, 'dim')}`,
` ${field('reason', reason)}`,
]),
]);
}
case 'trace':
return section(card, 'Trace', [
` ${field('message', event.message)}`,
Expand Down Expand Up @@ -794,6 +801,19 @@ function formatDuration(durationMs: number): string {
return `${durationMs.toFixed(2)}ms`;
}

function suppressionSummaryLines(
locations: readonly SuppressedLocation[] | undefined,
safe: (text: string) => string,
muted: (text: string) => string,
): string[] {
return (locations ?? []).flatMap(({ location, reason }) => {
const target = `${safe(location.filepath)}${
location.lines === undefined ? '' : `:${location.lines.start}-${location.lines.end}`
}${location.scopeId === undefined ? '' : ` ${safe(location.scopeId)}`}`;
return [muted(` Suppressed: ${target}`), ` ${safe(reason)}`];
});
}

export function renderCheckSummary(
reports: readonly RuleReport[],
redact: TextRedactor,
Expand All @@ -807,12 +827,17 @@ export function renderCheckSummary(
const skipped = reports.filter(report => report.status === 'skipped');
const verifications = reports.reduce((total, report) => total + report.evaluations.length, 0);
const violations = countViolations(reports);
const suppressed = reports.reduce(
(total, report) => total + report.evaluations.filter(evaluation => evaluation.result.status === 'suppressed').length,
0,
);
const lines = [
[
`Rules: ${reports.length}`,
paint(`Skipped: ${skipped.length}`, skipped.length > 0 ? 'yellow' : 'dim', color),
`Verifications: ${verifications}`,
paint(`Violations: ${violations}`, violations > 0 ? 'red' : 'green', color),
...(suppressed > 0 ? [paint(`Suppressed: ${suppressed}`, 'yellow', color)] : []),
`Time: ${formatDuration(options.durationMs)}`,
].join(' '),
'',
Expand All @@ -829,17 +854,15 @@ export function renderCheckSummary(
const entries: {
readonly name: string;
readonly subject: Subject<JsonValue>;
readonly message: string | undefined;
readonly status: 'pass' | 'violation';
readonly result: RuleReport['evaluations'][number]['result'];
}[] = [];
for (const report of reports) {
for (const evaluation of report.evaluations) {
if (!outputAll && evaluation.result.status !== 'violation') continue;
entries.push({
name: report.name,
subject: evaluation.subject,
message: evaluation.result.message,
status: evaluation.result.status,
result: evaluation.result,
});
}
}
Expand All @@ -849,17 +872,23 @@ export function renderCheckSummary(
const shown = outputAll ? entries : entries.slice(0, MAX_VIOLATIONS);
for (const entry of shown) {
const subject = entry.subject;
const prefix = entry.status === 'pass' ? `${paint('✓', 'green', color)} ` : '';
const result = entry.result;
const prefix = result.status === 'pass'
? `${paint('✓', 'green', color)} `
: result.status === 'suppressed'
? `${paint('[suppressed]', 'yellow', color)} `
: '';
const ruleName = `${prefix}${paint(
name(entry.name), entry.status === 'violation' ? 'red' : 'green', color,
name(entry.name), result.status === 'violation' ? 'red' : result.status === 'suppressed' ? 'yellow' : 'green', color,
)}`;
if (subject.kind === 'group') {
lines.push(
` ${ruleName}: ${muted(
`group ${subject.group.type} ${safe(subject.group.path)} (${subject.members.length} members)`,
)}`,
);
if (entry.message !== undefined) lines.push(` ${safe(entry.message)}`);
if (result.message !== undefined) lines.push(` ${safe(result.message)}`);
if (outputAll) lines.push(...suppressionSummaryLines(result.suppressedLocations, safe, muted));
const members = subject.members.slice(0, MAX_GROUP_MEMBERS);
for (const member of members) {
lines.push(
Expand All @@ -885,7 +914,8 @@ export function renderCheckSummary(
subject.lines === undefined ? '' : `:${subject.lines.start}-${subject.lines.end}`
}`;
lines.push(` ${ruleName}: ${muted(location)}`);
if (entry.message !== undefined) lines.push(` ${safe(entry.message)}`);
if (result.message !== undefined) lines.push(` ${safe(result.message)}`);
if (outputAll) lines.push(...suppressionSummaryLines(result.suppressedLocations, safe, muted));
lines.push(...excerptLines(subject, MAX_SOURCE_LINES, redact, color, outputAll));
}
if (!outputAll && entries.length > MAX_VIOLATIONS) {
Expand Down
2 changes: 2 additions & 0 deletions src/adapter/filesystem/project.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import type { Project } from '../../app/ports.js';
import { parseCsv } from '../scope/csv.js';
import { parseJson } from '../scope/json.js';
import { parseYaml } from '../scope/yaml.js';
import { loadSuppressions } from './suppressions.js';

const SKIPPED_ANY_DEPTH: ReadonlySet<string> = new Set([
'.git',
Expand Down Expand Up @@ -259,6 +260,7 @@ export function createProject(root: string, options: ProjectOptions = {}): Proje

return {
root: resolvedRoot,
readSuppressions: () => loadSuppressions(resolvedRoot),

glob: globFiles,

Expand Down
62 changes: 62 additions & 0 deletions src/adapter/filesystem/suppressions.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
import { constants } from 'node:fs';
import { lstat, open, type FileHandle } from 'node:fs/promises';
import { join } from 'node:path';

import type { SuppressionConfig } from '../../app/suppressions.js';
import { parseSuppressionConfig } from '../../app/suppressions.js';
import { parseYaml } from '../scope/yaml.js';

const CONFIG_PATH = '.coderuler/suppressions.yml';

function emptyConfig(): SuppressionConfig {
return new Map();
}

function errorCode(error: unknown): string | undefined {
return typeof error === 'object' && error !== null && 'code' in error && typeof error.code === 'string'
? error.code
: undefined;
}

async function readConfiguration(path: string): Promise<string | undefined> {
let handle: FileHandle | undefined;
try {
const stat = await lstat(path);
if (!stat.isFile() || stat.isSymbolicLink()) {
throw new Error('must be a regular, non-symlinked file');
}
handle = await open(path, constants.O_RDONLY | constants.O_NOFOLLOW);
if (!(await handle.stat()).isFile()) throw new Error('must be a regular file');
return await handle.readFile('utf8');
} catch (error) {
if (errorCode(error) === 'ENOENT') return undefined;
throw error;
} finally {
await handle?.close();
}
}

export async function loadSuppressions(root: string): Promise<SuppressionConfig> {
try {
const directory = join(root, '.coderuler');
let directoryStat;
try {
directoryStat = await lstat(directory);
} catch (error) {
if (errorCode(error) === 'ENOENT') return emptyConfig();
throw error;
}
if (directoryStat.isSymbolicLink() || !directoryStat.isDirectory()) {
throw new Error('.coderuler must be a regular directory');
}

const source = await readConfiguration(join(directory, 'suppressions.yml'));
if (source === undefined) return emptyConfig();
const documents = parseYaml(source);
if (documents.length === 0) return emptyConfig();
if (documents.length !== 1) throw new Error('must contain exactly one nonempty YAML document');
return parseSuppressionConfig(documents[0]);
} catch (error) {
throw new Error(`${CONFIG_PATH}: ${error instanceof Error ? error.message : String(error)}`, { cause: error });
}
}
15 changes: 12 additions & 3 deletions src/app/execute.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,8 @@ import type {
RunContext,
RunEvent,
} from './ports.js';
import { applySuppressions } from './suppressions.js';
import type { EvaluationResult, SuppressionConfig } from './suppressions.js';
import { createRunLsp } from './lsp.js';
import { mapConcurrent } from './concurrency.js';
import { evaluateAi } from './evaluators.js';
Expand Down Expand Up @@ -181,6 +183,12 @@ async function executeRun(
});
return error instanceof Error ? error : new Error(message);
};
let suppressions: SuppressionConfig;
try {
suppressions = await context.project.readSuppressions();
} catch (error) {
throw fail(undefined, 'suppression', error);
}

const statCache = new Map<string, Promise<{ readonly kind: 'file' | 'directory' } | undefined>>();

Expand Down Expand Up @@ -279,7 +287,7 @@ async function executeRun(
subject: Subject<JsonValue>,
evaluationId: number,
reportFailure: (error: unknown) => void,
): Promise<CheckResult> => {
): Promise<EvaluationResult> => {
plan.evaluationStartedAt ??= performance.now();
const { name, rule } = plan;
emit({ type: 'evaluation', evaluationId, name, subject });
Expand Down Expand Up @@ -378,12 +386,13 @@ async function executeRun(
}

const normalized: CheckResult = trackedUsage === undefined ? result : { ...result, usage: trackedUsage };
emit({ type: 'result', evaluationId, name, subject, result: normalized });
const effective = applySuppressions(suppressions, name, subject, normalized);
emit({ type: 'result', evaluationId, name, subject, result: effective });
context.emit?.({
type: 'progress', phase: 'evaluation', completed: ++completedEvaluations, total: tasks.length,
});
plan.evaluationFinishedAt = performance.now();
return normalized;
return effective;
};

const plans: Plan[] = [];
Expand Down
7 changes: 4 additions & 3 deletions src/app/ports.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
import type {
AiUsage,
Awaitable,
CheckResult,
GroupSubject,
JsonValue,
LineRange,
Expand All @@ -13,6 +12,7 @@ import type {
TokenPricing,
TokenUsage,
} from './rule.js';
import type { EvaluationResult, SuppressionConfig } from './suppressions.js';

export interface DiffChanges {
readonly changedFiles: ReadonlySet<string>;
Expand All @@ -28,6 +28,7 @@ export interface Project {
readJson(path: string): Promise<JsonValue>;
readYaml(path: string): Promise<readonly JsonValue[]>;
readCsv(path: string, options?: CsvOptions): Promise<readonly CsvRow[]>;
readSuppressions(): Promise<SuppressionConfig>;
}
export interface LspBackend extends LspTools {
close(): Promise<void>;
Expand Down Expand Up @@ -109,7 +110,7 @@ export interface RunContext {

export interface RuleEvaluation {
readonly subject: Subject;
readonly result: CheckResult;
readonly result: EvaluationResult;
}

export interface RuleReport {
Expand Down Expand Up @@ -180,7 +181,7 @@ export type RunEvent =
readonly evaluationId: number;
readonly name: string;
readonly subject: Subject;
readonly result: CheckResult;
readonly result: EvaluationResult;
}
| {
readonly type: 'trace';
Expand Down
Loading
Loading