Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
133 changes: 102 additions & 31 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,50 +3,121 @@
Thanks for your interest in improving Diskern! Contributions of all kinds
are welcome — bug reports, docs, rules for the safety database, and code.

**New here?** [docs/DEVELOPMENT.md](docs/DEVELOPMENT.md) gets the project
running on your machine, including what to install and what the build
errors mean. This file is about what to work on and how to land it.

## Ground rules

Diskern's [safety principles](README.md#principles-non-negotiable) are
non-negotiable. PRs that make scanning mutate state, hard-delete files, or
let a model/network call decide a safety verdict will not be accepted.
non-negotiable. PRs that make scanning mutate state, hard-delete files,
or let a model or network call decide a safety verdict will not be
accepted.

Two consequences worth stating plainly, because they come up:

- **A wrong `safe` verdict is a bug of a different class.** `safe` and
`review` are both *actionable* in the app — the UI offers to quarantine
either. A rule that matches more than it means doesn't produce a
cosmetic error, it offers to move someone's data. When in doubt use
`review`, and when still in doubt leave it `unknown`: `report::build`
drops unknown entries, so they never reach the user as an actionable
row.
- **The engine decides, the frontends display.** The CLI and the app both
call the same functions in `diskern-core` and neither contains
scanning or classification logic. A new capability goes into the engine
first, with tests, and the frontends render what it returns.

## Good first contributions

**Rules for the safety database.** This is the most useful thing you can
do and the easiest to start on. Diskern only knows what its rules tell
it, and the shipped set is deliberately tiny — thirteen rules covering
Chrome, Firefox and pip caches, Rust and Node build output, system logs
and crash dumps, installer packages, temp directories, and a few
protected Windows paths. Everything else on your disk classifies as
`unknown`, and `report::build` drops those, so it never reaches the user
at all.

If you know where an application on your platform keeps its regenerable
cache, that is a rule nobody else can write as well as you. Format,
verdict levels and the safety reasoning are in
[docs/RULES.md](docs/RULES.md); the rules themselves are JSON in
[`crates/diskern-core/rules/base.json`](crates/diskern-core/rules/base.json),
and adding one needs no Rust.

**Issues labelled `good first issue`** are scoped to a single file with
the reasoning already worked out.

## Getting started
**Documentation.** If something here or in `docs/` was wrong or missing
when you followed it, that is a bug report worth filing even if you
don't fix it.

```sh
# Engine + CLI — no GUI dependencies needed
cargo test -p diskern-core
cargo run -p diskern-cli -- scan ~/Downloads
## The loop

# Desktop app (needs Tauri v2 prerequisites)
cd app && npm install && npm run tauri dev
```
1. Fork, and branch from `main`. Branch names follow the commit prefixes
below: `fix/…`, `feat/…`, `docs/…`, `ci/…`.
2. Make the change, with a test that fails without it.
3. Run [what CI runs](docs/DEVELOPMENT.md#5-running-the-tests).
4. Open the PR.

See each section's README for details:
[`crates/diskern-core`](crates/diskern-core/README.md) ·
[`crates/diskern-cli`](crates/diskern-cli/README.md) ·
[`app`](app/README.md) · [`site`](site/README.md)
### Commits

## Before you open a PR
`type(scope): summary in the imperative`, where type is `feat`, `fix`,
`docs`, `test`, `refactor`, `perf`, `style`, `chore` or `ci`, and scope
is usually the module or crate (`rules`, `scanner`, `app`, `cli`).

1. `cargo fmt --all` — CI enforces formatting.
2. `cargo clippy --workspace` — fix new warnings.
3. `cargo test --workspace` — all tests green.
4. Keep commits small and focused; one logical change per commit.
One logical change per commit. The body matters more than the summary:
say what was wrong and why the fix is the right shape, not what the diff
already shows. `git log` in this repo is the reference — the useful
messages explain a decision someone would otherwise have to re-derive.

CI runs the same checks on every PR, plus a spell checker
([typos](https://github.com/crate-ci/typos), config in `_typos.toml`),
a markdown link checker, and lint/build of the site and app frontends.
A weekly audit workflow additionally scans dependencies for RustSec
advisories and docs for dead external links.
### Pull requests

## Adding safety rules
The [template](.github/PULL_REQUEST_TEMPLATE.md) asks for what and why.
Include the reasoning you'd want if you were reviewing it cold: what
breaks without the change, what you considered and rejected, and
anything you're unsure about — an explicit "I'm not certain this is the
right layer" gets a better review than silence.

Rules live in [`crates/diskern-core/rules/base.json`](crates/diskern-core/rules/base.json)
and are the main way to contribute — see [docs/RULES.md](docs/RULES.md)
for the format, verdict levels, and guidelines. Prefer conservative
verdicts: when in doubt, use `review` rather than `safe`.
If your change alters a verdict, say so in the description. Verdicts are
what the app acts on, so a rule or risk change is a behaviour change
even when the diff looks like data.

Before you open it:

1. `cargo fmt --all`
2. `cargo clippy -p diskern-core -p diskern-cli --all-targets -- -D warnings`
3. `cargo test -p diskern-core -p diskern-cli`
4. Update [CHANGELOG.md](CHANGELOG.md) under `Unreleased` if the change
is user-visible.

`--workspace` instead of `-p …` also builds the desktop app, which needs
the [platform webview
dependencies](docs/DEVELOPMENT.md#1-what-you-need). CI compiles the app
separately, so scoping to the two crates locally is fine.

## Where things go

| You want to… | Goes in |
| --- | --- |
| Teach Diskern about a new cache or artifact directory | [`rules/base.json`](crates/diskern-core/rules/base.json) |
| Change how a verdict is decided | `crates/diskern-core/src/{rules,risk,graph}.rs` |
| Change what a scan finds or how it reports | `crates/diskern-core/src/{scanner,dedup,report}.rs` |
| Change what quarantine does | `crates/diskern-core/src/actions.rs` — the only module that writes |
| Add a terminal flag or change CLI output | `crates/diskern-cli/src/main.rs` |
| Change the app's UI | `app/src/` |
| Expose engine behaviour to the app | `app/src-tauri/src/commands.rs`, then `app/src/` |

[docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) walks a scan through the
engine stage by stage if you want the fuller picture first.

## Reporting bugs

Open an issue with your OS, the command or app action you ran, and what
you expected vs. what happened. For security issues, see
[SECURITY.md](SECURITY.md) — please don't open a public issue.
you expected versus what happened. For a wrong verdict, the path that
was misclassified and what it actually is are the two things that make
it fixable — a `diskern scan <dir> --json` excerpt is ideal.

For security issues, see [SECURITY.md](SECURITY.md) — please don't open
a public issue.
15 changes: 10 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,9 @@ site/ landing page (GitHub Pages)
docs/ architecture and release docs
```

Project docs live in [docs/](docs/README.md) — architecture, the
[rules database](docs/RULES.md), an [FAQ](docs/FAQ.md), and release
Project docs live in [docs/](docs/README.md) — [running
locally](docs/DEVELOPMENT.md), architecture, the [rules
database](docs/RULES.md), an [FAQ](docs/FAQ.md), and release
instructions. Each section has its own README:
[core](crates/diskern-core/README.md) ·
[cli](crates/diskern-cli/README.md) ·
Expand All @@ -42,14 +43,18 @@ instructions. Each section has its own README:
## Development

```sh
# Engine + CLI (no GUI deps needed)
cargo test -p diskern-core
# Engine + CLI no GUI dependencies needed
cargo test -p diskern-core -p diskern-cli
cargo run -p diskern-cli -- scan ~/Downloads

# Desktop app (needs Tauri v2 prerequisites: https://v2.tauri.app/start/prerequisites/)
# Desktop app
cd app && npm install && npm run tauri dev
```

[docs/DEVELOPMENT.md](docs/DEVELOPMENT.md) has the full setup: what to
install per platform, how to run what CI runs, and what the build
failures mean.

## Releasing

See [docs/RELEASING.md](docs/RELEASING.md). Short version: set the two
Expand Down
27 changes: 23 additions & 4 deletions app/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,20 +6,39 @@ Tauri v2 desktop app: React frontend ([`src/`](src)) over the Rust engine
## Develop

Needs the [Tauri v2 prerequisites](https://v2.tauri.app/start/prerequisites/)
(Rust toolchain + platform WebView deps), then:
(Rust toolchain + platform WebView deps) — see
[docs/DEVELOPMENT.md](../docs/DEVELOPMENT.md) for the per-platform list
and the errors you get without them. Then:

```sh
npm install
npm run tauri dev
```

`npm run build` compiles the frontend alone, without touching Rust,
which is enough to check a JSX change.

## How it talks to the engine

The frontend never touches the filesystem itself. It invokes Tauri
commands defined in [`src-tauri/src/commands.rs`](src-tauri/src/commands.rs),
which call into [`diskern-core`](../crates/diskern-core). During a scan the
backend emits `scan-progress` events (~every 150ms) so the UI can show a
live file counter.
which call into [`diskern-core`](../crates/diskern-core). During a scan
the backend emits `scan-progress` events (~every 150ms) so the UI can show
a live file counter.

| Command | Does |
| --- | --- |
| `start_scan` | Read-only scan; returns a report, or `null` if cancelled |
| `cancel_scan` | Stops the scan in flight |
| `quarantine_finding` | Moves one file to quarantine, re-classifying server-side first |
| `list_quarantine` | Everything currently quarantined, read from the manifest |
| `restore_quarantined` | Puts one file back where it came from |
| `purge_quarantine` | Empties quarantine for good — the only deletion in the app |

Everything except `quarantine_finding` and `purge_quarantine` is
read-only. Quarantine is manifest-backed, so what was moved in one
session is still restorable in the next; the Quarantine panel renders
before any scan has been run for exactly that reason.

## Updater

Expand Down
17 changes: 15 additions & 2 deletions crates/diskern-cli/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,9 @@ diskern scan ~/Downloads --json
### Output

Findings are grouped by verdict (safest first), then by category within
each verdict, with the rule that matched printed underneath as the
evidence:
each verdict, with every reason printed underneath as the evidence — the
rule that matched, and anything that changed the verdict from the rule's
own:

```text
Scanned 84,213 files.
Expand All @@ -32,9 +33,21 @@ Safe to remove — 802 findings · 6.1 GB
Browser cache · 641 · 4.8 GB
412.0 MB /home/u/.cache/google-chrome/Default/Cache/data_2
matched rule chrome-cache: Chrome browser cache. …
not accessed in 210 days
… … … 636 more

Risky — not recommended — 3 findings · 0 B
Build artifacts · 3 · 0 B
0 B /home/u/work/api/node_modules/.package-lock.json
matched rule node-modules: Node.js dependencies. …
referenced by 3 projects
```

`referenced by 3 projects` is the impact graph: three live projects have
that `node_modules` on their dependency path, so the verdict drops from
`review` to `risky` and its bytes stop counting as reclaimable —
nothing will offer to move it.

| Flag | Default | Effect |
| ----------- | ------- | ------------------------------------------------------- |
| `--top N` | `5` | Findings shown per category; `0` shows every one. |
Expand Down
12 changes: 9 additions & 3 deletions crates/diskern-core/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,17 @@ and tests alike.
## Pipeline

```text
scanner ──► index ──► dedup ──► graph ──► rules + risk ──► report
(optional) ai narration
scanner ──► graph ──► rules + risk ──► dedup ──► report
(optional) ai narration
```

Dedup runs *after* classification: a duplicate set is an offer to keep
one copy and drop the rest, so entries nothing will act on have no place
in one — and hashing them is the most expensive way to produce a number
nobody can use. [`report::build_with`](src/report.rs) is the whole
sequence in one function.

## Modules

| Module | Purpose |
Expand Down
15 changes: 12 additions & 3 deletions crates/diskern-core/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,19 @@
//! ## Pipeline
//!
//! ```text
//! scanner ──► index ──► dedup ──► graph ──► rules + risk ──► report
//! │
//! (optional) ai narration
//! scanner ──► graph ──► rules + risk ──► dedup ──► report
//!
//! (optional) ai narration
//! ```
//!
//! Dedup runs *after* classification, not before: a duplicate set is an
//! offer to keep one copy and drop the rest, so entries nothing will act
//! on — protected system files, stores a live project references — have
//! no business in one, and hashing them is the most expensive way to
//! produce a number nobody can use. The graph runs first because the
//! verdict it feeds ([`risk::downgrade`]) is part of deciding that.
//!
//! [`report::build_with`] is where the whole sequence lives.

pub mod actions;
pub mod dedup;
Expand Down
80 changes: 79 additions & 1 deletion docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,87 @@ marketing site.
| `crates/diskern-cli` | `diskern` binary — engine from the terminal |
| `app/` | Tauri v2 desktop app (React frontend, Rust backend) |
| `site/` | Landing page, deployed to GitHub Pages |
| `docs/` | Project docs (this file, [RELEASING.md](RELEASING.md)) |
| `docs/` | Project docs — [running locally](DEVELOPMENT.md), [rules](RULES.md), [releasing](RELEASING.md) |
| `.github/workflows/` | CI, release builds, Pages deploy |

## A scan, end to end

One function orchestrates almost all of it:
[`report::build_with`](../crates/diskern-core/src/report.rs). Reading it
alongside this section is the fastest way into the engine.

```text
scanner ──► graph ──► rules + risk ──► dedup ──► report
```

**1. Walk.** [`scanner::scan`](../crates/diskern-core/src/scanner.rs)
walks the roots in parallel with `jwalk`, skipping excluded directories,
and returns a `FileEntry` per file: path, size, modified and accessed
times, whether it is a symlink. Metadata only — nothing is read or
hashed here, and nothing is written ever.

**2. Graph.**
[`graph::ImpactGraph::from_entries`](../crates/diskern-core/src/graph.rs)
makes one pass looking for two things: directories holding a project
marker (`Cargo.toml`, `package.json`, `pyproject.toml`), and directories
that are dependency stores (`target`, `node_modules`, a virtualenv). It
links each project to the store it owns, so the engine can later answer
"how many live projects reference this?".

**3. Classify.** [`rules::RulesDb::classify`](../crates/diskern-core/src/rules.rs)
matches the normalized path against the rule globs — first match wins,
which is why `protected` rules are listed first — yielding a `Category`
and a base `Verdict`. [`risk::downgrade`](../crates/diskern-core/src/risk.rs)
then applies the graph's answer. Evidence can only make a verdict *more*
cautious, never less, and `Protected` is final.

**4. Dedup.** [`dedup::find_duplicates_filtered`](../crates/diskern-core/src/dedup.rs)
buckets by size, BLAKE3-hashes only the files whose sizes collide (which
skips most of a real disk), then buckets by hash. It runs after
classification so that entries nothing will act on can sit it out — they
have no place in an offer to keep one copy and drop the rest, and
hashing them is the most expensive way to produce an unusable number.

**5. Report.** [`risk::assess`](../crates/diskern-core/src/risk.rs) adds
an informational score and per-file evidence, and each entry becomes a
`Finding` carrying its category, verdict, reclaimable bytes and the
`reasons` that justify them. The headline total counts findings in full
and adds only the duplicate copies nothing has counted yet.

Acting on a finding is a separate call:
[`actions::quarantine`](../crates/diskern-core/src/actions.rs) is the
only function in the crate that writes, it refuses `Risky` and
`Protected`, and it records every move in a manifest so a restore
survives the process exiting.

## Adding a feature

Work out which layer owns it before writing anything — the answer is
usually further down than it first looks.

**Does a rule cover it?** Teaching Diskern that some directory is a cache
is data, not code: add it to
[`rules/base.json`](../crates/diskern-core/rules/base.json) with a test.
No Rust, no new code paths, and it reaches both frontends at once.

**Does it change a verdict?** Then it belongs in `rules`, `risk` or
`graph`, and the constraint in [design decisions](#design-decisions)
applies: evidence may only make a verdict more cautious. Add the evidence
as a `reason` too — a verdict the user can't see the basis for is the
thing Diskern exists not to ship.

**Does it change what a scan finds or reports?** `scanner`, `dedup` and
`report`. Watch the cancellation flag: anything that loops over every
entry has to check it, or a Cancel arriving during that stage does
nothing.

**Only then, the frontends.** Add a Tauri command in
[`commands.rs`](../app/src-tauri/src/commands.rs) and register it in
[`lib.rs`](../app/src-tauri/src/lib.rs), or a flag in
[`main.rs`](../crates/diskern-cli/src/main.rs). Both should be thin
enough that the interesting part of your change is already tested in the
engine before either sees it.

## Design decisions

**Frontends are thin.** The CLI and the app both call the same engine
Expand Down
Loading
Loading