Document Forgery Detection with Error Level Analysis

Detects tampered identity documents using Error Level Analysis (ELA) and noise-residual forensics, validated by stratified cross-validation and a controlled ablation study that quantifies exactly what each feature group contributes.

Verified results (reproduce with python run_pipeline.py):

Metric	Value
Test suite	71 tests passing
Code coverage	96%
Cross-validated accuracy	88.3%
Cross-validated F1	0.860
ROC AUC	0.926
Precision	0.975
Improvement over pixel baseline	+56.1% F1
Feature extraction	59 ms/image

---

The forensic principle

When a JPEG is re-encoded, regions already compressed many times reconstruct with a low error, while freshly pasted or edited content reconstructs with a high error. Re-saving the image at a known quality and measuring the residual makes that mismatch measurable — this is Error Level Analysis.

authentic region:  capture → JPEG → JPEG → JPEG   (deep history, low residual)
tampered region:   fresh pixels ──────→ JPEG      (shallow history, high residual)
                                                   ↑ ELA sees the discontinuity

Controlling the confound

A naive version of this dataset is trivially solvable: if authentic images get one JPEG pass and tampered images get two, compression count alone leaks the label and the model learns a shortcut rather than detecting tampering. An early iteration of this project scored a suspicious F1 = 1.000 for exactly that reason.

The fix: both classes receive an identical compression history. Authentic and tampered images are aged through the same two low-quality cycles and the same final save. The only difference is the localised edit. The honest 0.860 F1 reported above is measured under that control.

Tampering simulated

Manipulation	Description
copy_move	A patch duplicated elsewhere in the same document
splice	Fresh, never-compressed content pasted from another document
text_replace	A field value overwritten — the classic identity-document edit

Feature set (17 forensic features)

ELA statistics (10) — ela_mean, ela_std, ela_max, ela_p95, ela_p99, ela_block_var_mean, ela_block_var_std, ela_block_max_ratio, ela_hot_block_frac, ela_edge_density

Block-wise statistics matter most: a forgery is localised, so the discriminating signal is the contrast between a hot block and its neighbours, not the image-wide average.

Noise residual (3) — noise_std, noise_block_var_std, noise_kurtosis Laplacian sharpness (2) — lap_var, lap_block_var_std Colour (2) — sat_mean, sat_std

Ablation study — what actually works

Feature group	F1	AUC	Features
ELA only	0.862	0.912	10
All features	0.860	0.926	17
No ELA	0.537	0.585	7
Noise only	0.540	0.579	3
Laplacian only	0.428	0.468	2

The finding: ELA carries essentially all of the signal. Removing it collapses F1 from 0.862 to 0.537 — barely above chance. Noise and Laplacian features add nothing on their own, though they do lift AUC slightly when combined with ELA.

This is the kind of result an ablation exists to surface: without it, one would wrongly credit the full 17-feature stack.

Quick start

python run_pipeline.py                          # full pipeline + ablation
python run_pipeline.py --n-per-class 100        # larger dataset
python run_pipeline.py --min-f1 0.75            # CI quality gate
python run_pipeline.py --importance             # permutation importance
python -m pytest tests/ -v --cov=src            # 71 tests

Architecture

src/synth.py         Document renderer + 3 tampering simulators
src/features.py      ELA, noise residual, Laplacian, colour features
src/classifier.py    Logistic regression, k-fold CV, ROC AUC (pure numpy)
run_pipeline.py      5-stage pipeline with ablation and quality gate
tests/               71 tests across features, classifier, generator

Everything — logistic regression, gradient descent, stratified k-fold, ROC AUC with tie handling, standardisation — is implemented in numpy. No scikit-learn, so the maths is auditable rather than opaque.

Methodology notes

• Scaling inside the fold. The StandardScaler is fitted only on each training split, so test-set statistics never leak into training. Fitting it once on the whole dataset would inflate the reported score.
• Stratified folds. Class balance is preserved in every fold, tested explicitly.
• Negative control. A test asserts that random labels score near chance (AUC < 0.80). If that ever failed, the pipeline would be leaking.
• Effect-size tests. The suite asserts Cohen's d > 0.8 on ela_max, so a regression that destroys the forensic signal fails the build even if the code still runs.
• High precision by design. At 0.975 precision the detector rarely flags an authentic document — the right trade-off when a false accusation is costlier than a missed forgery.

Tech stack

Python 3.13 · NumPy · Pillow · pytest · pytest-cov