Malware Detection

Malware binaries can be visualized as grayscale images ("byteplots") by mapping each byte to a pixel intensity. This project takes that idea one step further and makes it reversible by construction: the conversion preserves the byte stream, so the images a model evaluated can be reassembled into the original executable and hash-verified against the SHA-256 recorded at conversion time.

On top of that representation this project includes training of an unsupervised anomaly detector on benign software images only and evaluates whether previously unseen malware images are flagged as anomalies without showing the model a malware sample during training.

The notebook malware_detection.ipynb implements two complementary scorers:

1. Convolutional autoencoder (from scratch) — reconstruction error as the anomaly score. Trained exclusively on harvested benign PE tiles; anything it cannot reconstruct is an anomaly.

2. Frozen DINOv2 (ViT-B/14) + Mahalanobis distance — a frozen pretrained encoder produces embeddings; a shrinkage-estimated Gaussian is fit to the benign embedding distribution; distance to that Gaussian is the anomaly score.

Both are evaluated on the theZoo malware evaluation dataset, reporting ROC-AUC, AUPR, TPR at 1% FPR, operating-point metrics and per-family detection rates — all at file level, aggregated from per-tile scores, so no executable can leak across a split boundary.

---

Contents

Overview

• Where each stage runs
• Repository contents
• Datasets

Runbook

• 1. Environment
• 2. Harvest benign PEs (Windows)
• 3. Convert in the air-gapped VM
• 4. Train and evaluate (analysis host)
• 5. Round-trip verification (Air-gapped Linux VM)
• 6. Live demonstration (offline Windows VM)

Conversion VM pre-flight

• Conversion VM Pre-flight — Guide
• How to run it
• The checks
• Interpreting the conversion summary
• Reversibility check, before transfer
• Stop conditions
• Scope

Reference

• Converter CLI
• Environment variables
• Legal & ethical note
• License & Attribution
• Credits & citations

Where each stage runs

This is the load-bearing safety property of the project. Each stage is confined to one environment and the artefacts move in one direction.

#	Stage	Environment	Input	Output
1	Harvest benign PEs	Windows	System32 / SysWOW64	bin/ + harvest_index.csv
2	Convert binaries → tiles	Isolated VM (QEMU, air-gapped)	bin/, theZoo zips	benign_tiles/, malware_tiles/ (PNG + manifest.json)
3	Train + evaluate	Analysis host	tiles + manifests only	metrics JSON, checkpoints, figures
4	Reconstruct + SHA-256 verification	Isolated VM (QEMU, air-gapped)	tiles + manifests	hash comparison on screen
5	Live execution demo	Offline Windows VM	one reconstructed binary	screen recording

The host receives only images and manifests. No stage writes a reconstructed executable on the host: reconstruct --out is refused unless MALWARE_ALLOW_BINARY_WRITE=1 and --all cannot be combined with --out.

What the host does and does not hold

Stated precisely, because the project's credibility depends on it:

• The host neither executes malware nor runs reconstruction code. The notebook contains no reconstruction helpers; the converter and CLI tooling run inside the VM. The single binary write path in malware_converter.py stays gated (--out plus MALWARE_ALLOW_BINARY_WRITE=1).

• Inside the VM, reconstruction materializes the full executable in process RAM. That is intentional: it is how the hash comparison is computed. The payload is resident until the process exits.

• The VM tools disable their own core dump on import (RLIMIT_CORE=0, PR_SET_DUMPABLE=0), so a crash does not hand the payload to the system crash handler. Override for debugging only: MALWARE_KEEP_COREDUMPS=1.

• The host does hold malware bytes inside the tile PNGs (losslessly recoverable by design) and inside the Windows VM disk image after stage 5. These are containers, not executables.

• Expect antivirus on the host to react to the tile PNGs. Add an exclusion for the tile directories before copying them across, or the scanner may quarantine your dataset.

---

Repository contents

File	Purpose
scripts/malware_converter.py	Lossless PE ↔ 256×256 tiled byteplot converter. Interactive menu, or subcommands convert, reconstruct, verify, selftest.
malware_detection.ipynb	The educational walkthrough: concept, dataset integrity, EDA, training, metrics, model cards, round-trip verification, security boundary, conclusions. Self-contained — runs inline, no script import.
scripts/harvest_benign.ps1	Read-only PowerShell harvester for known-clean Windows PEs, run in Windows.

---

Datasets

Dataset	Role	Origin
Harvested benign PEs	Training / validation / test (benign only)	System32 + SysWOW64 via harvest_benign.ps1
theZoo	Malware evaluation dataset (Windows PE only)	Password-protected ZIPs, standard password infected, with .pass / .shasum sidecars
Kaggle MaleX 200K Balanced Byteplot	Not used. Documented as related work only	khansaurooj/malex-200k-balanced-byteplot

Why the Kaggle redistribution is excluded. The audit found the archive mixes representations: its benign/ half is bigram-DCT imagery while its malware/ half is spatial-domain byteplots. A model trained across the two halves can separate their generation pipelines rather than the binaries themselves and the bigram-DCT half cannot support the round-trip demonstration in any case — pair frequencies discard file positions, so the byte stream is not uniquely recoverable. The finding applies to the audited redistribution, not to the original MaleX dataset. The notebook documents the evidence.

Both active datasets pass through the same converter, with the same width, padding and hashing settings, so the training and evaluation representations are identical by construction.

theZoo also contains non-PE binaries (Linux/Android/BAT); the converter keeps only payloads whose first two bytes are MZ, so the evaluation domain matches the benign harvest.

---

Runbook

1. Environment

# PyTorch: https://pytorch.org/get-started/locally/
pip install -r requirements.txt

2. Harvest benign PEs (Windows)

powershell -ExecutionPolicy Bypass -File .\scripts\harvest_benign.ps1 -OutDir "H:\harvest" -MaxFiles 5000

Produces bin/ (MZ-verified copies, SHA-256-prefixed names) and harvest_index.csv (source path, size, SHA-256). Copy the folder out through a share, ISO mount, virtual disk, or USB passthrough. The script does not modify the source tree and refuses an output path inside a source root.

3. Convert in the air-gapped VM

The pre-flight requires each source to sit on a read-only mount — an accidental write to a source would corrupt the hash reference you compare against afterwards. The sources below live on the read-write shared tree, so expose them read-only first; the outputs stay on the read-write tree, which is why the tree itself is not remounted.

# expose the sources read-only (skip if a source already sits on a read-only mount)
sudo mkdir -p /mnt/harvest /mnt/thezoo
sudo mount -o ro,bind /shared/harvest /mnt/harvest
sudo mount -o ro,bind /shared/theZoo /mnt/thezoo

# benign
python scripts/malware_converter.py convert --benign /mnt/harvest/bin --out /shared/benign_tiles --preflight

# malware (network disabled)
python scripts/malware_converter.py convert --malware /mnt/thezoo/malware/Binaries --out /shared/malware_tiles --preflight

The converter decrypts each archive in memory, verifies the declared sidecar SHA-256 against the archive or the payload (skipping mismatches unless --allow-unverified), enforces per-member and total size budgets and writes tiles plus a manifest.

Verify the round trip before anything leaves the VM:

python scripts/malware_converter.py reconstruct --manifest /shared/malware_tiles/manifest.json --all

Also run the self-test anywhere — it runs on seeded random bytes and contains no malware:

python scripts/malware_converter.py selftest

4. Train and evaluate (analysis host)

export MALWARE_BENIGN_TILES=/shared/benign_tiles
export MALWARE_MALWARE_TILES=/shared/malware_tiles
jupyter notebook malware_detection.ipynb

The notebook performs EDA, deduplication, the file-level split, training, threshold calibration and evaluation. Round-trip verification runs in the VM (§5 below; stage 4 in the table above) — the notebook contains no reconstruction code.

On a fresh clone, with neither tile directory present, the notebook stops early with a clear error: it requires the two tile datasets produced by §2–§3. Both manifests must exist and contain converted entries before any metric is computed. The notebook picks up the datasets automatically the moment both manifests exist at the paths above; point the two environment variables at your copied directories before drawing any conclusion.

5. Round-trip verification (Air-gapped Linux VM)

In the air-gapped conversion VM, each converted sample's tiles are reassembled in RAM — without writing a file and the recomputed SHA-256 is compared against the sha256_binary recorded in the manifest. A full-dataset run and its output are documented in the notebook's Round-Trip Verification section; the pre-flight guide below applies to the same VM before anything is transferred.

6. Live demonstration (offline Windows VM)

The demonstration rebuilds and executes WannaCry inside an isolated Windows VM with networking off and antivirus disabled: the converter script, the manifest and the sample's tiles are copied into the guest, the sample is rebuilt there in PowerShell and a snapshot is taken before it is executed.

Conversion VM Pre-flight — Guide

The pre-flight is implemented in malware_converter.py

Each check states the condition it looks for, why that condition is a stop and how to remedy it.

Target environment: the air-gapped conversion VM. The network is intentionally cut; nothing below requires it.

---

How to run it

python scripts/malware_converter.py convert \
  --malware /path/to/theZoo/malware/Binaries \
  --out /path/to/output \
  --preflight-only

--preflight-only runs each check and stops. It converts nothing and creates no output directory.

python scripts/malware_converter.py convert \
  --malware /path/to/theZoo/malware/Binaries \
  --out /path/to/output \
  --preflight

--preflight runs the same checks and then converts only if all pass. On any failure it stops before conversion begins.

Exit codes

Exit	Meaning
0	all checks passed
1	at least one check failed; conversion was blocked

What the gate ensures

The checks run before the Converter object is constructed. Converter.__init__ creates the output directory tree immediately, so running the gate earlier is what makes the claim hold: on failure, no output directory is created and nothing is written.

A blocked pre-flight leaves the output path unchanged.

The gate is read-only

The pre-flight inspects /proc and /sys and reports. It does not change swap, remount, or alter permissions or any configuration. Where a check fails, the script prints the remedy and stops; the operator applies it.

---

Without either flag

Plain convert still runs a three-check fatal subset — hardening, output-directory writability and source-path existence and blocks on failure. A user who skips the full gate cannot convert with core dumps re-enabled or a nonexistent source path.

The full gate is available on demand.

---

The checks

The count is not fixed: the source mount and dataset checks run once per source given, so a --malware-only run reports fewer lines than a mixed --malware plus --benign run. The remaining checks are unconditional.

dependencies

The converter's only third-party imports are numpy and PIL. Both load at startup, so an absent package stops the converter before the pre-flight runs; this check records the loaded versions as evidence of the VM environment. Resolve any version gap through the QEMU shared folder, not the network.

swap

The only vector by which payload bytes can reach a disk-backed medium without anyone deliberately writing them there. Under memory pressure, anonymous pages holding decrypted bytes are paged out or compressed into zram. A conversion VM has the constrained RAM that creates such pressure.

Fails if any swap device is active.

sudo swapoff -a
swapon --show

Re-run the pre-flight afterwards. Note that compressed-RAM swap (zram) is still a fail here: it is memory, but it is memory the process does not control and the check cannot distinguish one form from another.

sudo ip link set <iface> down

source mount (one line per source)

Conversion must not alter the source dataset and one needs a pristine reference for hash comparison afterward. A read-write mount means an accidental write would corrupt the reference silently.

Fails if any source path sits on a read-write mount.

# malware source
sudo mount -o ro,bind /path/to/theZoo /mnt/thezoo

# benign source
sudo mount -o ro,bind /path/to/harvest /mnt/harvest

Re-run the conversion with the command's source pointing at the bind path (--malware /mnt/thezoo/…, --benign /mnt/harvest/…). Bind only the source tree: remounting a shared tree read-only would also cover the --out path and the disk headroom check requires it writable.

If the source is already read-only, no action is needed.

dataset (one line per source)

For a theZoo source, family_for() derives the family from the directory directly under Binaries/. If the path does not point where you think, families collapse and the per-family metrics derived later are silently wrong — with no error at conversion time.

For a benign source, the check instead confirms files are present and reports how they group into top-level entries, which is what becomes the family label.

Also confirms, for theZoo sources, the .shasum and .pass sidecars that drive hash verification and password lookup. A member whose computed hash matches neither the declared binary hash nor the archive hash is skipped, not converted, unless --allow-unverified is passed. Harvested benign files carry no declared hash, so their hash_verified field is false in the manifest by design and is not a failure.

Fails if a source is missing, holds no archives or files, or resolves to zero families.

A high skip count after conversion is a finding to investigate, not noise to accept.

Fails if the output path is unwritable, or holds less than 2x the source dataset size.

Default budgets are 64 MB per member, 2 GB total read and 32 members per archive. On a large theZoo the 2 GB cap will bind; the check says so when the source exceeds it. Decide deliberately whether to raise --max-total-mb or to convert family-by-family into separate --out directories, which also keeps each dataset manifest clean and independently auditable.

hardening

harden_process() runs at module scope, so it fires whenever malware_converter is imported or run as a command. It sets RLIMIT_CORE=(0,0) and PR_SET_DUMPABLE=0, closing the path by which a crash would hand a reconstructed payload to the system crash handler.

Fails if either setting is not in place, or if MALWARE_KEEP_COREDUMPS=1 is set in the environment. That variable re-enables core dumps — for debugging, not with live payloads.

MALWARE_ALLOW_BINARY_WRITE=1 is unrelated to this check: it is required only for reconstruct --out and is not needed for convert.

selftest

Round-trips seeded random bytes through the production encode/decode path with no malware involved. It separates an environment fault from a dataset fault, which is why it runs before samples rather than after.

Fails if any assertion fails. A non-zero result is a hard stop — diagnose the environment before touching samples.

---

Interpreting the conversion summary

After conversion, the summary block lists only nonzero statuses — zero-count rows are omitted entirely, so a missing line means zero, not "not checked."

Read each skipped_duplicate, skipped_too_large, skipped_non_pe and error row rather than ignoring it. Each carries a note field explaining itself.

---

Reversibility check, before transfer

The tile set must be shown reversible on the VM itself, before anything crosses the air gap. A set that does not reassemble is not worth transferring.

python scripts/malware_converter.py reconstruct \
  --manifest /path/to/output/manifest.json --all

Each sample line must begin with [MATCH]; any [MISMATCH] means do not transfer that tile set. Exit code must be 0.

--all verifies in memory and writes nothing, so it needs no opt-in. The closing message is accurate and worth reading literally: payload bytes do remain resident in the process until it exits. That is expected, it is how the hash comparison is computed and is the reason swap must be off before this step runs.

Confirm the output tree contains no binaries:

find /path/to/output -type f ! -name "*.png" ! -name "*.json"

This must print nothing. Transfer only manifest.json and images/.

---

Stop conditions

Signal	Action
Any [FAIL] line in the pre-flight report	remedy per the check above, re-run
Swap active	swapoff -a, re-verify
Missing numpy / PIL	resolve via shared folder
Hardening reports an override	unset MALWARE_KEEP_COREDUMPS, re-run
Selftest failure	stop; diagnose environment before touching samples
Reconstruct mismatch	do not transfer; re-run selftest, inspect affected tiles
Unexpected skip or error tally	read the note field, decide consciously

---

Scope

This covers the conversion VM only: producing a verifiable tile dataset from theZoo.

The later demonstration — reconstructing a binary from those tiles and executing it in an isolated Windows 11 VM with snapshot and revert is a separate protocol with its own pre-flight, run at a different stage of the project. It is referenced in the notebook's Security Boundary section and is not covered here.

Reconstruction writes to disk only under reconstruct --out with MALWARE_ALLOW_BINARY_WRITE=1 set, which is intended for the isolated execution VM and not for the conversion or analysis hosts.

# rebuild a single sample and write it to disk — VM ONLY
MALWARE_ALLOW_BINARY_WRITE=1 \
  python scripts/malware_converter.py reconstruct \
    --manifest malware_tiles/manifest.json --sample <sample_id> --out ./reconstructed.exe

python scripts/malware_converter.py verify --file ./reconstructed.exe \
  --manifest malware_tiles/manifest.json --sample <sample_id>

Matching hashes are indication of byte-for-byte identity, contingent on SHA-256 collision resistance. Alternatively --sha256 <hex> verifies against a hash you supply.

---

Converter CLI

python scripts/malware_converter.py                          # interactive numbered menu
python scripts/malware_converter.py selftest                 # seeded random-byte check (no malware)
python scripts/malware_converter.py convert --benign DIR  --out DIR
python scripts/malware_converter.py convert --malware DIR  --out DIR
python scripts/malware_converter.py reconstruct --manifest M --all
python scripts/malware_converter.py reconstruct --manifest M --sample ID [--out PATH]
python scripts/malware_converter.py verify --file F --manifest M --sample ID
python scripts/malware_converter.py verify --file F --sha256 HEX

convert options: --password, --allow-unverified, --max-member-mb, --max-total-mb, --max-members, --demo-families.

Reconstruction with --out additionally requires MALWARE_ALLOW_BINARY_WRITE=1 and a single --sample; the menu asks you to type WRITE to confirm.

---

Environment variables

Variable	Default	Meaning
MALWARE_SEED	42	Reproducibility seed
MALWARE_BENIGN_TILES	./data/benign_tiles	Benign tile directory (must contain manifest.json)
MALWARE_MALWARE_TILES	./data/malware_tiles	Malware tile directory (must contain manifest.json)
MALWARE_SPLIT_PATH	./split_data	File-level split root
MALWARE_OUTPUT_DIR	./malware_cnn_outputs	Checkpoints, metrics, reports
MALWARE_IMAGE_SIZE	256	Model input size (native tile size is 256)
MALWARE_AE_BATCH_SIZE	64	Training batch size
MALWARE_AE_MAX_EPOCHS	50	Epoch budget (early stopping applies)
MALWARE_NUM_WORKERS	4	DataLoader workers
MALWARE_ALLOW_BINARY_WRITE	—	1 permits reconstruct --out
MALWARE_KEEP_COREDUMPS	—	1 disables the core-dump hardening (debugging only)

---

Legal & ethical note

The contents of this repository are intended for educational and research purposes only. Users must ensure compliance with all applicable local and international laws when utilizing these tools. Material from this project must not be deployed to facilitate unauthorized or illicit activities. When handling sensitive or third-party data, adherence to relevant privacy regulations and institutional policies is required. See DISCLAIMER.md.

---

License & Attribution

This project is licensed under the Creative Commons Attribution 4.0 International (CC BY 4.0) license.

You are free to use, share, copy, modify and redistribute this material for any purpose (including commercial use), provided that proper attribution is given.

Attribution requirements

Any reuse, redistribution, or derivative work must include:

1. The creator's name: HelixCipher
2. A link to the original repository: https://github.com/HelixCipher/malware-detection
3. An indication of whether changes were made
4. A reference to the license (CC BY 4.0)

Example attribution

This work is based on Malware Detection by HelixCipher. Original source: https://github.com/HelixCipher/malware-detection Licensed under the Creative Commons Attribution 4.0 International (CC BY 4.0).

See ATTRIBUTION.md and LICENSE.

---

Credits & citations

• Byteplot visualization — L. Nataraj, S. Karthikeyan, G. Jacob, B. S. Manjunath, Malware Images: Visualization and Automatic Classification.

• Malware dataset — theZoo.

• Frozen encoder — M. Oquab et al., DINOv2: Learning Robust Visual Features without Supervision.

• Shrinkage covariance — O. Ledoit, M. Wolf, A well-conditioned estimator for large-dimensional covariance matrices.

• Related work, not used here — F. Qaiser, B. Tahir, M. A. Mughal, N. Shamim, ViPER: Vision-based Packing-Aware Encoder for Robust Malware Detection. ViPER is a supervised dual-head (malware class + packing state) LoRA-adapted ViT-B/14 reporting ROC-AUC 0.9260 on 200k PE byteplot images. This project is unsupervised and uses a frozen encoder with a benign-only Gaussian, so the two methods are not interchangeable.

MaleX and bigram-DCT representation — T. M. Mohammed, L. Nataraj, S. Chikkagoudar, S. Chandrasekaran, B. S. Manjunath, Malware Detection Using Frequency Domain-Based Image Visualization and Deep Learning. The MaleX dataset (~1M Windows executables) is described in this paper.

@inproceedings{mohammedmalware,
  title={Malware Detection Using Frequency Domain-Based Image Visualization and Deep Learning},
  author={Mohammed, Tajuddin Manhar and Nataraj, Lakshmanan and Chikkagoudar, Satish and Chandrasekaran, Shivkumar and Manjunath, BS},
  booktitle={Proceedings of the 54th Hawaii International Conference on System Sciences},
  pages={7132},
  year={2021}
}