Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -202,6 +202,28 @@ jobs:
- name: Test
run: npm test

# deploy/scan2-*.sh (SCAN-2, randprotocol/monorepo#16): shellcheck, then the offline tests (a
# fake explorer on 127.0.0.1, a fake journalctl). No host is contacted.
deploy-scripts:
name: Deploy scripts (SCAN-2)
runs-on: ubuntu-latest
timeout-minutes: 5
permissions:
contents: read

steps:
- name: Checkout
uses: actions/checkout@v4

- name: Shellcheck
run: shellcheck deploy/scan2-verify.sh deploy/scan2-log-audit.sh deploy/tests/*.sh

- name: scan2-verify against a fake explorer
run: bash deploy/tests/scan2-verify.test.sh

- name: scan2-log-audit against a fake journal
run: bash deploy/tests/scan2-log-audit.test.sh

# Docker build test
docker:
name: Docker Build
Expand Down
23 changes: 23 additions & 0 deletions deploy/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,3 +27,26 @@ validator e. Do not re-enable it until all of these hold:
`chain-activity.sh call` without it is refused. `PROVE=` (empty) would prove locally too.

Then: `systemctl daemon-reload && systemctl enable --now randscan-activity.timer`.

## SCAN-2: the history page never sends a nullifier (randprotocol/monorepo#16)

Since 66bc1a7 the history page downloads the public nullifier set (`GET /api/v1/nullifiers`, pages
of 5,000) and matches the wallet's own nullifiers in the browser; `POST /api/v1/nullifiers/lookup`
is removed and the frontend has no single-nullifier request (`frontend/tests/spent.test.mjs` fails
the build if one comes back). Two scripts make the owner's remaining steps checkable:

1. **After a deploy**, from any machine (read-only, three kinds of HTTP request):

deploy/scan2-verify.sh --record https://randscan.org

It passes only when the lookup route answers 404/405, the paged set is served, and no
JavaScript chunk the served history page references contains a nullifier lookup. Paste the
`SCAN-2 deploy check: … passed …` line into the issue as the deploy record.
2. **On the explorer host**, as root: `deploy/scan2-log-audit.sh` (a dry run: counts per source,
never prints a matching line), then `deploy/scan2-log-audit.sh --purge --before <deploy day>`
if it found any. `--purge` drops the whole journal older than that day (journald cannot delete
single lines) and the matching lines of the plain log files. The units ship with
`tower_http=warn` and the Caddyfile has no `log` directive, so a host that always ran this
configuration should report zero; the script is how that is shown rather than assumed.

Both are tested offline: `bash deploy/tests/scan2-verify.test.sh`, `bash deploy/tests/scan2-log-audit.test.sh`.
110 changes: 110 additions & 0 deletions deploy/scan2-log-audit.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
#!/usr/bin/env bash
# SCAN-2 (randprotocol/monorepo#16), owner follow-up: logs on the explorer host written before the
# fix may hold visitors' nullifiers. Before 66bc1a7 the history page POSTed them to
# /api/v1/nullifiers/lookup (bodies; a request log shows who asked and when) and the transaction
# opener asked GET /api/v1/nullifiers/<nullifier> per note (the nullifier is in the URL).
#
# Run ON the explorer host, as root. The default is a DRY RUN: it counts matching lines per source
# and prints the date range, and never prints a matching line (so a nullifier cannot leak through
# this script's output or a terminal log).
#
# deploy/scan2-log-audit.sh # dry run: counts only
# deploy/scan2-log-audit.sh --purge --before 2026-11-10
# rotates the journal, removes journal files older than --before (journald cannot delete
# single lines: the whole journal up to that date goes, for every unit), and deletes the
# matching LINES from the plain log files below. --before is the day the fixed build was
# deployed; it is required, there is no default.
#
# Sources: journald for the units in UNITS (default: randscan-api randscan-frontend caddy), and
# the plain files matched by LOG_GLOBS (default: /var/log/caddy/*.log /var/log/nginx/*.log
# /var/log/syslog /var/log/syslog.1). Compressed rotations (*.gz) are counted with zgrep and, on
# --purge, deleted whole when they match.
#
# Exit: 0 nothing found (or purged), 3 matches found in a dry run, 2 usage error.
# Tested with a fake journalctl and log files: deploy/tests/scan2-log-audit.test.sh.
set -u

purge=0
before=""
while [ $# -gt 0 ]; do
case "$1" in
--purge) purge=1 ;;
--before) shift; before=${1:-} ;;
-h|--help) sed -n '2,26p' "$0"; exit 0 ;;
*) echo "scan2-log-audit: unknown argument '$1'" >&2; exit 2 ;;
esac
shift
done
if [ "$purge" -eq 1 ]; then
case "$before" in
[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]) ;;
*) echo "scan2-log-audit: --purge needs --before YYYY-MM-DD (the day the fixed build went live)" >&2; exit 2 ;;
esac
fi

UNITS=${UNITS:-randscan-api randscan-frontend caddy}
LOG_GLOBS=${LOG_GLOBS:-/var/log/caddy/*.log /var/log/nginx/*.log /var/log/syslog /var/log/syslog.1}
JOURNALCTL=${JOURNALCTL:-journalctl}
# A lookup POST, or a /nullifiers/<64 hex> URL. The paged GET /nullifiers?after=<cursor> is the
# fixed build's own request and is not matched.
PATTERN='/api/v1/nullifiers/(lookup|[0-9a-fA-F]{64})'

found=0
report() { printf '%-9s %-44s %s\n' "$1" "$2" "$3"; }

for u in $UNITS; do
if ! command -v "$JOURNALCTL" >/dev/null 2>&1; then
report journal "$u" "skipped: no journalctl"; continue
fi
n=$("$JOURNALCTL" -u "$u" --no-pager -o short-iso 2>/dev/null | grep -cE "$PATTERN" || true)
if [ "${n:-0}" -gt 0 ]; then
first=$("$JOURNALCTL" -u "$u" --no-pager -o short-iso 2>/dev/null | grep -E "$PATTERN" | head -1 | cut -c1-10)
last=$("$JOURNALCTL" -u "$u" --no-pager -o short-iso 2>/dev/null | grep -E "$PATTERN" | tail -1 | cut -c1-10)
report journal "$u" "$n matching lines, $first .. $last"
found=$((found + n))
else
report journal "$u" "0 matching lines"
fi
done

files=""
# LOG_GLOBS is a list of globs on purpose: unquoted so the shell expands them.
# shellcheck disable=SC2086
for f in $LOG_GLOBS; do
[ -f "$f" ] || continue
files="$files $f"
for z in "$f".*.gz; do [ -f "$z" ] && files="$files $z"; done
done
filehits=""
for f in $files; do
case "$f" in
*.gz) n=$(zgrep -cE "$PATTERN" "$f" 2>/dev/null || true) ;;
*) n=$(grep -cE "$PATTERN" "$f" 2>/dev/null || true) ;;
esac
report file "$f" "${n:-0} matching lines"
if [ "${n:-0}" -gt 0 ]; then found=$((found + n)); filehits="$filehits $f"; fi
done

if [ "$purge" -eq 0 ]; then
if [ "$found" -gt 0 ]; then
echo "DRY RUN: $found matching lines. Re-run with --purge --before <deploy day> to remove them."
exit 3
fi
echo "DRY RUN: nothing found in the sources above."
exit 0
fi

echo "PURGE: journal files older than $before, and the matching lines of:${filehits:- (no plain file)}"
now=$(date -u +%s)
cut=$(date -u -d "$before" +%s 2>/dev/null || date -u -j -f '%F %T' "$before 00:00:00" +%s) || { echo "scan2-log-audit: cannot read the date '$before'" >&2; exit 2; }
age=$((now - cut))
[ "$age" -ge 1 ] || { echo "scan2-log-audit: --before $before is not in the past" >&2; exit 2; }
"$JOURNALCTL" --rotate || { echo "scan2-log-audit: journalctl --rotate failed" >&2; exit 1; }
"$JOURNALCTL" --vacuum-time="${age}s" || { echo "scan2-log-audit: journalctl --vacuum-time failed" >&2; exit 1; }
for f in $filehits; do
case "$f" in
*.gz) rm -f -- "$f" && echo "removed $f" ;;
*) grep -vE "$PATTERN" "$f" > "$f.scan2.tmp"; cat "$f.scan2.tmp" > "$f"; rm -f -- "$f.scan2.tmp"; echo "rewrote $f" ;;
esac
done
echo "PURGE done. Run the dry run again: it must report nothing (journal lines newer than $before, if any, mean the fixed build was not live on that day)."
91 changes: 91 additions & 0 deletions deploy/scan2-verify.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
#!/usr/bin/env bash
# SCAN-2 (randprotocol/monorepo#16): is the explorer at <base-url> the build that no longer takes
# a visitor's nullifiers? Read-only: three kinds of HTTP request, nothing written anywhere.
#
# deploy/scan2-verify.sh https://randscan.org # prints PASS/FAIL per check, exit 0/1
# deploy/scan2-verify.sh --record https://randscan.org # also prints a dated line for the issue
#
# Checks:
# 1. POST /api/v1/nullifiers/lookup answers 404 or 405 (the bulk lookup route is gone). The body
# sent names one all-zero nullifier, which belongs to no wallet.
# 2. GET /api/v1/nullifiers?limit=1 answers 200 with {"nullifiers": [...], "next": ...} (the
# paged public set the browser matches against).
# 3. The history page's JavaScript, as served, contains no "nullifiers/lookup" and no
# single-nullifier request builder: every /_next/static chunk the page references is
# downloaded and searched.
#
# Exit 0 only when all three pass; 2 on a usage error. Needs curl. Tested offline against a fake
# explorer: deploy/tests/scan2-verify.test.sh.
set -u

record=0
base=""
for a in "$@"; do
case "$a" in
--record) record=1 ;;
-h|--help) sed -n '2,20p' "$0"; exit 0 ;;
http://*|https://*) base=${a%/} ;;
*) echo "scan2-verify: unknown argument '$a'" >&2; exit 2 ;;
esac
done
[ -n "$base" ] || { echo "usage: scan2-verify.sh [--record] <base-url>" >&2; exit 2; }
command -v curl >/dev/null 2>&1 || { echo "scan2-verify: curl is not installed" >&2; exit 2; }

tmp=$(mktemp -d "${TMPDIR:-/tmp}/scan2-verify.XXXXXX")
trap 'rm -rf "$tmp"' EXIT
fail=0
pass() { echo "PASS $1"; }
bad() { echo "FAIL $1"; fail=1; }
get() { local c; c=$(curl -sS -m "${SCAN2_TIMEOUT:-20}" -o "$2" -w '%{http_code}' "$1" 2>"$tmp/err") || c=000; echo "$c"; }

# 1. the removed route
zero=0000000000000000000000000000000000000000000000000000000000000000
code=$(curl -sS -m "${SCAN2_TIMEOUT:-20}" -o "$tmp/lookup" -w '%{http_code}' -X POST \
-H 'content-type: application/json' --data "{\"nullifiers\":[\"$zero\"]}" \
"$base/api/v1/nullifiers/lookup" 2>"$tmp/err") || code=000
case "$code" in
404|405) pass "POST /api/v1/nullifiers/lookup is gone (HTTP $code)" ;;
000) bad "POST /api/v1/nullifiers/lookup: no answer ($(head -c 200 "$tmp/err"))" ;;
*) bad "POST /api/v1/nullifiers/lookup answered HTTP $code: the lookup route is still served" ;;
esac

# 2. the paged set
code=$(get "$base/api/v1/nullifiers?limit=1" "$tmp/page")
if [ "$code" = 200 ] && grep -q '"nullifiers"[[:space:]]*:' "$tmp/page" && grep -q '"next"[[:space:]]*:' "$tmp/page"; then
pass "GET /api/v1/nullifiers?limit=1 serves the paged set"
else
bad "GET /api/v1/nullifiers?limit=1 answered HTTP $code without {nullifiers, next}: the browser cannot match locally"
fi

# 3. the served frontend
code=$(get "$base/en/viewing" "$tmp/viewing.html")
if [ "$code" != 200 ]; then
bad "GET /en/viewing answered HTTP $code: the history page could not be read"
else
grep -oE '/_next/static/[A-Za-z0-9_./%@()~-]+\.js' "$tmp/viewing.html" | sort -u > "$tmp/chunks"
n=0; hits=""
while IFS= read -r c; do
[ -n "$c" ] || continue
code=$(get "$base$c" "$tmp/chunk.js")
[ "$code" = 200 ] || { bad "chunk $c answered HTTP $code"; continue; }
n=$((n + 1))
# the old client's two request builders: the bulk lookup, and /nullifiers/<encodeURIComponent(nf)>
if grep -qE 'nullifiers/lookup|/nullifiers/"\.concat\(encodeURIComponent|/nullifiers/\$\{encodeURIComponent' "$tmp/chunk.js"; then
hits="$hits $c"
fi
done < "$tmp/chunks"
if [ "$n" -eq 0 ]; then
bad "/en/viewing references no /_next/static chunk: nothing was checked"
elif [ -n "$hits" ]; then
bad "the served frontend can still send a nullifier (in:$hits)"
else
pass "the served history page ($n chunks) has no nullifier lookup"
fi
fi

if [ "$fail" -eq 0 ]; then
[ "$record" -eq 0 ] || echo "SCAN-2 deploy check: $base passed all three checks at $(date -u +%FT%TZ) (deploy/scan2-verify.sh)"
exit 0
fi
[ "$record" -eq 0 ] || echo "SCAN-2 deploy check: $base FAILED at $(date -u +%FT%TZ) (deploy/scan2-verify.sh)"
exit 1
50 changes: 50 additions & 0 deletions deploy/tests/fake_explorer.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
#!/usr/bin/env python3
"""A fake explorer for scan2-verify.test.sh: `fixed` is the build after SCAN-2, `old` the one before.

Prints its port on stdout, then serves until killed. Standard library only."""
import json, sys
from http.server import BaseHTTPRequestHandler, HTTPServer

MODE = sys.argv[1]
OLD_CHUNK = 'function a(e){return r("/nullifiers/".concat(encodeURIComponent(e)))}function b(n){return p("/nullifiers/lookup",{nullifiers:n})}'
NEW_CHUNK = 'function a(e){return r("/nullifiers".concat(q({after:null!=e?e:void 0,limit:5e3})))}'


class H(BaseHTTPRequestHandler):
def log_message(self, *a):
pass

def send(self, code, body, ctype="application/json"):
b = body.encode()
self.send_response(code)
self.send_header("content-type", ctype)
self.send_header("content-length", str(len(b)))
self.end_headers()
self.wfile.write(b)

def do_POST(self):
self.rfile.read(int(self.headers.get("content-length", 0)))
if self.path == "/api/v1/nullifiers/lookup" and MODE in ("old", "old-api"):
return self.send(200, json.dumps({"spent": []}))
self.send(405 if self.path.startswith("/api/v1/nullifiers") else 404, "{}")

def do_GET(self):
if self.path.startswith("/api/v1/nullifiers?"):
if MODE == "no-page":
return self.send(404, "{}")
return self.send(200, json.dumps({"nullifiers": [], "next": None}))
if self.path == "/en/viewing":
if MODE == "no-chunks":
return self.send(200, "<html></html>", "text/html")
return self.send(200, '<html><script src="/_next/static/chunks/app/viewing-1a2b.js"></script>'
'<script src="/_next/static/chunks/main-9f.js"></script></html>', "text/html")
if self.path == "/_next/static/chunks/app/viewing-1a2b.js":
return self.send(200, OLD_CHUNK if MODE in ("old", "old-frontend") else NEW_CHUNK, "text/javascript")
if self.path == "/_next/static/chunks/main-9f.js":
return self.send(200, "console.log(1)", "text/javascript")
self.send(404, "{}")


srv = HTTPServer(("127.0.0.1", 0), H)
print(srv.server_address[1], flush=True)
srv.serve_forever()
63 changes: 63 additions & 0 deletions deploy/tests/scan2-log-audit.test.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
#!/usr/bin/env bash
# Offline test of deploy/scan2-log-audit.sh: a fake journalctl on a private PATH and log files in a
# temp directory. Nothing outside the temp directory is read or written.
set -u
here=$(cd "$(dirname "$0")" && pwd -P)
audit="$here/../scan2-log-audit.sh"
tmp=$(mktemp -d "${TMPDIR:-/tmp}/scan2-log-test.XXXXXX")
trap 'rm -rf "$tmp"' EXIT
PASS=0; FAIL=0
ok() { PASS=$((PASS + 1)); echo " ok $1"; }
bad() { FAIL=$((FAIL + 1)); echo " FAIL $1"; [ ! -f "$tmp/out" ] || sed 's/^/ /' "$tmp/out"; }
NF=8c04aa11223344556677889900aabbccddeeff00112233445566778899aabbd1

mkdir -p "$tmp/bin" "$tmp/log"
cat > "$tmp/bin/journalctl" <<EOF
#!/usr/bin/env bash
echo "\$*" >> "$tmp/journalctl.calls"
case "\$*" in
*"-u randscan-api"*) cat "$tmp/journal.api" 2>/dev/null ;;
*--rotate*|*--vacuum-time*) ;;
esac
exit 0
EOF
chmod +x "$tmp/bin/journalctl"
run() { JOURNALCTL="$tmp/bin/journalctl" UNITS="randscan-api caddy" LOG_GLOBS="$tmp/log/*.log" "$audit" "$@" > "$tmp/out" 2>&1; }
seed() {
printf '%s\n' \
"2026-10-20T10:00:00+0000 host randscan-api[1]: GET /api/v1/nullifiers/$NF 200" \
"2026-10-21T10:00:00+0000 host randscan-api[1]: POST /api/v1/nullifiers/lookup 200" \
"2026-11-10T10:00:00+0000 host randscan-api[1]: GET /api/v1/nullifiers?after=$NF&limit=5000 200" > "$tmp/journal.api"
printf '%s\n' "a GET /api/v1/blocks 200" "b GET /api/v1/nullifiers/$NF 200" "c GET /api/v1/nullifiers?limit=5000 200" > "$tmp/log/access.log"
printf '%s\n' "z POST /api/v1/nullifiers/lookup 200" | gzip > "$tmp/log/access.log.1.gz"
printf '%s\n' "clean line" > "$tmp/log/other.log"
: > "$tmp/journalctl.calls"
}


seed; run; got=$?
if [ "$got" -eq 3 ] && grep -q 'DRY RUN: 4 matching lines' "$tmp/out"; then ok "the dry run counts 2 journal + 1 file + 1 gz lines and exits 3 (exit $got)"; else bad "the dry run counts 2 journal + 1 file + 1 gz lines and exits 3 (exit $got)"; fi
if grep -q 'randscan-api .*2 matching lines, 2026-10-20 .. 2026-10-21' "$tmp/out"; then ok "the journal's date range is reported"; else bad "the journal's date range is reported"; fi
if ! grep -q "$NF" "$tmp/out"; then ok "no nullifier appears in the output"; else bad "no nullifier appears in the output"; fi
if ! grep -qE -- '--rotate|--vacuum' "$tmp/journalctl.calls"; then ok "the dry run never rotates or vacuums"; else bad "the dry run never rotates or vacuums"; fi
if [ "$(wc -l < "$tmp/log/access.log" | tr -d ' ')" = 3 ] && [ -f "$tmp/log/access.log.1.gz" ]; then ok "the dry run leaves the files alone"; else bad "the dry run leaves the files alone"; fi

seed; run --purge; got=$?
if [ "$got" -eq 2 ]; then ok "--purge without --before is refused"; else bad "--purge without --before is refused"; fi
seed; run --purge --before tomorrow; got=$?
if [ "$got" -eq 2 ]; then ok "a junk --before is refused"; else bad "a junk --before is refused"; fi
seed; run --purge --before 2999-01-01; got=$?
if [ "$got" -eq 2 ] && ! grep -qE -- '--rotate' "$tmp/journalctl.calls"; then ok "a future --before is refused before anything is rotated (exit $got)"; else bad "a future --before is refused before anything is rotated (exit $got)"; fi

seed; run --purge --before 2026-11-01; got=$?
if [ "$got" -eq 0 ]; then ok "the purge exits 0 (exit $got)"; else bad "the purge exits 0 (exit $got)"; fi
if grep -q -- '--rotate' "$tmp/journalctl.calls" && grep -qE -- '--vacuum-time=[0-9]+s' "$tmp/journalctl.calls"; then ok "the purge rotates, then vacuums by age"; else bad "the purge rotates, then vacuums by age"; fi
if [ "$(cat "$tmp/log/access.log")" = "$(printf '%s\n' "a GET /api/v1/blocks 200" "c GET /api/v1/nullifiers?limit=5000 200")" ]; then ok "only the matching line left the plain log"; else bad "only the matching line left the plain log"; fi
if [ ! -e "$tmp/log/access.log.1.gz" ]; then ok "the matching compressed rotation is removed"; else bad "the matching compressed rotation is removed"; fi
if [ "$(cat "$tmp/log/other.log")" = "clean line" ]; then ok "a log with no match is untouched"; else bad "a log with no match is untouched"; fi

: > "$tmp/journal.api"; run; got=$?
if [ "$got" -eq 0 ] && grep -q 'DRY RUN: nothing found' "$tmp/out"; then ok "after the purge the dry run finds nothing (exit $got)"; else bad "after the purge the dry run finds nothing (exit $got)"; fi

echo "scan2-log-audit test: $PASS passed, $FAIL failed"
[ "$FAIL" -eq 0 ]
Loading
Loading