Add observability dashboards and host ingestion

.env.example

@@ -3,7 +3,7 @@
 LOCAL_ONLY=false
 APP_URL=https://dash.buffer.lol
 BIND_ADDRESS=127.0.0.1
-APP_PORT=3000
+APP_PORT=3001
 TZ=America/New_York
 
 # Database
@@ -32,15 +32,18 @@ ENFORCE_TRACKING_ORIGIN=true
 IPINFO_TOKEN=
 IPINFO_TIER=lite
 
-# Optional structured events from a trusted reverse proxy or host security agent.
+# Authenticated host collector. Keep the three ingestion switches aligned with the installed agent.
 ENABLE_LOG_INGESTION=false
+ENABLE_HTTP_INGESTION=false
+ENABLE_HOST_INGESTION=false
 INGESTION_SECRET=
 
 # Security
 RATE_LIMIT_TRACKING_PER_MINUTE=120
 RATE_LIMIT_ADMIN_PER_MINUTE=60
 
-# Server Monitoring Optional
+# Server monitoring. host is recommended; ENABLE_SERVER_METRICS remains a temporary legacy fallback.
+SERVER_METRICS_SOURCE=disabled
 ENABLE_SERVER_METRICS=false
 METRICS_INTERVAL_SECONDS=60
 CLEANUP_INTERVAL_HOURS=24

DEPLOYMENT.md

@@ -39,7 +39,9 @@ ANONYMIZE_IP=true
 TRUST_PROXY=true
 ENFORCE_TRACKING_ORIGIN=true
 FILTER_BOTS=false
-ENABLE_SERVER_METRICS=true
+SERVER_METRICS_SOURCE=host
+ENABLE_HTTP_INGESTION=true
+ENABLE_HOST_INGESTION=true
 DATA_RETENTION_DAYS=90
 ```
 
@@ -58,7 +60,7 @@ It checks secret strength and separation, the admin hash, HTTPS and loopback set
 ```bash
 scripts/deploy-production.sh .env
 docker compose ps
-curl -fsS http://127.0.0.1:3000/health
+curl -fsS http://127.0.0.1:3001/health
 ```
 
 Both `app` and `worker` should become healthy, `migrate` should exit successfully, and `postgres` should remain healthy.
@@ -67,13 +69,26 @@ The deploy script automatically backs up a running database before an update, re
 
 ## Caddy
 
-```caddy
-dash.buffer.lol {
-    encode zstd gzip
-    reverse_proxy 127.0.0.1:3000
-}
+Use [deploy/Caddyfile.example](deploy/Caddyfile.example) as the starting point. It enables strict trusted-proxy parsing for Cloudflare, overwrites upstream client-IP headers, and writes permission-restricted JSON access logs with 10 MiB rotation, seven rolled files, and seven-day retention. Re-check Cloudflare's published ranges before every proxy change.
+
+Validate the final configuration, then restart Caddy in a short maintenance window because changes to an existing file output may not take effect on reload:
+
+```bash
+sudo caddy validate --config /etc/caddy/Caddyfile
+sudo systemctl restart caddy
+```
+
+Install the host collector after setting the same strong ingestion secret in `.env` and `/etc/bufferdash-agent.env`:
+
+```bash
+sudo scripts/install-host-agent.sh
+sudoedit /etc/bufferdash-agent.env
+sudo scripts/install-host-agent.sh
+systemctl status bufferdash-host-agent
 ```
 
+The agent posts only to `http://127.0.0.1:3001`, strips queries before transmission, retains its inode/offset checkpoint only after successful ingestion, and reads VPS metrics from `/proc`, `/sys`, and the root filesystem.
+
 Allow only SSH, HTTP, and HTTPS through the VPS firewall. Port 3000 must remain bound to loopback, and PostgreSQL must not be exposed publicly.
 
 ## Connect buffer.lol
@@ -173,5 +188,7 @@ docker compose up -d
 - Confirm the live `buffer.lol/bufferdash.js` loader points to `dash.buffer.lol/tracker.js`.
 - Confirm a page view appears in BufferDash.
 - Confirm query strings containing test values do not appear in events.
-- Confirm PostgreSQL and port 3000 are unreachable from the public internet.
+- Confirm PostgreSQL and the configured application port are unreachable from the public internet.
+- Confirm Settings shows fresh `caddy` and `host` collectors, then compare Runtime against `top`, `free`, `df /`, and `/proc/uptime`.
+- Generate controlled 404 and 500 responses and confirm sanitized samples appear under HTTP.
 - Confirm backups exist off-host and can be restored.

README.md

@@ -11,12 +11,12 @@ BufferDash is a self-hosted, first-party web analytics dashboard with traffic-qu
 - Secure IP handling with optional anonymization and hashed IPs
 - Bot, unknown-path, failed-login, and rate-limit security signals
 - Protected admin dashboard with signed HTTP-only sessions and CSRF checks
-- Background retention cleanup and optional runtime metric collection
+- Background retention cleanup, Caddy HTTP diagnostics, and explicit VPS-host metrics
 - Docker Compose setup with PostgreSQL
 
 ## VPS Deployment
 
-BufferDash should run behind Caddy or Nginx at an HTTPS hostname such as `https://dash.buffer.lol`. PostgreSQL remains inside Docker, while the app binds only to `127.0.0.1:3000`.
+BufferDash should run behind Caddy at an HTTPS hostname such as `https://dash.buffer.lol`. PostgreSQL remains inside Docker, while the app binds only to loopback.
 
 ```bash
 git clone https://github.com/1337lean/bufferdash.git
@@ -60,7 +60,7 @@ Run the production preflight, deploy, and verify:
 scripts/production-check.sh .env
 scripts/deploy-production.sh .env
 docker compose ps
-curl -fsS http://127.0.0.1:3000/health
+curl -fsS http://127.0.0.1:3001/health
 ```
 
 The deploy command takes a database backup before updating an existing installation, applies migrations, waits for the app, worker, and database to become healthy, and fails if any production guardrail is missing.
@@ -122,6 +122,20 @@ window.bufferdash.track("tool_used", {
 });
 ```
 
+The convenience API is equivalent:
+
+```js
+window.bufferdash.trackTool("ping-checker", { mode: "tcp" });
+```
+
+For interactive elements, declarative tracking emits exactly one `tool_used` event per activation:
+
+```html
+<button data-bufferdash-tool="dns-lookup">Run lookup</button>
+```
+
+BufferDash does not infer tool usage from arbitrary clicks. Each tracked application must mark its primary tool action or call `trackTool()` explicitly.
+
 The tracker excludes form inputs, cookies, localStorage contents, passwords, URL fragments, and query strings by default. Add `data-include-query` only after auditing every tracked URL.
 
 ## GeoIP
@@ -143,8 +157,8 @@ GeoIP sends visitor IPs to the configured provider. Leave the token empty if tha
 - `ANONYMIZE_IP`
 - `ENFORCE_TRACKING_ORIGIN`
 - `IPINFO_TOKEN` and `IPINFO_TIER`
-- `ENABLE_LOG_INGESTION` and `INGESTION_SECRET`
-- `ENABLE_SERVER_METRICS`
+- `ENABLE_LOG_INGESTION`, `ENABLE_HTTP_INGESTION`, `ENABLE_HOST_INGESTION`, and `INGESTION_SECRET`
+- `SERVER_METRICS_SOURCE=host|container|disabled` (`ENABLE_SERVER_METRICS` is a temporary compatibility fallback)
 - `DATA_RETENTION_DAYS`
 
 ## Security Notes

app/(dashboard)/dashboard/page.tsx

@@ -7,11 +7,20 @@ import { getDashboardData, getRecentEvents } from "@/lib/data";
 import { compactDuration, numberFormat, shortDate } from "@/lib/format";
 import { maskIp } from "@/lib/ip";
 import { parseRange, rangeLabel } from "@/lib/range";
+import { parseTraffic, type SearchParams } from "@/lib/filters";
+import { TrafficToggle } from "@/components/TrafficToggle";
+import { StatusBadge } from "@/components/StatusBadge";
+import { InfoCallout } from "@/components/InfoCallout";
+import { env } from "@/lib/env";
 
-export default async function DashboardPage({ searchParams }: { searchParams: Promise<{ range?: string }> }) {
-  const range = parseRange((await searchParams).range);
-  const [data, recentEvents] = await Promise.all([getDashboardData(undefined, range), getRecentEvents(undefined, 10)]);
+export default async function DashboardPage({ searchParams }: { searchParams: Promise<SearchParams> }) {
+  const params = await searchParams;
+  const range = parseRange(params.range);
+  const traffic = parseTraffic(params.traffic, "human");
+  const [data, recentEvents] = await Promise.all([getDashboardData(undefined, range, traffic), getRecentEvents(undefined, 10, traffic)]);
   const { overview } = data;
+  const cloudflareNetworks = recentEvents.filter((event) => event.asn?.toUpperCase() === "AS13335" || event.isp?.toLowerCase().includes("cloudflare")).length;
+  const proxyWarning = recentEvents.length >= 5 && cloudflareNetworks / recentEvents.length > 0.5;
 
   return (
     <>
@@ -20,7 +29,9 @@ export default async function DashboardPage({ searchParams }: { searchParams: Pr
         title="Traffic, health, and security at a glance"
         description="A live command center for buffer.lol and any other site you add."
       />
-      <RangeSelector selected={range} basePath="/dashboard" />
+      <TrafficToggle selected={traffic} path="/dashboard" params={params} />
+      <RangeSelector selected={range} basePath="/dashboard" params={params} />
+      {proxyWarning && <InfoCallout title="Check trusted-proxy configuration" tone="warning">Most recent client networks resolve to Cloudflare. This can indicate that the origin is storing the proxy address instead of Caddy&apos;s parsed client IP.</InfoCallout>}
       <section className="metrics-grid">
         <MetricCard label="Unique visitors" value={numberFormat(overview.uniqueVisitors)} detail={rangeLabel(range)} />
         <MetricCard label="Page views" value={numberFormat(overview.pageViews)} detail={rangeLabel(range)} tone="orange" />
@@ -49,7 +60,10 @@ export default async function DashboardPage({ searchParams }: { searchParams: Pr
         <TopList title="Operating systems" rows={data.operatingSystems} />
         <TopList title="Devices" rows={data.devices} />
         <TopList title="Countries" rows={data.countries} />
-        <TopList title="Cities" rows={data.cities} />
+        {data.cities.length ? <TopList title="Cities" rows={data.cities} /> : <section className="panel"><div className="panel-header"><h2>Cities</h2></div>
+          <InfoCallout title={overview.pageViews === 0 ? "No traffic yet" : !env.ipinfoToken ? "City provider not configured" : env.ipinfoTier === "lite" ? "IPinfo Core required" : "No city data returned"}>
+            {overview.pageViews === 0 ? "City data will appear with new page views." : !env.ipinfoToken ? "Configure IPINFO_TOKEN and IPINFO_TIER=core for city analytics." : env.ipinfoTier === "lite" ? "IPinfo Lite supplies country and ASN; Core is needed for city and region." : "Core is configured, but recent events did not include a city."}
+          </InfoCallout></section>}
         <TopList title="Tools used" rows={data.topTools} />
       </section>
 
@@ -61,7 +75,7 @@ export default async function DashboardPage({ searchParams }: { searchParams: Pr
         <div className="table-wrap">
           <table>
             <thead>
-              <tr><th>Time</th><th>Site</th><th>Path</th><th>Visitor</th><th>Browser</th></tr>
+              <tr><th>Time</th><th>Site</th><th>Path</th><th>Visitor</th><th>Classification</th><th>Browser</th></tr>
             </thead>
             <tbody>
               {recentEvents.map((event) => (
@@ -70,10 +84,11 @@ export default async function DashboardPage({ searchParams }: { searchParams: Pr
                   <td>{event.site.name}</td>
                   <td>{event.path || event.type}</td>
                   <td>{maskIp(event.ipAddress)}</td>
+                  <td><StatusBadge isBot={event.isBot} botName={event.botName} asn={event.asn} isp={event.isp} /></td>
                   <td>{event.browser || "Unknown"}</td>
                 </tr>
               ))}
-              {recentEvents.length === 0 && <tr><td colSpan={5}>No events yet. Add a site and install the tracker.</td></tr>}
+              {recentEvents.length === 0 && <tr><td colSpan={6}>No events match this traffic view.</td></tr>}
             </tbody>
           </table>
         </div>

app/(dashboard)/http/page.tsx

@@ -0,0 +1,63 @@
+import { HttpStatusChart } from "@/components/Charts";
+import { DataTable } from "@/components/DataTable";
+import { DateRangeFilter } from "@/components/DateRangeFilter";
+import { FilterBar } from "@/components/FilterBar";
+import { InfoCallout } from "@/components/InfoCallout";
+import { MetricCard } from "@/components/MetricCard";
+import { PageHeader } from "@/components/PageHeader";
+import { Pagination } from "@/components/Pagination";
+import { StatusBadge } from "@/components/StatusBadge";
+import { TopList } from "@/components/TopList";
+import { TrafficToggle } from "@/components/TrafficToggle";
+import { parseDateWindow, parsePage, parsePageSize, parseTraffic, type SearchParams } from "@/lib/filters";
+import { compactDuration, numberFormat, shortDate } from "@/lib/format";
+import { getHttpPage } from "@/lib/list-data";
+import { maskIp } from "@/lib/ip";
+
+const one = (value: string | string[] | undefined) => Array.isArray(value) ? value[0] : value;
+
+export default async function HttpPage({ searchParams }: { searchParams: Promise<SearchParams> }) {
+  const params = await searchParams;
+  const window = parseDateWindow(params); const page = parsePage(params.page); const pageSize = parsePageSize(params.pageSize);
+  const traffic = parseTraffic(params.traffic, "all");
+  const statusValue = Number(one(params.status));
+  const status = Number.isInteger(statusValue) && statusValue >= 100 && statusValue <= 599 ? statusValue : undefined;
+  const statusCandidate = one(params.statusClass);
+  const statusClass = ["2xx", "3xx", "4xx", "5xx"].includes(statusCandidate || "") ? statusCandidate : undefined;
+  const filters = { host: one(params.host), method: one(params.method)?.toUpperCase(), statusClass, status, path: one(params.path) };
+  const data = await getHttpPage({ ...window, page, pageSize, traffic, ...filters });
+  const requests = data.summary.requests;
+  const rate4xx = requests ? data.summary.errors4xx / requests * 100 : 0; const rate5xx = requests ? data.summary.errors5xx / requests * 100 : 0;
+  const byTime = new Map<string, { time: string; "2xx": number; "3xx": number; "4xx": number; "5xx": number }>();
+  for (const row of data.timeline) { const key = new Date(row.bucket).toISOString(); const item = byTime.get(key) || { time: shortDate(new Date(row.bucket)), "2xx": 0, "3xx": 0, "4xx": 0, "5xx": 0 }; if (row.class in item) item[row.class as "2xx"] = row.count; byTime.set(key, item); }
+  const stale = !data.source || window.end.getTime() - data.source.lastSeenAt.getTime() > 150_000;
+  return <>
+    <PageHeader eyebrow="HTTP" title="Reverse-proxy diagnostics" description="First-party request aggregates and sanitized 4xx/5xx samples from Caddy." />
+    <DateRangeFilter path="/http" params={params} selected={window.range} from={window.from} to={window.to} />
+    <TrafficToggle selected={traffic} path="/http" params={params} />
+    <FilterBar><form action="/http" className="filter-form">
+      <input type="hidden" name="range" value={window.range} />{window.from && <input type="hidden" name="from" value={window.from} />}{window.to && <input type="hidden" name="to" value={window.to} />}<input type="hidden" name="traffic" value={traffic} />
+      <label><span>Host</span><select name="host" defaultValue={filters.host || ""}><option value="">All hosts</option>{data.hosts.map((host) => <option key={host}>{host}</option>)}</select></label>
+      <label><span>Method</span><select name="method" defaultValue={filters.method || ""}><option value="">All methods</option>{["GET","POST","PUT","PATCH","DELETE","HEAD","OPTIONS"].map((method) => <option key={method}>{method}</option>)}</select></label>
+      <label><span>Status class</span><select name="statusClass" defaultValue={filters.statusClass || ""}><option value="">All statuses</option>{["2xx","3xx","4xx","5xx"].map((value) => <option key={value}>{value}</option>)}</select></label>
+      <label><span>Exact status</span><input name="status" inputMode="numeric" defaultValue={filters.status} placeholder="502" /></label>
+      <label className="filter-search"><span>Path contains</span><input name="path" defaultValue={filters.path} placeholder="/api/…" /></label>
+      <label><span>Rows</span><select name="pageSize" defaultValue={pageSize}>{[25,50,100].map((value) => <option key={value}>{value}</option>)}</select></label>
+      <button className="primary-button" type="submit">Apply filters</button>
+    </form></FilterBar>
+    <InfoCallout title={stale ? "Collector stale" : "Collector healthy"} tone={stale ? "warning" : "info"}>{data.source ? <>Last Caddy batch {shortDate(data.source.lastSeenAt)}{data.source.hostname ? ` from ${data.source.hostname}` : ""}.</> : "No Caddy request batch has been received."} Cloudflare edge-only errors that never reach the VPS are outside this view.</InfoCallout>
+    <section className="metrics-grid">
+      <MetricCard label="Requests" value={numberFormat(requests)} detail="Requests reaching the VPS" />
+      <MetricCard label="4xx" value={numberFormat(data.summary.errors4xx)} detail={`${rate4xx.toFixed(1)}% of requests`} tone="orange" />
+      <MetricCard label="5xx" value={numberFormat(data.summary.errors5xx)} detail={`${rate5xx.toFixed(1)}% of requests`} tone="red" />
+      <MetricCard label="Average duration" value={compactDuration(data.summary.averageDuration)} detail={`Max ${compactDuration(data.summary.maximumDuration)}`} tone="green" />
+    </section>
+    <section className="panel span-full"><div className="panel-header"><h2>Status timeline</h2><span>2xx / 3xx / 4xx / 5xx</span></div><HttpStatusChart data={[...byTime.values()]} /></section>
+    <section className="dashboard-grid"><TopList title="Top failing paths" rows={data.paths} /><TopList title="Status codes" rows={data.statuses} /></section>
+    <section className="panel span-full"><div className="panel-header"><h2>Recent 4xx/5xx samples</h2><span>Sanitized; no queries, bodies, cookies, or authorization</span></div>
+      <DataTable label="HTTP error samples"><thead><tr><th>Time</th><th>Status</th><th>Host</th><th>Method</th><th>Path</th><th>Duration</th><th>Visitor</th><th>Classification</th><th>Proxy error</th></tr></thead><tbody>
+        {data.samples.map((sample) => <tr key={sample.id}><td>{shortDate(sample.occurredAt)}</td><td><span className={`status-badge ${sample.status >= 500 ? "error" : "warning"}`}>{sample.status >= 500 ? "Server error" : "Client error"} · {sample.status}</span></td><td>{sample.host}</td><td>{sample.method}</td><td className="wrap-cell" title={sample.path}>{sample.path}</td><td>{compactDuration(sample.durationMs)}</td><td>{maskIp(sample.ipAddress)}</td><td><StatusBadge isBot={sample.isBot} botName={sample.botName} /></td><td className="wrap-cell">{sample.proxyError || "—"}</td></tr>)}
+        {!data.samples.length && <tr><td colSpan={9}>No 4xx/5xx samples match these filters.</td></tr>}
+      </tbody></DataTable><Pagination path="/http" params={params} page={page} pageSize={pageSize} total={data.sampleCount} /></section>
+  </>;
+}