ostrail

Give ostrail a request ID (req-...) or a resource UUID (a server, volume, network, ...), and it SSHes into the hosts running your OpenStack services, pulls their journals for that time window, and prints every matching log line from every service — Nova, Neutron, Cinder, Keystone, whatever's out there — merged into one chronological, color-coded timeline. No more guessing which of a dozen services to check first, or which host it happened on.

$ ostrail 7ab93819-ede2-4c95-b5c0-ff524d0c3081 --since "15 minutes ago"

Searching 1 host(s) for '7ab93819-ede2-4c95-b5c0-ff524d0c3081' (since 15 minutes ago, until now) ...
3 matching line(s):

08-26 14:23:39  devstack     nova-conductor            WARN   WARNING nova.scheduler.utils [None req-3ccf19e4-... admin admin] [instance: 7ab93819-...] Setting instance to ERROR state.: nova.exception_Remote.NoValidHost_Remote: No valid host was found.
08-26 14:24:58  devstack     devstack@neutron-api.se…  ?      [pid: 380278|app: 0|req: 26/51] 192.168.122.38 () ... GET /networking/v2.0/ports?fields=security_groups&fields=device_id&device_id=7ab93819-... => generated 12 bytes in 57 msecs (HTTP/1.1 200)
08-26 14:24:59  devstack     devstack@n-api.service    ?      [pid: 379649|app: 0|req: 9/17] 192.168.122.1 () ... GET /compute/v2.1/servers/7ab93819-... => generated 3617 bytes in 165 msecs (HTTP/1.1 200)

(Real output, lightly trimmed for width, from an actual NoValidHost scheduling failure on a live DevStack cloud — the WARNING line is colored yellow, ERROR lines red, in a real terminal.)

How it works

OpenStack already threads a request ID through every service a call touches — that's not something ostrail adds, it's a real, existing global_request_id mechanism in oslo.log. A request into Nova that in turn calls Neutron shows up in both services' logs tagged with the same ID (Neutron additionally logs its own local request ID alongside it). Resource UUIDs work too, just as plain substrings in the log message rather than a structured tag — Nova tags lines [instance: <uuid>], Neutron logs a port's device_id directly in its request body dump, and so on.

The hard part was never "does the ID exist in the logs" — it's that there's no API to query them. So ostrail:

1. Logs in to Keystone and asks Nova (/os-services) and Neutron (/v2.0/agents) which hosts are running something, and
2. SSHes to each one and pulls journalctl -o json for the given time window, then
3. Decodes and filters every line for your ID after fetching it — not with a remote grep on the raw JSON, because journald encodes MESSAGE as an array of byte values instead of a string whenever a line contains raw control bytes, which happens routinely because some OpenStack services colorize their output with ANSI codes even when journald (not a real terminal) is capturing them. A substring that's plainly visible in the decoded text never appears as that substring in the raw JSON for such a line.
4. Merges every host's results and sorts by timestamp.

No assumption is made about which systemd unit any service runs under — DevStack, RDO, and Ubuntu-packaged deployments are all known to name their units differently, so ostrail just reads a host's entire journal rather than guessing a unit name. That said, testing so far has only been against a real DevStack deployment (which uses its own generic devstack@<code> naming regardless of the underlying distro, not RHEL's or Ubuntu's own package-native unit names) — the unit-name-agnostic design is meant to hold up against RDO/RHEL and Ubuntu-packaged clouds too, but that's an untested expectation, not a verified one yet.

Requirements

• A systemd-journald-based deployment (DevStack, RDO/Packstack, Ubuntu-packaged bare metal, OpenStack-Ansible on the host layer, ...). Containerized deployments (Kolla-Ansible, etc.) aren't supported yet — a container's logs usually don't reach the host's own journal, so there's nothing for journalctl to find there.
• SSH access to the hosts (key-based, already working - ostrail doesn't manage credentials for this part), and sudo journalctl access on each one. Production clouds routinely restrict this more tightly than a DevStack box does; if a host isn't reachable this way, ostrail reports it and moves on rather than pretending nothing happened there.

Usage

$ ostrail <ID>                                    # auto-discover hosts via Nova/Neutron
$ ostrail <ID> --hosts controller,compute-1        # search specific hosts instead
$ ostrail <ID> --since "1 hour ago" --until "30 minutes ago"
$ ostrail <ID> --min-level debug                   # include everything, not just warning+

Auto-discovery needs OpenStack credentials the same way the openstack CLI does — OS_* environment variables (source your openrc.sh) or clouds.yaml. There's no setup wizard here; if neither is found, pass --hosts directly and skip auth entirely.

Flags:

• --since <spec> / --until <spec>: the search window, in journalctl's own time syntax (default: the last 10 minutes)
• --hosts <a,b,c>: search these hosts instead of discovering them
• --min-level <debug|info|warning|error>: hide log lines below this level (default: warning). Lines ostrail couldn't classify (not in oslo.log's usual LEVEL logger [...] message shape - access logs, third-party libraries, ...) are always shown regardless, since they still matched the search and there's no safe way to guess their severity.
• --cacert <path>: a PEM file with extra CA certificates to trust (see below)

TLS and certificates

ostrail sends your OpenStack password and token to Keystone over HTTPS. It checks the server certificate with the operating system's own certificate store and verifier (through rustls-platform-verifier). It does not use a CA list built into the binary. So when your OS stops trusting a CA, ostrail stops trusting it too. You do not need a new ostrail build for that.

• On Windows and macOS, the OS verifier also uses the OS list of blocked certificates, and it checks if the server certificate was revoked.
• On Linux, ostrail uses the system CA bundle (for example /etc/ssl/certs). Keep the ca-certificates package up to date. There is no revocation check on Linux.

If your cloud uses a private CA, give ostrail the CA certificate as a PEM file. The file can hold more than one certificate. These certificates are trusted in addition to the OS store, not instead of it.

• --cacert <path> on the command line, or
• the OS_CACERT environment variable (the same one the openstack CLI uses), or
• cacert: for the cloud in clouds.yaml.

If more than one is set, --cacert wins, then OS_CACERT, then clouds.yaml. ostrail stops with an error if the file cannot be read or has no certificates.

There is no option to turn certificate checks off.

Quality checks

Every push runs these checks in GitHub Actions:

• Build and test on Linux, macOS and Windows.
• cargo fmt --check for formatting.
• cargo clippy with all warnings treated as errors.
• cargo audit against the RustSec advisory database.

A separate job runs cargo audit every Monday, even when the code does not change. New advisories for our dependencies show up there first.

Example: on 2026-10-22 the audit job found RUSTSEC-2026-0382 in rustls-webpki (distrusted intermediate certificates could be accepted). We updated the dependency the same day.

Install

• Cargo: cargo install ostrail
• Debian package: https://github.com/wabuntu/ostrail/tree/main/target/debian
• RPM package: https://github.com/wabuntu/ostrail/tree/main/target/release/rpmbuild/RPMS/x86_64
• Single binary: https://github.com/wabuntu/ostrail/tree/main/binaries