TestFlows™ Machine is a deterministic hypervisor and execution environment.

Record a run of your software, replay it exactly, and branch it from any point to explore what else could have happened.

Website · Documentation · Pricing · Examples · Python SDK · Discussions

Steer threads into the order a race needs, and inject faults into system calls, memory, disks, containers, clocks and networks. Debug with time travel: go back to the point of failure and run debugging commands right there, step backwards as well as forwards, add logging after the fact, or use GDB inside the machine. Time compression skips idle waits, so sleeps and timeouts take almost no time.

It runs in the cloud as a self-serve service, with a free plan and paid plans.

Note

Machine is in public beta. We're improving it quickly with feedback from users like you. Backward compatibility isn't guaranteed yet, and you may need to update the client and the SDK Core often.

Features

• Deterministic runs: Random numbers, Multiple vCPUs, Recording that repeats, Time compression
• Deterministic replay: Comparing entries, Comparing two machines
• Checkpoints and branches: Checkpoints, Starting a branch, Switching the machine, The tree, Marks
• Disks: Docker and Compose, Programs that need a runtime, Compose projects
• Steering programs: Focus, Holding threads, Plans, Interrupts, Machine time
• Fault injection: System call faults, Function faults, Memory faults, Resource exhaustion, Kernel faults, Disk faults, Container faults, Clock faults, Network faults
• Debugging: The machine's debugger, GDB inside the machine
• Inspecting a run: Commands in the guest, Copying files, Core dumps, Profiling, Interactive mode
• Reading the disk
• Scripting
• Python SDK

Install the client

On Linux (x86_64 and arm64), on a Mac with Apple Silicon, and on Windows inside WSL:

curl https://testflows.com/machine/install -fsS | bash

The installer signs you in, or creates your account, with a code sent to your email. It checks the client's checksum and signature, puts machine-env in ~/.local/bin, and puts machine beside it unless another program already has that name. macOS has its own /usr/bin/machine, so on a Mac start machine-env first, which gives you the client as machine. Run the same command again to update. See the download page for details and for checking the download yourself.

Sign up

The installer creates your account if you don't have one. You can also sign up on the sign-up page or with machine account signup. Then sign in from the terminal:

machine login

Every account starts on the free plan. For scripts and CI, create an API key with machine account api-keys create ci and put it in the TESTFLOWS_MACHINE_API_KEY environment variable instead of signing in.

Your first run

Save print("hello, world") as hello.py. Then create a session, build a disk that runs it, run it until the machine halts, and read the last lines it printed:

machine sessions create first --cpus 2
machine disks build --from python:3.12-slim --add hello.py hello -- python hello.py
machine create hello-run --disk hello
machine run hello-run --until halted
machine output hello-run -n -5 --plain

Getting started continues with checkpoints, branches and replay.

Examples

• examples/ has small programs that each hold a known kind of bug, such as data races, starvation and missed wakeups, and Compose projects. Build disks from them and explore them with branches and replay.
• sdk/python/examples/ has scripts that drive runs from Python.

Python SDK

Drive runs, branches and replays from Python:

pip3 install testflows.machine

The SDK runs on the SDK Core, which the client's installer installs. See the Python SDK for how to use it.

Documentation

• What is it?: Concepts
• Getting started: The client, Signing up, Your first run, Getting help
• Disks: Docker and Compose, Programs that need a runtime, Compose projects, Building disks on an ARM machine, Images at boot, Disk size
• Sessions: Idle sessions stop
• Running a machine: Creating a run, Random numbers, Multiple vCPUs, Driving it, Schedules, Recording that repeats, Time compression, Waiting, Inspecting a run, Sending input, Commands in the guest, Copying files, Core dumps, Profiling, Interactive mode, Stopping and starting
• Checkpoints and branches: Checkpoints, Starting a branch, Switching the machine, The tree, Marks
• Replay: Replay margins, Comparing entries, At the control prompt, From Python, Comparing two machines
• Steering programs: Looking at tasks, Focus, Holding threads, Plans, Interrupts, Machine time
• Fault injection: System call faults, Function faults, Memory faults, Resource exhaustion, Kernel faults, Disk faults, Faults that cycle, Stopping at a fault, Changing faults, What a fault misses, Container faults, Clock faults, Network faults
• Debugging: The machine's debugger, How many stops a run holds, GDB inside the machine
• Reading the disk: Unsynced writes
• Account and billing
• Scripting
• Commands
• Python SDK: Installing, Hello Machine, The same names, Signing in, Sessions, Disks, Runs, Faults, Container faults, Clock faults, Network faults, Stopping at an event, Debugging a task, Points in a run's history, Watching and following, Choosing entries, Cleaning up, Errors, Examples, License

Community

This repository is the home of the TestFlows Machine community: bug reports, feature requests, discussions, the examples and the Python SDK. The client, the SDK Core and the service are not open source.

• Report a bug. Search the existing issues first, then open an issue with the commands you ran, what you expected, what happened, and the output of machine --version.
• Request a feature. Open a feature request that describes what you're trying to do.
• Ask a question or share an idea in Discussions. See SUPPORT.md for other ways to get help.
• Contribute an example or an improvement to the Python SDK. See CONTRIBUTING.md.
• Report a security vulnerability privately, never in public. See SECURITY.md.

Warning

Issues and discussions are public. Never include an API key, a token, a password or other secrets, and remove them from any output you paste.

Everyone taking part in this community is expected to follow our Code of Conduct.

Legal

The Python SDK in sdk/ and the examples in examples/ are licensed under the Apache License 2.0; see the LICENSE file in each. No other file in this repository is licensed under it.

The TestFlows Machine client, the SDK Core and the service are provided under the Terms of Service and the Client EULA.

TestFlows and the TestFlows Logo are trademarks of Katteli Inc.