Unix: stdio hooks are installed in non-Unity processes, breaking shells in the launch chain #119

Summary

On Unix, Doorstop installs dup2_hook and fclose_hook into the main executable's PLT whenever no UnityPlayer module is present. Since LD_PRELOAD is inherited by every child process, that includes the launcher scripts that start the game — and both hooks deliberately suppress redirection of stdout/stderr, which is exactly what a shell uses to implement $(...) and >.

The result is that any /bin/sh in the launch chain silently stops working.

Reproduction

No Unity or Steam needed — any shell will do:

$ LD_PRELOAD=./libdoorstop.so DOORSTOP_ENABLED=1 DOORSTOP_TARGET_ASSEMBLY=/path/to/BepInEx.Preloader.dll \
    sh -c 'x="$(echo hello)"; echo "substitution: [$x]"; echo redir > /tmp/t; echo "redirect: [$(cat /tmp/t)]"'
hello
substitution: []
redirect: []

Expected:

substitution: [hello]
redirect: [redir]

Reproduced against master (8e66ca0) built from source, and against the shipped 4.5.0 release.

Real-world impact: BepInEx is broken on every Steam Linux Runtime game

Steam launches native Linux games through the Steam Linux Runtime:

_v2-entry-point --verb=waitforexitandrun -- scout-on-soldier-entry-point-v2 -- <game>

Both entry points are shell scripts. SteamLinuxRuntime_soldier/_v2-entry-point parses its arguments with:

getopt_temp="$(getopt -o '' --long "$getopt_temp" -n "$me" -- "$@")"
eval "set -- $getopt_temp"

With Doorstop preloaded that command substitution returns an empty string, so $@ is destroyed and the script exits at its own guard:

if [ "$#" -eq 0 ] || [ "$1" = -- ]; then
    log "Error: A command to run is required"
    usage 125
fi

The game never starts. Demonstrated with STRAFTAT (native Linux, Unity 2021.3.45f2, Mono):

# Doorstop environment set around the runtime chain
$ LD_PRELOAD=libdoorstop.so DOORSTOP_ENABLED=1 DOORSTOP_TARGET_ASSEMBLY=... \
    _v2-entry-point --verb=waitforexitandrun -- scout-on-soldier-entry-point-v2 -- STRAFTAT.x86_64
[27623]: Error: A command to run is required

This is, I believe, why Linux mod managers have to work around Doorstop rather than simply exporting its environment — e.g. ebkr/r2modmanPlus#1848.

Cause

doorstop_ctor() in src/nix/entrypoint.c:

void *unity_player = plthook_handle_by_name("UnityPlayer");

if (unity_player && PLTHOOK_OPEN_BY_HANDLE_OR_ADDRESS(&hook, unity_player) == 0) {
    LOG("Found UnityPlayer, hooking into it instead");
} else if (plthook_open(&hook, NULL) != 0) {   // <-- the main executable
    ...
}
...
plthook_replace(hook, "fclose", &fclose_hook, NULL);
plthook_replace(hook, "dup2",   &dup2_hook,   NULL);

In a shell there is no UnityPlayer, so the fallback patches the shell's own dup2/fclose. Both hooks are written for Unity's behaviour specifically and are unconditionally destructive elsewhere:

int dup2_hook(int od, int nd) {
    // Newer versions of Unity redirect stdout to player.log, we don't want that
    if (nd == fileno(stdout) || nd == fileno(stderr))
        return F_OK;    // silently refuses the redirection
    return dup2(od, nd);
}

Note the fallback itself is legitimate — older Unity games have no separate UnityPlayer.so and the player is the main executable — so it can't simply be removed; it needs to distinguish a player from a launcher.

Related

#110 proposed this same scoping ("scope Unix stdout/fclose hooks to UnityPlayer so inherited LD_PRELOAD no longer breaks shell command substitution or file redirection") but bundled a large set of unrelated cross-platform changes and was closed with a request to split it up. This issue covers only the stdio scoping; I have a focused PR for it.