fix(workflow): make _log_uncommitted_code safe in non-git CWDs - #2338
fix(workflow): make _log_uncommitted_code safe in non-git CWDs#2338Misty_Cirque (Cyber-Marty) wants to merge 3 commits into
Conversation
MLflowRecorder._log_uncommitted_code ran git via subprocess with shell=True and no stderr capture. In a non-git working directory (containers, CI sandboxes, /tmp), every R.start() leaked a multi-line 'usage: git diff --no-index ...' banner straight to the caller's stderr, bypassing qlib's logger, and logged three misleading INFO records for an environment that is perfectly legitimate. - probe with 'git rev-parse --is-inside-work-tree' and skip silently (DEBUG level) when CWD is not a git work tree - pass git commands as argument lists instead of shell=True strings (removes a needless injection footgun) - capture stderr on every call so git diagnostics can never leak to the caller's terminal Behavior inside a real git work tree is unchanged: the three artifacts (code_diff.txt / code_status.txt / code_cached.txt) are still logged. Fixes microsoft#2252 (secondary defect; the primary FileLock path fix is in PR microsoft#2337).
Parafee41 (koriyoshi2041)
left a comment
There was a problem hiding this comment.
The ordinary non-repo and worktree cases pass locally at e2c4cb26, but the probe has one uncovered Git state: in a freshly initialized bare repository, git rev-parse --is-inside-work-tree exits 0 and prints false. The current implementation then runs all three artifact commands; in a minimal bare repo I observed two INFO failures and one logged artifact, so the advertised silent skip does not hold.
Could the predicate also require result.stdout.strip() == b"true" and add a bare-repository regression beside the existing non-git case? That keeps the change bounded to detecting an actual work tree.
A bare repository makes 'git rev-parse --is-inside-work-tree' exit 0 while printing 'false', so the exit-code-only probe let bare repos through and the artifact commands still ran (two INFO failures and one misfiled artifact). Check the probe output as well: skip unless it prints exactly 'true'. Adds a bare-repository regression test alongside the existing non-git case; verified the new test fails against the previous probe implementation.
Companion to the bare-repository regression test: the probe must require stdout 'true', not merely a zero exit.
|
Good catch — thank you for testing this locally and for the precise failure description. You're right: Fixed in 6fcbb2d:
Thanks again — this made the fix genuinely correct rather than just quiet. |
Parafee41 (koriyoshi2041)
left a comment
There was a problem hiding this comment.
Verified the changed head at 6fcbb2dd: the probe now requires stdout true, the new bare-repository regression covers the exact missed state, and the focused workflow tests pass 3/3 locally. This resolves my review finding.
Summary
MLflowRecorder._log_uncommitted_code(qlib/workflow/recorder.py:362) ran its git commands viasubprocess.check_output(cmd, shell=True)with no stderr capture. In a non-git working directory (containers, CI sandboxes,/tmp, any non-repo checkout), everyR.start():usage: git diff --no-index ...banner straight to the caller's stderr, bypassing qlib's logger entirely — corrupting structured-stderr consumers (JSONL loggers, CI parsers, anything piping output throughjq);INFO - "Fail to log the uncommitted code of ..."records for what is a perfectly legitimate environment.The git repo in CWD is not load-bearing for experiment correctness — the feature is an optional reproducibility hook, so it should degrade quietly when its environment isn't available.
Fix
git rev-parse --is-inside-work-treeonce; on failure, skip atDEBUGlevel instead of running three doomed commands perR.start().shell=True: pass git commands as argument lists (["git", "diff"], ...). The commands use no shell features, andshell=Truewas a needless injection footgun.stdout=PIPE, stderr=PIPE), so git diagnostics can never bypass the logger.Behavior inside a real git work tree is unchanged: the three artifacts (
code_diff.txt/code_status.txt/code_cached.txt) are still logged to the recorder.Tests
New
tests/workflow_tests/test_log_uncommitted_code.py:INFOrecords (skip logs atDEBUG), no exceptionAll new and existing workflow tests pass locally (
tests/test_workflow.py+ both new files, 7 passed).Fixes #2252 (secondary defect). The primary defect —
FileLockpath built CWD-relative — is fixed in PR #2337.