Repository navigation
Timestamp fingerprinting is slow for large glob patterns (e.g. **/*.gql) #2746
Description
Description
When using method: timestamp with glob patterns like **/*.gql that match thousands of files across a large directory tree, the fingerprinting step is significantly slower than expected (~6.6s for ~5000 matching files in a monorepo).
What did you do?
Created a task with method: timestamp and source globs like:
sources:
- /path/to/monorepo/schema/**/*.gql
- /path/to/monorepo/app/src/**/*.gql
- /path/to/monorepo/app/scripts/codegen/**/*
generates:
- /tmp/output.txtWhat did you expect to happen?
The up-to-date check should be fast since it only needs to compare file modification times.
What happened instead?
The check takes ~6.6s because:
-
Globs()usesexecext.ExpandFieldswhich shells out to expand glob patterns into a complete file list before any timestamp comparison. For**/*.gqlpatterns matching thousands of files, this is very expensive (~2.7s per call). -
Value()is called redundantly: IncompiledTask()(variables.go:184), theValue()call is not gated byevaluateShVars. SinceRunTaskcalls bothFastCompiledTaskandCompiledTasksequentially (task.go:142andtask.go:159),Value()runs twice — adding ~2.7s of unnecessary overhead. -
No short-circuiting in
IsUpToDate: All globs are fully expanded into a file list before any timestamp comparison begins. For the common "up-to-date" case, this means stating every single matching file even though only one newer file would be enough to decide.
Version
Latest main (commit 54bdcba)
Operating system
macOS (Darwin 25.3.0), but the issue is platform-independent — it affects any large directory tree with ** glob patterns.
Activity
trulede commentedon Mar 22, 2026
The succinct problem is, essentially (and because) files normally don't change in your example, that you want to reliably detect 1 changed file out of 5000 when often the are actually no changed files. This means, statistically, that most often you would have to scan a large portion of the files for a single change (50%), and if normally you don't expect files to change, actually you would have to scan all the files every time (100%).
To do that Task will have to glob all those files, and then compare the timestamps of each. The call to
value()might be redundant, in some cases, and I believe there is a more generic strategy to deal with that - but its a longer term adjustment.So, returning to the original (succinct) problem. You could possibly use a
statuscommand to check if any file has changed, that would prevent the unnecessary globing in most cases. Try a command like this, and see if it is significantly faster than Task. That would establish a useful benchmark.find /your/directory -type f -name "*.txt" -newermt "2024-01-01"With that benchmark you can determine if a status command would be of benefit.
trulede commentedon Apr 13, 2026
@ItaiYosephi do you have any more feedback. Otherwise I would close the issue with a suggestion to use a status command to bypass the task mechanism, especially if you don't expect files to change.
trulede commentedon Apr 18, 2026
I suggest to use either
findorgitto detect changes files, configuring the commands in Task withiforpreconditions. If that is faster, then great! At least when no files have changed, you should avoid the overhead of Tasks mechanism.