diff --git a/COVERAGE.md b/COVERAGE.md index 44b8e06..f898f80 100644 --- a/COVERAGE.md +++ b/COVERAGE.md @@ -64,4 +64,6 @@ go-statistics/models (item 6, waves W0–W12), then items 7–22 of [PLAN.md](PL ## Rules for every slice +Speed is checked by the [performance campaign](docs/performance-campaign.md): 2081 public functions, 234 of them timed against their reference today, a shared harness, the go-simd kernels to add, and the waves. + Port the reference exactly and validate by equality where the reference is deterministic; measure against the reference on one core and treat any slower row as an open defect; use go-simd kernels without FMA and in the reference's reduction order where bit-parity is claimed; 100 % coverage, fuzzing, README, landing and docs for each release. diff --git a/README.md b/README.md index 2214236..9beea10 100644 --- a/README.md +++ b/README.md @@ -46,3 +46,6 @@ and lists what its host must provide; `bib/c-cpp.md` quotes both. `go run ./tools/rcount [--cran PACKAGES] ` counts the primitives, internals, grammar, opcodes, lines per language and native call sites of an R release tarball, and with a CRAN `PACKAGES` index how much of CRAN needs compiled code: the figures `bib/r.md` quotes. +`go run ./tools/perfinventory [--list] ...` counts the public functions of each repository +and how they are timed (against a reference, alone, or not at all): the inventory of the +[performance campaign](docs/performance-campaign.md). diff --git a/docs/performance-campaign.md b/docs/performance-campaign.md new file mode 100644 index 0000000..60a2281 --- /dev/null +++ b/docs/performance-campaign.md @@ -0,0 +1,313 @@ +# Performance campaign + +Goal: show, for every public function of the go-scikit stack, that it is at least as fast as its +reference (NumPy, SciPy, statsmodels, mpmath, or the C library it ports) **on one core**, and +close every gap that the measurement finds — the algorithm first, then a SIMD kernel from +[go-simd](https://github.com/go-simd) generated by [go-asmgen](https://github.com/go-asmgen/asmgen). +This page is the design: what exists, the shared tool, the kernels, the order. No optimisation is +done here. + +Written 2026-11-07. Everything below cites a file that was opened, at the commit given in +[Sources](#sources). + +## 1. Rules + +1. **One core on both sides.** Go under `GOMAXPROCS=1`, set outside the timed interval; the + reference with `OMP_NUM_THREADS`, `OPENBLAS_NUM_THREADS`, `MKL_NUM_THREADS` and + `VECLIB_MAXIMUM_THREADS` at 1 (the set go-simd/floats `bench/numpycmp` already uses). An + all-cores column may be shown, never used as the verdict. +2. **The same data on both sides**, generated by the same formula (go-statistics/stats and + go-numerics/special use `lo + (hi-lo)·frac(0.618…·i)`), and checked by hashing the arguments + the reference receives (stats does it with `-perf.datahash`). +3. **The same statistic on both sides.** Each sample is a loop calibrated to the same minimum + duration, timed inside its own process (the pipe is not counted, as go-bignum/bigfloat's + `oracle/mpmath.go` already does); a round is one sample per side; the verdict is on the + median over rounds, with the quartiles kept. Today the harnesses differ: signal reports the + best of 5, stats the median of 7, special the median of 5, and floats' `numpycmp` compares + Go's mean ns/op with `timeit`'s best of 5 — two different statistics in one ratio. +4. **Interleaved rounds**, alternating the order (Go then reference, then reference then Go), so + that a change in the host's load hits both sides alike. +5. **Load gate.** The 1-minute load is read before every round and stored with it. At or above + 8 the tool waits (or refuses, by flag); a run whose rounds were not all under the gate gives + no verdict. Allocation counts are exact under any load and are checked everywhere; times are + not. (While this page was written the load went from 6.6 to 136: a timing campaign on this + machine is a matter of waiting for a window, and the tool must do the waiting.) +6. **Verdict.** `ratio = Go / reference` (median). *Slower* when the Go first quartile is above + the reference third quartile; *tie* when the quartile ranges overlap; *faster* otherwise. A + slower row is an open defect, filed (section 5). +7. **Bit parity is a property of a kernel, not of a speed-up.** A kernel claimed bit-identical to + NumPy or SciPy runs without FMA and in the reference's order (floats' `SumPairwise` keeps + NumPy's eight accumulators as lanes; its `*NoFMA` kernels round every product and sum as NumPy's + temporaries do). A fused or reordered kernel is allowed only where the result is validated by + a tolerance, and its table row says so. + +## 2. Inventory + +### 2.1 Count + +[`tools/perfinventory`](../tools/perfinventory/main.go) counts, per repository, the exported +functions and exported methods of exported types in the public packages (no `internal`, `cmd`, +`tools`, `bench`, `benchmarks`, `examples`, `testdata` element, no `main`; `Error`, `String`, +`Format`, the `Marshal` family, `Unwrap` and `Is` left out). Every module of the repository is +loaded with its tests through `golang.org/x/tools/go/packages`, so each use is resolved by the +type checker. A function is + +- **ref** when it is used in timing code (a benchmark, a function calling `time.Now` or + `time.Since`, a `perf*` file that times, a harness package that times) whose code — string + literals, identifiers, `import "C"`, not comments — names a reference; +- **noted** when it is timed and the reference is named only in comments: the comparison is run + by a script outside Go (`run.sh`, `bench_numpy.py`, …); +- **bench** when it is timed and no reference is named; +- **none** otherwise. + +```sh +go run ./tools/perfinventory --list ... +``` + +on clones of the commits listed in [Sources](#sources): + +| repository | modules | public functions | ref | noted | bench only | none | external timing scripts | +|---|---:|---:|---:|---:|---:|---:|---:| +| go-numerics-optimize | 8 | 57 | 1 | 21 | 0 | 35 | 4 | +| go-numerics-special | 2 | 223 | 74 | 0 | 1 | 148 | 1 | +| go-numerics-integrate | 1 | 27 | 0 | 18 | 0 | 9 | 1 | +| go-statistics-stats | 1 | 828 | 112 | 4 | 2 | 710 | 0 | +| go-statistics-models | 0 | 0 | 0 | 0 | 0 | 0 | 0 | +| go-sparse-sparse | 1 | 138 | 14 | 2 | 1 | 121 | 0 | +| go-approx-interp | 1 | 44 | 0 | 0 | 7 | 37 | 0 | +| go-approx-poly | 1 | 34 | 0 | 0 | 4 | 30 | 0 | +| go-dsp-signal | 1 | 113 | 26 | 0 | 0 | 87 | 0 | +| go-fft-fft | 7 | 161 | 0 | 15 | 23 | 123 | 6 | +| go-ndarray-ndarray | 2 | 364 | 0 | 54 | 5 | 305 | 3 | +| go-bignum-bigfloat | 1 | 53 | 3 | 0 | 1 | 49 | 0 | +| go-geometry-predicates | 2 | 14 | 4 | 0 | 9 | 1 | 0 | +| go-simd-floats | 2 | 25 | 0 | 17 | 0 | 8 | 0 | +| **total** | | **2081** | **234** | **131** | **53** | **1663** | **15** | + +Denominator: **2081** public functions. **234 (11 %)** are timed by Go code that drives the +reference, 131 more are timed by a Go benchmark that an outside script pairs with the reference, +53 are timed with no reference, and **1663 (80 %) are not timed at all.** go-statistics/models has +no code yet (README and LICENSE only). + +What the count is not: + +- **A lower bound where a harness reaches functions indirectly.** stats' `docs/perf.md` says its + harness times 355 cases covering every public operation; the tool sees 112 functions referenced + directly, because the cases reach most distribution methods through tables and interfaces, + which the direct-reference rule does not credit. The shared tool removes the ambiguity: its + case registry names the function each case measures (section 3). +- **Not a statement that the comparison is right.** A *ref* row says the code drives a reference, + not that it does so on one core or with the same statistic (see 2.2). + +### 2.2 The harnesses that exist + +| repository | harness (opened) | one core | interleaved | load gate | notes | +|---|---|---|---|---|---| +| go-dsp/signal | `perf_test.go`, `docs/perf.md` | yes | yes | checked before each case (< 8) | best of 5; the model to generalise | +| go-statistics/stats | `perf_scipy_test.go`, `perf_test.go`, `perf_all_test.go`, `docs/perf.md` | yes (+ all-cores column) | yes, case by case | load in the table header only | median of 7; SciPy through `uv`, one JSON line per request; 355 cases, **236 slower than SciPy**, 51 of them because of go-numerics/special and 25 because of go-ndarray's generator (its `docs/perf.md`) | +| go-numerics/special | `nd/perf_test.go`, `bench/scipy_server.py`, `docs/perf.md` | scalar column yes | yes | load in the header (7.67 at the last run) | the Go scalar loop is **0.19× to 0.40×** SciPy's speed on every row read (Gamma … ErfInv) | +| go-ndarray/ndarray | `docs/perf.md`, `benchmarks/` (`bench_numpy.py`, `gen_report.sh`, `merge_report.py`) | **no**: multicore Go against single-threaded NumPy | — | — | NumPy 2.2.4, not the pinned 2.5.x; shell and Python report tooling | +| go-fft/fft | `docs/perf.md`, `BENCHMARKS.md`, `benchmarks/run.sh`, `report.py`, `cbench/` | — | — | — | FFTW native, numpy, scipy, gonum; shell and Python report tooling | +| go-simd/floats | `bench/numpycmp` (Go, cobra) | yes | yes, per row | refuses at ≥ 8, **once at start** | mean vs best-of-5 (rule 3) | +| go-geometry/predicates | `bench/` module (cgo + `predicates.c`) | yes | C and Go sub-benchmarks side by side in one benchmark | none | gap 1.4–3.6× (issue #4) | +| go-numerics/optimize, integrate | `benchmarks/run.sh` + `bench_scipy.py` + `summarize.py`; `bench/compare.sh` + `scipy_bench.py`; `tools/*bench` | not checked | not checked | not checked | shell and Python drivers | +| go-sparse/sparse | `cmd/perfbench`, `ldl/perf_test.go` | not opened | | | AMD-ordered refactor 11–21 % slower than QDLDL (issue #10) | +| go-approx/interp, poly; go-bignum/bigfloat | Go benchmarks; `bigfloat/cmd/bench` + `oracle/mpmath.go` | not opened beyond the count | | | | + +Five repositories drive their comparison with shell or Python scripts. Those scripts are the +logic of the measurement, so they move into the Go tool (one tested implementation) and the +scripts are deleted when each repository migrates. + +## 3. The shared tool: `refbench` + +### 3.1 Home + +**Proposed: a new public repository `go-scikit/refbench`, created with `ghnew -public`.** It is a +library first: every repository registers its cases in Go, in a `perf_test.go`, as stats, +special and signal already do, so the code has to be importable from test files, small, and +free of cgo. go-fleettools is a collection of operations commands; making every numerical +library's test build depend on it is the wrong direction. What already exists is reused, not +rewritten: + +- the load reader and waiter: `github.com/go-fleettools/fleettools/loadavg` (`Avg1`, `Wait`, + darwin and Linux), imported; if its module weight matters, it moves to its own module rather + than being copied; +- the alternation and the fail-stop rule of `go-fleettools cmd/abbench` (a child that fails stops + the run: a series with a hole is biased, not smaller), and `cmd/hostload`'s `/proc/stat` busy + check on Linux hosts; +- stats' JSON-lines Python worker (`perf_scipy_test.go`: one `uv` process, one request per line, + arguments cached per `(setup, n)`, a `hash` request to check the data); +- signal's report format, and floats' `numpycmp` table. + +### 3.2 Design + +- **Case registry** (Go): name, the public function it measures (a function value, so the name is + resolved by the compiler and the inventory can match it), sizes, input formula, Go body, + reference call, parity class (`exact` / `tolerance` / `none`) and the expected allocations + per call. +- **Python references** through one long-lived worker started as + `uv run --no-project --python 3.12.12 --with-requirements refs.txt python -I -c `, + where `refs.txt` is committed in the repository with exact versions and hashes (NumPy, SciPy, + statsmodels, mpmath; the versions in [COVERAGE.md](../COVERAGE.md) — numpy 2.5.3 there, + 2.5.4 in floats and stats: one version is chosen and pinned everywhere). The worker reports + `numpy.__version__`, `scipy.__version__` and the BLAS in use; they go in the table header. +- **C references** through a subprocess speaking the same JSON-lines protocol, built by the + tool from committed sources with the flags the reference requires (predicates.c: no FMA + contraction), so the library never needs cgo. A repository that already has a cgo module + (predicates `bench/`) may keep it, timed through the same runner. +- **Runner**: calibrated loops on both sides, rounds interleaved and alternated, the load read + and recorded before every round, `--on-load=wait|refuse`, `--max-load 8` by default. +- **Outputs**: `docs/perf-table.md` (generated: date, commit, Go version, reference versions, + machine, load min/median/max, one row per case with both medians, the ratio, the verdict, the + parity class) and `docs/perf.json` (the same plus every round's samples and load, schema + versioned). `docs/perf.md` stays the hand-written method and analysis page — eleven of the + repositories already have one, and stats and special already generate a `perf-table.md` beside + it, so the names follow the existing convention rather than a new `performance.md`. +- **CI mode** (`go test -run TestPerf -perf.ci`), on every pull request, no Python, no timing: + 1. each case runs once and its allocations per call must equal `docs/perf.json`'s (a change + either way fails: an increase is a regression, a decrease must be recorded); + 2. `docs/perf-table.md` must be what `docs/perf.json` generates (no hand edits); + 3. every public function (the inventory's rule, section 2.1) is registered or listed in + `docs/perf-exempt.md` with a reason — the denominator is a gate, not a note. +- **Timing mode** runs only on the quiet machine (this darwin/arm64 host, load < 8) and, as + indicative, on a scheduled amd64 runner whose rows are labelled as such (floats' README + already reports CI EPYC numbers this way). Timing never runs in a pull request's checks. + +## 4. go-simd and go-asmgen + +### 4.1 What there is + +- **go-simd/floats** (v0.7.0 is what optimize, sparse, poly and signal require): `Dot`, `Sum`, + `Min`, `Max`, `Distance`, `CosineSimilarity` (float64 and float32); fused `Axpy`, `Xpby`, + `AxpyDot`, `Scale`, `Mul`, `Axpby3`, `Clamp`, `MaxAbs`; NumPy-exact `AxpyNoFMA`, `XpbyNoFMA`, + `Axpby3NoFMA`, `AxpyDiffNoFMA` and `SumPairwise`. Kernels generated by go-asmgen for amd64 + (AVX2+FMA) and arm64 (NEON); the other four architectures run portable loops until a kernel is + measured faster on real silicon. +- **go-asmgen**: ABI0 builders for six architectures, SIMD through `Raw`, CPU feature probes, + the instructions `cmd/asm` lacks as checked `WORD`s (arm64 multiply-by-element, ppc64le VSX + float64/float32 arithmetic, s390x float32 vector ops, loong64 vector FMA), and go-fft's kernel + builders (`fft/amd64`, `fft/arm64`, `fft/riscv64`, `fft/s390x`, including the complex products). + Its README lists typed vector loads and vector types as the main remaining items. +- **go-simd/highway** (a fork of go-highway): not used by the stack; needs `GOEXPERIMENT=simd` + on amd64 and documents about 4 ULP for its math, so it fits neither the stable-Go rule nor + bit parity. Not adopted. +- **go-simd/histogram**: a byte-value histogram; nothing in the stack's `histogram` (float + binning) maps to it. + +### 4.2 Which hot loops use what today + +From the call sites of `floats.` in the stack's sources: + +| kernel | used by | +|---|---| +| `Mul`, `MaxAbs`, `Clamp`, `Axpby3` | optimize QP (ADMM step, scaling, polishing, checks) | +| `Dot`, `Axpy`, `Scale`, `Xpby` | optimize trust-constr and its dense linear algebra; sparse `iter` (CG); poly QR; signal (`linalg`, `resample`, `spectral`) | +| `AxpyDot` | sparse `iter` (CG residual update) | +| `AxpyDiffNoFMA`, `SumPairwise` | optimize differential evolution (bit-identical to SciPy) | +| `Sum` | signal (`spectral.go`: the segment mean) | + +`floats.Sum` adds in lane order, not NumPy's pairwise order; where signal's spectral results are +compared with SciPy by equality, that mean should be `SumPairwise` — to check in W1. + +go-ndarray does not import go-simd/floats. It has its own go-asmgen kernels in +`internal/kernels`: `Add`, `Sub`, `Mul`, `Div`, `Sqrt`, `Sum` (NumPy's pairwise order), `Max`, +`Min`, `Clip`, `Dot1D`, `Exp`, `Log`, float32 `Bin32`, `Max32`, `Min32`, `AddBlocks8F32`, +`Exp32`, `Log32`, axis reductions and GEMM (float64 and float32). Several of these do what a +floats kernel does (`Mul`, pairwise `Sum`, `Max`/`Min`, `Dot1D`, `Clip`/`Clamp`): two +implementations of one operation, against the rule that a kernel lives once, in the lowest +library that every user can import. go-simd/floats has no dependency, so it is that library. + +### 4.3 Kernels to add or move + +Each in one place; *parity* says how it is validated. + +| # | kernel | lowest home | serves | parity | +|---|---|---|---|---| +| K1 | Move ndarray's 1-D kernels down: `Add`, `Sub`, `Div`, `Sqrt`, `Max`/`Min` with NumPy's NaN rule, float32 `Bin32`, extrema and pairwise sum (`AddBlocks8F32`), `Exp`/`Log` float64 and `Exp32`/`Log32`; ndarray then calls floats and deletes its copies of `Mul`, `Sum`, `Dot1D`, `Clip` | go-simd/floats | ndarray ufuncs and reductions, and through it stats, special `nd`, optimize | exact for the arithmetic, `Sqrt`, extrema and pairwise sums; `Exp`/`Log`: ULP-bounded (ndarray's tests), labelled tolerance against NumPy | +| K2 | Elementwise polynomial evaluation over an array of points: Horner, and Clenshaw for Chebyshev/Legendre series, unfused | go-simd/floats | poly (`polyval`, `chebval`, `legval`, …); special's rational approximations in its array path; stats pdf/cdf through special | exact: the lanes run over different points, so each point sees NumPy's order; for special, exact against its own scalar code | +| K3 | Gathered dot product `Σ val[k]·x[idx[k]]` (a CSR row) | go-simd/floats | sparse CSR mat-vec, sparse `iter`, optimize QP on sparse KKT systems | tolerance (lanes reorder SciPy's sequential sum); the sequential unfused loop stays as the exact path | +| K4 | Complex elementwise products (`CMul`, conjugate product, `|z|²`) over `[]complex128` | go-simd/floats, emitted by go-asmgen's existing `fft/*` complex-product builders under floats' symbols (go-fft keeps its internal kernels until it switches) | signal (`fftconvolve`, `hilbert`, frequency responses), ndarray complex ufuncs, special `cmplx` arrays | exact against the Go scalar formula; against NumPy to be established by its oracle, since a C compiler may contract the products | +| K5 | `SearchSorted` over many points, and the linear-interpolation step of `np.interp` in NumPy's operation order | go-simd/floats | interp (`interp1d`, `np.interp`), ndarray `searchsorted`, stats empirical CDFs | exact (indices are integers; the step is unfused, in NumPy's order) | +| K6 | Multi-term linear combination `dst = y + h·Σ aⱼ·kⱼ` (Runge–Kutta stages) | go-simd/floats | integrate (`solve_ivp` RK23, RK45, DOP853) | tolerance (SciPy forms it with a matrix product, whose order BLAS chooses); fused allowed, labelled | +| K7 | Vector `cos`/`sin` | go-simd/floats, beside K1's `Exp`/`Log` | signal windows and spectral code, ndarray ufuncs | tolerance (ULP-bounded) | +| K8 | Batched floating-point filter for `orient2d`/`incircle`/`orient3d`/`insphere` over arrays of point records, falling back to the exact stages per lane | go-geometry/predicates (its only user) | predicates, later Delaunay (COVERAGE.md item 7) | exact: the filter is conservative, the sign is the exact predicate's | + +Not kernels, but the same rule: + +- **Double-double arithmetic exists twice**: `go-numerics/special/dd` and + `go-statistics/stats/internal/dd` have the same package comment and the same `zz_exptable.go` + and `zz_logtable.go`. stats already depends on special, so stats imports `special/dd` and + deletes its copy. +- **Scatter-add** (the `y[idx[k]] -= a·val[k]` of sparse LDL, where QDLDL leads by 11–21 %) has no + vector form on AVX2 or NEON, which have gathers and no scatter (go-simd/histogram's README + works through the same limit). The QDLDL gap is algorithmic first. +- **special's 0.19×–0.40× rows** are scalar against scalar: SciPy's ufunc loops are one-thread C + calling scalar code too. They are profiled before any kernel is written (the cost of + `math.Exp`/`math.Log` against the C library's, bounds checks, coefficient loops); K1 and K2 are + what follows if the profile points at elementary functions or polynomial evaluation. +- **GEMM stays in go-ndarray**: every user of dense matrix products already imports it. + +### 4.4 go-asmgen extensions + +| # | extension | needed by | +|---|---|---| +| A1 | The typed vector-load helper and vector types its README lists as the main remaining items | K1–K7: each new kernel is otherwise `Raw` text for two architectures | +| A2 | Check that `cmd/asm` encodes the AVX2 gathers (`VGATHERDPD`/`VGATHERQPD`) used by K3; if not, a `WORD` entry in `internal/gap` with its references, as for the other missing instructions. arm64 has no gather: K3 there is unrolled scalar loads feeding vector multiply-adds, nothing to add | K3 | +| A3 | Let the `fft/*` complex-product builders emit under a caller's symbol and ABI for a plain slice signature (the builders already take the caller's symbol and constants) | K4 | +| A4 | Kernels for ppc64le, s390x, riscv64 and loong64 only after a measurement on real silicon shows a gain, as floats' dispatch rule requires; the instructions are there (VSX float64 `XV*DP`, s390x `VF*SB`, loong64 vector FMA) | later; outside this campaign's single machine | + +## 5. Order of the campaign + +### Waves + +| wave | content | where it runs | +|---|---|---| +| W0 | Create `go-scikit/refbench` (library, runner, Python and C workers, table and JSON writers, CI mode, the inventory rule as a gate); pin `refs.txt`; this page and `tools/perfinventory` | CI for the tool's own tests | +| W1 | Migrate the three harnesses closest to the rules, to prove the tool on known numbers: go-dsp/signal, go-simd/floats (`numpycmp` retired), go-geometry/predicates (C worker; issue #4) | quiet machine for the tables; CI mode from the first PR | +| W2 | The two largest gaps: go-numerics/special first (stats inherits 51 slower rows from it), then go-statistics/stats (236 slower rows) | quiet machine | +| W3 | go-ndarray (re-measured on one core against the pinned NumPy; draws measured for the first time), go-fft (its `run.sh`/`report.py` folded into the tool; FFTW through the C worker), go-sparse (QDLDL through the C worker; issue #10) | quiet machine | +| W4 | go-numerics/optimize and integrate (their shell drivers folded in), go-approx/interp and poly, go-bignum/bigfloat (mpmath worker), go-statistics/models when its first code lands | quiet machine | +| K | Kernel work (section 4.3) starts from the issues the waves file, in the order the tables rank them by total time lost; every kernel PR re-runs the affected repositories' tables | quiet machine; CI checks parity and allocations | + +Each wave registers every public function of its repositories (or exempts it with a reason), +so that by its end the repository's row in section 2.1 is all *ref* and *exempt*; then it +measures; then it files. + +### A slower row becomes an issue + +`refbench issues` (a command of the same repository) reads `docs/perf.json` from a valid run: + +- for each *slower* row it opens or updates one issue per case in that repository, titled + `perf: is x slower than on one core`, labelled `perf`, with the row, + the machine, the load of every round, the reference versions and the commit; +- for each row that is no longer slower in a valid run it comments the new row and closes the + issue; +- a run that fails the load gate files nothing. + +go-fleettools has a `ghissue` command; it is read before this one is written, and reused if it +does the job. + +## Sources + +Opened for this page, at these commits: + +- go-scikit/roadmap: [COVERAGE.md](../COVERAGE.md). +- go-simd/floats `5fcc534`: `README.md`, `bench/numpycmp/main.go`, `main_test.go`. +- go-asmgen/asmgen `ddd6d3c`: `README.md`. +- go-simd/highway `318554d`: `README.md`, `go.mod`. go-simd/histogram `1a229c4`: `README.md`. +- go-fleettools/fleettools `98c3142`: `cmd/abbench/main.go` (package comment), `cmd/hostload/main.go` + (comments), `loadavg/` (function list). +- go-statistics/stats `50a43bb`: `docs/perf.md`, `perf_scipy_test.go`, function lists of + `perf_test.go`, `perf_all_test.go`, `kde_test.go`, `internal/dd/dd.go`. +- go-numerics/special `672b3cf`: `docs/perf.md`, `dd/dd.go`. +- go-ndarray/ndarray `a1745f5`: `docs/perf.md`, function list of `internal/kernels`. +- go-fft/fft `0684da9`: `docs/perf.md`. go-dsp/signal `6879c23`: `docs/perf.md`. +- go-geometry/predicates `625791f`: `bench/cref.go`, `bench/cref_test.go`, `bench/go.mod`. +- go.mod files of every repository (dependencies on go-simd, go-asmgen and each other), and the + `floats.` call sites of optimize `5bd0f0b`, sparse `0bcaac0`, poly `3de2769`, signal `6879c23`, + stats. +- The inventory also read integrate `d4587b4`, models `62a5776`, interp `23a47aa`, bigfloat + `6cd7f4c` through `tools/perfinventory`. + +Issues go-geometry/predicates#4 and go-sparse/sparse#10 are cited from the campaign brief; the +GitHub API timed out when they were opened for this page. diff --git a/go.mod b/go.mod index 28d63d1..f210493 100644 --- a/go.mod +++ b/go.mod @@ -5,10 +5,13 @@ go 1.27 require ( github.com/spf13/cobra v1.10.2 github.com/tetratelabs/wazero v1.12.0 + golang.org/x/tools v0.52.0 ) require ( github.com/inconshreveable/mousetrap v1.1.0 // indirect github.com/spf13/pflag v1.0.9 // indirect - golang.org/x/sys v0.44.0 // indirect + golang.org/x/mod v0.42.0 // indirect + golang.org/x/sync v0.24.0 // indirect + golang.org/x/sys v0.49.0 // indirect ) diff --git a/go.sum b/go.sum index d348db3..43c047b 100644 --- a/go.sum +++ b/go.sum @@ -1,4 +1,6 @@ github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g= +github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI= +github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8= github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw= github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM= @@ -9,6 +11,12 @@ github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An github.com/tetratelabs/wazero v1.12.0 h1:DuWcpNu/FzgEXgGBDp8J1Spc+CWOvvtvVyjKlaZopYU= github.com/tetratelabs/wazero v1.12.0/go.mod h1:LvKtzl2RqO4gyF27BiXU+nKAjcV8f38U+kP/q2vgxh0= go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg= -golang.org/x/sys v0.44.0 h1:ildZl3J4uzeKP07r2F++Op7E9B29JRUy+a27EibtBTQ= -golang.org/x/sys v0.44.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= +golang.org/x/mod v0.42.0 h1:ICeL3t2L30ljSPHHxDnR4fqVcBt/zR2zJAb9juavmQg= +golang.org/x/mod v0.42.0/go.mod h1:u5fzIrRInkL8MaO1cbRkUHT64v303YIT5KEqpNr7jRs= +golang.org/x/sync v0.24.0 h1:iqiDrGoDgyuvAKrhLQthUEzU/R378N78gXbBjxed7oA= +golang.org/x/sync v0.24.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0= +golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA= +golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og= +golang.org/x/tools v0.52.0 h1:ryxg4U63oaUYk7uOLTF37n43f4qgtGvqV4Ioit1BaEs= +golang.org/x/tools v0.52.0/go.mod h1:ItcP5UNaVGG+Yc4eIYmaFLNJ4soEopkx5Y2d4lcEurs= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= diff --git a/tools/perfinventory/main.go b/tools/perfinventory/main.go new file mode 100644 index 0000000..77b628b --- /dev/null +++ b/tools/perfinventory/main.go @@ -0,0 +1,470 @@ +// Command perfinventory counts, in each repository given as a directory, the +// exported functions and methods of its public packages, and how each is +// measured. docs/performance-campaign.md quotes its output. +// +// go run ./tools/perfinventory [--list] ... +// +// The rules, which the campaign document restates: +// +// - Every directory holding a go.mod is a module, except under testdata, +// vendor and dot-directories; each is loaded with its tests through +// golang.org/x/tools/go/packages, so names are resolved by the type +// checker, not by spelling. +// - Public: an exported function, or an exported method of an exported +// named type, declared in a non-test file of a non-main package whose +// import path has no internal, cmd, tools, bench, benchmarks, examples +// or testdata element. Methods that satisfy a standard interface +// (Error, String, Format, the Marshal family, Unwrap, Is) are left out. +// - A use is a direct reference resolved to that function (generic +// instantiations to their origin). A call through an interface, a +// function value built elsewhere or a helper is not a use. +// - A use is timed when it sits in a benchmark (named Benchmark… or taking +// a *testing.B), in a function that calls time.Now or time.Since, in a +// package-level declaration of a file that has one of those, or anywhere +// in a harness package (an import path with a bench, benchmarks, cmd or +// tools element) that times something. +// - The classes, best first: ref (timed, and the file's code — string +// literals, identifiers, the import "C" — names a reference +// implementation, or the use is in a harness package whose code does); +// noted (timed, and the reference is named only in the file's comments: +// the comparison is run by a script outside Go); bench (timed, no +// reference named); none. +// - Non-Go files (.py, .sh, .c, .jl, .r) that contain a timing word are +// counted per repository as external harnesses: their pairing with Go +// functions is not machine-readable. +package main + +import ( + "fmt" + "go/ast" + "go/token" + "go/types" + "io" + "io/fs" + "os" + "path/filepath" + "sort" + "strings" + + "github.com/spf13/cobra" + "golang.org/x/tools/go/packages" +) + +// osExit is replaced by tests. +var osExit = os.Exit + +var ( + referenceMarkers = []string{"numpy", "scipy", "mpmath", "statsmodels", "fftw", + "pocketfft", "qdldl", "osqp", "shewchuk"} + // literalMarkers are how a reference shows in a string literal that is + // code (a Python program, a script path), not prose ("SciPy %g"). + literalMarkers = []string{"import numpy", "import scipy", "import mpmath", "import statsmodels", + "np.", "scipy.", "mpmath.", "statsmodels.", "scipy_", "numpy_", "mpmath_", "fftw", + "pocketfft", "qdldl", "osqp"} + externalTiming = []string{"perf_counter", "timeit", "clock_gettime", "time.time(", + "time_ns", "chrono::", "@btime", "system.time"} + privateElems = map[string]bool{"internal": true, "cmd": true, "tools": true, "bench": true, + "benchmarks": true, "examples": true, "testdata": true} + plumbing = map[string]bool{"Error": true, "String": true, "GoString": true, "Format": true, + "MarshalJSON": true, "UnmarshalJSON": true, "MarshalText": true, "UnmarshalText": true, + "MarshalBinary": true, "UnmarshalBinary": true, "Unwrap": true, "Is": true} + externalExts = map[string]bool{".py": true, ".sh": true, ".c": true, ".jl": true, ".r": true} +) + +// Class is how a public function is measured. +type Class int + +// The classes, from worst to best. +const ( + None Class = iota + Bench + Noted + Ref +) + +func (c Class) String() string { return [...]string{"none", "bench", "noted", "ref"}[c] } + +// Repo is the tally of one repository. +type Repo struct { + Dir string + Modules int + Funcs map[string]Class // key: import/path.Func or import/path.Type.Method + Why map[string]string // the file that gave the class + ExternalHarness int + ExternalExamples []string +} + +// Count returns how many functions are in class c. +func (r Repo) Count(c Class) int { + n := 0 + for _, v := range r.Funcs { + if v == c { + n++ + } + } + return n +} + +// containsAny reports whether s contains one of subs. +func containsAny(s string, subs []string) bool { + for _, x := range subs { + if strings.Contains(s, x) { + return true + } + } + return false +} + +// hasElem reports whether the import path has one of the elements. +func hasElem(path string, elems map[string]bool) bool { + for _, e := range strings.Split(path, "/") { + if elems[e] { + return true + } + } + return false +} + +var harnessElems = map[string]bool{"bench": true, "benchmarks": true, "cmd": true, "tools": true} + +// public reports whether a package with this import path and name is public. +func public(path, name string) bool { + return name != "main" && !hasElem(path, privateElems) +} + +// skipDir reports whether a walk must not enter the directory (root excepted). +func skipDir(name string) bool { + return name == "testdata" || name == "vendor" || strings.HasPrefix(name, ".") +} + +// modules returns the directories under root that hold a go.mod, and the +// number of non-Go files carrying a timing word (with up to 3 of their +// paths, relative to root). +func modules(root string) (mods []string, ext int, examples []string, err error) { + err = filepath.WalkDir(root, func(p string, d fs.DirEntry, err error) error { + if err != nil { + return err + } + if d.IsDir() { + if p != root && skipDir(d.Name()) { + return filepath.SkipDir + } + return nil + } + if d.Name() == "go.mod" { + mods = append(mods, filepath.Dir(p)) + return nil + } + if !externalExts[strings.ToLower(filepath.Ext(p))] { + return nil + } + b, err := os.ReadFile(p) + if err != nil { + return err + } + if containsAny(strings.ToLower(string(b)), externalTiming) { + ext++ + if len(examples) < 3 { + rel, _ := filepath.Rel(root, p) + examples = append(examples, rel) + } + } + return nil + }) + return mods, ext, examples, err +} + +// funcKey names a function: path.Name, or path.Type.Name for a method of a +// named type; ok is false for a method of anything else. +func funcKey(f *types.Func) (string, bool) { + f = f.Origin() + if f.Pkg() == nil { + return "", false + } + sig := f.Type().(*types.Signature) + if sig.Recv() == nil { + return f.Pkg().Path() + "." + f.Name(), true + } + t := sig.Recv().Type() + if p, isPtr := t.(*types.Pointer); isPtr { + t = p.Elem() + } + n, isNamed := t.(*types.Named) + if !isNamed { + return "", false + } + return f.Pkg().Path() + "." + n.Obj().Name() + "." + f.Name(), true +} + +// declare adds the public functions of pkg to funcs. +func declare(pkg *packages.Package, funcs map[string]Class) { + if strings.Contains(pkg.ID, "[") || strings.HasSuffix(pkg.PkgPath, "_test") || + !public(pkg.PkgPath, pkg.Name) { + return + } + scope := pkg.Types.Scope() + for _, name := range scope.Names() { + obj := scope.Lookup(name) + if !obj.Exported() { + continue + } + switch o := obj.(type) { + case *types.Func: + k, _ := funcKey(o) + funcs[k] = None + case *types.TypeName: + n, isNamed := o.Type().(*types.Named) + if !isNamed || o.IsAlias() { + continue + } + for i := 0; i < n.NumMethods(); i++ { + if m := n.Method(i); m.Exported() && !plumbing[m.Name()] { + k, _ := funcKey(m) + funcs[k] = None + } + } + } + } +} + +// benchmarkFunc reports whether fd is a Go benchmark: named Benchmark… or +// taking a *testing.B. +func benchmarkFunc(fd *ast.FuncDecl) bool { + if strings.HasPrefix(fd.Name.Name, "Benchmark") { + return true + } + for _, f := range fd.Type.Params.List { + if s, isStar := f.Type.(*ast.StarExpr); isStar { + if sel, isSel := s.X.(*ast.SelectorExpr); isSel && sel.Sel.Name == "B" { + return true + } + } + } + return false +} + +// callsClock reports whether n references time.Now or time.Since. +func callsClock(n ast.Node, info *types.Info) bool { + found := false + ast.Inspect(n, func(n ast.Node) bool { + if id, isIdent := n.(*ast.Ident); isIdent { + if f, isF := info.Uses[id].(*types.Func); isF && f.Pkg() != nil && + f.Pkg().Path() == "time" && (f.Name() == "Now" || f.Name() == "Since") { + found = true + } + } + return !found + }) + return found +} + +// codeNamesReference reports whether the code of file (string literals, +// identifiers, an import "C"), comments excluded, names a reference. +func codeNamesReference(file *ast.File) bool { + found := false + ast.Inspect(file, func(n ast.Node) bool { + switch x := n.(type) { + case *ast.ImportSpec: + found = found || x.Path.Value == `"C"` + case *ast.BasicLit: + found = found || (x.Kind == token.STRING && containsAny(strings.ToLower(x.Value), literalMarkers)) + case *ast.Ident: + low := strings.ToLower(x.Name) + found = found || containsAny(low, referenceMarkers) || strings.HasPrefix(low, "_cfunc_") + } + return !found + }) + return found +} + +// commentsNameReference reports whether a comment of file names a reference. +func commentsNameReference(file *ast.File) bool { + for _, g := range file.Comments { + if containsAny(strings.ToLower(g.Text()), referenceMarkers) { + return true + } + } + return false +} + +// raise sets r.Funcs[k] to c, and r.Why[k] to the file, when c is better +// than what it holds. +func raise(r *Repo, k string, c Class, file string) { + if old, ok := r.Funcs[k]; ok && c > old { + r.Funcs[k] = c + r.Why[k] = file + } +} + +// use records, for every file of pkg, the public functions it references. +func use(pkg *packages.Package, r *Repo) { + harness := hasElem(pkg.PkgPath, harnessElems) + pkgRef, pkgTimes := false, false + for _, f := range pkg.Syntax { + pkgRef = pkgRef || codeNamesReference(f) + pkgTimes = pkgTimes || callsClock(f, pkg.TypesInfo) + } + for i, file := range pkg.Syntax { + rel, _ := filepath.Rel(r.Dir, pkg.CompiledGoFiles[i]) + perfFile := strings.HasPrefix(filepath.Base(rel), "perf") + best := Bench + switch { + case codeNamesReference(file) || ((harness || perfFile) && pkgRef): + best = Ref + case commentsNameReference(file): + best = Noted + } + // A harness package or a perf* file is a measurement throughout + // (its case tables are built in init or helpers); elsewhere only + // the timing functions and the package-level tables of a file that + // times count. + whole := (harness && pkgTimes) || (perfFile && callsClock(file, pkg.TypesInfo)) + fileTimes := whole + for _, decl := range file.Decls { + if fd, isFunc := decl.(*ast.FuncDecl); isFunc && (benchmarkFunc(fd) || callsClock(fd, pkg.TypesInfo)) { + fileTimes = true + } + } + for _, decl := range file.Decls { + timed := whole + if fd, isFunc := decl.(*ast.FuncDecl); isFunc { + timed = timed || benchmarkFunc(fd) || callsClock(fd, pkg.TypesInfo) + } else { + timed = timed || fileTimes + } + if !timed { + continue + } + ast.Inspect(decl, func(n ast.Node) bool { + if id, isIdent := n.(*ast.Ident); isIdent { + if f, isF := pkg.TypesInfo.Uses[id].(*types.Func); isF { + if k, ok := funcKey(f); ok { + raise(r, k, best, rel) + } + } + } + return true + }) + } + } +} + +// load is replaced by tests that need a loader failure. +var load = packages.Load + +// inventory tallies one repository. +func inventory(dir string) (Repo, error) { + if abs, err := filepath.Abs(dir); err == nil { + dir = abs + } + r := Repo{Dir: dir, Funcs: map[string]Class{}, Why: map[string]string{}} + mods, ext, examples, err := modules(dir) + if err != nil { + return r, err + } + r.Modules, r.ExternalHarness, r.ExternalExamples = len(mods), ext, examples + var all []*packages.Package + for _, m := range mods { + cfg := &packages.Config{ + Mode: packages.NeedName | packages.NeedFiles | packages.NeedCompiledGoFiles | + packages.NeedSyntax | packages.NeedTypes | packages.NeedTypesInfo, + Dir: m, + Tests: true, + } + pkgs, err := load(cfg, "./...") + if err != nil { + return r, fmt.Errorf("%s: %w", m, err) + } + if n := packages.PrintErrors(pkgs); n > 0 { + return r, fmt.Errorf("%s: %d package errors", m, n) + } + all = append(all, pkgs...) + } + for _, p := range all { + declare(p, r.Funcs) + } + for _, p := range all { + use(p, &r) + } + return r, nil +} + +func report(w io.Writer, dirs []string, list bool) error { + var tot [4]int + total, ext := 0, 0 + fmt.Fprintln(w, "| repository | modules | public functions | ref | noted | bench only | none | external timing scripts |") + fmt.Fprintln(w, "|---|---:|---:|---:|---:|---:|---:|---:|") + var repos []Repo + for _, d := range dirs { + r, err := inventory(d) + if err != nil { + return err + } + repos = append(repos, r) + var c [4]int + for i := range c { + c[i] = r.Count(Class(i)) + tot[i] += c[i] + } + total += len(r.Funcs) + ext += r.ExternalHarness + fmt.Fprintf(w, "| %s | %d | %d | %d | %d | %d | %d | %d |\n", filepath.Base(r.Dir), r.Modules, + len(r.Funcs), c[Ref], c[Noted], c[Bench], c[None], r.ExternalHarness) + } + fmt.Fprintf(w, "| **total** | | %d | %d | %d | %d | %d | %d |\n", total, tot[Ref], tot[Noted], tot[Bench], tot[None], ext) + if !list { + return nil + } + for _, r := range repos { + fmt.Fprintf(w, "\n## %s\n", filepath.Base(r.Dir)) + for _, e := range r.ExternalExamples { + fmt.Fprintf(w, "external\t%s\n", e) + } + byFile := map[string]map[Class]int{} + for k, f := range r.Why { + if byFile[f] == nil { + byFile[f] = map[Class]int{} + } + byFile[f][r.Funcs[k]]++ + } + files := make([]string, 0, len(byFile)) + for f := range byFile { + files = append(files, f) + } + sort.Strings(files) + for _, f := range files { + fmt.Fprintf(w, "file\t%s\tref %d\tnoted %d\tbench %d\n", f, byFile[f][Ref], byFile[f][Noted], byFile[f][Bench]) + } + keys := make([]string, 0, len(r.Funcs)) + for k := range r.Funcs { + keys = append(keys, k) + } + sort.Strings(keys) + for _, k := range keys { + fmt.Fprintf(w, "%s\t%s\t%s\n", r.Funcs[k], k, r.Why[k]) + } + } + return nil +} + +// RootCmd is the perfinventory command. +func RootCmd() *cobra.Command { + var list bool + cmd := &cobra.Command{ + Use: "perfinventory [--list] ...", + Short: "Count public functions timed against a reference, timed alone, or not timed", + Args: cobra.MinimumNArgs(1), + SilenceUsage: true, + RunE: func(cmd *cobra.Command, args []string) error { + return report(cmd.OutOrStdout(), args, list) + }, + } + cmd.Flags().BoolVar(&list, "list", false, "also list every public function with its class and file") + return cmd +} + +func main() { + if err := RootCmd().Execute(); err != nil { + osExit(1) + } +} diff --git a/tools/perfinventory/main_test.go b/tools/perfinventory/main_test.go new file mode 100644 index 0000000..9213361 --- /dev/null +++ b/tools/perfinventory/main_test.go @@ -0,0 +1,297 @@ +package main + +import ( + "bytes" + "errors" + "os" + "path/filepath" + "strings" + "testing" + + "golang.org/x/tools/go/packages" +) + +// plant writes files under a fresh directory and returns it. +func plant(t *testing.T, files map[string]string) string { + t.Helper() + root := t.TempDir() + for name, body := range files { + p := filepath.Join(root, filepath.FromSlash(name)) + if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(p, []byte(body), 0o644); err != nil { + t.Fatal(err) + } + } + return root +} + +// repoFiles is a repository with one function of every class, a harness +// module, private packages and external scripts. +var repoFiles = map[string]string{ + "go.mod": "module example.com/r\n\ngo 1.27\n", + "r.go": `package r + +import "errors" + +// Ref is timed against the reference by perf_test.go. +func Ref(x float64) float64 { return x } + +// Noted is timed by a benchmark whose comment names SciPy. +func Noted(x float64) float64 { return x } + +// Bench is timed by a plain benchmark. +func Bench(x float64) float64 { return x } + +// Untimed has a test but no timing. +func Untimed(x float64) float64 { return x } + +// Harness is timed only by the harness module. +func Harness(x float64) float64 { return x } + +// Gen is generic; its uses resolve to the origin. +func Gen[T any](x T) T { return x } + +// T has methods. +type T struct{} + +// M is a method. +func (T) M() float64 { return 0 } + +// P has a pointer receiver. +func (*T) P() float64 { return 0 } + +// Error is plumbing, not in the denominator. +func (T) Error() string { return "" } + +func (T) private() {} + +// A is an alias, whose methods are T's. +type A = T + +// F is a named func type: no methods. +type F func() + +var Err = errors.New("x") + +func helper() {} +`, + "internal/k/k.go": "package k\n\nfunc K() {}\n", + "cmd/tool/main.go": "package main\n\nfunc Exported() {}\n\nfunc main() {}\n", + "r_internal_test.go": "package r\n\nimport \"testing\"\n\nfunc TestUntimed(t *testing.T) { Untimed(1); helper(); T{}.private() }\n", + "perf_test.go": `package r_test + +import ( + "testing" + "time" + + "example.com/r" +) + +const program = "import numpy as np" + +// cases are read by the timing loop. +var cases = []func(float64) float64{r.Ref} + +func TestPerf(t *testing.T) { + t0 := time.Now() + for _, c := range cases { + c(1) + } + _ = time.Since(t0) + _ = program +} +`, + "bench_test.go": `package r_test + +import ( + "testing" + + "example.com/r" +) + +// BenchmarkNoted is mirrored by a SciPy script. +func BenchmarkNoted(b *testing.B) { + for range b.N { + r.Noted(1) + } +} +`, + "plain_test.go": `package r_test + +import ( + "errors" + "testing" + + "example.com/r" +) + +var table = []float64{1} + +func run(b *testing.B, n *int, k int) { + var anon interface{ M() float64 } = r.T{} + _ = anon.M() + _ = errors.Unwrap(r.Err).Error + _ = r.Bench(table[0]) + r.Gen(1.0) + r.A{}.M() + (&r.T{}).P() +} + +func BenchmarkPlain(b *testing.B) { run(b, nil, 0) } + +func TestNotTimed(t *testing.T) { r.Untimed(1) } +`, + "bench/go.mod": "module example.com/r/bench\n\ngo 1.27\n\nrequire example.com/r v0.0.0\n\nreplace example.com/r => ../\n", + "bench/b.go": "package bench\n\n// Server is the reference process.\nconst Server = \"bench/scipy_server.py\"\n", + "bench/b_test.go": `package bench + +import ( + "testing" + "time" + + "example.com/r" +) + +func TestHarness(t *testing.T) { + _ = time.Now() + r.Harness(1) +} +`, + "scripts/ref.py": "import time\nt = time.perf_counter()\n", + "scripts/a.py": "import time\ntime.perf_counter()\n", + "scripts/b.sh": "python -m timeit 'x'\n", + "scripts/c.c": "clock_gettime(0, 0);\n", + "scripts/notes.sh": "echo nothing timed\n", + "scripts/notes.txt": "perf_counter in a text file is not a script\n", + "testdata/x/go.mod": "module skipped\n", + ".hidden/go.mod": "module skipped\n", + "vendor/v/go.mod": "module skipped\n", +} + +func TestInventory(t *testing.T) { + root := plant(t, repoFiles) + r, err := inventory(root) + if err != nil { + t.Fatal(err) + } + want := map[string]Class{ + "example.com/r.Ref": Ref, + "example.com/r.Harness": Ref, + "example.com/r.Noted": Noted, + "example.com/r.Bench": Bench, + "example.com/r.Gen": Bench, + "example.com/r.T.M": Bench, + "example.com/r.T.P": Bench, + "example.com/r.Untimed": None, + } + for k, c := range want { + if got, ok := r.Funcs[k]; !ok || got != c { + t.Errorf("%s: %v (declared %v), want %v", k, got, ok, c) + } + } + if len(r.Funcs) != len(want) { + t.Errorf("funcs = %v, want exactly %v", r.Funcs, want) + } + if r.Modules != 2 || r.ExternalHarness != 4 || len(r.ExternalExamples) != 3 { + t.Errorf("modules %d, external %d %v", r.Modules, r.ExternalHarness, r.ExternalExamples) + } + if r.Why["example.com/r.Ref"] != "perf_test.go" || r.Why["example.com/r.Harness"] != filepath.Join("bench", "b_test.go") { + t.Errorf("why = %v", r.Why) + } +} + +func run(t *testing.T, args ...string) (string, error) { + t.Helper() + var out bytes.Buffer + cmd := RootCmd() + cmd.SetArgs(args) + cmd.SetOut(&out) + cmd.SetErr(&out) + err := cmd.Execute() + return out.String(), err +} + +func TestReport(t *testing.T) { + root := plant(t, repoFiles) + out, err := run(t, "--list", root) + if err != nil { + t.Fatal(err) + } + for _, s := range []string{ + "| " + filepath.Base(root) + " | 2 | 8 | 2 | 1 | 4 | 1 | 4 |", + "| **total** | | 8 | 2 | 1 | 4 | 1 | 4 |", + "file\tperf_test.go\tref 1\tnoted 0\tbench 0\n", + "none\texample.com/r.Untimed\t\n", + "external\t", + } { + if !strings.Contains(out, s) { + t.Errorf("output lacks %q:\n%s", s, out) + } + } + out, err = run(t, root) + if err != nil || strings.Contains(out, "## ") { + t.Errorf("without --list: %v\n%s", err, out) + } +} + +func TestErrors(t *testing.T) { + if _, err := run(t, filepath.Join(t.TempDir(), "absent")); err == nil { + t.Error("a missing directory was counted") + } + broken := plant(t, map[string]string{ + "go.mod": "module example.com/b\n\ngo 1.27\n", + "b.go": "package b\n\nfunc F() int { return \"no\" }\n", + }) + if _, err := run(t, broken); err == nil || !strings.Contains(err.Error(), "package errors") { + t.Errorf("a package that does not type-check: %v", err) + } + saved := load + defer func() { load = saved }() + load = func(*packages.Config, ...string) ([]*packages.Package, error) { return nil, errors.New("boom") } + if _, err := run(t, plant(t, repoFiles)); err == nil || !strings.Contains(err.Error(), "boom") { + t.Errorf("a loader failure: %v", err) + } +} + +func TestUnreadableScript(t *testing.T) { + root := plant(t, map[string]string{"x.py": "perf_counter"}) + p := filepath.Join(root, "x.py") + if err := os.Chmod(p, 0); err != nil { + t.Fatal(err) + } + defer os.Chmod(p, 0o644) + if _, err := os.ReadFile(p); err == nil { + t.Skip("running with privileges that ignore file modes") + } + if _, err := inventory(root); err == nil { + t.Error("an unreadable script was skipped") + } +} + +func TestClassString(t *testing.T) { + if Ref.String() != "ref" || None.String() != "none" || Noted.String() != "noted" || Bench.String() != "bench" { + t.Error("class names") + } +} + +func TestMain_(t *testing.T) { + saved, savedExit, savedOut := os.Args, osExit, os.Stdout + defer func() { os.Args, osExit, os.Stdout = saved, savedExit, savedOut }() + devnull, err := os.OpenFile(os.DevNull, os.O_WRONLY, 0) + if err != nil { + t.Fatal(err) + } + defer devnull.Close() + os.Stdout = devnull + code := -1 + osExit = func(c int) { code = c } + os.Args = []string{"perfinventory", plant(t, map[string]string{"README": "no module"})} + main() + if code != -1 { + t.Fatalf("exit %d on an empty repository", code) + } + os.Args = []string{"perfinventory"} + main() + if code != 1 { + t.Fatalf("exit %d without an argument, want 1", code) + } +}