Skip to content

Commit f685074

Browse files
authored
Merge pull request #222 from CoreTrace/fix/structured-file-positions
fix: locate structured-file findings at their own line, or by pointer
2 parents d1e1ea9 + 8ce2575 commit f685074

25 files changed

Lines changed: 1485 additions & 118 deletions

‎docs/plugins.md‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -320,8 +320,12 @@ any analysis result: `SSAAnalysis` (`coretrace_python.ir.ssa`) for the code,
320320
## Findings
321321

322322
A `Finding` (`coretrace_python.findings`) has a `rule_id`, a `message`, a `Severity`, a
323-
`Confidence`, a `SourceSpan` (file, start and end line and column), the enclosing
324-
`function` and string `metadata`. Rule ids are lower-case words joined by dashes.
323+
`Confidence`, a location, the enclosing `function` and string `metadata`. Rule ids are
324+
lower-case words joined by dashes. The location (`span`) is a `SourceSpan` (file, start
325+
and end line and column) when the line is established, or a `FileLocation` (file and the
326+
JSON pointer of a value) for a value of a structured file whose line cannot be: a plugin
327+
that reads a finding's line checks which it has, and never invents a line for a
328+
`FileLocation` (`coretrace_python.source.positions` places JSON and TOML values).
325329

326330
## Testing a plugin
327331

‎docs/usage.md‎

Lines changed: 23 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -215,7 +215,8 @@ The comment also works in requirements files, where a line
215215
`pyyaml==5.3.1 # coretrace: ignore[vulnerable-dependency]` silences that requirement's
216216
finding. Suppressed findings are kept apart: the text report counts them, the JSON report
217217
lists them under `suppressed`, the SARIF log marks them as suppressed in source, and
218-
they never affect the exit status.
218+
they never affect the exit status. A comment only suppresses a finding located at a
219+
verified line; a finding with a file location (see Reports) is never suppressed this way.
219220

220221
## Adopting the analyzer on an existing code base
221222

@@ -232,7 +233,16 @@ above it does not make it new; a change to the line itself does. Commit the file
232233
shrink it as findings are fixed: an entry without a matching finding is simply unused.
233234
Baselined findings are counted in the text report, listed under `baselined` in the JSON
234235
report and marked `baselineState: unchanged` in the SARIF log, where new results are
235-
marked `new`.
236+
marked `new`. A finding with a file location is recognised by the JSON pointer of its
237+
value and a digest of the value (never the value itself) instead of the text of a line,
238+
so a new value at the same place is a new finding.
239+
240+
Baselines are written in schema 2. A schema 1 file, written by an earlier version, is
241+
still read and its entries match exactly as they did; but it recorded a finding in a
242+
JSON, TOML or lock file with the text of its key's first line, which was its own only
243+
when the key was not repeated. A finding such an entry cannot identify with certainty is
244+
reported as new, and the check then warns on standard error: review the new findings,
245+
delete the baseline and run the check again to record it in schema 2.
236246

237247
## Reports
238248

@@ -267,6 +277,17 @@ the directory of the checked file; a path outside it is printed as it is.
267277
}
268278
```
269279

280+
Lines are counted as Python counts them, by `\n`, `\r\n` or a lone `\r` (never U+2028
281+
or a form feed), in every report, suppression and baseline. A value of a JSON or TOML
282+
file, a package of a lock file, or a requirement of `pyproject.toml`, is located at its own line,
283+
which a separate component establishes from the value's structural path and checks
284+
against the value. When it cannot (a dotted TOML key, an inline table, a multi-line
285+
value), the finding gets a file location instead of a guessed line: `line`, `column`,
286+
`end_line` and `end_column` are `null`, and `pointer` gives the JSON pointer of the value
287+
in the file (`"/tool/app/api_token"`). The SARIF log gives such a result no `region`,
288+
with the pointer under the location's `properties`, and the text report shows the file
289+
alone (`settings.toml: high hardcoded-credential: …`).
290+
270291
`root` is the directory the paths are relative to. A file's `status` is `analysed`,
271292
`ambiguous` (analysed, but another file has its module name, see above), `syntax-error`
272293
or `unreadable`; only `analysed` files count in `files_analysed`.

‎src/coretrace_python/bundled/dependency/vulnerable_dependency/vulnerable_dependency.py‎

Lines changed: 23 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -3,10 +3,11 @@
33
from __future__ import annotations
44

55
from collections.abc import Sequence
6+
from pathlib import PurePath
67
from typing import ClassVar
78

89
from coretrace_python.analysis import AnyAnalysis
9-
from coretrace_python.dependency import DependencyAnalysis
10+
from coretrace_python.dependency import Advisory, DependencyAnalysis, Requirement
1011
from coretrace_python.dependency.correlation import (
1112
affected_entries,
1213
affected_symbols,
@@ -44,11 +45,7 @@ def analyze_project(self, ctx: ProjectContext) -> Sequence[Finding]:
4445
findings.append(
4546
Finding(
4647
rule_id="vulnerable-dependency",
47-
message=(
48-
f"{advisory.id}: {advisory.package} {advisory.vulnerable} is "
49-
f"{'required' if pinned else 'allowed'} by {requirement.name}"
50-
f"{requirement.specifier or ''}: {advisory.summary}"
51-
),
48+
message=_message(ctx, advisory, requirement),
5249
severity=advisory.severity,
5350
confidence=Confidence.HIGH if pinned else Confidence.MEDIUM,
5451
span=requirement.span,
@@ -58,6 +55,26 @@ def analyze_project(self, ctx: ProjectContext) -> Sequence[Finding]:
5855
return findings
5956

6057

58+
def _message(ctx: ProjectContext, advisory: Advisory, requirement: Requirement) -> str:
59+
"""The evidence: for a locked package, the version the lock file pins and the
60+
packages that require it; for a declared requirement, the specifier that allows or
61+
requires a vulnerable version."""
62+
63+
if requirement.locked is None:
64+
verb = "required" if requirement.pinned is not None else "allowed"
65+
return (
66+
f"{advisory.id}: {advisory.package} {advisory.vulnerable} is {verb} by "
67+
f"{requirement.name}{requirement.specifier or ''}: {advisory.summary}"
68+
)
69+
lock = PurePath(str(requirement.span.source_id)).name
70+
dependents = sorted(ctx.dependencies.required_by(requirement.name) or ())
71+
required = f" (required by {', '.join(dependents)})" if dependents else ""
72+
return (
73+
f"{advisory.id}: {advisory.package} {requirement.locked}, pinned in {lock}, "
74+
f"is in {advisory.vulnerable}{required}: {advisory.summary}"
75+
)
76+
77+
6178
def _ruled_out(ctx: ProjectContext) -> dict[str, list[str]]:
6279
"""By advisory, the calls to its entry points whose arguments contradict one of its
6380
conditions: the evidence that the requirement, imported, is not reached there."""

‎src/coretrace_python/cache.py‎

Lines changed: 22 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -41,9 +41,9 @@
4141
UnknownTarget,
4242
)
4343
from coretrace_python.semantic.symbols import SymbolId
44-
from coretrace_python.source import SourceId, SourceSpan
44+
from coretrace_python.source import FileLocation, Location, SourceId, SourceSpan
4545

46-
CACHE_FORMAT = 13
46+
CACHE_FORMAT = 14
4747

4848

4949
@dataclass(frozen=True)
@@ -185,6 +185,24 @@ def _encode_span(span: SourceSpan) -> list[Any]:
185185
return [str(span.source_id), span.start_line, span.start_column, span.end_line, span.end_column]
186186

187187

188+
def _encode_location(location: Location) -> Any:
189+
if isinstance(location, FileLocation):
190+
return {
191+
"file": str(location.source_id),
192+
"pointer": location.pointer,
193+
"digest": location.digest,
194+
}
195+
return _encode_span(location)
196+
197+
198+
def _decode_location(data: Any) -> Location:
199+
if isinstance(data, Mapping):
200+
return FileLocation(
201+
SourceId(_string(data["file"])), _string(data["pointer"]), _string(data["digest"])
202+
)
203+
return _decode_span(data)
204+
205+
188206
def _decode_span(data: Any) -> SourceSpan:
189207
file, line, column, end_line, end_column = data
190208
return SourceSpan(
@@ -217,7 +235,7 @@ def _encode_finding(finding: Finding) -> dict[str, Any]:
217235
"message": finding.message,
218236
"severity": finding.severity.value,
219237
"confidence": finding.confidence.value,
220-
"span": _encode_span(finding.span),
238+
"span": _encode_location(finding.span),
221239
"function": finding.function,
222240
"metadata": dict(finding.metadata),
223241
}
@@ -230,7 +248,7 @@ def _decode_finding(data: Mapping[str, Any]) -> Finding:
230248
_string(data["message"]),
231249
Severity(data["severity"]),
232250
Confidence(data["confidence"]),
233-
_decode_span(data["span"]),
251+
_decode_location(data["span"]),
234252
None if function is None else _string(function),
235253
{_string(k): _string(v) for k, v in data["metadata"].items()},
236254
)

‎src/coretrace_python/cli.py‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -274,7 +274,11 @@ def main(argv: list[str] | None = None) -> int:
274274
baselined = None
275275
if args.baseline is not None:
276276
if args.baseline.is_file():
277-
findings, baselined = Baseline.load(args.baseline).partition(findings, root)
277+
baseline = Baseline.load(args.baseline)
278+
findings, baselined = baseline.partition(findings, root)
279+
notice = baseline.transition_notice(findings)
280+
if notice is not None:
281+
print(f"coretrace: {args.baseline}: {notice}", file=sys.stderr)
278282
else:
279283
Baseline.of(findings, root).save(args.baseline)
280284
findings, baselined = (), findings

‎src/coretrace_python/dependency/graph.py‎

Lines changed: 51 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -13,15 +13,16 @@
1313
import re
1414
import tomllib
1515
from collections.abc import Iterable, Iterator, Mapping
16-
from dataclasses import dataclass
16+
from dataclasses import dataclass, replace
1717
from pathlib import PurePath, PurePosixPath
1818
from types import MappingProxyType
1919
from typing import Any, ClassVar
2020

2121
from coretrace_python.analysis import Analysis, AnalysisContext
2222
from coretrace_python.findings import Severity
2323
from coretrace_python.semantic.symbols import SymbolId
24-
from coretrace_python.source import SourceFile, SourceId, SourceSpan
24+
from coretrace_python.source import Location, SourceFile, SourceId, SourceSpan, lines_of
25+
from coretrace_python.source.positions import Pointer, TomlPositions, file_location
2526

2627
_REQUIREMENT = re.compile(r"^\s*([A-Za-z0-9][A-Za-z0-9._-]*)\s*(\[[^\]]*\])?\s*(.*)$")
2728
_CLAUSE = re.compile(r"^(===|==|!=|<=|>=|~=|<|>|\^)\s*([0-9][0-9A-Za-z.*+!-]*)$")
@@ -115,11 +116,16 @@ def _lower_bounds(specifier: str) -> list[Version]:
115116

116117
@dataclass(frozen=True)
117118
class Requirement:
119+
"""``span`` is where the requirement is written: a verified line, or, for a locked
120+
package whose entry cannot be placed, a file location. ``locked`` is the version a
121+
lock file pins, as it writes it; None for a declared requirement."""
122+
118123
name: str
119124
specifier: str
120-
span: SourceSpan
125+
span: Location
121126
pinned: Version | None = None
122127
optional: bool = False
128+
locked: str | None = None
123129

124130
@classmethod
125131
def parse(cls, text: str, source_id: SourceId, line: int, optional: bool = False) -> Requirement | None:
@@ -313,12 +319,14 @@ def merge(self, other: DependencyGraph) -> DependencyGraph:
313319
if current is None:
314320
merged[name] = requirement
315321
continue
322+
declared = current if current.specifier else requirement
316323
merged[name] = Requirement(
317324
name,
318325
current.specifier or requirement.specifier,
319-
current.span if current.specifier else requirement.span,
326+
declared.span,
320327
requirement.pinned or current.pinned,
321328
current.optional and requirement.optional,
329+
declared.locked,
322330
)
323331
dependents = dict(self._dependents)
324332
for name, found in other._dependents.items():
@@ -341,7 +349,7 @@ def parse_dependencies(source: SourceFile) -> DependencyGraph:
341349

342350
def _parse_requirements_txt(source: SourceFile) -> DependencyGraph:
343351
found: dict[str, Requirement] = {}
344-
for number, line in enumerate(source.text.splitlines(), start=1):
352+
for number, line in enumerate(lines_of(source.text), start=1):
345353
stripped = line.strip()
346354
if not stripped or stripped.startswith(("#", "-")):
347355
continue
@@ -362,25 +370,32 @@ def _parse_toml(source: SourceFile, extract: Any, dependents: Any = None) -> Dep
362370
return DependencyGraph(found, dependents=None if dependents is None else dependents(data))
363371

364372

365-
def _line_of(source: SourceFile, key: str, default: int = 1) -> int:
366-
for number, line in enumerate(source.text.splitlines(), start=1):
367-
if key in line:
368-
return number
369-
return default
373+
def _pyproject_requirements(data: Mapping[str, Any], source: SourceFile) -> list[Requirement]:
374+
"""The requirements ``pyproject.toml`` declares (PEP 621 and Poetry), each located at
375+
its own array element or key, which the position component checks, or given a file
376+
location when it cannot be placed."""
370377

378+
positions = TomlPositions(source.text)
371379

372-
def _pyproject_requirements(data: Mapping[str, Any], source: SourceFile) -> list[Requirement]:
373-
found: list[Requirement] = []
380+
def declared(text: str, path: Pointer, value: object, optional: bool = False) -> Requirement | None:
381+
requirement = Requirement.parse(text, source.source_id, 1, optional)
382+
if requirement is None:
383+
return None
384+
place = positions.locate(path, value)
385+
location: Location = (
386+
SourceSpan(source.source_id, *place)
387+
if place is not None
388+
else file_location(source.source_id, path, text)
389+
)
390+
return replace(requirement, span=location)
391+
392+
found: list[Requirement | None] = []
374393
project = data.get("project", {})
375-
for text in project.get("dependencies", []) or []:
376-
requirement = Requirement.parse(text, source.source_id, _line_of(source, text))
377-
if requirement is not None:
378-
found.append(requirement)
379-
for group in (project.get("optional-dependencies", {}) or {}).values():
380-
for text in group or []:
381-
requirement = Requirement.parse(text, source.source_id, _line_of(source, text), True)
382-
if requirement is not None:
383-
found.append(requirement)
394+
for index, text in enumerate(project.get("dependencies", []) or []):
395+
found.append(declared(text, ("project", "dependencies", index), text))
396+
for group, texts in (project.get("optional-dependencies", {}) or {}).items():
397+
for index, text in enumerate(texts or []):
398+
found.append(declared(text, ("project", "optional-dependencies", group, index), text, True))
384399
poetry = data.get("tool", {}).get("poetry", {})
385400
for section, optional in (("dependencies", False), ("dev-dependencies", True)):
386401
for name, spec in (poetry.get(section, {}) or {}).items():
@@ -389,23 +404,29 @@ def _pyproject_requirements(data: Mapping[str, Any], source: SourceFile) -> list
389404
specifier = spec.get("version", "") if isinstance(spec, dict) else str(spec)
390405
if specifier == "*":
391406
specifier = ""
392-
requirement = Requirement.parse(
393-
f"{name}{specifier}", source.source_id, _line_of(source, name), optional
394-
)
395-
if requirement is not None:
396-
found.append(requirement)
397-
return found
407+
found.append(declared(f"{name}{specifier}", ("tool", "poetry", section, name), spec, optional))
408+
return [requirement for requirement in found if requirement is not None]
398409

399410

400411
def _lock_requirements(data: Mapping[str, Any], source: SourceFile) -> list[Requirement]:
412+
"""The packages a lock file pins, each located at the ``name`` of its own
413+
``[[package]]`` entry, which the position component checks."""
414+
401415
found: list[Requirement] = []
402-
for package in data.get("package", []) or []:
416+
positions = TomlPositions(source.text)
417+
for index, package in enumerate(data.get("package", []) or []):
403418
name, version = package.get("name"), package.get("version")
404419
if not isinstance(name, str) or not isinstance(version, str):
405420
continue
406-
line = _line_of(source, f'name = "{name}"')
421+
path = ("package", index, "name")
422+
place = positions.locate(path, name)
423+
location: Location = (
424+
SourceSpan(source.source_id, *place)
425+
if place is not None
426+
else file_location(source.source_id, path, name)
427+
)
407428
found.append(
408-
Requirement(normalize(name), "", SourceSpan(source.source_id, line, 1), Version.parse(version))
429+
Requirement(normalize(name), "", location, Version.parse(version), locked=version)
409430
)
410431
return found
411432

‎src/coretrace_python/dependency/vex.py‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@
2626
from coretrace_python.dependency.sbom import purl
2727
from coretrace_python.findings import Component, Finding
2828
from coretrace_python.findings.coverage import Coverage
29+
from coretrace_python.source import SourceSpan
2930
from coretrace_python.taint import FILTER_FUNCTIONS
3031

3132
CONTEXT = "https://openvex.dev/ns/v0.2.0"
@@ -179,10 +180,12 @@ def _reached(findings: Sequence[Finding], root: Path) -> str:
179180
level = finding.metadata.get("level")
180181
if level in REACHED:
181182
path = _relative(str(finding.span.source_id), root)
182-
place = (path, finding.span.start_line, str(finding.metadata.get("symbol")))
183+
line = finding.span.start_line if isinstance(finding.span, SourceSpan) else 0
184+
place = (path, line, str(finding.metadata.get("symbol")))
183185
levels[place] = max(level, levels.get(place, level), key=REACHED.index)
184186
return "; ".join(
185-
f"{path}:{line} {symbol} ({level})" for (path, line, symbol), level in sorted(levels.items())
187+
f"{path}{f':{line}' if line else ''} {symbol} ({level})"
188+
for (path, line, symbol), level in sorted(levels.items())
186189
)
187190

188191

0 commit comments

Comments
 (0)