mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
Marius CiepluchandClaude Opus 5.5
800204de87
Reran benchmarks/harnessx_removal_equality.py record from a git archive export ofb9afb66(tasks 1-6 landed; same venv freeze hash as the baseline, harnessx not importable, no API keys, empty config) and compared it against 2026-09-30-harnessx-removal-baseline: equal=True differences=0 All 13 output files (pairs.json, benchmark, standard scan, improve --dry-run) are raw-byte identical to the baseline. The suite map (suite-map.json) accounts for 29 deleted ids, 2 renamed ids and 83 added ids; each deleted or renamed id cites the design's Tests table and commitcbadc38orb9afb66. Every surviving test id keeps its outcome (1740 passed, 38 skipped incl. the README xfail). Instrument note: a first record failed test_push_hook twice because the recording shell's PATH lacked .venv/bin (the hook execs `ousast`); the same two tests fail atb9afb66without it and pass with it. The committed record has .venv/bin on PATH. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
16 lines
794 B
JSON
16 lines
794 B
JSON
{
|
|
"baseline": "benchmarks/measurements/2026-09-30-harnessx-removal-baseline",
|
|
"candidate": "benchmarks/measurements/2026-09-30-harnessx-removal-equality/candidate",
|
|
"differences": [],
|
|
"equal": true,
|
|
"exclusions": [
|
|
"JSON keys: created_at, finished, generated_at, runtime_seconds, scan_id, started, timestamp",
|
|
"JSON keys ending in: _seconds, _ms",
|
|
"run directory names / run ids (<timestamp>-<8 hex>, benchmark.py create_benchmark_run, run.py create_scan_run) -> <RUN_ID>",
|
|
"trace/events.jsonl as a whole (timings); never copied",
|
|
"the export's absolute root path -> <ROOT>; the children's TMPDIR -> <TMP>",
|
|
"suite: only the outcome per test id is kept (junit durations dropped)",
|
|
"env.json is a provenance record (commit, freeze hash, instrument checks), not compared"
|
|
]
|
|
}
|