mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-22 06:29:29 +00:00
bench(harnessx-removal): deterministic baseline without the extra (task 1)
Equality script (design section 3) records suite outcomes, `ousast pairs --json`, the python-vulnerable quick benchmark, one standard scan and `improve --dry-run` from a frozen export with the harnessx extra absent and no model reachable. Two independent records are byte-identical (determinism.json). Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
2ed6d9a036
commit
ca5caf735f
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"baseline": "this directory (record run A)",
|
||||
"candidate": "record run B: second independent record from a fresh export of the same tree, same venv; not committed",
|
||||
"differences": [],
|
||||
"equal": true,
|
||||
"exclusions": [
|
||||
"JSON keys: created_at, finished, generated_at, runtime_seconds, scan_id, started, timestamp",
|
||||
"JSON keys ending in: _seconds, _ms",
|
||||
"run directory names / run ids (<timestamp>-<8 hex>, benchmark.py create_benchmark_run, run.py create_scan_run) -> <RUN_ID>",
|
||||
"trace/events.jsonl as a whole (timings); never copied",
|
||||
"the export's absolute root path -> <ROOT>; the children's TMPDIR -> <TMP>",
|
||||
"suite: only the outcome per test id is kept (junit durations dropped)",
|
||||
"env.json is a provenance record (commit, freeze hash, instrument checks), not compared"
|
||||
],
|
||||
"raw_bytes_identical": "all 15 files incl. env.json (diff -r empty)"
|
||||
}
|
||||
Reference in New Issue
Block a user