marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

bench(harnessx-removal): deterministic outputs byte-identical after the removal (task 7)

Browse Source 
Reran benchmarks/harnessx_removal_equality.py record from a git archive
export of b9afb66 (tasks 1-6 landed; same venv freeze hash as the
baseline, harnessx not importable, no API keys, empty config) and
compared it against 2026-09-30-harnessx-removal-baseline:

  equal=True differences=0

All 13 output files (pairs.json, benchmark, standard scan, improve
--dry-run) are raw-byte identical to the baseline. The suite map
(suite-map.json) accounts for 29 deleted ids, 2 renamed ids and 83
added ids; each deleted or renamed id cites the design's Tests table and
commit cbadc38 or b9afb66. Every surviving test id keeps its outcome
(1740 passed, 38 skipped incl. the README xfail).

Instrument note: a first record failed test_push_hook twice because the
recording shell's PATH lacked .venv/bin (the hook execs `ousast`); the
same two tests fail at b9afb66 without it and pass with it. The
committed record has .venv/bin on PATH.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-09-30 12:40:57 +02:00
co-authored by Claude Opus 5.5
parent b9afb66250
commit 800204de87
19 changed files with 21046 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.kiro/specs/harnessx-removal/tasks.md
+1 -1
View File
@@ -50,7 +50,7 @@ and passed its fixture test. Every task is a commit gated on the full suite's ow
reference search test (Req 2.4).
- _Requirements: 2.1, 2.2, 2.3, 2.4_
- [ ] 7. Equality proof
- [x] 7. Equality proof
- Rerun the baseline script; commit the comparison record next to the baseline; any difference outside the
exclusion list blocks the merge.
- _Requirements: 3.2_
benchmarks/measurements/2026-09-30-harnessx-removal-equality/README
+16
View File
@@ -0,0 +1,16 @@
Equality proof for harnessx-removal (task 7, Requirement 3.2), recorded 2026-09-30.
candidate/ `harnessx_removal_equality.py record` from a `git archive` export of b9afb66 (tasks 1-6 landed),
same venv (freeze sha256 0e5ad419... identical to the baseline's), no API keys, empty config.
suite-map.json the suite ids deleted (29), renamed (2) and added (83) since the baseline, each deleted or renamed
id justified by the design's Tests table and commit cbadc38 or b9afb66.
comparison.json `compare ../2026-09-30-harnessx-removal-baseline candidate --suite-map suite-map.json`:
equal=True differences=0.
Beyond the comparison: every output file (pairs, benchmark, scan, improve) is raw-byte identical to the baseline
(`diff -rq` differs only in suite.json and env.json; determinism.json is baseline-only), and every surviving test id keeps its outcome.
Instrument note: the first record failed the two test_push_hook cases (exit 1, prior-input never written) because
the recording shell's PATH lacked .venv/bin, so the hook's `exec ousast` could not start. Same failure at b9afb66
in the worktree, gone with .venv/bin on PATH (5 passed). The recorded run has .venv/bin on PATH; the baseline
passed these tests, so its PATH had `ousast` too.
benchmarks/measurements/2026-09-30-harnessx-removal-equality/candidate/README
+1
View File
@@ -0,0 +1 @@
Excluded from comparison: JSON keys: created_at, finished, generated_at, runtime_seconds, scan_id, started, timestamp; JSON keys ending in: _seconds, _ms; run directory names / run ids (<timestamp>-<8 hex>, benchmark.py create_benchmark_run, run.py create_scan_run) -> <RUN_ID>; trace/events.jsonl as a whole (timings); never copied; the export's absolute root path -> <ROOT>; the children's TMPDIR -> <TMP>; suite: only the outcome per test id is kept (junit durations dropped); env.json is a provenance record (commit, freeze hash, instrument checks), not compared
benchmarks/measurements/2026-09-30-harnessx-removal-equality/candidate/benchmark/benchmark_result.json
+181
View File
@@ -0,0 +1,181 @@
{
"baseline_deltas": [],
"benchmark_name": "python-vulnerable",
"benchmark_run_id": "<RUN_ID>",
"calibration_records": [
{
"cwe": "CWE-89",
"evidence": "execute runs a SQL query string built from request input on a prior line",
"failed_stage": "benchmark_ground_truth_matching",
"next_improvement_candidate": "language_hunter",
"path": "app.py",
"reason": "no OpenUltraSAST finding matched the expected benchmark vulnerability",
"rule_id": null,
"vulnerability_class": "sql injection"
}
],
"false_positives": [],
"language": "python_web",
"metrics": {
"actual_findings_total": 12,
"artifact_links": {
"findings": "<ROOT>/benchmarks/fixtures/python-vulnerable/.openultrasast/runs/<RUN_ID>/findings.json",
"manifest": "<ROOT>/benchmarks/fixtures/python-vulnerable/.openultrasast/runs/<RUN_ID>/manifest.json",
"report": "<ROOT>/benchmarks/fixtures/python-vulnerable/.openultrasast/runs/<RUN_ID>/report.md",
"sarif": "<ROOT>/benchmarks/fixtures/python-vulnerable/.openultrasast/runs/<RUN_ID>/report.sarif"
},
"expected_findings_total": 13,
"false_positive_findings_total": 0,
"matched_findings_total": 12,
"missed_findings_total": 1,
"model_usage": {},
"per_rule": {
"python-flask-debug": {
"false_positives": 0,
"matched": 1,
"missed": 0
},
"python-os-command": {
"false_positives": 0,
"matched": 1,
"missed": 0
},
"python-path-traversal": {
"false_positives": 0,
"matched": 1,
"missed": 0
},
"python-shell-true": {
"false_positives": 0,
"matched": 1,
"missed": 0
},
"python-sql-injection": {
"false_positives": 0,
"matched": 2,
"missed": 0
},
"python-ssrf": {
"false_positives": 0,
"matched": 1,
"missed": 0
},
"python-ssti": {
"false_positives": 0,
"matched": 1,
"missed": 0
},
"python-unsafe-deserialization": {
"false_positives": 0,
"matched": 2,
"missed": 0
},
"python-unsafe-eval": {
"false_positives": 0,
"matched": 1,
"missed": 0
},
"python-weak-hash": {
"false_positives": 0,
"matched": 1,
"missed": 0
}
},
"precision": 1.0,
"recall": 0.9230769230769231
},
"misses": [
{
"cwe": "CWE-89",
"evidence": "execute runs a SQL query string built from request input on a prior line",
"path": "app.py",
"reason": "no OpenUltraSAST finding matched the expected benchmark vulnerability",
"rule_id": null,
"vulnerability_class": "sql injection"
}
],
"mode": "quick",
"rule_recommendations": [
{
"action": "keep",
"false_positives": 0,
"matched": 1,
"missed": 0,
"rationale": "rule matched expected findings with no false positives",
"rule_id": "python-flask-debug"
},
{
"action": "keep",
"false_positives": 0,
"matched": 1,
"missed": 0,
"rationale": "rule matched expected findings with no false positives",
"rule_id": "python-os-command"
},
{
"action": "keep",
"false_positives": 0,
"matched": 1,
"missed": 0,
"rationale": "rule matched expected findings with no false positives",
"rule_id": "python-path-traversal"
},
{
"action": "keep",
"false_positives": 0,
"matched": 1,
"missed": 0,
"rationale": "rule matched expected findings with no false positives",
"rule_id": "python-shell-true"
},
{
"action": "keep",
"false_positives": 0,
"matched": 2,
"missed": 0,
"rationale": "rule matched expected findings with no false positives",
"rule_id": "python-sql-injection"
},
{
"action": "keep",
"false_positives": 0,
"matched": 1,
"missed": 0,
"rationale": "rule matched expected findings with no false positives",
"rule_id": "python-ssrf"
},
{
"action": "keep",
"false_positives": 0,
"matched": 1,
"missed": 0,
"rationale": "rule matched expected findings with no false positives",
"rule_id": "python-ssti"
},
{
"action": "keep",
"false_positives": 0,
"matched": 2,
"missed": 0,
"rationale": "rule matched expected findings with no false positives",
"rule_id": "python-unsafe-deserialization"
},
{
"action": "keep",
"false_positives": 0,
"matched": 1,
"missed": 0,
"rationale": "rule matched expected findings with no false positives",
"rule_id": "python-unsafe-eval"
},
{
"action": "keep",
"false_positives": 0,
"matched": 1,
"missed": 0,
"rationale": "rule matched expected findings with no false positives",
"rule_id": "python-weak-hash"
}
],
"scan_run_dir": "<ROOT>/benchmarks/fixtures/python-vulnerable/.openultrasast/runs/<RUN_ID>"
}
benchmarks/measurements/2026-09-30-harnessx-removal-equality/candidate/benchmark/calibration_records.json
+12
View File
@@ -0,0 +1,12 @@
[
{
"cwe": "CWE-89",
"evidence": "execute runs a SQL query string built from request input on a prior line",
"failed_stage": "benchmark_ground_truth_matching",
"next_improvement_candidate": "language_hunter",
"path": "app.py",
"reason": "no OpenUltraSAST finding matched the expected benchmark vulnerability",
"rule_id": null,
"vulnerability_class": "sql injection"
}
]
benchmarks/measurements/2026-09-30-harnessx-removal-equality/candidate/benchmark/external_baseline_deltas.json
+1
View File
@@ -0,0 +1 @@
[]
benchmarks/measurements/2026-09-30-harnessx-removal-equality/candidate/benchmark/stdout.txt
+11
View File
@@ -0,0 +1,11 @@
scan_id=<RUN_ID>
run_dir=<ROOT>/benchmarks/fixtures/python-vulnerable/.openultrasast/runs/<RUN_ID>
file_targets=2
ranked_targets=2
findings=12
calibrations_applied=0
benchmark_run_id=<RUN_ID>
benchmark_run_dir=<ROOT>/benchmarks/fixtures/python-vulnerable/.openultrasast/benchmarks/<RUN_ID>
benchmark_expected=13
benchmark_matched=12
benchmark_missed=1
benchmarks/measurements/2026-09-30-harnessx-removal-equality/candidate/env.json
+71
View File
@@ -0,0 +1,71 @@
{
"child_env": {
"OPENULTRASAST_SKIP_DOTENV": "1",
"PYTHONHASHSEED": "0",
"api_keys": "every *_API_KEY unset",
"config": "empty"
},
"exclusions": [
"JSON keys: created_at, finished, generated_at, runtime_seconds, scan_id, started, timestamp",
"JSON keys ending in: _seconds, _ms",
"run directory names / run ids (<timestamp>-<8 hex>, benchmark.py create_benchmark_run, run.py create_scan_run) -> <RUN_ID>",
"trace/events.jsonl as a whole (timings); never copied",
"the export's absolute root path -> <ROOT>; the children's TMPDIR -> <TMP>",
"suite: only the outcome per test id is kept (junit durations dropped)",
"env.json is a provenance record (commit, freeze hash, instrument checks), not compared"
],
"freeze_sha256": "0e5ad41948a751e1c60f72738c48f5ac482b24b67e51c77369ab3be51fa418eb",
"harnessx_importable": false,
"inputs": {
"benchmarks/manifests/java-spring-boot-vulnerable.toml": 1032,
"benchmarks/manifests/python-vulnerable.toml": 2771,
"benchmarks/pairs/catalog.toml": 6052
},
"instrument_checks": {
"benchmark": {
"expected": 13,
"findings": 12,
"matched": 12,
"missed": 1
},
"benchmark_files": {
"benchmark_result.json": 5896,
"calibration_records.json": 406,
"external_baseline_deltas.json": 3
},
"improve": {
"journal_bytes": null,
"rounds": 1
},
"openultrasast_module": "<ROOT>/src/openultrasast/__init__.py",
"pairs": {
"detected_vuln": 57,
"labeled_expected": 342,
"labeled_matched": 82,
"outcomes": 317,
"pair_correct": 32,
"pairs": 317
},
"scan": {
"file_targets": 2,
"findings": 17
},
"scan_files": {
"findings.json": 20469,
"fusion.json": 6284,
"harness.json": 189,
"manifest.json": 13266,
"report.sarif": 40310,
"verification.json": 10573
},
"suite": {
"error": 0,
"failure": 0,
"passed": 1740,
"skipped": 38
}
},
"python": "3.11.13",
"tree_ish": "b9afb662506dbe73de931b666400de5485443c19",
"tree_sitter": true
}
benchmarks/measurements/2026-09-30-harnessx-removal-equality/candidate/improve/stdout.txt
+5
View File
@@ -0,0 +1,5 @@
manifest=benchmarks/manifests/java-spring-boot-vulnerable.toml
target=<ROOT>/benchmarks/fixtures/java-spring-boot-vulnerable
round 1: no_proposals | recall 100.00%->100.00% fp 0.00%->0.00% score 73->73 | edits: -
rounds=1 accepted=0
dry_run=true (no ledger or mechanism store written)
benchmarks/measurements/2026-09-30-harnessx-removal-equality/candidate/pairs.json
+17216
View File
File diff suppressed because it is too large Load Diff
benchmarks/measurements/2026-09-30-harnessx-removal-equality/candidate/scan/findings.json
+630
View File
@@ -0,0 +1,630 @@
{
"findings": [
{
"confidence": "medium",
"evidence_level": "static_corroboration",
"finding_id": "python-flask-debug:app.py:81",
"function_name": null,
"line": 81,
"path": "app.py",
"proposed_snippet": null,
"ranking_priority": 1.5,
"rationale": "Static pattern python-flask-debug (CWE-489) matched debug on line 81; manual verification still required.",
"reachability_conditions": [],
"reachability_evidence": [],
"reachability_status": "unknown",
"rung": "suspicion",
"severity": "medium",
"status": "enabled",
"tags": [
"crypto",
"deserialization",
"filesystem_entry",
"insecure_default",
"syscall_entry"
],
"title": "Flask debug mode enabled needs review"
},
{
"confidence": "medium",
"evidence_level": "static_corroboration",
"finding_id": "python-os-command:app.py:17",
"function_name": "ping",
"line": 17,
"path": "app.py",
"proposed_snippet": null,
"ranking_priority": 3.8,
"rationale": "Static pattern python-os-command (CWE-78) matched os.system on line 17; manual verification still required.",
"reachability_conditions": [],
"reachability_evidence": [
{
"access_declared": false,
"access_evidence": [
"app.get('/ping')"
],
"access_level": "public",
"conditions": [],
"defines_handler": true,
"end_line": 17,
"function_name": "ping",
"kind": "route",
"line": 15,
"name": "ping",
"path": "app.py",
"provenance": "entrypoint:route:public",
"rationale": "route surface classified as public at http_request",
"registered_by": "",
"trust_boundary": "http_request"
}
],
"reachability_status": "reachable",
"rung": "suspicion",
"severity": "critical",
"status": "enabled",
"tags": [
"crypto",
"deserialization",
"filesystem_entry",
"syscall_entry"
],
"title": "Python OS command execution needs review"
},
{
"confidence": "medium",
"evidence_level": "static_corroboration",
"finding_id": "python-path-traversal:app.py:55",
"function_name": "read",
"line": 55,
"path": "app.py",
"proposed_snippet": null,
"ranking_priority": 3.8,
"rationale": "Static pattern python-path-traversal (CWE-22) matched open on line 55; manual verification still required.",
"reachability_conditions": [],
"reachability_evidence": [
{
"access_declared": false,
"access_evidence": [
"app.get('/read')"
],
"access_level": "public",
"conditions": [],
"defines_handler": true,
"end_line": 55,
"function_name": "read",
"kind": "route",
"line": 54,
"name": "read",
"path": "app.py",
"provenance": "entrypoint:route:public",
"rationale": "route surface classified as public at http_request",
"registered_by": "",
"trust_boundary": "http_request"
}
],
"reachability_status": "reachable",
"rung": "suspicion",
"severity": "medium",
"status": "enabled",
"tags": [
"crypto",
"deserialization",
"filesystem_entry",
"syscall_entry"
],
"title": "Python file open with external input needs review"
},
{
"confidence": "medium",
"evidence_level": "static_corroboration",
"finding_id": "python-shell-true:app.py:23",
"function_name": "run",
"line": 23,
"path": "app.py",
"proposed_snippet": null,
"ranking_priority": 3.8,
"rationale": "Static pattern python-shell-true (CWE-78) matched subprocess.check_output on line 23; manual verification still required.",
"reachability_conditions": [],
"reachability_evidence": [
{
"access_declared": false,
"access_evidence": [
"app.get('/run')"
],
"access_level": "public",
"conditions": [],
"defines_handler": true,
"end_line": 23,
"function_name": "run",
"kind": "route",
"line": 21,
"name": "run",
"path": "app.py",
"provenance": "entrypoint:route:public",
"rationale": "route surface classified as public at http_request",
"registered_by": "",
"trust_boundary": "http_request"
}
],
"reachability_status": "reachable",
"rung": "suspicion",
"severity": "critical",
"status": "enabled",
"tags": [
"crypto",
"deserialization",
"filesystem_entry",
"syscall_entry"
],
"title": "Subprocess shell execution needs review"
},
{
"confidence": "medium",
"evidence_level": "static_corroboration",
"finding_id": "python-sql-injection:app.py:44",
"function_name": "user",
"line": 44,
"path": "app.py",
"proposed_snippet": null,
"ranking_priority": 3.8,
"rationale": "Static pattern python-sql-injection (CWE-89) matched .execute on line 44; manual verification still required.",
"reachability_conditions": [],
"reachability_evidence": [
{
"access_declared": false,
"access_evidence": [
"app.get('/user')"
],
"access_level": "public",
"conditions": [],
"defines_handler": true,
"end_line": 44,
"function_name": "user",
"kind": "route",
"line": 42,
"name": "user",
"path": "app.py",
"provenance": "entrypoint:route:public",
"rationale": "route surface classified as public at http_request",
"registered_by": "",
"trust_boundary": "http_request"
}
],
"reachability_status": "reachable",
"rung": "suspicion",
"severity": "high",
"status": "enabled",
"tags": [
"crypto",
"deserialization",
"filesystem_entry",
"injection",
"syscall_entry"
],
"title": "Python SQL string composition needs review"
},
{
"confidence": "medium",
"evidence_level": "static_corroboration",
"finding_id": "python-sql-injection:app.py:50",
"function_name": "report",
"line": 50,
"path": "app.py",
"proposed_snippet": null,
"ranking_priority": 3.8,
"rationale": "Static pattern python-sql-injection (CWE-89) matched .execute on line 50; manual verification still required.",
"reachability_conditions": [],
"reachability_evidence": [
{
"access_declared": false,
"access_evidence": [
"app.get('/report')"
],
"access_level": "public",
"conditions": [],
"defines_handler": true,
"end_line": 50,
"function_name": "report",
"kind": "route",
"line": 48,
"name": "report",
"path": "app.py",
"provenance": "entrypoint:route:public",
"rationale": "route surface classified as public at http_request",
"registered_by": "",
"trust_boundary": "http_request"
}
],
"reachability_status": "reachable",
"rung": "suspicion",
"severity": "high",
"status": "enabled",
"tags": [
"crypto",
"deserialization",
"filesystem_entry",
"injection",
"syscall_entry"
],
"title": "Python SQL string composition needs review"
},
{
"confidence": "medium",
"evidence_level": "static_corroboration",
"finding_id": "python-ssrf:app.py:60",
"function_name": "fetch",
"line": 60,
"path": "app.py",
"proposed_snippet": null,
"ranking_priority": 3.8,
"rationale": "Static pattern python-ssrf (CWE-918) matched requests.get on line 60; manual verification still required.",
"reachability_conditions": [],
"reachability_evidence": [
{
"access_declared": false,
"access_evidence": [
"app.get('/fetch')"
],
"access_level": "public",
"conditions": [],
"defines_handler": true,
"end_line": 60,
"function_name": "fetch",
"kind": "route",
"line": 59,
"name": "fetch",
"path": "app.py",
"provenance": "entrypoint:route:public",
"rationale": "route surface classified as public at http_request",
"registered_by": "",
"trust_boundary": "http_request"
}
],
"reachability_status": "reachable",
"rung": "suspicion",
"severity": "medium",
"status": "enabled",
"tags": [
"crypto",
"deserialization",
"filesystem_entry",
"network_entry",
"syscall_entry"
],
"title": "Python outbound request with external input needs review"
},
{
"confidence": "medium",
"evidence_level": "static_corroboration",
"finding_id": "python-ssti:app.py:70",
"function_name": "render",
"line": 70,
"path": "app.py",
"proposed_snippet": null,
"ranking_priority": 3.8,
"rationale": "Static pattern python-ssti (CWE-94) matched render_template_string on line 70; manual verification still required.",
"reachability_conditions": [],
"reachability_evidence": [
{
"access_declared": false,
"access_evidence": [
"app.get('/render')"
],
"access_level": "public",
"conditions": [],
"defines_handler": true,
"end_line": 70,
"function_name": "render",
"kind": "route",
"line": 69,
"name": "render",
"path": "app.py",
"provenance": "entrypoint:route:public",
"rationale": "route surface classified as public at http_request",
"registered_by": "",
"trust_boundary": "http_request"
}
],
"reachability_status": "reachable",
"rung": "suspicion",
"severity": "medium",
"status": "enabled",
"tags": [
"crypto",
"deserialization",
"filesystem_entry",
"injection",
"syscall_entry"
],
"title": "Python server-side template injection needs review"
},
{
"confidence": "medium",
"evidence_level": "static_corroboration",
"finding_id": "python-unsafe-deserialization:app.py:33",
"function_name": "load",
"line": 33,
"path": "app.py",
"proposed_snippet": null,
"ranking_priority": 3.8,
"rationale": "Static pattern python-unsafe-deserialization (CWE-502) matched pickle.loads on line 33; manual verification still required.",
"reachability_conditions": [],
"reachability_evidence": [
{
"access_declared": false,
"access_evidence": [
"app.get('/load')"
],
"access_level": "public",
"conditions": [],
"defines_handler": true,
"end_line": 33,
"function_name": "load",
"kind": "route",
"line": 32,
"name": "load",
"path": "app.py",
"provenance": "entrypoint:route:public",
"rationale": "route surface classified as public at http_request",
"registered_by": "",
"trust_boundary": "http_request"
}
],
"reachability_status": "reachable",
"rung": "suspicion",
"severity": "medium",
"status": "enabled",
"tags": [
"crypto",
"deserialization",
"filesystem_entry",
"syscall_entry"
],
"title": "Unsafe Python deserialization needs review"
},
{
"confidence": "medium",
"evidence_level": "static_corroboration",
"finding_id": "python-unsafe-deserialization:app.py:38",
"function_name": "config",
"line": 38,
"path": "app.py",
"proposed_snippet": null,
"ranking_priority": 3.8,
"rationale": "Static pattern python-unsafe-deserialization (CWE-502) matched yaml.load on line 38; manual verification still required.",
"reachability_conditions": [],
"reachability_evidence": [
{
"access_declared": false,
"access_evidence": [
"app.get('/config')"
],
"access_level": "public",
"conditions": [],
"defines_handler": true,
"end_line": 38,
"function_name": "config",
"kind": "route",
"line": 37,
"name": "config",
"path": "app.py",
"provenance": "entrypoint:route:public",
"rationale": "route surface classified as public at http_request",
"registered_by": "",
"trust_boundary": "http_request"
}
],
"reachability_status": "reachable",
"rung": "suspicion",
"severity": "medium",
"status": "enabled",
"tags": [
"crypto",
"deserialization",
"filesystem_entry",
"syscall_entry"
],
"title": "Unsafe Python deserialization needs review"
},
{
"confidence": "medium",
"evidence_level": "static_corroboration",
"finding_id": "python-unsafe-eval:app.py:28",
"function_name": "calc",
"line": 28,
"path": "app.py",
"proposed_snippet": null,
"ranking_priority": 3.8,
"rationale": "Static pattern python-unsafe-eval (CWE-95) matched eval on line 28; manual verification still required.",
"reachability_conditions": [],
"reachability_evidence": [
{
"access_declared": false,
"access_evidence": [
"app.get('/calc')"
],
"access_level": "public",
"conditions": [],
"defines_handler": true,
"end_line": 28,
"function_name": "calc",
"kind": "route",
"line": 27,
"name": "calc",
"path": "app.py",
"provenance": "entrypoint:route:public",
"rationale": "route surface classified as public at http_request",
"registered_by": "",
"trust_boundary": "http_request"
}
],
"reachability_status": "reachable",
"rung": "suspicion",
"severity": "critical",
"status": "enabled",
"tags": [
"crypto",
"deserialization",
"filesystem_entry",
"syscall_entry"
],
"title": "Dynamic Python execution needs review"
},
{
"confidence": "medium",
"evidence_level": "static_corroboration",
"finding_id": "python-weak-hash:app.py:65",
"function_name": "token",
"line": 65,
"path": "app.py",
"proposed_snippet": null,
"ranking_priority": 3.8,
"rationale": "Static pattern python-weak-hash (CWE-327) matched md5 on line 65; manual verification still required.",
"reachability_conditions": [],
"reachability_evidence": [
{
"access_declared": false,
"access_evidence": [
"app.get('/token')"
],
"access_level": "public",
"conditions": [],
"defines_handler": true,
"end_line": 65,
"function_name": "token",
"kind": "route",
"line": 64,
"name": "token",
"path": "app.py",
"provenance": "entrypoint:route:public",
"rationale": "route surface classified as public at http_request",
"registered_by": "",
"trust_boundary": "http_request"
}
],
"reachability_status": "reachable",
"rung": "suspicion",
"severity": "medium",
"status": "enabled",
"tags": [
"crypto",
"deserialization",
"filesystem_entry",
"syscall_entry"
],
"title": "Python weak hash needs review"
},
{
"confidence": "medium",
"evidence_level": "static_corroboration",
"finding_id": "overlay-coverage:app.py:77:execute",
"function_name": null,
"line": 77,
"path": "app.py",
"proposed_snippet": null,
"ranking_priority": 0.0,
"rationale": "uninventoried sink execute reached from request",
"reachability_conditions": [],
"reachability_evidence": [],
"reachability_status": "unknown",
"rung": "suspicion",
"severity": "high",
"status": "enabled",
"tags": [
"overlay-originated"
],
"title": "Overlay coverage: execute"
},
{
"confidence": "medium",
"evidence_level": "static_corroboration",
"finding_id": "overlay-coverage:safe.py:11:os.system",
"function_name": null,
"line": 11,
"path": "safe.py",
"proposed_snippet": null,
"ranking_priority": 0.0,
"rationale": "uninventoried sink os.system reached from parameter",
"reachability_conditions": [],
"reachability_evidence": [],
"reachability_status": "unknown",
"rung": "suspicion",
"severity": "critical",
"status": "enabled",
"tags": [
"overlay-originated"
],
"title": "Overlay coverage: os.system"
},
{
"confidence": "low",
"evidence_level": "suspicion",
"finding_id": "obligation:protected_read:app.py:44:identity_constraint",
"function_name": "user",
"line": 44,
"path": "app.py",
"proposed_snippet": null,
"ranking_priority": 0.3,
"rationale": "protected_read on users in user is reached without a dominating identity_constraint. Why the obligation exists: the operation fact alone (function-local mode). Label: function_local.",
"reachability_conditions": [],
"reachability_evidence": [],
"reachability_status": "unknown",
"rung": "suspicion",
"severity": "high",
"status": "enabled",
"tags": [
"obligation:protected_read",
"discharger:identity_constraint",
"obligation_evidence:function_local",
"resource:users"
],
"title": "Obligation not discharged: protected_read on users without identity_constraint"
},
{
"confidence": "low",
"evidence_level": "suspicion",
"finding_id": "obligation:protected_read:app.py:50:identity_constraint",
"function_name": "report",
"line": 50,
"path": "app.py",
"proposed_snippet": null,
"ranking_priority": 0.3,
"rationale": "protected_read on reports in report is reached without a dominating identity_constraint. Why the obligation exists: the operation fact alone (function-local mode). Label: function_local.",
"reachability_conditions": [],
"reachability_evidence": [],
"reachability_status": "unknown",
"rung": "suspicion",
"severity": "high",
"status": "enabled",
"tags": [
"obligation:protected_read",
"discharger:identity_constraint",
"obligation_evidence:function_local",
"resource:reports"
],
"title": "Obligation not discharged: protected_read on reports without identity_constraint"
},
{
"confidence": "low",
"evidence_level": "suspicion",
"finding_id": "obligation:protected_read:app.py:77:identity_constraint",
"function_name": "search",
"line": 77,
"path": "app.py",
"proposed_snippet": null,
"ranking_priority": 0.3,
"rationale": "protected_read on a resource in search is reached without a dominating identity_constraint. Why the obligation exists: the operation fact alone (function-local mode). Label: function_local.",
"reachability_conditions": [],
"reachability_evidence": [],
"reachability_status": "unknown",
"rung": "suspicion",
"severity": "high",
"status": "enabled",
"tags": [
"obligation:protected_read",
"discharger:identity_constraint",
"obligation_evidence:function_local"
],
"title": "Obligation not discharged: protected_read on a resource without identity_constraint"
}
]
}
benchmarks/measurements/2026-09-30-harnessx-removal-equality/candidate/scan/fusion.json
+192
View File
@@ -0,0 +1,192 @@
[
{
"decision_source": "deterministic-reconciler",
"degradations": [],
"disposition": "accepted",
"finding_id": "python-os-command:app.py:17",
"model_ids": [],
"panels": [
{
"confidence": 0.75,
"disposition": "accepted",
"false_positive_case": "Reachability or sanitizer evidence could still neutralize the sink.",
"leaning": "vulnerability",
"model_id": null,
"rationale": "reachability=reachable, severity=critical, verified=True",
"role": "panel_a",
"vulnerability_case": "Static corroboration exists and function-level reachability evidence is attached."
},
{
"confidence": 0.6,
"disposition": "accepted",
"false_positive_case": "Impact and exploitability still require deeper verifier or dynamic evidence.",
"leaning": "false_positive",
"model_id": null,
"rationale": "reachable=True, rejected=False, verified=True",
"role": "panel_b",
"vulnerability_case": "A tainted sink shape exists in the source."
}
],
"triggered": true,
"triggers": [
"gates_risky_fix",
"severity:critical"
],
"votes": {
"accepted": 2
},
"warnings": []
},
{
"decision_source": "deterministic-reconciler",
"degradations": [],
"disposition": "accepted",
"finding_id": "python-shell-true:app.py:23",
"model_ids": [],
"panels": [
{
"confidence": 0.75,
"disposition": "accepted",
"false_positive_case": "Reachability or sanitizer evidence could still neutralize the sink.",
"leaning": "vulnerability",
"model_id": null,
"rationale": "reachability=reachable, severity=critical, verified=True",
"role": "panel_a",
"vulnerability_case": "Static corroboration exists and function-level reachability evidence is attached."
},
{
"confidence": 0.6,
"disposition": "accepted",
"false_positive_case": "Impact and exploitability still require deeper verifier or dynamic evidence.",
"leaning": "false_positive",
"model_id": null,
"rationale": "reachable=True, rejected=False, verified=True",
"role": "panel_b",
"vulnerability_case": "A tainted sink shape exists in the source."
}
],
"triggered": true,
"triggers": [
"gates_risky_fix",
"severity:critical"
],
"votes": {
"accepted": 2
},
"warnings": []
},
{
"decision_source": "deterministic-reconciler",
"degradations": [],
"disposition": "accepted",
"finding_id": "python-sql-injection:app.py:44",
"model_ids": [],
"panels": [
{
"confidence": 0.75,
"disposition": "accepted",
"false_positive_case": "Reachability or sanitizer evidence could still neutralize the sink.",
"leaning": "vulnerability",
"model_id": null,
"rationale": "reachability=reachable, severity=high, verified=True",
"role": "panel_a",
"vulnerability_case": "Static corroboration exists and function-level reachability evidence is attached."
},
{
"confidence": 0.6,
"disposition": "accepted",
"false_positive_case": "Impact and exploitability still require deeper verifier or dynamic evidence.",
"leaning": "false_positive",
"model_id": null,
"rationale": "reachable=True, rejected=False, verified=True",
"role": "panel_b",
"vulnerability_case": "A tainted sink shape exists in the source."
}
],
"triggered": true,
"triggers": [
"gates_risky_fix",
"severity:high"
],
"votes": {
"accepted": 2
},
"warnings": []
},
{
"decision_source": "deterministic-reconciler",
"degradations": [],
"disposition": "accepted",
"finding_id": "python-sql-injection:app.py:50",
"model_ids": [],
"panels": [
{
"confidence": 0.75,
"disposition": "accepted",
"false_positive_case": "Reachability or sanitizer evidence could still neutralize the sink.",
"leaning": "vulnerability",
"model_id": null,
"rationale": "reachability=reachable, severity=high, verified=True",
"role": "panel_a",
"vulnerability_case": "Static corroboration exists and function-level reachability evidence is attached."
},
{
"confidence": 0.6,
"disposition": "accepted",
"false_positive_case": "Impact and exploitability still require deeper verifier or dynamic evidence.",
"leaning": "false_positive",
"model_id": null,
"rationale": "reachable=True, rejected=False, verified=True",
"role": "panel_b",
"vulnerability_case": "A tainted sink shape exists in the source."
}
],
"triggered": true,
"triggers": [
"gates_risky_fix",
"severity:high"
],
"votes": {
"accepted": 2
},
"warnings": []
},
{
"decision_source": "deterministic-reconciler",
"degradations": [],
"disposition": "accepted",
"finding_id": "python-unsafe-eval:app.py:28",
"model_ids": [],
"panels": [
{
"confidence": 0.75,
"disposition": "accepted",
"false_positive_case": "Reachability or sanitizer evidence could still neutralize the sink.",
"leaning": "vulnerability",
"model_id": null,
"rationale": "reachability=reachable, severity=critical, verified=True",
"role": "panel_a",
"vulnerability_case": "Static corroboration exists and function-level reachability evidence is attached."
},
{
"confidence": 0.6,
"disposition": "accepted",
"false_positive_case": "Impact and exploitability still require deeper verifier or dynamic evidence.",
"leaning": "false_positive",
"model_id": null,
"rationale": "reachable=True, rejected=False, verified=True",
"role": "panel_b",
"vulnerability_case": "A tainted sink shape exists in the source."
}
],
"triggered": true,
"triggers": [
"gates_risky_fix",
"severity:critical"
],
"votes": {
"accepted": 2
},
"warnings": []
}
]
benchmarks/measurements/2026-09-30-harnessx-removal-equality/candidate/scan/harness.json
+11
View File
@@ -0,0 +1,11 @@
{
"contract_mode": "strict",
"model_roles": {
"hunter": null,
"patcher": null,
"ranker": null,
"verifier": null
},
"processor_versions": {},
"prompt_hashes": {}
}
benchmarks/measurements/2026-09-30-harnessx-removal-equality/candidate/scan/manifest.json
+461
View File
@@ -0,0 +1,461 @@
{
"artifacts": {
"complexity_map": "complexity_map.json",
"findings": "findings.json",
"fusion": "fusion.json",
"markdown": "report.md",
"overlay": "overlay.json",
"sarif": "report.sarif",
"score": "score.json",
"trajectories": "traces/hunter_trajectories.jsonl",
"verification": "verification.json"
},
"complexity": {
"heuristic_only": true,
"hotspot_count": 14
},
"degradations": [
{
"detail": "no path records; obligations evaluated within entry-point handlers only",
"files": [
"app.py",
"safe.py"
],
"language": "python",
"reason": "obligations_function_local",
"stage": "obligations"
},
{
"detail": "no hunter client or model",
"reason": "intent_adjudication_unavailable",
"stage": "obligations"
},
{
"reason": "cpg_unavailable",
"stage": "model"
},
{
"reason": "hunter_model_unavailable",
"stage": "map"
}
],
"findings": [
{
"artifact_refs": {
"findings_json": "findings.json",
"markdown": "report.md",
"sarif": "report.sarif",
"trajectories_jsonl": "traces/hunter_trajectories.jsonl",
"verification_json": "verification.json"
},
"evidence_level": "static_corroboration",
"finding_id": "python-flask-debug:app.py:81",
"line": 81,
"path": "app.py",
"rung": "suspicion",
"severity": "medium",
"verification_status": "rejected"
},
{
"artifact_refs": {
"findings_json": "findings.json",
"markdown": "report.md",
"sarif": "report.sarif",
"trajectories_jsonl": "traces/hunter_trajectories.jsonl",
"verification_json": "verification.json"
},
"evidence_level": "static_corroboration",
"finding_id": "python-os-command:app.py:17",
"line": 17,
"path": "app.py",
"rung": "suspicion",
"severity": "critical",
"verification_status": "accepted"
},
{
"artifact_refs": {
"findings_json": "findings.json",
"markdown": "report.md",
"sarif": "report.sarif",
"trajectories_jsonl": "traces/hunter_trajectories.jsonl",
"verification_json": "verification.json"
},
"evidence_level": "static_corroboration",
"finding_id": "python-path-traversal:app.py:55",
"line": 55,
"path": "app.py",
"rung": "suspicion",
"severity": "medium",
"verification_status": "accepted"
},
{
"artifact_refs": {
"findings_json": "findings.json",
"markdown": "report.md",
"sarif": "report.sarif",
"trajectories_jsonl": "traces/hunter_trajectories.jsonl",
"verification_json": "verification.json"
},
"evidence_level": "static_corroboration",
"finding_id": "python-shell-true:app.py:23",
"line": 23,
"path": "app.py",
"rung": "suspicion",
"severity": "critical",
"verification_status": "accepted"
},
{
"artifact_refs": {
"findings_json": "findings.json",
"markdown": "report.md",
"sarif": "report.sarif",
"trajectories_jsonl": "traces/hunter_trajectories.jsonl",
"verification_json": "verification.json"
},
"evidence_level": "static_corroboration",
"finding_id": "python-sql-injection:app.py:44",
"line": 44,
"path": "app.py",
"rung": "suspicion",
"severity": "high",
"verification_status": "accepted"
},
{
"artifact_refs": {
"findings_json": "findings.json",
"markdown": "report.md",
"sarif": "report.sarif",
"trajectories_jsonl": "traces/hunter_trajectories.jsonl",
"verification_json": "verification.json"
},
"evidence_level": "static_corroboration",
"finding_id": "python-sql-injection:app.py:50",
"line": 50,
"path": "app.py",
"rung": "suspicion",
"severity": "high",
"verification_status": "accepted"
},
{
"artifact_refs": {
"findings_json": "findings.json",
"markdown": "report.md",
"sarif": "report.sarif",
"trajectories_jsonl": "traces/hunter_trajectories.jsonl",
"verification_json": "verification.json"
},
"evidence_level": "static_corroboration",
"finding_id": "python-ssrf:app.py:60",
"line": 60,
"path": "app.py",
"rung": "suspicion",
"severity": "medium",
"verification_status": "accepted"
},
{
"artifact_refs": {
"findings_json": "findings.json",
"markdown": "report.md",
"sarif": "report.sarif",
"trajectories_jsonl": "traces/hunter_trajectories.jsonl",
"verification_json": "verification.json"
},
"evidence_level": "static_corroboration",
"finding_id": "python-ssti:app.py:70",
"line": 70,
"path": "app.py",
"rung": "suspicion",
"severity": "medium",
"verification_status": "accepted"
},
{
"artifact_refs": {
"findings_json": "findings.json",
"markdown": "report.md",
"sarif": "report.sarif",
"trajectories_jsonl": "traces/hunter_trajectories.jsonl",
"verification_json": "verification.json"
},
"evidence_level": "static_corroboration",
"finding_id": "python-unsafe-deserialization:app.py:33",
"line": 33,
"path": "app.py",
"rung": "suspicion",
"severity": "medium",
"verification_status": "accepted"
},
{
"artifact_refs": {
"findings_json": "findings.json",
"markdown": "report.md",
"sarif": "report.sarif",
"trajectories_jsonl": "traces/hunter_trajectories.jsonl",
"verification_json": "verification.json"
},
"evidence_level": "static_corroboration",
"finding_id": "python-unsafe-deserialization:app.py:38",
"line": 38,
"path": "app.py",
"rung": "suspicion",
"severity": "medium",
"verification_status": "accepted"
},
{
"artifact_refs": {
"findings_json": "findings.json",
"markdown": "report.md",
"sarif": "report.sarif",
"trajectories_jsonl": "traces/hunter_trajectories.jsonl",
"verification_json": "verification.json"
},
"evidence_level": "static_corroboration",
"finding_id": "python-unsafe-eval:app.py:28",
"line": 28,
"path": "app.py",
"rung": "suspicion",
"severity": "critical",
"verification_status": "accepted"
},
{
"artifact_refs": {
"findings_json": "findings.json",
"markdown": "report.md",
"sarif": "report.sarif",
"trajectories_jsonl": "traces/hunter_trajectories.jsonl",
"verification_json": "verification.json"
},
"evidence_level": "static_corroboration",
"finding_id": "python-weak-hash:app.py:65",
"line": 65,
"path": "app.py",
"rung": "suspicion",
"severity": "medium",
"verification_status": "accepted"
},
{
"artifact_refs": {
"findings_json": "findings.json",
"markdown": "report.md",
"sarif": "report.sarif",
"trajectories_jsonl": "traces/hunter_trajectories.jsonl",
"verification_json": "verification.json"
},
"evidence_level": "static_corroboration",
"finding_id": "overlay-coverage:app.py:77:execute",
"line": 77,
"path": "app.py",
"rung": "suspicion",
"severity": "high",
"verification_status": "not_run"
},
{
"artifact_refs": {
"findings_json": "findings.json",
"markdown": "report.md",
"sarif": "report.sarif",
"trajectories_jsonl": "traces/hunter_trajectories.jsonl",
"verification_json": "verification.json"
},
"evidence_level": "static_corroboration",
"finding_id": "overlay-coverage:safe.py:11:os.system",
"line": 11,
"path": "safe.py",
"rung": "suspicion",
"severity": "critical",
"verification_status": "not_run"
},
{
"artifact_refs": {
"findings_json": "findings.json",
"markdown": "report.md",
"sarif": "report.sarif",
"trajectories_jsonl": "traces/hunter_trajectories.jsonl",
"verification_json": "verification.json"
},
"evidence_level": "suspicion",
"finding_id": "obligation:protected_read:app.py:44:identity_constraint",
"line": 44,
"path": "app.py",
"rung": "suspicion",
"severity": "high",
"verification_status": "not_run"
},
{
"artifact_refs": {
"findings_json": "findings.json",
"markdown": "report.md",
"sarif": "report.sarif",
"trajectories_jsonl": "traces/hunter_trajectories.jsonl",
"verification_json": "verification.json"
},
"evidence_level": "suspicion",
"finding_id": "obligation:protected_read:app.py:50:identity_constraint",
"line": 50,
"path": "app.py",
"rung": "suspicion",
"severity": "high",
"verification_status": "not_run"
},
{
"artifact_refs": {
"findings_json": "findings.json",
"markdown": "report.md",
"sarif": "report.sarif",
"trajectories_jsonl": "traces/hunter_trajectories.jsonl",
"verification_json": "verification.json"
},
"evidence_level": "suspicion",
"finding_id": "obligation:protected_read:app.py:77:identity_constraint",
"line": 77,
"path": "app.py",
"rung": "suspicion",
"severity": "high",
"verification_status": "not_run"
}
],
"fusion": [
{
"decision_source": "deterministic-reconciler",
"disposition": "accepted",
"finding_id": "python-os-command:app.py:17",
"triggers": [
"gates_risky_fix",
"severity:critical"
]
},
{
"decision_source": "deterministic-reconciler",
"disposition": "accepted",
"finding_id": "python-shell-true:app.py:23",
"triggers": [
"gates_risky_fix",
"severity:critical"
]
},
{
"decision_source": "deterministic-reconciler",
"disposition": "accepted",
"finding_id": "python-sql-injection:app.py:44",
"triggers": [
"gates_risky_fix",
"severity:high"
]
},
{
"decision_source": "deterministic-reconciler",
"disposition": "accepted",
"finding_id": "python-sql-injection:app.py:50",
"triggers": [
"gates_risky_fix",
"severity:high"
]
},
{
"decision_source": "deterministic-reconciler",
"disposition": "accepted",
"finding_id": "python-unsafe-eval:app.py:28",
"triggers": [
"gates_risky_fix",
"severity:critical"
]
}
],
"model": {
"by_rung": {},
"cost_usd": 0.0,
"coverage": [],
"family_coverage": [],
"model_calls": 0,
"partitions": [],
"query_seconds_by_kind": {},
"question_outcomes": [],
"regions_scanned": 0,
"regions_unasked": 0,
"regions_unjudged": 0,
"scope": null,
"seconds": 0.0,
"skipped": "cpg_unavailable",
"unjudged_sample": []
},
"obligations": {
"degradations": [
{
"detail": "no path records; obligations evaluated within entry-point handlers only",
"files": [
"app.py",
"safe.py"
],
"language": "python",
"reason": "obligations_function_local",
"stage": "obligations"
},
{
"detail": "no hunter client or model",
"reason": "intent_adjudication_unavailable",
"stage": "obligations"
}
],
"findings_by_label": {
"function_local": 3
},
"operations": 4,
"policy_version": null,
"sets": [
{
"handlers": 1,
"module": "app",
"resource": "reports"
},
{
"handlers": 1,
"module": "app",
"resource": "users"
}
],
"sibling_sets": 2,
"under_populated": 2
},
"provenance": {
"agent_commits": 0,
"sampled_commits": 0,
"signals": [
"git:unavailable"
],
"value": "human"
},
"score": {
"by_category": {
"Code Injection": 2,
"Code Quality": 1,
"Command or Argument Injection": 2,
"Cryptographic Issues": 1,
"Directory Traversal": 1,
"Encapsulation": 2,
"Information Leakage": 1,
"SQL Injection": 2
},
"gate": {
"block_severity_reachable": 5,
"blocking": false,
"min_score": 80,
"passed": false
},
"max_severity": 5,
"out_of_scope_dynamic_only": 0,
"penalty_total": 105.6,
"project_score": 17,
"unmapped_cwe": []
},
"stages": {
"completed": [
"static",
"map"
],
"requested": [
"static",
"map"
],
"skipped": []
},
"target": "<ROOT>/benchmarks/fixtures/python-vulnerable"
}
benchmarks/measurements/2026-09-30-harnessx-removal-equality/candidate/scan/stdout.txt
+6
View File
@@ -0,0 +1,6 @@
scan_id=<RUN_ID>
run_dir=<ROOT>/benchmarks/fixtures/python-vulnerable/.openultrasast/runs/<RUN_ID>
file_targets=2
ranked_targets=2
findings=17
calibrations_applied=1
benchmarks/measurements/2026-09-30-harnessx-removal-equality/candidate/scan/verification.json
+280
View File
@@ -0,0 +1,280 @@
{
"verifications": [
{
"context_sources": [
"evidence_level",
"finding_id",
"function_name",
"line",
"path",
"reachability_conditions",
"reachability_evidence",
"reachability_status",
"severity",
"tags",
"title"
],
"counter_case": "No function-level attacker-reachable entry point currently reaches this line.",
"evidence_level": "static_corroboration",
"finding_id": "python-flask-debug:app.py:81",
"pro_case": "A static pattern matched the source line.",
"required_next_step": "Add reachability evidence before prioritizing a fix.",
"status": "rejected",
"tie_breaker": "Provide call-graph, route, CLI, parser, or dynamic evidence linking attacker input to this function.",
"verified": false
},
{
"context_sources": [
"evidence_level",
"finding_id",
"function_name",
"line",
"path",
"reachability_conditions",
"reachability_evidence",
"reachability_status",
"severity",
"tags",
"title"
],
"counter_case": "Impact and exploitability still require deeper verifier or dynamic evidence.",
"evidence_level": "static_corroboration",
"finding_id": "python-os-command:app.py:17",
"pro_case": "Static corroboration exists and function-level reachability evidence is attached.",
"required_next_step": "Proceed to standard verifier or dynamic reproduction if severity warrants it.",
"status": "accepted",
"tie_breaker": "Confirm attacker control, sanitizer absence, and impact in the next verification depth.",
"verified": true
},
{
"context_sources": [
"evidence_level",
"finding_id",
"function_name",
"line",
"path",
"reachability_conditions",
"reachability_evidence",
"reachability_status",
"severity",
"tags",
"title"
],
"counter_case": "Impact and exploitability still require deeper verifier or dynamic evidence.",
"evidence_level": "static_corroboration",
"finding_id": "python-path-traversal:app.py:55",
"pro_case": "Static corroboration exists and function-level reachability evidence is attached.",
"required_next_step": "Proceed to standard verifier or dynamic reproduction if severity warrants it.",
"status": "accepted",
"tie_breaker": "Confirm attacker control, sanitizer absence, and impact in the next verification depth.",
"verified": true
},
{
"context_sources": [
"evidence_level",
"finding_id",
"function_name",
"line",
"path",
"reachability_conditions",
"reachability_evidence",
"reachability_status",
"severity",
"tags",
"title"
],
"counter_case": "Impact and exploitability still require deeper verifier or dynamic evidence.",
"evidence_level": "static_corroboration",
"finding_id": "python-shell-true:app.py:23",
"pro_case": "Static corroboration exists and function-level reachability evidence is attached.",
"required_next_step": "Proceed to standard verifier or dynamic reproduction if severity warrants it.",
"status": "accepted",
"tie_breaker": "Confirm attacker control, sanitizer absence, and impact in the next verification depth.",
"verified": true
},
{
"context_sources": [
"evidence_level",
"finding_id",
"function_name",
"line",
"path",
"reachability_conditions",
"reachability_evidence",
"reachability_status",
"severity",
"tags",
"title"
],
"counter_case": "Impact and exploitability still require deeper verifier or dynamic evidence.",
"evidence_level": "static_corroboration",
"finding_id": "python-sql-injection:app.py:44",
"pro_case": "Static corroboration exists and function-level reachability evidence is attached.",
"required_next_step": "Proceed to standard verifier or dynamic reproduction if severity warrants it.",
"status": "accepted",
"tie_breaker": "Confirm attacker control, sanitizer absence, and impact in the next verification depth.",
"verified": true
},
{
"context_sources": [
"evidence_level",
"finding_id",
"function_name",
"line",
"path",
"reachability_conditions",
"reachability_evidence",
"reachability_status",
"severity",
"tags",
"title"
],
"counter_case": "Impact and exploitability still require deeper verifier or dynamic evidence.",
"evidence_level": "static_corroboration",
"finding_id": "python-sql-injection:app.py:50",
"pro_case": "Static corroboration exists and function-level reachability evidence is attached.",
"required_next_step": "Proceed to standard verifier or dynamic reproduction if severity warrants it.",
"status": "accepted",
"tie_breaker": "Confirm attacker control, sanitizer absence, and impact in the next verification depth.",
"verified": true
},
{
"context_sources": [
"evidence_level",
"finding_id",
"function_name",
"line",
"path",
"reachability_conditions",
"reachability_evidence",
"reachability_status",
"severity",
"tags",
"title"
],
"counter_case": "Impact and exploitability still require deeper verifier or dynamic evidence.",
"evidence_level": "static_corroboration",
"finding_id": "python-ssrf:app.py:60",
"pro_case": "Static corroboration exists and function-level reachability evidence is attached.",
"required_next_step": "Proceed to standard verifier or dynamic reproduction if severity warrants it.",
"status": "accepted",
"tie_breaker": "Confirm attacker control, sanitizer absence, and impact in the next verification depth.",
"verified": true
},
{
"context_sources": [
"evidence_level",
"finding_id",
"function_name",
"line",
"path",
"reachability_conditions",
"reachability_evidence",
"reachability_status",
"severity",
"tags",
"title"
],
"counter_case": "Impact and exploitability still require deeper verifier or dynamic evidence.",
"evidence_level": "static_corroboration",
"finding_id": "python-ssti:app.py:70",
"pro_case": "Static corroboration exists and function-level reachability evidence is attached.",
"required_next_step": "Proceed to standard verifier or dynamic reproduction if severity warrants it.",
"status": "accepted",
"tie_breaker": "Confirm attacker control, sanitizer absence, and impact in the next verification depth.",
"verified": true
},
{
"context_sources": [
"evidence_level",
"finding_id",
"function_name",
"line",
"path",
"reachability_conditions",
"reachability_evidence",
"reachability_status",
"severity",
"tags",
"title"
],
"counter_case": "Impact and exploitability still require deeper verifier or dynamic evidence.",
"evidence_level": "static_corroboration",
"finding_id": "python-unsafe-deserialization:app.py:33",
"pro_case": "Static corroboration exists and function-level reachability evidence is attached.",
"required_next_step": "Proceed to standard verifier or dynamic reproduction if severity warrants it.",
"status": "accepted",
"tie_breaker": "Confirm attacker control, sanitizer absence, and impact in the next verification depth.",
"verified": true
},
{
"context_sources": [
"evidence_level",
"finding_id",
"function_name",
"line",
"path",
"reachability_conditions",
"reachability_evidence",
"reachability_status",
"severity",
"tags",
"title"
],
"counter_case": "Impact and exploitability still require deeper verifier or dynamic evidence.",
"evidence_level": "static_corroboration",
"finding_id": "python-unsafe-deserialization:app.py:38",
"pro_case": "Static corroboration exists and function-level reachability evidence is attached.",
"required_next_step": "Proceed to standard verifier or dynamic reproduction if severity warrants it.",
"status": "accepted",
"tie_breaker": "Confirm attacker control, sanitizer absence, and impact in the next verification depth.",
"verified": true
},
{
"context_sources": [
"evidence_level",
"finding_id",
"function_name",
"line",
"path",
"reachability_conditions",
"reachability_evidence",
"reachability_status",
"severity",
"tags",
"title"
],
"counter_case": "Impact and exploitability still require deeper verifier or dynamic evidence.",
"evidence_level": "static_corroboration",
"finding_id": "python-unsafe-eval:app.py:28",
"pro_case": "Static corroboration exists and function-level reachability evidence is attached.",
"required_next_step": "Proceed to standard verifier or dynamic reproduction if severity warrants it.",
"status": "accepted",
"tie_breaker": "Confirm attacker control, sanitizer absence, and impact in the next verification depth.",
"verified": true
},
{
"context_sources": [
"evidence_level",
"finding_id",
"function_name",
"line",
"path",
"reachability_conditions",
"reachability_evidence",
"reachability_status",
"severity",
"tags",
"title"
],
"counter_case": "Impact and exploitability still require deeper verifier or dynamic evidence.",
"evidence_level": "static_corroboration",
"finding_id": "python-weak-hash:app.py:65",
"pro_case": "Static corroboration exists and function-level reachability evidence is attached.",
"required_next_step": "Proceed to standard verifier or dynamic reproduction if severity warrants it.",
"status": "accepted",
"tie_breaker": "Confirm attacker control, sanitizer absence, and impact in the next verification depth.",
"verified": true
}
]
}
benchmarks/measurements/2026-09-30-harnessx-removal-equality/candidate/suite.json
+1780
View File
File diff suppressed because it is too large Load Diff
benchmarks/measurements/2026-09-30-harnessx-removal-equality/comparison.json
+15
View File
@@ -0,0 +1,15 @@
{
"baseline": "benchmarks/measurements/2026-09-30-harnessx-removal-baseline",
"candidate": "benchmarks/measurements/2026-09-30-harnessx-removal-equality/candidate",
"differences": [],
"equal": true,
"exclusions": [
"JSON keys: created_at, finished, generated_at, runtime_seconds, scan_id, started, timestamp",
"JSON keys ending in: _seconds, _ms",
"run directory names / run ids (<timestamp>-<8 hex>, benchmark.py create_benchmark_run, run.py create_scan_run) -> <RUN_ID>",
"trace/events.jsonl as a whole (timings); never copied",
"the export's absolute root path -> <ROOT>; the children's TMPDIR -> <TMP>",
"suite: only the outcome per test id is kept (junit durations dropped)",
"env.json is a provenance record (commit, freeze hash, instrument checks), not compared"
]
}
benchmarks/measurements/2026-09-30-harnessx-removal-equality/suite-map.json
+156
View File
@@ -0,0 +1,156 @@
{
"added": [
"tests.test_config::test_retired_agentic_section_loads_with_one_warning",
"tests.test_config::test_retired_llm_keys_fail_naming_the_replacement[[fusion]\\ndecider_model = \"gpt-4o\"\\n-`[fusion] decider_model`-`agree` task]",
"tests.test_config::test_retired_llm_keys_fail_naming_the_replacement[[fusion]\\npanel_model = \"gpt-4o\"\\n-`[fusion] panel_model`-`agree` task]",
"tests.test_config::test_retired_llm_keys_fail_naming_the_replacement[[models]\\nverifier = \"gpt-4o\"\\n-`[models] verifier`-`verify` + `agree`]",
"tests.test_config::test_the_cli_exits_2_on_a_retired_key",
"tests.test_gate::test_governance_planes_import_only_stdlib_and_declared_dependencies",
"tests.test_improve_memory::test_a_reverted_memory_proposal_is_not_retried_on_the_same_evidence",
"tests.test_improve_memory::test_default_improve_output_matches_the_recorded_baseline",
"tests.test_improve_memory::test_default_round_journal_has_no_evidence_and_no_sidecar",
"tests.test_improve_memory::test_disputed_families_are_advisory_and_never_reach_the_validator",
"tests.test_improve_memory::test_guard_drops_holdout_pair_repositories_and_gated_cases",
"tests.test_improve_memory::test_guard_drops_the_qualification_population_and_reports_it",
"tests.test_improve_memory::test_improve_memory_with_an_empty_store_has_no_proposals",
"tests.test_improve_memory::test_m1_counts_fixed_pin_alerts_in_the_fix_range_and_ignores_disputes",
"tests.test_improve_memory::test_m1_demotes_a_rule_with_false_alerts_across_repositories",
"tests.test_improve_memory::test_m1_never_demotes_a_rule_with_an_agreed_hit",
"tests.test_improve_memory::test_m1_skips_rules_the_ruleset_does_not_know_or_that_are_not_enabled",
"tests.test_improve_memory::test_m1_stays_silent_below_threshold[one-repository]",
"tests.test_improve_memory::test_m1_stays_silent_below_threshold[two-alerts]",
"tests.test_improve_memory::test_m2_promotes_a_shadow_rule_that_hits_declared_sites_nothing_else_catches",
"tests.test_improve_memory::test_m2_stays_silent_below_threshold_or_when_already_caught",
"tests.test_improve_memory::test_m2_stays_silent_on_a_fixed_pin_alert_or_rejected_candidates",
"tests.test_improve_memory::test_provenance_round_trips_journal_sidecar_store",
"tests.test_plane_loop::test_alerts_attribute_functions_and_mark_the_fix_on_both_pins",
"tests.test_plane_loop::test_every_case_ends_in_remember_and_the_loop_is_model_free",
"tests.test_plane_loop::test_fix_ranges_new_side_are_the_fixed_pins_lines",
"tests.test_plane_loop::test_the_loop_runs_as_one_plane_run_and_the_gate_rejects_a_demotion_that_costs_recall",
"tests.test_plane_memory::test_a_failing_select_on_one_object_falls_back_for_that_object",
"tests.test_plane_memory::test_a_malformed_row_fails_naming_it_and_writes_nothing[fake-minio-fallback]",
"tests.test_plane_memory::test_a_malformed_row_fails_naming_it_and_writes_nothing[fake-minio-select]",
"tests.test_plane_memory::test_a_malformed_row_fails_naming_it_and_writes_nothing[file]",
"tests.test_plane_memory::test_a_malformed_row_fails_naming_it_and_writes_nothing[minio]",
"tests.test_plane_memory::test_a_repeated_id_replaces_the_stored_row[fake-minio-fallback]",
"tests.test_plane_memory::test_a_repeated_id_replaces_the_stored_row[fake-minio-select]",
"tests.test_plane_memory::test_a_repeated_id_replaces_the_stored_row[file]",
"tests.test_plane_memory::test_a_repeated_id_replaces_the_stored_row[minio]",
"tests.test_plane_memory::test_configure_turns_versioning_on_and_expires_runs_only",
"tests.test_plane_memory::test_facts_that_no_longer_match_their_hash_are_dropped[fake-minio-fallback]",
"tests.test_plane_memory::test_facts_that_no_longer_match_their_hash_are_dropped[fake-minio-select]",
"tests.test_plane_memory::test_facts_that_no_longer_match_their_hash_are_dropped[file]",
"tests.test_plane_memory::test_facts_that_no_longer_match_their_hash_are_dropped[minio]",
"tests.test_plane_memory::test_file_store_refuses_below_one_gib_free_naming_its_path",
"tests.test_plane_memory::test_identical_facts_are_stored_once_and_read_back_by_content[fake-minio-fallback]",
"tests.test_plane_memory::test_identical_facts_are_stored_once_and_read_back_by_content[fake-minio-select]",
"tests.test_plane_memory::test_identical_facts_are_stored_once_and_read_back_by_content[file]",
"tests.test_plane_memory::test_identical_facts_are_stored_once_and_read_back_by_content[minio]",
"tests.test_plane_memory::test_ingest_is_idempotent_by_the_index[fake-minio-fallback]",
"tests.test_plane_memory::test_ingest_is_idempotent_by_the_index[fake-minio-select]",
"tests.test_plane_memory::test_ingest_is_idempotent_by_the_index[file]",
"tests.test_plane_memory::test_ingest_is_idempotent_by_the_index[minio]",
"tests.test_plane_memory::test_minio_settings_name_missing_keys_never_values",
"tests.test_plane_memory::test_minio_without_the_sdk_names_the_extra",
"tests.test_plane_memory::test_open_store_selects_the_backend",
"tests.test_plane_memory::test_presigned_urls_name_the_object[fake-minio-fallback]",
"tests.test_plane_memory::test_presigned_urls_name_the_object[fake-minio-select]",
"tests.test_plane_memory::test_presigned_urls_name_the_object[file]",
"tests.test_plane_memory::test_presigned_urls_name_the_object[minio]",
"tests.test_plane_memory::test_row_objects_carry_tags_and_a_kind_tag_skips_objects_without_that_kind",
"tests.test_plane_memory::test_rows_filter_by_repo_pin_kind_and_where_with_their_object_version[fake-minio-fallback]",
"tests.test_plane_memory::test_rows_filter_by_repo_pin_kind_and_where_with_their_object_version[fake-minio-select]",
"tests.test_plane_memory::test_rows_filter_by_repo_pin_kind_and_where_with_their_object_version[file]",
"tests.test_plane_memory::test_rows_filter_by_repo_pin_kind_and_where_with_their_object_version[minio]",
"tests.test_plane_memory::test_seed_never_overwrites_a_task_directory_and_drops_tampered_facts",
"tests.test_plane_memory::test_seed_recomputes_on_a_changed_pin_candidate_set_or_image[change0]",
"tests.test_plane_memory::test_seed_recomputes_on_a_changed_pin_candidate_set_or_image[change1]",
"tests.test_plane_memory::test_seed_recomputes_on_a_changed_pin_candidate_set_or_image[change2]",
"tests.test_plane_memory::test_seed_reuses_facts_for_an_unchanged_key_and_marks_the_task_done",
"tests.test_plane_memory::test_select_is_pushed_down_when_the_probe_answers",
"tests.test_plane_memory::test_where_sql_quotes_values_and_refuses_odd_fields",
"tests.test_plane_memory::test_without_select_the_same_rows_come_from_a_local_filter_logged_once[raises]",
"tests.test_plane_memory::test_without_select_the_same_rows_come_from_a_local_filter_logged_once[wrong]",
"tests.test_plane_reconciler::test_mark_done_sets_the_state_under_the_run_lock",
"tests.test_plane_remember::test_a_malformed_delivered_row_fails_naming_file_and_line",
"tests.test_plane_remember::test_ingest_without_remember_outputs_is_a_no_op",
"tests.test_plane_remember::test_plane_remember_cli_reads_population_from_the_run_and_reports_counts",
"tests.test_plane_remember::test_plane_run_seeds_before_and_ingests_after",
"tests.test_plane_remember::test_re_ingest_of_the_same_run_is_a_no_op",
"tests.test_plane_remember::test_remember_turns_the_recorded_case_into_rows",
"tests.test_plane_remember::test_rows_carry_every_final_the_tiebreak_and_alerts",
"tests.test_plane_remember::test_the_task_entrypoint_writes_memory_and_passes_facts_through",
"tests.test_removed_plane_references::test_no_harnessx_outside_retirement_notes",
"tests.test_removed_plane_references::test_the_readme_names_the_removed_plane_only_in_its_retirement_note",
"tests.test_removed_plane_references::test_the_retirement_notes_exist"
],
"added_from": "tasks 2-6 of harnessx-removal: test_plane_memory, test_plane_remember, test_plane_loop, test_improve_memory, test_plane_reconciler (mark_done), test_removed_plane_references, and the replacements named above",
"deleted": [
"tests.test_cli_hx_dispatch::test_degradation_recorded_when_models_set_but_extra_absent",
"tests.test_cli_hx_dispatch::test_manifest_has_no_harnessx_degradations_without_models",
"tests.test_cli_hx_dispatch::test_standard_scan_falls_back_to_run_hunter_pool",
"tests.test_config::test_harnessx_defaults_when_section_absent",
"tests.test_config::test_load_config_reads_harnessx_section",
"tests.test_fusion::test_dispatch_records_degradation_when_llm_requested_but_extra_absent",
"tests.test_gate::test_governance_scoring_benchmark_planes_never_import_harnessx",
"tests.test_gate::test_harnessx_stage_stays_within_tolerance_of_baseline",
"tests.test_gate::test_planes_have_no_module_level_harnessx_import",
"tests.test_harness_ext::test_build_provider_constructs_selected_provider[anthropic-AnthropicProvider]",
"tests.test_harness_ext::test_build_provider_constructs_selected_provider[openai-OpenAIProvider]",
"tests.test_harness_ext::test_build_provider_rejects_unknown_provider_before_importing",
"tests.test_harness_ext::test_core_dependencies_are_pyyaml_only",
"tests.test_harness_ext::test_harnessx_is_a_sha_pinned_optional_extra",
"tests.test_harness_ext::test_has_harnessx_probes_without_importing",
"tests.test_harness_ext::test_importing_openultrasast_does_not_import_harnessx",
"tests.test_harness_ext::test_require_harnessx_raises_with_install_guidance_when_absent",
"tests.test_harness_ext::test_supported_providers_cover_anthropic_and_openai",
"tests.test_hunter_harness::test_build_sast_raises_without_extra",
"tests.test_hunter_harness::test_build_sast_returns_harness_config_offline",
"tests.test_hunter_harness::test_extract_findings_parses_json_array_and_tolerates_garbage",
"tests.test_hunter_harness::test_module_import_does_not_pull_harnessx",
"tests.test_hunter_harness::test_orchestrator_builds_agent_offline",
"tests.test_hunter_harness::test_orchestrator_construction_raises_without_extra",
"tests.test_slot_contract::test_host_under_harnessx_disables_the_model",
"tests.test_verify_judge::test_dispatch_does_not_import_harnessx_on_fallback",
"tests.test_verify_judge::test_dispatch_falls_back_to_structural_by_default",
"tests.test_verify_judge::test_dispatch_stays_structural_when_model_unset_or_extra_absent",
"tests.test_verify_judge::test_verdict_mapping_is_evidence_gated"
],
"justification": {
"tests.test_cli_hx_dispatch::test_degradation_recorded_when_models_set_but_extra_absent": "file deleted: exercised only the HarnessX dispatch (design Tests table; commit cbadc38)",
"tests.test_cli_hx_dispatch::test_manifest_has_no_harnessx_degradations_without_models": "file deleted: exercised only the HarnessX dispatch (design Tests table; commit cbadc38)",
"tests.test_cli_hx_dispatch::test_standard_scan_falls_back_to_run_hunter_pool": "file deleted: exercised only the HarnessX dispatch (design Tests table; commit cbadc38)",
"tests.test_config::test_harnessx_defaults_when_section_absent": "replaced by test_retired_agentic_section_loads_with_one_warning and test_retired_llm_keys_fail_naming_the_replacement (design Tests table; commit cbadc38)",
"tests.test_config::test_load_config_reads_harnessx_section": "replaced by test_retired_agentic_section_loads_with_one_warning and test_retired_llm_keys_fail_naming_the_replacement (design Tests table; commit cbadc38)",
"tests.test_fusion::test_dispatch_records_degradation_when_llm_requested_but_extra_absent": "deleted: asserted the extra-absent degradation of the retired LLM panels (design Tests table; commit cbadc38)",
"tests.test_gate::test_governance_scoring_benchmark_planes_never_import_harnessx": "the live-extra tolerance test is deleted and the two never-import tests are replaced by test_governance_planes_import_only_stdlib_and_declared_dependencies (design Tests table; commit b9afb66)",
"tests.test_gate::test_harnessx_stage_stays_within_tolerance_of_baseline": "the live-extra tolerance test is deleted and the two never-import tests are replaced by test_governance_planes_import_only_stdlib_and_declared_dependencies (design Tests table; commit b9afb66)",
"tests.test_gate::test_planes_have_no_module_level_harnessx_import": "the live-extra tolerance test is deleted and the two never-import tests are replaced by test_governance_planes_import_only_stdlib_and_declared_dependencies (design Tests table; commit b9afb66)",
"tests.test_harness_ext::test_build_provider_constructs_selected_provider[anthropic-AnthropicProvider]": "file deleted: exercised only HarnessX (design Tests table; commit b9afb66)",
"tests.test_harness_ext::test_build_provider_constructs_selected_provider[openai-OpenAIProvider]": "file deleted: exercised only HarnessX (design Tests table; commit b9afb66)",
"tests.test_harness_ext::test_build_provider_rejects_unknown_provider_before_importing": "file deleted: exercised only HarnessX (design Tests table; commit b9afb66)",
"tests.test_harness_ext::test_core_dependencies_are_pyyaml_only": "file deleted: exercised only HarnessX (design Tests table; commit b9afb66)",
"tests.test_harness_ext::test_harnessx_is_a_sha_pinned_optional_extra": "file deleted: exercised only HarnessX (design Tests table; commit b9afb66)",
"tests.test_harness_ext::test_has_harnessx_probes_without_importing": "file deleted: exercised only HarnessX (design Tests table; commit b9afb66)",
"tests.test_harness_ext::test_importing_openultrasast_does_not_import_harnessx": "file deleted: exercised only HarnessX (design Tests table; commit b9afb66)",
"tests.test_harness_ext::test_require_harnessx_raises_with_install_guidance_when_absent": "file deleted: exercised only HarnessX (design Tests table; commit b9afb66)",
"tests.test_harness_ext::test_supported_providers_cover_anthropic_and_openai": "file deleted: exercised only HarnessX (design Tests table; commit b9afb66)",
"tests.test_hunter_harness::test_build_sast_raises_without_extra": "file deleted: exercised only HarnessX (design Tests table; commit b9afb66)",
"tests.test_hunter_harness::test_build_sast_returns_harness_config_offline": "file deleted: exercised only HarnessX (design Tests table; commit b9afb66)",
"tests.test_hunter_harness::test_extract_findings_parses_json_array_and_tolerates_garbage": "file deleted: exercised only HarnessX (design Tests table; commit b9afb66)",
"tests.test_hunter_harness::test_module_import_does_not_pull_harnessx": "file deleted: exercised only HarnessX (design Tests table; commit b9afb66)",
"tests.test_hunter_harness::test_orchestrator_builds_agent_offline": "file deleted: exercised only HarnessX (design Tests table; commit b9afb66)",
"tests.test_hunter_harness::test_orchestrator_construction_raises_without_extra": "file deleted: exercised only HarnessX (design Tests table; commit b9afb66)",
"tests.test_scan_pipeline_integrity::test_quick_scan_is_unaffected_by_the_extra": "renamed to the behaviour it asserts; has_harnessx monkeypatch dropped (design Tests table; commit cbadc38)",
"tests.test_scan_pipeline_integrity::test_standard_scan_extra_absent_matches_deterministic_baseline": "renamed to the behaviour it asserts; has_harnessx monkeypatch dropped (design Tests table; commit cbadc38)",
"tests.test_slot_contract::test_host_under_harnessx_disables_the_model": "deleted with host_under_harnessx (design Tests table; commit cbadc38)",
"tests.test_verify_judge::test_dispatch_does_not_import_harnessx_on_fallback": "file deleted with verify_judge.py (design Tests table; commit cbadc38)",
"tests.test_verify_judge::test_dispatch_falls_back_to_structural_by_default": "file deleted with verify_judge.py (design Tests table; commit cbadc38)",
"tests.test_verify_judge::test_dispatch_stays_structural_when_model_unset_or_extra_absent": "file deleted with verify_judge.py (design Tests table; commit cbadc38)",
"tests.test_verify_judge::test_verdict_mapping_is_evidence_gated": "file deleted with verify_judge.py (design Tests table; commit cbadc38)"
},
"renamed": {
"tests.test_scan_pipeline_integrity::test_quick_scan_is_unaffected_by_the_extra": "tests.test_scan_pipeline_integrity::test_quick_scan_records_no_degradation",
"tests.test_scan_pipeline_integrity::test_standard_scan_extra_absent_matches_deterministic_baseline": "tests.test_scan_pipeline_integrity::test_standard_scan_with_a_hunter_model_keeps_deterministic_findings"
}
}