marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-11-13 17:58:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(access-control-prover): 13.3 four-arm development regression

Browse Source
This commit is contained in:
Marius Ciepluch
2026-10-30 12:37:19 +01:00
parent 9c95f842f7
commit d063167b28
6 changed files with 1214 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
benchmarks/measurements/2026-10-30-access-control-four-arm/NOTES.md
+125
View File
@@ -0,0 +1,125 @@
# Access-control four-arm development regression (task 13.3)
Date: 2026-10-30. Spec: `.kiro/specs/access-control-prover/` task 13.3 (Reqs 5.1-5.7, 6.1, 7.3, 11.8, 12.9).
Result role: **development regression evidence only** -- population used to choose facts and analyzer
behavior; not an admission claim (Req 5.7, 12.9).
## Setup
- Arms: four registered per Req 5.5 (old/new analyzer crossed with old/new fact set).
- Baseline bundle: analyzer revision `c5341d79a21bfae49ee157bdac19f967754f0be4` (the 2026-10-28 gate-1
commit), fact artifact digest `3eb5bf699434`, materialised under
`~/ousast-results/prover/ac-four-arm-1/bundles/baseline/`.
- Candidate bundle: analyzer revision `9c95f842f7263710ec7d106794b37eb94a45425d` (HEAD at run time),
same fact artifact digest `3eb5bf699434`.
- Fact identity, proven before any arm ran: the 18 ruleset TOMLs under `src/openultrasast/ruleset` are
byte-identical between the baseline revision and HEAD (manifest digest `e7db4031d3d5`, disk == HEAD ==
baseline ref). The amendment cycle is therefore analyzer-only, and the cross arms are input-identical to
their parents by digest proof:
- `baseline-analyzer/candidate-facts` == `baseline-analyzer/baseline-facts` (identical inputs),
- `candidate-analyzer/baseline-facts` == `candidate-analyzer/candidate-facts` (identical inputs).
Determinism is spot-verified by re-proving sampled changes and diffing judgments (below).
- Population: `benchmarks/valid/train/pool-train.toml`, freeze-verified through `load_slice` (the same
frozen 23-repo pool as the 2026-10-28 run; 483 changes = 23 fix pushes + 460 ordinary pushes).
- Clones: `~/ousast-results/valid/train/hist/<owner__name>`, repos mapping
`~/ousast-results/valid/train/repos_hist_pool23.json`.
- Nonzero input reads: the runner records a byte table per change (`instrument.file_bytes`,
`changed_bytes`, `hook_bytes_read`), refuses zero-byte reads of tracked files, and allows a file git
holds empty only as a counted `empty_files` entry -- the same instrument contract as the 2026-10-28 run.
Every adapted baseline record kept `hook_bytes_read > 0`; the summary refuses on any zero-byte entry.
## Execution
- `baseline-analyzer/baseline-facts`: the immutable 2026-10-28 records themselves
(`~/ousast-results/prover/ac-train-1/`, 483 records, 0 instrument failures at run time), adapted by
`benchmarks.valid.arm_regression.adapt_baseline_record` with an explicit `vocabulary: binary` marker:
unguarded := findings count, discharged := recorded discharge count, indeterminate := 0 (the baseline
analyzer could not express it). Research decision: the 2026-10-28 numbers remain the immutable baseline;
a rerun on the same repositories is regression evidence.
- `candidate-analyzer/candidate-facts`: executed 2026-10-30 from the working tree (clean, HEAD), strict
judgment lane (`judgment_mode=True`, three-way vocabulary), two resumable shards over the sorted change
ids (round-robin), records under `~/ousast-results/prover/ac-four-arm-1/cacfon/`.
- Cross arms: registered aliases by proven input identity (digest proof above) plus a determinism
spot-check that re-proves sampled changes (all 23 fix pushes were covered by the executed arm; the
sample re-proves the judgment lane reproduces identical judgments for identical inputs).
- Summary assembly: `arm_regression.summarize_arm` (masked repository identities, Req 5.4),
`arm_regression.four_arm_development` over `development_runner.four_arm_report` (Req 11.8), metric
headline through `benchmarks.valid.metrics` with the same scorer contract as the baseline summary
(Wilson intervals; `push.admission.defects` derived from strict-lane findings), budget result per
Req 6.1 published as development evidence (the summary itself refuses only on instrument failure).
## Corrections during the run (instrument honesty)
1. First probe run reported all-zero coverage in 0.0 s: the probe had skipped the sparse-checkout
disable step, so worktrees contained no files. Fixed by replicating the harness; the probe then read
4,096-6,170 files per repo and found real entry points.
2. Two shard restarts: the first record shape missed the scorer contract (`instrument.hook_exit`,
`push.admission.defects`); 35 stale-shape records were deleted rather than patched -- back-filling
instrument fields into written records would fabricate proof.
3. `tests/test_admission_scoring.py` imported `benchmarks` without the `sys.path.insert` its sibling
test files have; it had only ever passed when an earlier-collected file fixed the path first. Fixed
with the sibling pattern.
4. Review round 1 rejected the first summary for a false baseline provenance stamp
(`bundle_from` stamps the working-tree HEAD, so the baseline bundle claimed revision `9c95f842` instead
of `c5341d7`), for a missing covered/uncovered recall split, and for shard-scoped population digests.
Remediated by rebuilding both bundle manifests with explicit revisions and honest lane configurations,
re-adapting the baseline records from the corrected manifest, publishing the authoritative
full-population digest (`999594274a4e`, asserted in assembly) with a per-record stamp disclosure, and
adding the recall split. No arm was re-executed; no written record was back-filled.
## Result
All numbers from `summary.json` (this directory); raw records under `~/ousast-results/prover/ac-four-arm-1/`.
**Development regression evidence only.**
| metric | immutable 2026-10-28 baseline | candidate arm 2026-10-30 |
|---|---|---|
| recall (fix pushes) | 15/23 = 65.2% (Wilson 44.9-81.2) | 8/23 = 34.8% (Wilson 18.8-55.1) |
| covered-repo recall | 15/21 | 8/21 = 38.1% (Wilson 20.8-59.1) |
| uncovered-repo recall | 0/2 | 0/2 (separately reported, Req 7.3) |
| location recall | 55/93 = 59.1% (Wilson 49.0-68.6) | 25/93 = 26.9% (Wilson 18.9-36.7) |
| false alarms / 100 pushes | 69 on 460 = 15.0 (Wilson 12.0-18.6) | 24 on 460 = 5.2 (Wilson 3.5-7.6) |
| precision | 15/84 = 17.9% | 8/32 = 25.0% |
| instrument failures | 0 | 0 |
| per-change seconds | mean 27.9, max 1063 | mean 17.4, median 1.1, max 240 |
| budget gate (120 s) | exceeded (development evidence) | failed: 20 changes over, max 240 s |
| uncovered repositories | 2 | 2 (masked: `1ca62ca671af`, `c92157c966cc`) |
Four-arm attribution (Req 11.8): fact-only lift **0 on every judgment kind** -- the fact inputs are
byte-identical (digest `e7db4031d3d5`), so the entire movement is analyzer-attributed. Pair-level
totals: unguarded 340 -> 94 (-246), discharged 130 -> 0 (-130), indeterminate 0 -> 102 (+102).
Per-change transitions (baseline binary flag -> strict three-way): flagged->unguarded 29,
flagged->indeterminate 25, flagged->none 30 (84 baseline flags reconciled exactly), quiet->unguarded 3,
quiet->indeterminate 8, quiet->none 388. `discharged` is 0 across the whole strict run: on this
population the strict lane never proves a complete-path discharge -- every judged pair is unguarded or
indeterminate.
Reading: the amended analyzer buys judgment honesty at a large recall cost on this development
population. It retracts 55 of 84 binary alarms (25 to indeterminate -- incomplete evidence, visible;
30 to no judged pairs at all), finds 3 new ones, and cuts false alarms roughly threefold
(15.0 -> 5.2 per 100). The stage funnel names the bottleneck: of 4,943,304 recorded call resolutions,
only 589,520 (11.9%) are complete; 1,451,325 are incomplete and 2,908,559 unresolved. Exact, finite
call resolution (task 8.3) starves pair formation upstream of judgment; the strict dominance and
witness rules then never reach a definite discharge. This mirrors the pre-registered risk
("indeterminate inflation ... report the stage funnel and reason distribution; improve the dominant
reason rather than weakening the lattice", research.md) and the 2026-10-28 ablation's levers
(call resolution first). Development-only conclusion for the next amendment cycle: grow entry facts
and call-resolution coverage; the three-way lane itself behaves exactly as specified -- no pair is
ever silently clean, and every refusal is visible in the funnel.
Runtime: the executed arm ran as two resumable shards, 64.2 + 73.9 minutes wall (4 cores, 2 workers),
483/483 records, every record with nonzero-read proof (`hook_bytes_read` > 0, no zero-byte entries,
`hook_exit` 0). Determinism spot-check: 6 sampled changes (3 fix pushes, 3 from the largest
repositories) re-proved to identical judgments, findings and status.
## Reproduction
- Bundles: `/tmp/opencode/bundles_133.py` (fact manifests, identity proof, materialisation through
`benchmarks.valid.run_manifests`), corrected by `/tmp/opencode/fix_bundles_133.py` (explicit revisions).
- Executed arm: `/tmp/opencode/run_shard_133.py <i> <n>` (shardable, resumable; module:
`benchmarks.valid.arm_regression.run_arm`, strict judgment lane).
- Baseline adaptation: `/tmp/opencode/adapt_bab_133.py` (`adapt_baseline_record`, binary vocabulary).
- Determinism: `/tmp/opencode/determinism_133.py`.
- Summary: `/tmp/opencode/assemble_133.py` (`summarize_arm`, `four_arm_development`, scorer-compatible
metrics via `benchmarks.valid.metrics` and `benchmarks.unseen.score`).
benchmarks/measurements/2026-10-30-access-control-four-arm/summary.json
+489
View File
@@ -0,0 +1,489 @@
{
"arm_execution": {
"baseline-analyzer/baseline-facts": "immutable 2026-10-28 records, adapted (vocabulary: binary)",
"baseline-analyzer/candidate-facts": "alias of baseline arm by proven input identity",
"candidate-analyzer/baseline-facts": "alias of candidate arm by proven input identity",
"candidate-analyzer/candidate-facts": "executed 2026-10-30 (vocabulary: three-way)"
},
"bundles": {
"baseline": {
"acquisition_artifact": null,
"acquisition_record_digest": "none",
"analyzer_revision": "c5341d79a21bfae49ee157bdac19f967754f0be4",
"configuration_digest": "bc4d444e2bc4",
"fact_artifact": "facts-baseline.json",
"fact_set_digest": "3eb5bf699434"
},
"candidate": {
"acquisition_artifact": null,
"acquisition_record_digest": "none",
"analyzer_revision": "9c95f842f7263710ec7d106794b37eb94a45425d",
"configuration_digest": "be6d0f688598",
"fact_artifact": "facts-candidate.json",
"fact_set_digest": "3eb5bf699434"
}
},
"candidate_arm": {
"budget": {
"budget_seconds": 120.0,
"gate": "failed",
"overruns": [
{
"id": "099364c093061a7d02d2e728a503b753441bd455ae6ae5f6970326d1815ee6f4",
"seconds": 160.65696239499812
},
{
"id": "1d77199cf964d7f168f3f0c157646f52aef7e3931b8da2f9fa829e1e32931964",
"seconds": 237.99327803600318
},
{
"id": "3a66c58b6c2b345f622f00820bd833932c5fa02ccc0d03626ac95f9b179baf55",
"seconds": 160.20379009899625
},
{
"id": "514850869e2c19bfc55dbc75b2c1e1766249e12f04385b1de7b4d116b6b209fa",
"seconds": 239.97186829699785
},
{
"id": "58a2ffe30e0bc6a166a01fe016ebf43b0c0f3385b0f4a2e8578d3e039371c17e",
"seconds": 135.12179498800106
},
{
"id": "59327d70f193338a3ae990177f6fa8aece8e6913e6e5d3a27888d383edb56165",
"seconds": 195.88553729499836
},
{
"id": "5c73c7dac092be000911ad6d4327156dbdc7a02d3d22545cfe47a25b67ce76ec",
"seconds": 201.59726526400482
},
{
"id": "5c9a822532e45241cb40e12ba6437837bf0301a6b2bead1261fbaa90e028f211",
"seconds": 238.12383715200122
},
{
"id": "720017c40d9dd8ad541f00c2577b2d1676ba3dfcaa997166a1065dbb36b42231",
"seconds": 136.7938399220002
},
{
"id": "75f92ee98982d750540239481ee8aa4258ec3cc3cb64cce6ca19835d9fcf2d99",
"seconds": 195.32054610600608
},
{
"id": "a505c341e4c5e363b3050d1523bd8f6868c10e3cd5ed383d3c8e155b49bf9a70",
"seconds": 168.9383434449992
},
{
"id": "bada9b8aaceaff26178d1940f7f6032b24d213071c54c30d2e7ce929e557df90",
"seconds": 170.55653589800204
},
{
"id": "c0fca5070cee86d26832638230b364f45d245c6e1a77333d38ee393832291e13",
"seconds": 238.88118421599938
},
{
"id": "c1859d8c2376e25dd46024b336bb8106a702f8139437934cf48d282cf0f396cb",
"seconds": 203.85092047799844
},
{
"id": "c2c52c1bb6b1f0d70b29b7b64009d6ca85c6e4d3659a195a904d577f6b02c886",
"seconds": 193.02569297700393
},
{
"id": "da89557ec3cada71aed38482d8269959d56e9a37877f66d69cf5b3905d0cbcff",
"seconds": 238.55912208399968
},
{
"id": "e0b9c856901ff36c6e54a0a73752b06f6a9996930702c18593c25936709d4c79",
"seconds": 162.15434529800405
},
{
"id": "e934b0222e338b768bfc594668434850f18e3a7ee35a76fb35c19898d4924349",
"seconds": 162.4909854649959
},
{
"id": "ea9ed8aad6259c9ea8ee1f8227a981710e8b4b1da838e74d6e68075455e78631",
"seconds": 163.5947257539956
},
{
"id": "f664ff98347b54ace35a50a499c77cfe3c6e06158e78f76f20b9d9317a29bb94",
"seconds": 160.8364557419991
}
]
},
"development_only": true,
"instrument_failures": 0,
"judgment_counts": {
"discharged": 0,
"indeterminate": 102,
"unguarded": 94
},
"per_repository": {
"03d0eff98aa6": {
"changes": 21,
"discharged": 0,
"frameworks_recognised": 44,
"status": "covered",
"unguarded": 2
},
"0bdd00190b32": {
"changes": 21,
"discharged": 0,
"frameworks_recognised": 6179,
"status": "covered",
"unguarded": 3
},
"1ca62ca671af": {
"changes": 21,
"discharged": 0,
"frameworks_recognised": 0,
"status": "uncovered",
"unguarded": 0
},
"34b17c3454d8": {
"changes": 21,
"discharged": 0,
"frameworks_recognised": 828,
"status": "covered",
"unguarded": 13
},
"3caf85f05a65": {
"changes": 21,
"discharged": 0,
"frameworks_recognised": 559,
"status": "covered",
"unguarded": 23
},
"4b7075e3d5a1": {
"changes": 21,
"discharged": 0,
"frameworks_recognised": 93,
"status": "covered",
"unguarded": 11
},
"4dde9ea84248": {
"changes": 21,
"discharged": 0,
"frameworks_recognised": 100,
"status": "covered",
"unguarded": 3
},
"4e75cb1165d3": {
"changes": 21,
"discharged": 0,
"frameworks_recognised": 65,
"status": "covered",
"unguarded": 3
},
"69f3c5cdd90a": {
"changes": 21,
"discharged": 0,
"frameworks_recognised": 115,
"status": "covered",
"unguarded": 7
},
"6a4eac96c6ae": {
"changes": 21,
"discharged": 0,
"frameworks_recognised": 108,
"status": "covered",
"unguarded": 4
},
"6c1d234f9e23": {
"changes": 21,
"discharged": 0,
"frameworks_recognised": 21,
"status": "covered",
"unguarded": 0
},
"6c92cca94d4b": {
"changes": 21,
"discharged": 0,
"frameworks_recognised": 470,
"status": "covered",
"unguarded": 2
},
"760591cb57ba": {
"changes": 21,
"discharged": 0,
"frameworks_recognised": 87,
"status": "covered",
"unguarded": 5
},
"78d80100d68f": {
"changes": 21,
"discharged": 0,
"frameworks_recognised": 98,
"status": "covered",
"unguarded": 1
},
"89c10adb0673": {
"changes": 21,
"discharged": 0,
"frameworks_recognised": 91,
"status": "covered",
"unguarded": 0
},
"9047b2efc25b": {
"changes": 21,
"discharged": 0,
"frameworks_recognised": 86,
"status": "covered",
"unguarded": 0
},
"ba37e9ee7b7b": {
"changes": 21,
"discharged": 0,
"frameworks_recognised": 163,
"status": "covered",
"unguarded": 1
},
"c5bd586fae3f": {
"changes": 21,
"discharged": 0,
"frameworks_recognised": 33,
"status": "covered",
"unguarded": 6
},
"c690d767b023": {
"changes": 21,
"discharged": 0,
"frameworks_recognised": 29,
"status": "covered",
"unguarded": 5
},
"c92157c966cc": {
"changes": 21,
"discharged": 0,
"frameworks_recognised": 0,
"status": "uncovered",
"unguarded": 0
},
"d8eda62c6cc2": {
"changes": 21,
"discharged": 0,
"frameworks_recognised": 58,
"status": "covered",
"unguarded": 5
},
"e89a8c0f281c": {
"changes": 21,
"discharged": 0,
"frameworks_recognised": 375,
"status": "covered",
"unguarded": 0
},
"ed1f033972e0": {
"changes": 21,
"discharged": 0,
"frameworks_recognised": 355,
"status": "covered",
"unguarded": 0
}
},
"population_role": "development",
"seconds_per_change": {
"maximum": 239.97186829699785,
"mean": 17.395369377060025,
"median": 1.0545286380001926,
"n": 483
},
"total_changes": 483,
"uncovered_repositories": [
"1ca62ca671af",
"c92157c966cc"
]
},
"comparison_to_immutable_baseline": {
"current": {
"covered_recall": {
"count": 8,
"n": 21,
"rate": 0.38095238095238093,
"wilson95": [
0.20750767797982506,
0.5912170418449848
]
},
"false_alarms_per_100_pushes": {
"cluster95": [
2.608695652173913,
8.478260869565217
],
"count": 24,
"icc": 0.06521124844323971,
"n": 460,
"rate": 5.217391304347826,
"wilson95": [
3.5308056950422455,
7.645768376796072
]
},
"instrument_failures": 0,
"location_recall": {
"count": 25,
"n": 93,
"rate": 0.26881720430107525,
"wilson95": [
0.1892118216186081,
0.36676412260370955
]
},
"precision": {
"count": 8,
"n": 32,
"rate": 0.25,
"wilson95": [
0.1325224398262155,
0.42106891770246624
]
},
"recall_on_fix_changes": {
"count": 8,
"n": 23,
"rate": 0.34782608695652173,
"wilson95": [
0.18811054636982238,
0.5511002309310836
]
},
"uncovered_recall": {
"count": 0,
"n": 2,
"rate": 0.0,
"wilson95": [
0,
0.6576280471103807
]
}
},
"immutable_baseline": {
"false_alarms_per_100_pushes": {
"cluster95": [
8.695652173913043,
21.521739130434785
],
"count": 69,
"icc": 0.144640754781103,
"n": 460,
"rate": 15.0,
"wilson95": [
12.027398310244092,
18.552351328425665
]
},
"instrument_failures": 0,
"location_recall": {
"count": 55,
"n": 93,
"rate": 0.5913978494623656,
"wilson95": [
0.4897977403042717,
0.6857466538403933
]
},
"precision": {
"count": 15,
"n": 84,
"rate": 0.17857142857142858,
"wilson95": [
0.1113109090339025,
0.2739461901081896
]
},
"recall_on_fix_changes": {
"count": 15,
"n": 23,
"rate": 0.6521739130434783,
"wilson95": [
0.4488997690689164,
0.8118894536301776
]
}
},
"note": "regression evidence against the immutable 2026-10-28 baseline; development population only, never an untouched-population result"
},
"date": "2026-10-30",
"development_only": true,
"fact_inputs_identical": {
"fact_set_digest": "e7db4031d3d5",
"identical": true
},
"four_arm": {
"arms": [
"baseline-analyzer/baseline-facts",
"baseline-analyzer/candidate-facts",
"candidate-analyzer/baseline-facts",
"candidate-analyzer/candidate-facts"
],
"attribution": {
"analyzer_only_lift": {
"discharged": -130,
"indeterminate": 102,
"unguarded": -246
},
"arms": [
"baseline-analyzer/baseline-facts",
"baseline-analyzer/candidate-facts",
"candidate-analyzer/baseline-facts",
"candidate-analyzer/candidate-facts"
],
"fact_only_lift": {
"discharged": 0,
"indeterminate": 0,
"unguarded": 0
},
"funnel_delta_fact_only": {
"seconds": 0.0
},
"judgment_delta_analyzer_only": {
"discharged": -130,
"indeterminate": 102,
"unguarded": -246
},
"judgment_delta_fact_only": {
"discharged": 0,
"indeterminate": 0,
"unguarded": 0
},
"transitions": [
"flagged->indeterminate",
"flagged->none",
"flagged->unguarded",
"quiet->indeterminate",
"quiet->none",
"quiet->unguarded",
"same"
]
},
"fact_inputs_identical": true,
"per_change_transitions": {
"flagged->indeterminate": 25,
"flagged->none": 30,
"flagged->unguarded": 29,
"quiet->indeterminate": 8,
"quiet->none": 388,
"quiet->unguarded": 3
}
},
"population": {
"changes": 483,
"freeze_verified": true,
"per_record_stamp_note": "executed-arm records carry the shard-scoped population digest of the subset their worker processed (8b7be14b70df shard 0, 27fb15c10491 shard 1); this field is the authoritative full-population digest over all 483 change ids. Baseline-arm adapted records carry no population stamp from the immutable 2026-10-28 run; disclosed rather than back-filled.",
"pool": "benchmarks/valid/train/pool-train.toml",
"population_digest": "999594274a4e"
},
"population_role": "development",
"stage_funnel": {
"entry_points": 17046,
"judgment_counts": {
"discharged": 0,
"indeterminate": 102,
"unguarded": 94
},
"resolution": {
"complete_calls": 589520,
"incomplete_calls": 1451325,
"unresolved_calls": 2908559
}
}
}