mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-11-13 01:48:17 +00:00
feat(access-control-prover): 13.4 admission measurement
This commit is contained in:
Marius Ciepluch
@@ -0,0 +1,119 @@
|
||||
# Access-control prover admission measurement (Task 13.4)
|
||||
|
||||
First admission run of the amended access-control prover arm on the untouched pilot
|
||||
population, consumed through the reviewed `valid-detection-benchmark` manifest. Result:
|
||||
**NO-GO** — the development score does not generalize to the admission population.
|
||||
|
||||
## What ran
|
||||
|
||||
| Input | Value |
|
||||
|---|---|
|
||||
| Admission manifest | `manifest.json` (this directory), consumed and validated by `benchmarks/valid/admission_run.py` |
|
||||
| Analyzer revision | `d063167b28ed7db2335737fdb0dae0555225ea85` (HEAD at freeze) |
|
||||
| Fact set | 18 ruleset TOMLs, manifest digest `e7db4031d3d5` (unchanged since the 2026-10-30 four-arm run), bundle artifact digest `3eb5bf699434` |
|
||||
| Configuration | `(arm=access-control-admission, judgment_mode=strict)`, digest `095b5a4905ad` |
|
||||
| Population | `benchmarks/valid/pilot/pool-valid.toml`, freeze digest `c866c0e812c0`, held-out guard 0, labels 31 confirmed / 0 unchecked |
|
||||
| Population contents | 12 repositories, 252 changes (12 vulnerability twins + 240 reviewed ordinary) |
|
||||
| Isolation | No pilot repository referenced by any fact artifact (verified at freeze); facts trace to pool-train clones and approved documentation only |
|
||||
| Criteria | Pre-registered in `criteria.json` BEFORE the run: min_fix_recall 0.5, min_pair_correctness 0.5, min_verified_location_recall 0.5, min_precision 0.25, max_flagged_ordinary_per_100 10.0, max_indeterminate_rate 0.5 |
|
||||
|
||||
The freeze script verified facts-on-disk == facts-at-HEAD, re-derived the pool freeze
|
||||
digest, checked the held-out guard, proved isolation, materialised the bundle and wrote
|
||||
the manifest and criteria before any prover execution (Req 12.1 ordering).
|
||||
|
||||
## Result
|
||||
|
||||
| Metric | Value |
|
||||
|---|---|
|
||||
| Decision | **NO-GO** |
|
||||
| Failed gates | `min_fix_recall: 0.0 < 0.5`, `min_pair_correctness: 0.0 < 0.5`, `min_precision: 0.0 < 0.25`, `min_verified_location_recall: not measured (n=0)` |
|
||||
| Pairs | n=12, correct=0 |
|
||||
| fix_recall | 0/12 (Wilson95 0.0-0.243) |
|
||||
| flagged_ordinary_per_100 | 2/240 = 0.83 (Wilson95 0.23-2.99) |
|
||||
| precision | 0/14 |
|
||||
| indeterminate_rate | 3/264 sides = 1.1% (Wilson95 0.4-3.3) — the three fixed-side locator misses below |
|
||||
| coverage | 4/12 repositories (0.333) |
|
||||
| runtime | mean 2.8s, max 7.1s per pair; max ordinary change 7.7s (every change budget-gated at 120s, gate passed) |
|
||||
| instrument failures | 0 (all 264 records prove nonzero input reads) |
|
||||
|
||||
Fixed-side twin protocol: 9 of 12 labelled functions were located by name at the fix
|
||||
revision (e.g. dbhub `scanTokenMySQL` shifts from lines 135-143 vulnerable to 158-165
|
||||
fixed); the remaining 3 are **locator misses** — the function name is still present at
|
||||
the fix but the declaration scan cannot name it (md-editor `fenceGroup`, i18next
|
||||
`extendOptionsWithDefaults`, webpack `chartData`, as const-arrows/getters/import-moved
|
||||
syntax). Locator misses score `indeterminate` on the fixed side and are never converted
|
||||
to clean results (design.md:371); a function genuinely absent from the fixed revision
|
||||
would count as quiet ("removed"). No pair was affected: all 3 fixed records carry empty
|
||||
evidence and the vulnerable sides were not detected.
|
||||
|
||||
## Reading
|
||||
|
||||
The zero is a **recognition failure, not a judgment failure**. On 9 of 12 repositories
|
||||
the prover recognized no framework at the vulnerable sites, and every vulnerable-side
|
||||
judgment funnel came back all-zero: no obligation pair was ever formed, so no guard
|
||||
evidence and no unguarded evidence existed. Under the pair vocabulary these sites score
|
||||
as completed-quiet ("discharged" per the documented quiet-or-discharged mapping, Req
|
||||
12.5); the indeterminate rate of 1.1% counts exactly the three fixed-side locator
|
||||
misses — nothing on the vulnerable side was incomplete; the engine simply had no
|
||||
knowledge to apply. The obligation facts were promoted from pool-train
|
||||
clones (Jenkins/Stapler-shaped Java, plus Spring/Servlet documentation); the admission
|
||||
population is TypeScript/JavaScript/Python MCP-and-web ecosystems. Four repositories
|
||||
had any recognition at all (fastapi-users 117, malla 161, jupyterlab-lsp 10 framework
|
||||
hits, gitlab-mcp 2 on ordinary changes only); none produced a site finding.
|
||||
|
||||
The false-alarm side improved against development (0.83 vs 5.2 flagged/100) and the
|
||||
budget gate passed everywhere (max 7.1s vs 120s), but with zero recall the arm cannot
|
||||
enter pre-push admission. This is exactly the generalisation check Requirement 12
|
||||
exists to force: the development evidence (recall 34.8% on pool-train) does not
|
||||
transfer to untouched repositories.
|
||||
|
||||
## Verification
|
||||
|
||||
- Instrument: 264/264 records carry nonzero `hook_bytes_read`; zero zero-byte reads;
|
||||
0 instrument failures; `status=prover_covered` everywhere.
|
||||
- Determinism: 4/4 spot re-proofs (2 vulnerable sites + 2 ordinary) byte-identical on
|
||||
findings, funnel, status and read counts; the independent reviewer re-proved 4/4
|
||||
more, including both flagged ordinary records.
|
||||
- Unit tests: 24/24 (`tests/test_admission_run.py`). Statics: ruff check, format,
|
||||
C901 and mypy (6-file anchor set) all exit 0.
|
||||
- Keyword regression (handover-documented expression): 682 passed + 2 skipped, exit 0.
|
||||
The full suite still fails `test_no_reserved_repository_is_referenced_by_the_development_tree`
|
||||
— a pre-existing clean-HEAD failure from committed 2026-10-12 artifacts, documented
|
||||
at the 2026-10-30 four-arm run and outside this task's boundary.
|
||||
- Boundary: no fact, strategy or threshold file was modified (`git status` shows only
|
||||
this measurement directory plus the runner and its tests).
|
||||
|
||||
## Corrections (round-1 review)
|
||||
|
||||
The first review round rejected the task on two points, both remediated:
|
||||
|
||||
1. The fixed-side protocol mapped every locator miss to `fixed_operation="removed"`
|
||||
and thence to a clean `discharged` judgment — an inflationary path that contradicts
|
||||
design.md:371. Remediated by the three-state protocol (located / missed / removed,
|
||||
`classify_fixed_site`), with locator misses scored `indeterminate`. All 12 fixed
|
||||
sides were re-proved under the new stamps (19s); the three misses are disclosed
|
||||
above and the report was re-assembled from the resumable records.
|
||||
2. Ordinary changes are now individually budget-gated (design.md:370), and the
|
||||
regression count was re-measured under the documented keyword expression.
|
||||
|
||||
No measurement number changed except `indeterminate_rate` (0 -> 3/264, the honest
|
||||
locator-miss accounting); the decision remains NO-GO on the same four gates.
|
||||
|
||||
## Reproduction
|
||||
|
||||
Records: `~/ousast-results/prover/ac-admission-1/{vulnerable,fixed,ordinary}/<change-id>.json`
|
||||
(resumable by input digest; `population_digest` on every record is the authoritative
|
||||
freeze digest `c866c0e812c0`).
|
||||
|
||||
```sh
|
||||
.venv/bin/python /tmp/opencode/freeze_134.py # verify + author manifest + criteria
|
||||
.venv/bin/python /tmp/opencode/run_134.py # prove population, write report.json
|
||||
.venv/bin/python /tmp/opencode/determinism_134.py # 4/4 reproducibility spot-check
|
||||
```
|
||||
|
||||
The run is resumable: re-running `run_134.py` after the round-1 protocol fix reused
|
||||
the 252 unchanged records and re-proved only the 12 fixed sides.
|
||||
|
||||
Smoke trail: `/tmp/opencode/smoke_134.py` proved one twin pair end-to-end before the
|
||||
full run (fixed-side function located at shifted lines by name, e.g. dbhub
|
||||
`scanTokenMySQL` 135-143 vulnerable vs 158-165 fixed).
|
||||
+20
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"c/rules.toml": "88ed02da9fef6f170b912f601df12d3040b990d962b28ce4021f3f335442f713",
|
||||
"families.toml": "5b4c0487c755e8a280e33bbcfb1f497dc40d8703b5961c6f9322a17a3291881a",
|
||||
"frameworks.toml": "cbb9f791e25d73cc215315b2dbfe9654d2ccd5747852e2a0d11ee45ea3012642",
|
||||
"groovy/rules.toml": "ed0ab48e2da2ad9d084a2e5f55ac929e94ea381000d674c160843f79c352396a",
|
||||
"java/rules.toml": "5934e10f595f907f37629d3b05cb56744f70d3c5cb534231b4b3f13a6510c234",
|
||||
"javascript/rules.toml": "075ea7439d65082ecacdf9d9b98397ee278ed93c11ebf552687603d8fb171ce3",
|
||||
"obligations/java.toml": "9125cb707312a72ac766cd3a10e1a09063ac4a2861487ebcb342eadb620e9108",
|
||||
"obligations/javascript.toml": "14670c32a3622b213eb553256a7ffb38c40ffb3398012eb6f02a1e4da04d9a33",
|
||||
"obligations/php.toml": "a7bcbd9bedace740567c5900db91acaa1d0aeb2049dea3101f518acb934827ce",
|
||||
"obligations/python.toml": "f65cb000dbf9e777dce0f2888b0dca39b531b1624dcc3026cc2d55461647b983",
|
||||
"obligations/typescript.toml": "5b9a9bf1bb4c1846a518f724db161b4b5aa533451bc32aaf84ecde851d824bdf",
|
||||
"php/rules.toml": "e839846d760b57c7d65e4fcad6c206e1f1aa852d0ec4bbb9b0b459e761467115",
|
||||
"python/rules.toml": "e323456fd46b5c3e4092a92463695458a620830b6d9b5f05a4ade1e14d78a5c9",
|
||||
"semantic/c.toml": "2d4bbc842596fc17a70e51e528b57af5961ef05515259a57a203faa18018fa4e",
|
||||
"semantic/java.toml": "e7f5b73ffc0fe71d7a9096bdf705ee8cc8e5659e2715cdb1705bd59f6c31dced",
|
||||
"semantic/javascript.toml": "91c6aa895021aff755c4bf4b4ed4f4c7d8a991d260f1f9d9363be68f2134dfd5",
|
||||
"semantic/php.toml": "8c7585c1241bd904558389f66decf88c27554107fc7c6e3fa1b0855797e8ea0c",
|
||||
"semantic/python.toml": "97fd7cfd79ef90a3d51255d951c9f4e1ccabe09d5b6b2574f4ff788556977ab0"
|
||||
}
|
||||
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"acquisition_artifact": null,
|
||||
"acquisition_record_digest": "none",
|
||||
"analyzer_revision": "d063167b28ed7db2335737fdb0dae0555225ea85",
|
||||
"configuration_digest": "095b5a4905ad",
|
||||
"fact_artifact": "facts-admission.json",
|
||||
"fact_set_digest": "3eb5bf699434"
|
||||
}
|
||||
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"max_flagged_ordinary_per_100": 10.0,
|
||||
"max_indeterminate_rate": 0.5,
|
||||
"min_fix_recall": 0.5,
|
||||
"min_pair_correctness": 0.5,
|
||||
"min_precision": 0.25,
|
||||
"min_verified_location_recall": 0.5
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"c/rules.toml": "88ed02da9fef6f170b912f601df12d3040b990d962b28ce4021f3f335442f713",
|
||||
"families.toml": "5b4c0487c755e8a280e33bbcfb1f497dc40d8703b5961c6f9322a17a3291881a",
|
||||
"frameworks.toml": "cbb9f791e25d73cc215315b2dbfe9654d2ccd5747852e2a0d11ee45ea3012642",
|
||||
"groovy/rules.toml": "ed0ab48e2da2ad9d084a2e5f55ac929e94ea381000d674c160843f79c352396a",
|
||||
"java/rules.toml": "5934e10f595f907f37629d3b05cb56744f70d3c5cb534231b4b3f13a6510c234",
|
||||
"javascript/rules.toml": "075ea7439d65082ecacdf9d9b98397ee278ed93c11ebf552687603d8fb171ce3",
|
||||
"obligations/java.toml": "9125cb707312a72ac766cd3a10e1a09063ac4a2861487ebcb342eadb620e9108",
|
||||
"obligations/javascript.toml": "14670c32a3622b213eb553256a7ffb38c40ffb3398012eb6f02a1e4da04d9a33",
|
||||
"obligations/php.toml": "a7bcbd9bedace740567c5900db91acaa1d0aeb2049dea3101f518acb934827ce",
|
||||
"obligations/python.toml": "f65cb000dbf9e777dce0f2888b0dca39b531b1624dcc3026cc2d55461647b983",
|
||||
"obligations/typescript.toml": "5b9a9bf1bb4c1846a518f724db161b4b5aa533451bc32aaf84ecde851d824bdf",
|
||||
"php/rules.toml": "e839846d760b57c7d65e4fcad6c206e1f1aa852d0ec4bbb9b0b459e761467115",
|
||||
"python/rules.toml": "e323456fd46b5c3e4092a92463695458a620830b6d9b5f05a4ade1e14d78a5c9",
|
||||
"semantic/c.toml": "2d4bbc842596fc17a70e51e528b57af5961ef05515259a57a203faa18018fa4e",
|
||||
"semantic/java.toml": "e7f5b73ffc0fe71d7a9096bdf705ee8cc8e5659e2715cdb1705bd59f6c31dced",
|
||||
"semantic/javascript.toml": "91c6aa895021aff755c4bf4b4ed4f4c7d8a991d260f1f9d9363be68f2134dfd5",
|
||||
"semantic/php.toml": "8c7585c1241bd904558389f66decf88c27554107fc7c6e3fa1b0855797e8ea0c",
|
||||
"semantic/python.toml": "97fd7cfd79ef90a3d51255d951c9f4e1ccabe09d5b6b2574f4ff788556977ab0"
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"acquisition_record_digest": "none",
|
||||
"analyzer_revision": "d063167b28ed7db2335737fdb0dae0555225ea85",
|
||||
"configuration_digest": "095b5a4905ad",
|
||||
"fact_set_digest": "3eb5bf699434",
|
||||
"label_isolated": true,
|
||||
"labels": [],
|
||||
"population_digest": "c866c0e812c0b54476c7168fdb0e4a20652de0d30485270afecbe51e5c35e424",
|
||||
"population_role": "admission",
|
||||
"produced_by": "valid-detection-benchmark",
|
||||
"reviewed": true,
|
||||
"spent": false
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
{
|
||||
"coverage": 0.3333333333333333,
|
||||
"criteria_pre_registered": {
|
||||
"max_flagged_ordinary_per_100": 10.0,
|
||||
"max_indeterminate_rate": 0.5,
|
||||
"min_fix_recall": 0.5,
|
||||
"min_pair_correctness": 0.5,
|
||||
"min_precision": 0.25,
|
||||
"min_verified_location_recall": 0.5
|
||||
},
|
||||
"decision": "NO-GO",
|
||||
"failed_gates": [
|
||||
"min_fix_recall: 0.0 < 0.5",
|
||||
"min_pair_correctness: 0.0 < 0.5",
|
||||
"min_precision: 0.0 < 0.25",
|
||||
"min_verified_location_recall: not measured (n=0) vs 0.5"
|
||||
],
|
||||
"fix_recall": {
|
||||
"count": 0,
|
||||
"n": 12,
|
||||
"rate": 0.0,
|
||||
"wilson95": [
|
||||
0,
|
||||
0.2425007574992425
|
||||
]
|
||||
},
|
||||
"flagged_ordinary_per_100": {
|
||||
"count": 2,
|
||||
"n": 240,
|
||||
"rate": 0.8333333333333334,
|
||||
"wilson95": [
|
||||
0.2288235363161974,
|
||||
2.9870346273441455
|
||||
]
|
||||
},
|
||||
"indeterminate_rate": {
|
||||
"count": 3,
|
||||
"n": 264,
|
||||
"rate": 0.011363636363636364,
|
||||
"wilson95": [
|
||||
0.003871976570054633,
|
||||
0.03287212890123884
|
||||
]
|
||||
},
|
||||
"instrument_failures": 0,
|
||||
"manifest": {
|
||||
"acquisition_record_digest": "none",
|
||||
"analyzer_revision": "d063167b28ed7db2335737fdb0dae0555225ea85",
|
||||
"configuration_digest": "095b5a4905ad",
|
||||
"fact_set_digest": "3eb5bf699434",
|
||||
"population_digest": "c866c0e812c0b54476c7168fdb0e4a20652de0d30485270afecbe51e5c35e424",
|
||||
"population_role": "admission"
|
||||
},
|
||||
"pair_correctness": {
|
||||
"count": 0,
|
||||
"n": 12,
|
||||
"rate": 0.0,
|
||||
"wilson95": [
|
||||
0,
|
||||
0.2425007574992425
|
||||
]
|
||||
},
|
||||
"pairs": {
|
||||
"correct": 0,
|
||||
"n": 12
|
||||
},
|
||||
"precision": {
|
||||
"count": 0,
|
||||
"n": 14,
|
||||
"rate": 0.0,
|
||||
"wilson95": [
|
||||
0,
|
||||
0.21531701192718142
|
||||
]
|
||||
},
|
||||
"runtime": {
|
||||
"max_seconds": 7.112766742007807,
|
||||
"mean_seconds": 2.8931797610857757,
|
||||
"n": 12
|
||||
},
|
||||
"verified_location_recall": {
|
||||
"count": 0,
|
||||
"n": 0,
|
||||
"rate": null,
|
||||
"wilson95": [
|
||||
null,
|
||||
null
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,329 @@
|
||||
"""Admission measurement runner (Task 13.4, Reqs 12.1-12.9).
|
||||
|
||||
Consumes a reviewed admission manifest for the valid-detection-benchmark
|
||||
pilot population (authored and attested by the benchmark workflow, never by
|
||||
this module), proves both sides of every vulnerable/fixed twin plus every
|
||||
reviewed ordinary change on the strict judgment lane with nonzero-read
|
||||
instrumentation, and returns the paired admission report. Judgment mapping
|
||||
onto the three-way vocabulary: ``unguarded`` is a finding at the labelled
|
||||
site; ``indeterminate`` is incomplete site evidence (an incomplete trace, a
|
||||
degraded site file, or a fixed-side locator miss); every other completed
|
||||
site is ``discharged`` -- Req 12.5's "quiet or discharged" behaviour. The
|
||||
runner never modifies facts, strategies or thresholds, and never produces,
|
||||
mutates or re-labels the manifest it consumes.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import tomllib
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from benchmarks.unseen.replay import canonical
|
||||
from benchmarks.valid.admission_scoring import AdmissionCase, OrdinaryChange, admission_report
|
||||
from benchmarks.valid.arm_regression import BundleDigests, _input_digest, _prove_one
|
||||
from benchmarks.valid.prover_admission import AdmissionManifest, validate_admission_manifest
|
||||
from openultrasast.preprocess import detect_language
|
||||
from openultrasast.source import canonical_language, ranges
|
||||
|
||||
__all__ = [
|
||||
"admission_ordinary",
|
||||
"admission_pair_cases",
|
||||
"assemble_admission_report",
|
||||
"coverage_share",
|
||||
"fixed_side_change",
|
||||
"load_admission_population",
|
||||
"locate_function",
|
||||
"run_admission",
|
||||
"site_judgment",
|
||||
]
|
||||
|
||||
ARM_NAME = "access-control-admission"
|
||||
|
||||
|
||||
def _site_traces(record: dict[str, Any], path: str, start: int, end: int) -> list[dict[str, Any]]:
|
||||
"""Traces whose operation site falls inside the labelled range."""
|
||||
matched = []
|
||||
for trace in record.get("traces", ()):
|
||||
fact_id = str(trace.get("fact_id", ""))
|
||||
marker = f":{path}:"
|
||||
index = fact_id.find(marker)
|
||||
if index < 0:
|
||||
continue
|
||||
line_text = fact_id[index + len(marker) :].split(":", 1)[0]
|
||||
if line_text.isdigit() and start <= int(line_text) <= end:
|
||||
matched.append(trace)
|
||||
return matched
|
||||
|
||||
|
||||
def site_judgment(record: dict[str, Any], *, path: str, start: int, end: int) -> str:
|
||||
"""Three-way judgment of one labelled site (Req 12.5 pair semantics)."""
|
||||
findings = record.get("findings", ())
|
||||
if any(finding.get("path") == path and start <= int(finding.get("line", 0)) <= end for finding in findings):
|
||||
return "unguarded"
|
||||
judgments = [trace["abstract_state"]["path_judgment"] for trace in _site_traces(record, path, start, end)]
|
||||
if any(judgment.get("judgment") == "unguarded" for judgment in judgments):
|
||||
return "unguarded"
|
||||
incomplete = any(
|
||||
judgment.get("judgment") == "indeterminate" or not judgment.get("completeness", {}).get("is_complete", True)
|
||||
for judgment in judgments
|
||||
)
|
||||
if incomplete:
|
||||
return "indeterminate"
|
||||
if any(degradation.get("path") == path for degradation in record.get("degradations", ())):
|
||||
return "indeterminate"
|
||||
return "discharged"
|
||||
|
||||
|
||||
def locate_function(functions: Any, *, name: str) -> tuple[int, int] | None:
|
||||
"""Inclusive line range of ``name`` among function ranges, or None."""
|
||||
for function in functions:
|
||||
label = function[0] if isinstance(function, tuple) else function.name
|
||||
if label == name:
|
||||
start = function[1] if isinstance(function, tuple) else function.start
|
||||
end = function[2] if isinstance(function, tuple) else function.end
|
||||
return int(start), int(end)
|
||||
return None
|
||||
|
||||
|
||||
def fixed_side_change(change: dict[str, Any], *, fixed_lines: tuple[int, int] | None, fixed_operation: str) -> dict[str, Any]:
|
||||
"""The twin proof target: the fix commit checked out with its own site lines.
|
||||
|
||||
``fixed_operation`` is "located" with lines, or one of "missed" (the
|
||||
function name is present at the fix but the locator could not name it --
|
||||
const-arrows, getters, moved imports) and "removed" (the name is absent
|
||||
from the fixed revision entirely). Only "removed" counts as quiet.
|
||||
"""
|
||||
site = str(change.get("path") or change["files"][0]["path"])
|
||||
fixed: dict[str, Any] = {
|
||||
**change,
|
||||
"base": str(change["head"]),
|
||||
"head": str(change["base"]),
|
||||
"fixed_operation": fixed_operation,
|
||||
"files": [{"path": site, "head_lines": [list(fixed_lines)]}] if fixed_lines else [],
|
||||
}
|
||||
if fixed_lines:
|
||||
fixed["fixed_lines"] = list(fixed_lines)
|
||||
return fixed
|
||||
|
||||
|
||||
def classify_fixed_site(lines: tuple[int, int] | None, *, name_in_text: bool) -> str:
|
||||
"""Located, or missed (name present, locator failed) vs removed (name absent)."""
|
||||
if lines is not None:
|
||||
return "located"
|
||||
return "missed" if name_in_text else "removed"
|
||||
|
||||
|
||||
def _fixed_judgment(record: dict[str, Any]) -> str:
|
||||
if record.get("fixed_operation") == "removed":
|
||||
return "discharged"
|
||||
if "fixed_lines" not in record:
|
||||
return "indeterminate"
|
||||
start, end = (int(value) for value in record["fixed_lines"])
|
||||
return site_judgment(record, path=str(record["path"]), start=start, end=end)
|
||||
|
||||
|
||||
def admission_pair_cases(vulnerable: tuple[dict[str, Any], ...], fixed: tuple[dict[str, Any], ...]) -> tuple[AdmissionCase, ...]:
|
||||
"""Build one scored twin pair per vulnerable case (Req 12.5)."""
|
||||
fixed_by_id = {str(record["id"]): record for record in fixed}
|
||||
cases = []
|
||||
for record in vulnerable:
|
||||
counterpart = fixed_by_id[str(record["id"])]
|
||||
start, end = (int(value) for value in record["function_lines"])
|
||||
path = str(record["path"])
|
||||
verified = any(
|
||||
finding.get("path") == path and start <= int(finding.get("line", 0)) <= end and finding.get("family") == record.get("family")
|
||||
for finding in record.get("findings", ())
|
||||
)
|
||||
cases.append(
|
||||
AdmissionCase(
|
||||
case_id=str(record["id"]),
|
||||
vulnerable_judgment=site_judgment(record, path=path, start=start, end=end),
|
||||
fixed_judgment=_fixed_judgment(counterpart),
|
||||
location_verified=bool(verified),
|
||||
seconds=float(record.get("seconds", 0.0)) + float(counterpart.get("seconds", 0.0)),
|
||||
)
|
||||
)
|
||||
return tuple(cases)
|
||||
|
||||
|
||||
def admission_ordinary(records: tuple[dict[str, Any], ...]) -> tuple[OrdinaryChange, ...]:
|
||||
"""Reviewed ordinary changes form the false-alarm denominator (Req 12.4)."""
|
||||
changes = []
|
||||
for record in records:
|
||||
if record.get("findings"):
|
||||
judgment = "unguarded"
|
||||
elif int(record.get("stage_funnel", {}).get("judgment", {}).get("indeterminate", 0)) > 0:
|
||||
judgment = "indeterminate"
|
||||
else:
|
||||
judgment = "discharged"
|
||||
changes.append(OrdinaryChange(change_id=str(record["id"]), judgment=judgment, mislabeled_defect=False))
|
||||
return tuple(changes)
|
||||
|
||||
|
||||
def coverage_share(records: tuple[dict[str, Any], ...]) -> float:
|
||||
"""Share of population repositories with any recognised framework."""
|
||||
per_repository: dict[str, int] = {}
|
||||
for record in records:
|
||||
recognised = record.get("stage_funnel", {}).get("entry_recognition", {}).get("frameworks_recognised", {})
|
||||
total = sum(recognised.values()) if isinstance(recognised, dict) else int(recognised)
|
||||
per_repository[str(record["repo"])] = per_repository.get(str(record["repo"]), 0) + int(total)
|
||||
if not per_repository:
|
||||
return 0.0
|
||||
covered = sum(1 for total in per_repository.values() if total > 0)
|
||||
return covered / len(per_repository)
|
||||
|
||||
|
||||
def load_admission_population(
|
||||
payload: dict[str, object], pool_path: Path, freeze_path: Path
|
||||
) -> tuple[tuple[dict[str, Any], ...], tuple[dict[str, Any], ...]]:
|
||||
"""Validate the manifest and the reviewed freeze before opening labels (Req 12.1)."""
|
||||
manifest = validate_admission_manifest(payload)
|
||||
pool = tomllib.loads(Path(pool_path).read_text(encoding="utf-8"))
|
||||
freeze = json.loads(Path(freeze_path).read_text(encoding="utf-8"))
|
||||
digest = hashlib.sha256(canonical(pool)).hexdigest()
|
||||
if digest != freeze.get("freeze_digest"):
|
||||
raise ValueError(f"population: pool digest {digest[:12]} does not match the reviewed freeze")
|
||||
if freeze.get("guard_result") != 0:
|
||||
raise ValueError("guard: held-out guard failed for this population")
|
||||
if manifest.population_digest != digest:
|
||||
raise ValueError("population: manifest names a different population than the reviewed freeze")
|
||||
changes = tuple(
|
||||
{**change, "id": _change_identity(repository, change), "repo": str(repository["id"])}
|
||||
for repository in pool.get("repository", [])
|
||||
for change in repository.get("changes", [])
|
||||
)
|
||||
vulnerable = tuple(change for change in changes if change.get("kind") == "vulnerability")
|
||||
ordinary = tuple(change for change in changes if change.get("kind") == "ordinary")
|
||||
return vulnerable, ordinary
|
||||
|
||||
|
||||
def _change_identity(repository: dict[str, Any], change: dict[str, Any]) -> str:
|
||||
"""The pool's canonical change identity, as ``replay.load_slice`` derives it."""
|
||||
payload = [repository["id"], change["kind"], change["base"], change["head"]]
|
||||
return hashlib.sha256(canonical(payload)).hexdigest()
|
||||
|
||||
|
||||
def assemble_admission_report(
|
||||
manifest: AdmissionManifest,
|
||||
vulnerable: tuple[dict[str, Any], ...],
|
||||
fixed: tuple[dict[str, Any], ...],
|
||||
ordinary: tuple[dict[str, Any], ...],
|
||||
*,
|
||||
criteria: dict[str, float] | None,
|
||||
budget_seconds: float = 120.0,
|
||||
) -> dict[str, Any]:
|
||||
"""Bridge measured records to the paired admission report (Reqs 12.7, 12.8).
|
||||
|
||||
Every change is budget-gated: pairs by their combined seconds, ordinary
|
||||
changes individually (design.md:370 "fails any change over 120 seconds").
|
||||
"""
|
||||
failures = sum(1 for record in (*vulnerable, *fixed, *ordinary) if record.get("status") == "instrument_failure")
|
||||
report = admission_report(
|
||||
manifest,
|
||||
admission_pair_cases(vulnerable, fixed),
|
||||
admission_ordinary(ordinary),
|
||||
coverage=coverage_share((*vulnerable, *fixed, *ordinary)),
|
||||
instrument_failures=failures,
|
||||
budget_seconds=budget_seconds,
|
||||
criteria=criteria,
|
||||
)
|
||||
overruns = [
|
||||
f"budget:{record['id']}:{float(record.get('seconds', 0.0)):g}s>{budget_seconds:g}s"
|
||||
for record in ordinary
|
||||
if float(record.get("seconds", 0.0)) > budget_seconds
|
||||
]
|
||||
if overruns:
|
||||
report["failed_gates"] = [*report["failed_gates"], *overruns]
|
||||
report["decision"] = "NO-GO"
|
||||
return report
|
||||
|
||||
|
||||
def _resume_or_prove(
|
||||
bundle: BundleDigests,
|
||||
population: str,
|
||||
change: dict[str, Any],
|
||||
repo_path: Path,
|
||||
side_dir: Path,
|
||||
) -> dict[str, Any]:
|
||||
from benchmarks.valid.prover_arm import _write_json
|
||||
|
||||
side_dir.mkdir(parents=True, exist_ok=True)
|
||||
path = side_dir / f"{change['id']}.json"
|
||||
if path.is_file():
|
||||
previous: dict[str, Any] = json.loads(path.read_text(encoding="utf-8"))
|
||||
if previous.get("input_digest") == _input_digest(change, ARM_NAME):
|
||||
return previous
|
||||
raise ValueError(f"existing record for {change['id']} does not match this arm's inputs")
|
||||
record = _prove_one(bundle, ARM_NAME, population, change, repo_path)
|
||||
_write_json(path, record)
|
||||
return record
|
||||
|
||||
|
||||
def run_admission(
|
||||
*,
|
||||
payload: dict[str, object],
|
||||
pool_path: Path,
|
||||
freeze_path: Path,
|
||||
repos: dict[str, str],
|
||||
bundle: BundleDigests,
|
||||
out_dir: Path,
|
||||
criteria: dict[str, float] | None,
|
||||
budget_seconds: float = 120.0,
|
||||
) -> dict[str, Any]:
|
||||
"""Consume the reviewed manifest, prove the population, return GO/NO-GO."""
|
||||
vulnerable, ordinary = load_admission_population(payload, pool_path, freeze_path)
|
||||
manifest = validate_admission_manifest(payload)
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
vulnerable_records = tuple(
|
||||
_resume_or_prove(bundle, manifest.population_digest, change, Path(repos[change["repo"]]).expanduser(), out_dir / "vulnerable")
|
||||
for change in vulnerable
|
||||
)
|
||||
fixed_records = tuple(
|
||||
_resume_or_prove(
|
||||
bundle,
|
||||
manifest.population_digest,
|
||||
_fixed_side_change(Path(repos[change["repo"]]).expanduser(), change),
|
||||
Path(repos[change["repo"]]).expanduser(),
|
||||
out_dir / "fixed",
|
||||
)
|
||||
for change in vulnerable
|
||||
)
|
||||
ordinary_records = tuple(
|
||||
_resume_or_prove(bundle, manifest.population_digest, change, Path(repos[change["repo"]]).expanduser(), out_dir / "ordinary")
|
||||
for change in ordinary
|
||||
)
|
||||
return assemble_admission_report(
|
||||
manifest, vulnerable_records, fixed_records, ordinary_records, criteria=criteria, budget_seconds=budget_seconds
|
||||
)
|
||||
|
||||
|
||||
def _fixed_side_change(repo_path: Path, change: dict[str, Any]) -> dict[str, Any]:
|
||||
lines, operation = _fixed_site(repo_path, change)
|
||||
return fixed_side_change(change, fixed_lines=lines, fixed_operation=operation)
|
||||
|
||||
|
||||
def _fixed_site(repo_path: Path, change: dict[str, Any]) -> tuple[tuple[int, int] | None, str]:
|
||||
"""(lines, operation) for the labelled function at the fixed revision.
|
||||
|
||||
A locator miss (the name is present but ``ranges`` could not name it) is
|
||||
"missed" and can never become a clean judgment (design.md:371).
|
||||
"""
|
||||
from benchmarks.valid.catalog import local_git
|
||||
|
||||
git = local_git(repo_path)
|
||||
text = git.runner(
|
||||
["git", "-C", str(repo_path), "show", f"{change['base']}:{change['path']}"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
).stdout
|
||||
name = str(change.get("function"))
|
||||
try:
|
||||
functions = ranges(str(change["path"]), text, canonical_language(detect_language(repo_path / change["path"])))
|
||||
except ValueError:
|
||||
functions = ()
|
||||
lines = locate_function(functions, name=name)
|
||||
return lines, classify_fixed_site(lines, name_in_text=name in text)
|
||||
Reference in New Issue
Block a user