mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
feat(constrained-detector): the ceiling, measured offline across every slice
Group 1 complete. Tasks 1.2 and 1.3: `candidates.ceiling`, `ousast learning
candidates`, and the stop-or-go reading. No model is called anywhere in this
group, so the condition that could have ended the feature was reachable for
free.
Verdict: go.
agent-vfc (deciding) 89.7% ceiling against 44.4% round-zero detection
vfc (deciding) 84.7%, baseline still running, non-gating
vfc-js (deciding) 52.9% -- see below
github (deciding) 100.0%
vibe-py (development)100.0% against 83.3%
The margin is largest on agent-vfc, which is the slice the tool is actually
for.
The measurement forced a shape correction before the detector existed.
config_secrets first read 63.6% with a bind-and-literal shape, because a
permissive default is as often an argument to a call -- cors({origin: '*'}),
app.use(session({secure: false})) -- as it is a binding. Adding call sites to
that family's shape took it to 100% and recovered four agent-vfc rows.
Carry-forward, and not this feature's to fix: vfc-js sits at 52.9% because
eight of seventeen rows label a function the JavaScript CST names `<anon>` --
`module.exports.publish = function () {}` and its relatives. That is a
function-naming gap, it caps the existing hunter on that slice exactly as
much, and it is the same class as the `_declarator_start` defects found in
learning-harness task 4.1.
The ruleset arm of `ceiling` is kept so the 12.1% that rejected it stays a
measurement rather than a remembered number.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
co-authored by
Claude Opus 5
Claude Opus 5
parent
8d771901bf
commit
78b4f33a8e
@@ -1,20 +1,20 @@
|
||||
# Implementation Plan
|
||||
|
||||
- [ ] 1. The ceiling, offline: what a candidate detector could possibly find
|
||||
- [ ] 1.1 Candidate enumeration from the semantic IR, shaped by family
|
||||
- [x] 1. The ceiling, offline: what a candidate detector could possibly find
|
||||
- [x] 1.1 Candidate enumeration from the semantic IR, shaped by family
|
||||
- `learning/candidates.py`: `Candidate` (path, line, kind, name, text, argument texts and names, the binding chain the IR resolved, the guards in scope, the enclosing function, and an `id` of `path:line:name`); `CandidateKind` of call, bind, operation, config; `FAMILY_SHAPE` mapping each family to the kinds that can carry its bug; `enumerate_candidates` returning a `CandidateSet` with the candidates, the count discarded past the bound, and a reason of empty, `unsupported_language` or `no_candidate`.
|
||||
- Call and bind candidates come from `FunctionIR.calls` and `.binds`; operation candidates for access control come from the obligations facts with the decorators and in-body guards in scope; config candidates are bindings whose value is a literal in a configuration position. The pattern ruleset is never the generator.
|
||||
- Ordering is `(path, line, name)` and the bound is a constant, so the batches a later task forms are determined by the input alone.
|
||||
- Observable: on the shared obligations fixture every family shape yields the sites its docstring names; a file the IR cannot parse yields `unsupported_language` and no candidates; a labeled function with no calls or binds yields `no_candidate`; enumerating twice returns an identical sequence.
|
||||
- _Requirements: 1.1, 1.2, 1.4, 1.5, 1.6_
|
||||
|
||||
- [ ] 1.2 The ceiling report and the `learning candidates` command
|
||||
- [x] 1.2 The ceiling report and the `learning candidates` command
|
||||
- `candidates.ceiling(cases, taxonomy)` returns, per family and per slice, the number of labeled functions for which at least one candidate was enumerated, the sites-per-function distribution, and the rows that yielded nothing with their reason; `ousast learning candidates --catalog --slice --json` prints it and writes the artifact. No model is called at any point.
|
||||
- Observable: the command runs offline with no endpoint configured; the artifact reproduces the recorded web-slice figures (ruleset 12.1%, IR 96.6%, median 10 sites) and adds the deciding slices; running it twice produces a byte-identical artifact.
|
||||
- _Requirements: 1.3, 3.5_
|
||||
- _Depends: 1.1_
|
||||
|
||||
- [ ] 1.3 The stop-or-go reading of the ceiling
|
||||
- [x] 1.3 The stop-or-go reading of the ceiling
|
||||
- The artifact states, per deciding slice, the ceiling against that slice's recorded detection rate from round zero, and names the comparison as the condition on which this design continues. The reading is written into the spec's Implementation Notes with the numbers, whichever way it goes.
|
||||
- Observable: the artifact carries agent-vfc, vfc-js, vfc and github ceilings beside 44.4% and the other recorded rates, and the notes state whether any deciding slice fails the condition.
|
||||
- _Requirements: 1.3, 3.5, 3.7_
|
||||
@@ -100,3 +100,23 @@
|
||||
- Task 2.1 is the only task that touches `scoring.py`, and its observable is that nothing changes. The recorded round-zero artifacts are the pin; if they cannot be reproduced exactly, the refactor is wrong and not the numbers.
|
||||
- `(P)` marks the two tasks with no dependency on a model call: 2.1 and 4.1 can be built and reviewed while phase 1 runs.
|
||||
- Task 3.3 is the one that would have caught the defect this whole spec exists for. It belongs in the suite, run repeatedly, not in a one-off measurement.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- Task 1.1 (2026-09-06, RED first: 13 failing tests in `tests/test_candidate_enumeration.py`; then green). `learning/candidates.py`. Two things the fixtures taught: the IR emits a chained call twice at one line, once bare and once carrying the keyword arguments, so deduplication keeps the richest — a bare `filter_by` and `filter_by(user_id=owner)` are the same site and only one of them says what the constraint was. And `leaky` and `constrained` in the shared obligations fixture **both** discharge through `filter_by`; what separates them is that one binding chain ends at `request.args.get('owner')` and the other at `request.user.id`. The test asserts that contrast, which is the judgement the model is being narrowed to.
|
||||
- Task 1.2 (2026-09-06, RED first: 3 failing tests; then green). `candidates.ceiling(cases, taxonomy, generator="ir"|"ruleset")` and `ousast learning candidates`. The `ruleset` arm exists so the comparison that rejected it stays a measurement rather than a remembered number. The command calls no model and its artifact is byte-identical on a second run.
|
||||
- Task 1.3 (2026-09-06). **Verdict: go.** Measured offline across every slice, `benchmarks/measurements/2026-09-06-candidate-ceiling-all-slices.json`:
|
||||
|
||||
| slice | ceiling | round-zero detection | headroom |
|
||||
|---|---|---|---|
|
||||
| agent-vfc (deciding) | **89.7%** | 44.4% | +45.3 |
|
||||
| vfc (deciding, non-gating) | 84.7% | pending | — |
|
||||
| vfc-js (deciding) | 52.9% | none recorded | see below |
|
||||
| github (deciding) | 100.0% | none recorded | — |
|
||||
| vibe-py (development) | 100.0% | 83.3% | +16.7 |
|
||||
|
||||
The margin is largest on agent-vfc, which is the slice that matters most and the one the tool is actually for. Per family: config_secrets, deserialization, output_encoding and untrusted_destination 100%; access_control 90%; path 87.5%; memory 86.4%; injection 84.6%; prototype 71.4%; `unknown` 0% of two rows, which is what abstention looks like.
|
||||
|
||||
- **A shape correction the measurement forced, before the detector existed.** `config_secrets` first read 63.6% with a bind-and-literal shape. A permissive default is as often an argument to a call — `cors({origin: '*'})`, `app.use(session({secure: false}))` — as it is a binding; adding call sites to that family's shape took it to 100% and recovered `new-erp-final-app-6e9644`, `zoonk-origin-bfbf67`, `melodix-playerprovider-089931` and `registry-proxy-worker-426847`.
|
||||
- **Carry-forward, and it is not this feature's to fix.** vfc-js sits at 52.9% because eight of seventeen rows label a function the JavaScript CST names `<anon>`: `module.exports.publish = function () {}` and its relatives. The gap is in function *naming*, not in candidate generation, and it caps the existing hunter on that slice exactly as much. Same class as the `_declarator_start` defects found during learning-harness task 4.1.
|
||||
- vfc's 21 `unsupported_language` rows are C and C++ files the IR declines; they are the non-gating memory slice and are already excluded from every web number.
|
||||
|
||||
@@ -0,0 +1,381 @@
|
||||
{
|
||||
"command": "uv run ousast learning candidates --slice all",
|
||||
"deciding_slices": {
|
||||
"agent-vfc": {
|
||||
"ceiling": 0.897,
|
||||
"headroom": 0.453,
|
||||
"round_zero_detection": 0.444,
|
||||
"verdict": "go"
|
||||
},
|
||||
"github": {
|
||||
"ceiling": 1.0,
|
||||
"round_zero_detection": null,
|
||||
"verdict": "go"
|
||||
},
|
||||
"vfc": {
|
||||
"ceiling": 0.847,
|
||||
"note": "baseline still running; non-gating",
|
||||
"round_zero_detection": null,
|
||||
"verdict": "go"
|
||||
},
|
||||
"vfc-js": {
|
||||
"ceiling": 0.529,
|
||||
"note": "eight of seventeen rows label a function the JavaScript CST names `<anon>`: `module.exports.x = function () {}` and friends. The bug is in function naming, not in candidate generation, and it caps this slice for the existing hunter too.",
|
||||
"round_zero_detection": null,
|
||||
"verdict": "blocked_by_a_naming_gap"
|
||||
}
|
||||
},
|
||||
"development_slice": {
|
||||
"vibe-py": {
|
||||
"ceiling": 1.0,
|
||||
"headroom": 0.167,
|
||||
"round_zero_detection": 0.833
|
||||
}
|
||||
},
|
||||
"gaps": [
|
||||
[
|
||||
"agent-vfc",
|
||||
"edutrace-page-ee7ba2",
|
||||
"no_candidate"
|
||||
],
|
||||
[
|
||||
"agent-vfc",
|
||||
"lensops-kubernetesmanagement-fe4910",
|
||||
"no_candidate"
|
||||
],
|
||||
[
|
||||
"agent-vfc",
|
||||
"money-track-category-icons-4d4a5b",
|
||||
"no_candidate"
|
||||
],
|
||||
[
|
||||
"vfc",
|
||||
"chromium-cve-2019-5786",
|
||||
"no_candidate"
|
||||
],
|
||||
[
|
||||
"vfc",
|
||||
"chromium-cve-2022-2162",
|
||||
"no_candidate"
|
||||
],
|
||||
[
|
||||
"vfc",
|
||||
"curl-cve-2020-8284",
|
||||
"unsupported_language"
|
||||
],
|
||||
[
|
||||
"vfc",
|
||||
"curl-cve-2020-8286",
|
||||
"unsupported_language"
|
||||
],
|
||||
[
|
||||
"vfc",
|
||||
"curl-cve-2021-22890",
|
||||
"unsupported_language"
|
||||
],
|
||||
[
|
||||
"vfc",
|
||||
"curl-cve-2021-22901-ossl_connect_step1",
|
||||
"unsupported_language"
|
||||
],
|
||||
[
|
||||
"vfc",
|
||||
"curl-cve-2022-27775",
|
||||
"unsupported_language"
|
||||
],
|
||||
[
|
||||
"vfc",
|
||||
"curl-cve-2022-27776",
|
||||
"unsupported_language"
|
||||
],
|
||||
[
|
||||
"vfc",
|
||||
"curl-cve-2022-27778",
|
||||
"unsupported_language"
|
||||
],
|
||||
[
|
||||
"vfc",
|
||||
"curl-cve-2022-27782-curl_init_userdefined",
|
||||
"unsupported_language"
|
||||
],
|
||||
[
|
||||
"vfc",
|
||||
"curl-cve-2022-27782-nss_setup_connect",
|
||||
"unsupported_language"
|
||||
],
|
||||
[
|
||||
"vfc",
|
||||
"curl-cve-2022-27782-ossl_connect_step1",
|
||||
"unsupported_language"
|
||||
],
|
||||
[
|
||||
"vfc",
|
||||
"curl-cve-2023-38545",
|
||||
"unsupported_language"
|
||||
],
|
||||
[
|
||||
"vfc",
|
||||
"curl-cve-2023-38545-do_socks5",
|
||||
"unsupported_language"
|
||||
],
|
||||
[
|
||||
"vfc",
|
||||
"openssl-cve-2021-3449-init_sig_algs_cert",
|
||||
"unsupported_language"
|
||||
],
|
||||
[
|
||||
"vfc",
|
||||
"openssl-cve-2021-3712-ia5ncasecmp",
|
||||
"no_candidate"
|
||||
],
|
||||
[
|
||||
"vfc",
|
||||
"openssl-cve-2022-0778",
|
||||
"no_candidate"
|
||||
],
|
||||
[
|
||||
"vfc",
|
||||
"openssl-cve-2022-1292",
|
||||
"unsupported_language"
|
||||
],
|
||||
[
|
||||
"vfc",
|
||||
"openssl-cve-2022-2068-hash_dir",
|
||||
"unsupported_language"
|
||||
],
|
||||
[
|
||||
"vfc",
|
||||
"openssl-cve-2022-3358",
|
||||
"unsupported_language"
|
||||
],
|
||||
[
|
||||
"vfc",
|
||||
"openssl-cve-2022-3358-evp_cipher_init_internal",
|
||||
"unsupported_language"
|
||||
],
|
||||
[
|
||||
"vfc",
|
||||
"openssl-cve-2023-0216",
|
||||
"no_candidate"
|
||||
],
|
||||
[
|
||||
"vfc",
|
||||
"openssl-cve-2023-5363",
|
||||
"unsupported_language"
|
||||
],
|
||||
[
|
||||
"vfc",
|
||||
"openssl-cve-2024-2511",
|
||||
"unsupported_language"
|
||||
],
|
||||
[
|
||||
"vfc",
|
||||
"openssl-cve-2025-69421",
|
||||
"no_candidate"
|
||||
],
|
||||
[
|
||||
"vfc",
|
||||
"openssl-cve-2026-14456-expect_quic_for_value",
|
||||
"unsupported_language"
|
||||
],
|
||||
[
|
||||
"vfc",
|
||||
"openssl-cve-2026-31790-rsasve_generate",
|
||||
"unsupported_language"
|
||||
],
|
||||
[
|
||||
"vfc-js",
|
||||
"apex-publish-static-files-2-0-0",
|
||||
"no_candidate"
|
||||
],
|
||||
[
|
||||
"vfc-js",
|
||||
"bestzip-2-1-6",
|
||||
"no_candidate"
|
||||
],
|
||||
[
|
||||
"vfc-js",
|
||||
"clamscan-1-2-0",
|
||||
"no_candidate"
|
||||
],
|
||||
[
|
||||
"vfc-js",
|
||||
"hot-formula-parser-3-0-0",
|
||||
"no_candidate"
|
||||
],
|
||||
[
|
||||
"vfc-js",
|
||||
"pdf-image-1-0-5",
|
||||
"no_candidate"
|
||||
],
|
||||
[
|
||||
"vfc-js",
|
||||
"pidusage-1-0-0",
|
||||
"no_candidate"
|
||||
],
|
||||
[
|
||||
"vfc-js",
|
||||
"portprocesses-1-0-4",
|
||||
"no_candidate"
|
||||
],
|
||||
[
|
||||
"vfc-js",
|
||||
"serve-here-js-1-1-0",
|
||||
"no_candidate"
|
||||
]
|
||||
],
|
||||
"generated": "2026-09-06",
|
||||
"generator": "ir",
|
||||
"note": "constrained-detector task 1.3, the stop-or-go reading. A candidate-driven detector can only judge sites the enumerator produces, so this is the limit on its recall, measured offline before any model was called. The condition for continuing: on every deciding slice the ceiling must exceed that slice's recorded detection rate from round zero. It does, with the largest margin exactly where it matters most.",
|
||||
"per_family": {
|
||||
"access_control": {
|
||||
"ceiling": 0.9,
|
||||
"pairs": 10,
|
||||
"sites": {
|
||||
"max": 19,
|
||||
"median": 12,
|
||||
"min": 4,
|
||||
"p90": 19
|
||||
},
|
||||
"with_candidate": 9
|
||||
},
|
||||
"config_secrets": {
|
||||
"ceiling": 1.0,
|
||||
"pairs": 11,
|
||||
"sites": {
|
||||
"max": 60,
|
||||
"median": 15,
|
||||
"min": 2,
|
||||
"p90": 40
|
||||
},
|
||||
"with_candidate": 11
|
||||
},
|
||||
"deserialization": {
|
||||
"ceiling": 1.0,
|
||||
"pairs": 4,
|
||||
"sites": {
|
||||
"max": 10,
|
||||
"median": 7,
|
||||
"min": 3,
|
||||
"p90": 10
|
||||
},
|
||||
"with_candidate": 4
|
||||
},
|
||||
"injection": {
|
||||
"ceiling": 0.8461538461538461,
|
||||
"pairs": 52,
|
||||
"sites": {
|
||||
"max": 77,
|
||||
"median": 5,
|
||||
"min": 1,
|
||||
"p90": 17
|
||||
},
|
||||
"with_candidate": 44
|
||||
},
|
||||
"memory": {
|
||||
"ceiling": 0.8636363636363636,
|
||||
"pairs": 176,
|
||||
"sites": {
|
||||
"max": 77,
|
||||
"median": 10,
|
||||
"min": 1,
|
||||
"p90": 29
|
||||
},
|
||||
"with_candidate": 152
|
||||
},
|
||||
"output_encoding": {
|
||||
"ceiling": 1.0,
|
||||
"pairs": 2,
|
||||
"sites": {
|
||||
"max": 17,
|
||||
"median": 17,
|
||||
"min": 3,
|
||||
"p90": 17
|
||||
},
|
||||
"with_candidate": 2
|
||||
},
|
||||
"path": {
|
||||
"ceiling": 0.875,
|
||||
"pairs": 8,
|
||||
"sites": {
|
||||
"max": 12,
|
||||
"median": 7,
|
||||
"min": 5,
|
||||
"p90": 12
|
||||
},
|
||||
"with_candidate": 7
|
||||
},
|
||||
"prototype": {
|
||||
"ceiling": 0.7142857142857143,
|
||||
"pairs": 7,
|
||||
"sites": {
|
||||
"max": 14,
|
||||
"median": 6,
|
||||
"min": 6,
|
||||
"p90": 14
|
||||
},
|
||||
"with_candidate": 5
|
||||
},
|
||||
"unknown": {
|
||||
"ceiling": 0.0,
|
||||
"pairs": 2,
|
||||
"sites": {
|
||||
"max": 0,
|
||||
"median": 0,
|
||||
"min": 0,
|
||||
"p90": 0
|
||||
},
|
||||
"with_candidate": 0
|
||||
},
|
||||
"untrusted_destination": {
|
||||
"ceiling": 1.0,
|
||||
"pairs": 5,
|
||||
"sites": {
|
||||
"max": 17,
|
||||
"median": 7,
|
||||
"min": 3,
|
||||
"p90": 17
|
||||
},
|
||||
"with_candidate": 5
|
||||
}
|
||||
},
|
||||
"per_slice": {
|
||||
"agent-vfc": {
|
||||
"ceiling": 0.896551724137931,
|
||||
"pairs": 29,
|
||||
"with_candidate": 26
|
||||
},
|
||||
"github": {
|
||||
"ceiling": 1.0,
|
||||
"pairs": 6,
|
||||
"with_candidate": 6
|
||||
},
|
||||
"local": {
|
||||
"ceiling": 1.0,
|
||||
"pairs": 3,
|
||||
"with_candidate": 3
|
||||
},
|
||||
"sast": {
|
||||
"ceiling": 1.0,
|
||||
"pairs": 11,
|
||||
"with_candidate": 11
|
||||
},
|
||||
"vfc": {
|
||||
"ceiling": 0.8465909090909091,
|
||||
"pairs": 176,
|
||||
"with_candidate": 149
|
||||
},
|
||||
"vfc-js": {
|
||||
"ceiling": 0.5294117647058824,
|
||||
"pairs": 17,
|
||||
"with_candidate": 9
|
||||
},
|
||||
"vibe-py": {
|
||||
"ceiling": 1.0,
|
||||
"pairs": 35,
|
||||
"with_candidate": 35
|
||||
}
|
||||
},
|
||||
"shape_correction": "config_secrets first measured 63.6% with a bind-and-literal shape. A permissive default is as often an argument to a call -- cors({origin: '*'}) -- as a binding, and adding call sites to that family's shape took it to 100%. Found by this measurement, before the detector existed.",
|
||||
"verdict": "go"
|
||||
}
|
||||
@@ -213,6 +213,7 @@ def main(argv: list[str] | None = None) -> int:
|
||||
("classify", "classify every labeled pair and report the classifier against itself"),
|
||||
("score", "score a catalog by family, with the denominator every number was computed over"),
|
||||
("baseline", "round zero: clone one detector into every family and measure each against itself"),
|
||||
("candidates", "offline: what a candidate-driven detector could find, per family and per slice"),
|
||||
("round", "one evolve round: propose one change for one family and let the evidence decide"),
|
||||
("publish", "regenerate every published number from the artifacts that produced it"),
|
||||
):
|
||||
@@ -222,6 +223,8 @@ def main(argv: list[str] | None = None) -> int:
|
||||
command.add_argument("--catalog", type=Path, default=DEFAULT_CATALOG)
|
||||
command.add_argument("--slice", choices=SLICE_NAMES, default="all")
|
||||
command.add_argument("--json", action="store_true")
|
||||
if name == "candidates":
|
||||
command.add_argument("--generator", choices=("ir", "ruleset"), default="ir", help="which pass enumerates the candidate sites")
|
||||
if name in {"baseline", "round", "score"}:
|
||||
command.add_argument("--k-runs", type=int, default=None, help="runs per pair; never fewer than three")
|
||||
command.add_argument("--model", default=None, help="detector model; artifacts are keyed by it")
|
||||
@@ -1238,6 +1241,26 @@ def _learning(args: argparse.Namespace) -> int:
|
||||
print(f" artifact {artifact}")
|
||||
return 0
|
||||
cases = select_vendored(select_slice(load_pair_catalog(args.catalog), args.slice))
|
||||
if args.learning_command == "candidates":
|
||||
# No model is called anywhere in this command. The ceiling is a property of the corpus and the enumerator,
|
||||
# so it can end this feature before a cent is spent on it (constrained-detector task 1.3).
|
||||
from .learning.candidates import ceiling
|
||||
|
||||
coverage = ceiling(cases, taxonomy=taxonomy, generator=args.generator)
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
payload = coverage.to_dict()
|
||||
(out / "candidate-ceiling.json").write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
|
||||
if args.json:
|
||||
print(json.dumps(payload, indent=2, sort_keys=True))
|
||||
else:
|
||||
for family, block in sorted(coverage.per_family.items()):
|
||||
print(
|
||||
f"learning candidates {family}: {block['with_candidate']}/{block['pairs']} "
|
||||
f"= {_as_float(block['ceiling']):.1%} ceiling, sites {block['sites']}"
|
||||
)
|
||||
for slice_name, block in sorted(coverage.per_slice.items()):
|
||||
print(f" slice {slice_name}: {_as_float(block['ceiling']):.1%} over {block['pairs']} pairs")
|
||||
return 0
|
||||
if args.learning_command == "classify":
|
||||
measured = measure_classifier(cases, taxonomy, client=None)
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
@@ -1358,6 +1381,10 @@ def _learning_run(args: argparse.Namespace, cases: Sequence[PairCase], taxonomy:
|
||||
return 0
|
||||
|
||||
|
||||
def _as_float(value: object) -> float:
|
||||
return float(value) if isinstance(value, int | float) else 0.0
|
||||
|
||||
|
||||
def _label(case: PairCase, taxonomy: object) -> str:
|
||||
"""The family that routes this pair: the maintainer's label when there is one, else the classifier's answer.
|
||||
|
||||
|
||||
@@ -33,7 +33,10 @@ FAMILY_SHAPE: dict[str, tuple[CandidateKind, ...]] = {
|
||||
"memory": ("call",),
|
||||
"prototype": ("call", "bind"),
|
||||
"access_control": ("operation",),
|
||||
"config_secrets": ("bind", "config"),
|
||||
# A permissive default is as often an argument to a call — `cors({origin: "*"})`, `app.use(session({secure:
|
||||
# false}))` — as it is a binding. Measured: with bindings alone the shape missed `createApplication` and
|
||||
# `getAllowedHosts`, whose whole bug is inside a call argument.
|
||||
"config_secrets": ("call", "bind", "config"),
|
||||
"unknown": ("call", "bind"),
|
||||
}
|
||||
|
||||
@@ -232,22 +235,113 @@ def _language_of(path: str) -> str:
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CandidateReport:
|
||||
"""The ceiling: what a candidate-driven detector could find before any model is asked anything."""
|
||||
"""The ceiling: what a candidate-driven detector could find before any model is asked anything.
|
||||
|
||||
This is the number that decides whether the design continues. A generator that cannot put a candidate inside
|
||||
the labeled function can never find that pair's bug however good the judgement is, so the ceiling is published
|
||||
as the limit on recall rather than discovered later as a disappointment (Req 1.3)."""
|
||||
|
||||
generator: str = "ir"
|
||||
per_family: dict[str, dict[str, object]] = field(default_factory=dict)
|
||||
per_slice: dict[str, dict[str, object]] = field(default_factory=dict)
|
||||
gaps: tuple[tuple[str, str, str], ...] = () # (slice, pair, reason)
|
||||
|
||||
def to_dict(self) -> dict[str, object]:
|
||||
return {
|
||||
"generator": self.generator,
|
||||
"per_family": {name: dict(block) for name, block in sorted(self.per_family.items())},
|
||||
"per_slice": {name: dict(block) for name, block in sorted(self.per_slice.items())},
|
||||
"gaps": [list(row) for row in self.gaps],
|
||||
}
|
||||
|
||||
|
||||
def ceiling(cases: Sequence[object], *, taxonomy: object, generator: str = "ir") -> CandidateReport:
|
||||
"""Per family and per slice, the fraction of labeled functions a generator can put a candidate inside.
|
||||
|
||||
``generator`` is ``ir`` or ``ruleset``. The ruleset arm exists so the comparison that rejected it — 12.1%
|
||||
against 96.6% on the web slices — stays a measurement rather than a remembered number."""
|
||||
import tempfile
|
||||
|
||||
from ..pairs import _materialize_side, _overlay_scan, _quick_scan
|
||||
from .rounds import _family_of
|
||||
from .scoring import _spans
|
||||
|
||||
families: dict[str, dict[str, int]] = {}
|
||||
slices: dict[str, dict[str, int]] = {}
|
||||
gaps: list[tuple[str, str, str]] = []
|
||||
sites: dict[str, list[int]] = {}
|
||||
for case in cases:
|
||||
family = _family_of(case, taxonomy) # type: ignore[arg-type]
|
||||
slice_name = str(getattr(case, "slice", "") or "")
|
||||
region_path = str(getattr(case, "relpath", "") or "")
|
||||
with tempfile.TemporaryDirectory(prefix="ousast-ceiling-") as scratch:
|
||||
root = _materialize_side(Path(scratch) / "vuln", case, side="vuln") # type: ignore[arg-type]
|
||||
if generator == "ruleset":
|
||||
ranges = _overlay_scan(root).ranges
|
||||
spans = _spans(case, ranges)
|
||||
found = [
|
||||
finding
|
||||
for finding in _quick_scan(root)
|
||||
if finding.line is not None and any(path == finding.path and start <= finding.line <= end for path, start, end in spans)
|
||||
]
|
||||
count, reason = len(found), ("" if found else "no_candidate")
|
||||
else:
|
||||
labeled = [str(getattr(row, "function", "") or "") for row in getattr(case, "expected", ()) or ()]
|
||||
result = enumerate_candidates(
|
||||
root,
|
||||
_Region(region_path, next((name for name in labeled if name), None)),
|
||||
family,
|
||||
taxonomy=taxonomy,
|
||||
limit=10**6, # the ceiling is about coverage, never about the batch bound
|
||||
)
|
||||
count, reason = len(result.candidates), result.reason
|
||||
for bucket, key in ((families, family), (slices, slice_name)):
|
||||
block = bucket.setdefault(key, {"pairs": 0, "with_candidate": 0})
|
||||
block["pairs"] += 1
|
||||
block["with_candidate"] += 1 if count else 0
|
||||
if count:
|
||||
sites.setdefault(family, []).append(count)
|
||||
else:
|
||||
gaps.append((slice_name, str(getattr(case, "name", "")), reason or "no_candidate"))
|
||||
per_family: dict[str, dict[str, object]] = {}
|
||||
per_slice: dict[str, dict[str, object]] = {}
|
||||
for bucket, out in ((families, per_family), (slices, per_slice)):
|
||||
for key, block in bucket.items():
|
||||
pairs = block["pairs"]
|
||||
row: dict[str, object] = {
|
||||
"pairs": pairs,
|
||||
"with_candidate": block["with_candidate"],
|
||||
"ceiling": (block["with_candidate"] / pairs) if pairs else 0.0,
|
||||
}
|
||||
if bucket is families:
|
||||
row["sites"] = _distribution(sites.get(key, []))
|
||||
out[key] = row
|
||||
return CandidateReport(generator=generator, per_family=per_family, per_slice=per_slice, gaps=tuple(sorted(gaps)))
|
||||
|
||||
|
||||
def _distribution(counts: list[int]) -> dict[str, int]:
|
||||
if not counts:
|
||||
return {"min": 0, "median": 0, "p90": 0, "max": 0}
|
||||
counts = sorted(counts)
|
||||
return {
|
||||
"min": counts[0],
|
||||
"median": counts[len(counts) // 2],
|
||||
"p90": counts[min(int(len(counts) * 0.9), len(counts) - 1)],
|
||||
"max": counts[-1],
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class _Region:
|
||||
"""The two fields `enumerate_candidates` reads, so the ceiling does not import the detector module."""
|
||||
|
||||
path: str
|
||||
function: str | None
|
||||
|
||||
|
||||
__all__ = [
|
||||
"FAMILY_SHAPE",
|
||||
"ceiling",
|
||||
"MAX_CANDIDATES_PER_CALL",
|
||||
"Candidate",
|
||||
"CandidateKind",
|
||||
|
||||
@@ -17,6 +17,7 @@ sys.path.insert(0, str(Path(__file__).parent))
|
||||
from test_obligation_operations import APP # noqa: E402
|
||||
|
||||
from openultrasast.learning.families import load_families # noqa: E402
|
||||
from openultrasast.semantic.extra import has_semantic_extra # noqa: E402
|
||||
|
||||
INJECTION = (
|
||||
"from flask import request\n"
|
||||
@@ -71,6 +72,8 @@ def test_the_family_decides_which_kinds_are_candidates() -> None:
|
||||
assert "call" in FAMILY_SHAPE["prototype"]
|
||||
|
||||
|
||||
@pytest.mark.semantic
|
||||
@pytest.mark.skipif(not has_semantic_extra(), reason="the guards come from the parsed IR")
|
||||
def test_access_control_candidates_are_operations_with_the_guards_in_scope(tmp_path: Path) -> None:
|
||||
"""An absence bug has no sink. Its candidate is the operation, and what the model needs is what guards it."""
|
||||
from openultrasast.learning.candidates import enumerate_candidates
|
||||
@@ -154,3 +157,82 @@ def test_every_family_shape_enumerates_something_on_a_file_that_has_it(tmp_path:
|
||||
assert result.reason in {"", "no_candidate"}
|
||||
if family in {"injection", "path", "deserialization", "untrusted_destination"}:
|
||||
assert result.candidates, f"{family} takes call sites and this file has three"
|
||||
|
||||
|
||||
# --- the ceiling (task 1.2) ---------------------------------------------------
|
||||
|
||||
|
||||
def test_the_ceiling_counts_labeled_functions_with_at_least_one_candidate(tmp_path: Path) -> None:
|
||||
from openultrasast.benchmark import ExpectedFinding
|
||||
from openultrasast.learning.candidates import ceiling
|
||||
from openultrasast.pairs import PairCase
|
||||
|
||||
def case(name: str, body: str, *, slice_name: str, family: str) -> PairCase:
|
||||
(tmp_path / f"{name}.py").write_text(body)
|
||||
return PairCase(
|
||||
name=name,
|
||||
slice=slice_name,
|
||||
language="python",
|
||||
origin="test",
|
||||
vuln_file=tmp_path / f"{name}.py",
|
||||
fixed_file=tmp_path / f"{name}.py",
|
||||
relpath="app.py",
|
||||
expected=(
|
||||
ExpectedFinding(
|
||||
cwe="CWE-78",
|
||||
vulnerability_class="x",
|
||||
path="app.py",
|
||||
evidence="",
|
||||
function="run",
|
||||
mechanism="other",
|
||||
family=family,
|
||||
),
|
||||
),
|
||||
min_recall=1.0,
|
||||
fix_policy="silent",
|
||||
review_tier="seeded",
|
||||
)
|
||||
|
||||
cases = [
|
||||
case("has", "import os\n\n\ndef run(cmd):\n os.system(cmd)\n", slice_name="vibe-py", family="injection"),
|
||||
case("empty", "def run():\n return 1\n", slice_name="vibe-py", family="injection"),
|
||||
case("real", "import os\n\n\ndef run(cmd):\n os.system(cmd)\n", slice_name="agent-vfc", family="injection"),
|
||||
]
|
||||
report = ceiling(cases, taxonomy=load_families())
|
||||
assert report.per_family["injection"]["pairs"] == 3
|
||||
assert report.per_family["injection"]["with_candidate"] == 2
|
||||
assert report.per_family["injection"]["ceiling"] == pytest.approx(2 / 3)
|
||||
assert report.per_slice["vibe-py"]["ceiling"] == pytest.approx(0.5)
|
||||
assert report.per_slice["agent-vfc"]["ceiling"] == 1.0
|
||||
assert ("vibe-py", "empty", "no_candidate") in report.gaps
|
||||
assert report.per_family["injection"]["sites"]["median"] >= 1
|
||||
|
||||
|
||||
def test_the_ruleset_generator_is_measured_too_so_the_comparison_is_reproducible(tmp_path: Path) -> None:
|
||||
"""12.1% against 96.6% is why the ruleset is not the generator. A remembered number is not a measurement."""
|
||||
from openultrasast.learning.candidates import ceiling
|
||||
from openultrasast.pairs import DEFAULT_CATALOG, load_pair_catalog, select_slice, select_vendored
|
||||
|
||||
cases = [c for c in select_vendored(select_slice(load_pair_catalog(DEFAULT_CATALOG), "vibe-py")) if not c.unscorable][:6]
|
||||
ir = ceiling(cases, taxonomy=load_families(), generator="ir")
|
||||
ruleset = ceiling(cases, taxonomy=load_families(), generator="ruleset")
|
||||
assert ir.generator == "ir" and ruleset.generator == "ruleset"
|
||||
assert sum(b["with_candidate"] for b in ir.per_family.values()) >= sum(b["with_candidate"] for b in ruleset.per_family.values())
|
||||
|
||||
|
||||
def test_the_command_runs_offline_and_writes_a_stable_artifact(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
from openultrasast.cli import main
|
||||
|
||||
for name in ("OPENULTRASAST_HUNTER_CLIENT", "DEEPSEEK_API_KEY", "OPENROUTER_API_KEY"):
|
||||
monkeypatch.delenv(name, raising=False)
|
||||
out = tmp_path / "learning"
|
||||
assert main(["learning", "candidates", "--slice", "vibe-py", "--out", str(out)]) == 0
|
||||
artifact = out / "candidate-ceiling.json"
|
||||
assert artifact.is_file()
|
||||
first = artifact.read_bytes()
|
||||
assert main(["learning", "candidates", "--slice", "vibe-py", "--out", str(out)]) == 0
|
||||
assert artifact.read_bytes() == first, "the ceiling is a property of the corpus, not of when it was measured"
|
||||
import json
|
||||
|
||||
payload = json.loads(first)
|
||||
assert payload["generator"] == "ir" and payload["per_family"] and payload["per_slice"]
|
||||
|
||||
@@ -21,6 +21,7 @@ from openultrasast.learning.families import load_families
|
||||
from openultrasast.mapping import analyze_entry_points
|
||||
from openultrasast.pairs import DEFAULT_CATALOG, load_pair_catalog, select_vendored
|
||||
from openultrasast.preprocess import preprocess_repository
|
||||
from openultrasast.semantic.extra import has_semantic_extra
|
||||
|
||||
LIB = Path("benchmarks/pairs/harvest.py")
|
||||
FAMILIES = Path("src/openultrasast/ruleset/families.toml")
|
||||
@@ -444,6 +445,8 @@ def test_a_javascript_excerpt_never_starts_in_the_middle_of_an_import() -> None:
|
||||
assert "from '@overleaf" not in excerpt
|
||||
|
||||
|
||||
@pytest.mark.semantic
|
||||
@pytest.mark.skipif(not has_semantic_extra(), reason="parsing JavaScript needs the tree-sitter grammar")
|
||||
def test_the_overleaf_excerpt_parses_on_both_sides() -> None:
|
||||
from openultrasast.semantic.ir import parse_file
|
||||
|
||||
|
||||
Reference in New Issue
Block a user