mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-19 13:10:03 +00:00
Marius CiepluchandClaude Opus 4.8
a43e6293f5
Widened the banks label-mislocation finding across all 49 public pool vulns via DeepSeek (advisory + head code + pool label -> sound/unsound), $0.038 total. Result: 23/49 (47%) sound; 53% unsound (25 pre-exist in base, 16 sink outside the changed lines) -- a change-scoped hook correctly stays quiet there. Systematic by family: injection 10/13 sound, but path 1/7, output_encoding 1/8, config_secrets 1/4. Implications: record-07's floor (catch 3/92) is scored against a corpus ~half unfair to a delta hook; the honest catchable denominator is the sound subset (caveat added to record 07 NOTES). Fair pilot re-score: only bedrock + graphiti (injection) of the 5 are sound and the engine missed both = the real detection gaps for the LLM-guided hunter. Also: GEPA-style reflective-retry (future spec) + GLM cyber model (future spec) noted; DeepSeek is the working default. Artifact: benchmarks/measurements/2026-10-12-pool-p1-label-audit/. Co-Authored-By: Claude Opus 4.8 <[email protected]>
2.5 KiB
2.5 KiB
Record 07 — slice-1 unseen-repo baseline (G0 exit)
Measured 2026-10-11. Commit: e716e3beab
Image: ghcr.io/norandom/ousast-replay-task@sha256:a24fd03c6ccac0852d495e8a1931e630fffc866ca9e82dd81011533da413f783
Pool: benchmarks/unseen/pool-p1.toml (100 frozen never-seen repos, freeze_digest 3371c67d…), slice 1, 1300 changes.
Ledger: benchmarks/unseen/usage-p1.jsonl — one baseline row (this is G0's exit).
Headline (DEFAULT arm, docker lane)
- catch (recall): 3/92 = 3.3% (Wilson95 1.1%–9.2%)
- false alarm: 3.1% (cluster95 1.6%–5.4%, icc 0.25 — repo clustering real)
- alerts_only: 0/1154 = 0% (no hard alerts; catches are advisory/quick stream)
- coverage: quick_only 1246, engine_covered 0, unanalysable 0, instrument_failure 54 (4.2%, < 5% guard)
- by family (catch): deserialization 1/4, injection 2/20, all others 0
- unseen gate: not_run (no paired production arm; baseline only)
Caveats (read before citing)
- Machine-label: labels are SZZ-derived from fix commits, 100 unchecked by a human. Catch/FP are vs machine labels.
- Regex-only floor: model="disabled", max_model_calls=0, cost_usd=0 — NO LLM; and engine_covered=0 — the Joern engine never completed within the 30s default budget (the ~200s fixed-overhead wall, see .kiro/specs/engine-change-scoped-analysis/brief.md). So this is the quick pattern-rule tier ALONE, the floor, not the product. It is NOT the ≥90%/<10% gate and must not be read as release-grade detection.
- Docker lane, local (not the AX/kube lane; the AX completion path is operator-owned).
- Sizing: 100 vulns is ~±10% for one overall catch number; per-family (deser=4, config=10) is far too thin for family-level decisions — needs slices 2+.
- Raw per-change records streamed to the RustFS store; local subdirs pruned-as-we-go (records + inputs.json kept).
- CORPUS-LABEL CAVEAT (added 2026-10-12): an LLM audit of the 49 public pool vulns found only 47% (23/49)
of labels are sound (sink in the changed lines AND change-introduced); 53% pre-exist in base or are mislocated,
where a change-scoped hook correctly stays quiet. So this catch rate is scored against a corpus ~half of which
is unfair to a delta hook — the honest catchable denominator is the sound subset, and the floor should be
recomputed on it. By family the mislocation is systematic (path 1/7, output_encoding 1/8 sound; injection 10/13).
See
benchmarks/measurements/2026-10-12-pool-p1-label-audit/.