mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-19 13:10:03 +00:00
Marius CiepluchandClaude Opus 4.8
a43e6293f5
Widened the banks label-mislocation finding across all 49 public pool vulns via DeepSeek (advisory + head code + pool label -> sound/unsound), $0.038 total. Result: 23/49 (47%) sound; 53% unsound (25 pre-exist in base, 16 sink outside the changed lines) -- a change-scoped hook correctly stays quiet there. Systematic by family: injection 10/13 sound, but path 1/7, output_encoding 1/8, config_secrets 1/4. Implications: record-07's floor (catch 3/92) is scored against a corpus ~half unfair to a delta hook; the honest catchable denominator is the sound subset (caveat added to record 07 NOTES). Fair pilot re-score: only bedrock + graphiti (injection) of the 5 are sound and the engine missed both = the real detection gaps for the LLM-guided hunter. Also: GEPA-style reflective-retry (future spec) + GLM cyber model (future spec) noted; DeepSeek is the working default. Artifact: benchmarks/measurements/2026-10-12-pool-p1-label-audit/. Co-Authored-By: Claude Opus 4.8 <[email protected]>
32 lines
2.5 KiB
Markdown
32 lines
2.5 KiB
Markdown
# Record 07 — slice-1 unseen-repo baseline (G0 exit)
|
||
|
||
Measured 2026-10-11. Commit: e716e3beab8b01f63ac59dd49586b5904231e2a4
|
||
Image: ghcr.io/norandom/ousast-replay-task@sha256:a24fd03c6ccac0852d495e8a1931e630fffc866ca9e82dd81011533da413f783
|
||
Pool: benchmarks/unseen/pool-p1.toml (100 frozen never-seen repos, freeze_digest 3371c67d…), slice 1, 1300 changes.
|
||
Ledger: benchmarks/unseen/usage-p1.jsonl — one `baseline` row (this is G0's exit).
|
||
|
||
## Headline (DEFAULT arm, docker lane)
|
||
- catch (recall): 3/92 = 3.3% (Wilson95 1.1%–9.2%)
|
||
- false alarm: 3.1% (cluster95 1.6%–5.4%, icc 0.25 — repo clustering real)
|
||
- alerts_only: 0/1154 = 0% (no hard alerts; catches are advisory/quick stream)
|
||
- coverage: quick_only 1246, engine_covered 0, unanalysable 0, instrument_failure 54 (4.2%, < 5% guard)
|
||
- by family (catch): deserialization 1/4, injection 2/20, all others 0
|
||
- unseen gate: not_run (no paired production arm; baseline only)
|
||
|
||
## Caveats (read before citing)
|
||
- **Machine-label**: labels are SZZ-derived from fix commits, 100 unchecked by a human. Catch/FP are vs machine labels.
|
||
- **Regex-only floor**: model="disabled", max_model_calls=0, cost_usd=0 — NO LLM; and engine_covered=0 — the Joern
|
||
engine never completed within the 30s default budget (the ~200s fixed-overhead wall, see
|
||
.kiro/specs/engine-change-scoped-analysis/brief.md). So this is the quick pattern-rule tier ALONE, the floor,
|
||
not the product. It is NOT the ≥90%/<10% gate and must not be read as release-grade detection.
|
||
- **Docker lane, local** (not the AX/kube lane; the AX completion path is operator-owned).
|
||
- **Sizing**: 100 vulns is ~±10% for one overall catch number; per-family (deser=4, config=10) is far too thin for
|
||
family-level decisions — needs slices 2+.
|
||
- Raw per-change records streamed to the RustFS store; local subdirs pruned-as-we-go (records + inputs.json kept).
|
||
- **CORPUS-LABEL CAVEAT (added 2026-10-12):** an LLM audit of the 49 public pool vulns found only **47% (23/49)
|
||
of labels are sound** (sink in the changed lines AND change-introduced); 53% pre-exist in base or are mislocated,
|
||
where a change-scoped hook *correctly* stays quiet. So this catch rate is scored against a corpus ~half of which
|
||
is unfair to a delta hook — the honest catchable denominator is the sound subset, and the floor should be
|
||
recomputed on it. By family the mislocation is systematic (path 1/7, output_encoding 1/8 sound; injection 10/13).
|
||
See `benchmarks/measurements/2026-10-12-pool-p1-label-audit/`.
|