mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-18 12:39:28 +00:00
Marius CiepluchandClaude Opus 4.8
81a63cf5ac
Written after five evolve rounds produced zero decisions distinguishable from noise, and after reading Clearwing, Big Sleep, and the AIxCC 2025 finals. The premise we got wrong: OpenUltraSAST treats the LLM as a noisy oracle to average (K runs), floor (noise budget), gate (sign test) and slowly improve (evolve loop). 120 runs produced 120 distinct trajectories -- the variance is the loop's freedom to search, not the model's uncertainty. And the premise is empirically false as a claim about LLMs: Big Sleep found a real SQLite zero-day in Nov 2024; AIxCC finalists uncovered 18 real 0-days across 54M LOC, with the 2nd-place system using non-reasoning LLMs at 90% accuracy and $181/point. The lever is architecture, not prompt optimization. What the winners share, and Clearwing is the open instance of: the LLM is a judgement layer on an execution substrate, and truth comes from the sandbox -- a crash, a sanitizer report, a patch that stops the crash -- not from a score against a labeled corpus. Clearwing's poc_runner (apply diff, recompile, re-run, does-the-crash-still-happen) replaces our entire noise architecture. We built K-runs because we had no ground truth; they generate their own. The course: adopt Clearwing as the substrate (the maintainer already forks it), delete the noise architecture rather than fix it, and contribute the one thing Clearwing lacks -- the web/logic detection path. Our IR candidate enumerator (12.1% ruleset vs 96.6% IR, measured) and our obligation/absence-bug checker reach the dangerous web bugs that never crash a sandbox, which Clearwing's crash-oracle model cannot. The open research problem is a differential oracle for those classes: the vulnerable-parent/fix-commit pair as ground truth, with a request the fixed version rejects and the vulnerable one serves. Recommendation is explicitly not to rebuild: eighteen defects this session, all ours, against a 19k-line battle-tested MIT substrate. Not approved. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
28 lines
2.0 KiB
JSON
28 lines
2.0 KiB
JSON
{
|
|
"feature_name": "constrained-detector",
|
|
"created_at": "2026-09-06T14:30:00Z",
|
|
"updated_at": "2026-09-08T00:00:00Z",
|
|
"language": "en",
|
|
"phase": "tasks",
|
|
"approvals": {
|
|
"requirements": {
|
|
"generated": true,
|
|
"approved": true
|
|
},
|
|
"design": {
|
|
"generated": true,
|
|
"approved": true
|
|
},
|
|
"tasks": {
|
|
"generated": true,
|
|
"approved": false
|
|
}
|
|
},
|
|
"ready_for_implementation": false,
|
|
"depends_on": [
|
|
"learning-harness"
|
|
],
|
|
"approval_note": "2026-09-06 brief drafted at the maintainer's direction ('I'd rather spend time to design the exact DSPy style interface and evolutionary / statistical prompt optimization than fixing piece by piece'). Not approved; requirements not generated.; 2026-09-06 renamed from detector-optimizer to constrained-detector after the trajectory measurement: the brief's own conclusion is that the optimiser is not the next thing to build, and the spec should be named for what it delivers; 2026-09-06 requirements generated at the maintainer's direction, with the maintainer's emphasis recorded as Requirement 3 ('don't forget the emphasis on the real world bugs from the larger projects to avoid overfitting. we otherwise benchmark for cases that are not relevant'); 2026-09-06 requirements approved by the maintainer in session ('approve the requirements and run the design phase'); 2026-09-06 design approved by the maintainer in session ('approve the design and generate the tasks'); 2026-09-08 SUPERSEDED by execution-grounded-detection: this session's measurements showed the noise architecture (K-runs, per-family floor, regression budget, sign test, evolve loop) is a statistical treatment for a structural problem, and that the field (Big Sleep, AIxCC finals, Clearwing) grounds detection in execution rather than in scoring against a labeled corpus. The taxonomy, the IR candidate enumerator (96.6% ceiling) and the obligation checker survive as the web/logic detection path to contribute to a Clearwing-based substrate.",
|
|
"superseded_by": "execution-grounded-detection"
|
|
}
|