mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-18 12:39:28 +00:00
Marius CiepluchandClaude Opus 4.8
81a63cf5ac
Written after five evolve rounds produced zero decisions distinguishable from noise, and after reading Clearwing, Big Sleep, and the AIxCC 2025 finals. The premise we got wrong: OpenUltraSAST treats the LLM as a noisy oracle to average (K runs), floor (noise budget), gate (sign test) and slowly improve (evolve loop). 120 runs produced 120 distinct trajectories -- the variance is the loop's freedom to search, not the model's uncertainty. And the premise is empirically false as a claim about LLMs: Big Sleep found a real SQLite zero-day in Nov 2024; AIxCC finalists uncovered 18 real 0-days across 54M LOC, with the 2nd-place system using non-reasoning LLMs at 90% accuracy and $181/point. The lever is architecture, not prompt optimization. What the winners share, and Clearwing is the open instance of: the LLM is a judgement layer on an execution substrate, and truth comes from the sandbox -- a crash, a sanitizer report, a patch that stops the crash -- not from a score against a labeled corpus. Clearwing's poc_runner (apply diff, recompile, re-run, does-the-crash-still-happen) replaces our entire noise architecture. We built K-runs because we had no ground truth; they generate their own. The course: adopt Clearwing as the substrate (the maintainer already forks it), delete the noise architecture rather than fix it, and contribute the one thing Clearwing lacks -- the web/logic detection path. Our IR candidate enumerator (12.1% ruleset vs 96.6% IR, measured) and our obligation/absence-bug checker reach the dangerous web bugs that never crash a sandbox, which Clearwing's crash-oracle model cannot. The open research problem is a differential oracle for those classes: the vulnerable-parent/fix-commit pair as ground truth, with a request the fixed version rejects and the vulnerable one serves. Recommendation is explicitly not to rebuild: eighteen defects this session, all ours, against a 19k-line battle-tested MIT substrate. Not approved. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
32 lines
2.8 KiB
JSON
32 lines
2.8 KiB
JSON
{
|
|
"feature_name": "learning-harness",
|
|
"created_at": "2026-09-06T06:32:41Z",
|
|
"updated_at": "2026-09-08T00:00:00Z",
|
|
"language": "en",
|
|
"phase": "implementation",
|
|
"approvals": {
|
|
"requirements": {
|
|
"generated": true,
|
|
"approved": true
|
|
},
|
|
"design": {
|
|
"generated": true,
|
|
"approved": true
|
|
},
|
|
"tasks": {
|
|
"generated": true,
|
|
"approved": true
|
|
}
|
|
},
|
|
"ready_for_implementation": true,
|
|
"depends_on": [
|
|
"pair-corpus-honesty",
|
|
"corpus-seeded-mechanisms",
|
|
"authorization-obligations",
|
|
"tree-sitter-overlay-extra",
|
|
"harnessx-self-improving-rulesets"
|
|
],
|
|
"approval_note": "2026-09-06 requirements approved by the maintainer in session ('approve the requirements and run the design phase'); 2026-09-06 design approved by the maintainer in session ('approve the design and generate the tasks'); 2026-09-06 tasks approved by the maintainer in session ('approve the tasks and go on with the implementation'); 2026-09-06 requirements amended by the maintainer in session ('amend the requirements to untrusted_destination and note the flat taxonomy'): Req 1.1 names untrusted destination instead of server-side request forgery, and Req 2.7 and 3.7 require every agreement and family figure to state that hierarchical partial credit could not apply while the taxonomy is flat; 2026-09-06 Req 6.1 amended by the maintainer in session ('drop the skills set from the requirement'): a detector configuration has no skills set, because the skill router belongs to the tool's earlier shape and a family's prompt, checklist, tools and counterexamples already carry what a round evolves; 2026-09-06 design amended during task 4.1 (recorded in design.md): a recipe may name cross-file registration documents, because connexion registers by operationId in an OpenAPI file and express in a router module, so Req 10.2 is unreachable from a single-file excerpt for those rows; 2026-09-06 task 5.3 amended by the maintainer in session (asked and answered): the first evolve rounds target `injection` rather than `access_control`, because under the split and review-tier rules access_control has one train teacher and four holdout pairs and teaches zero obligation shapes, while injection has twelve teachers and ten holdout pairs; 2026-09-08 SUPERSEDED by execution-grounded-detection: this session's measurements showed the noise architecture (K-runs, per-family floor, regression budget, sign test, evolve loop) is a statistical treatment for a structural problem, and that the field (Big Sleep, AIxCC finals, Clearwing) grounds detection in execution rather than in scoring against a labeled corpus. The taxonomy, the IR candidate enumerator (96.6% ceiling) and the obligation checker survive as the web/logic detection path to contribute to a Clearwing-based substrate.",
|
|
"superseded_by": "execution-grounded-detection"
|
|
}
|