mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-18 20:49:28 +00:00
docs(learning-harness): record the maintainer's scope and decisions; initialize the spec and generate requirements
Scope: catch the ten taxonomy families reliably on real-world and generated application code, including forgotten authorization checks; not browser engines, interpreters or memory-safety fuzzing. Decisions: verifier-aligned families, a measured per-family noise floor as the regression budget, a cheap model on an independent chat endpoint with embeddings elsewhere, canary-oracle verifiers for the four most dangerous families first, the obligations lever folded into the access-control family's first round, web families before the C slice. Eleven requirements in EARS form; awaiting approval. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
ea05e3357c
commit
3ebcb7bd67
@@ -90,13 +90,18 @@ Principle for the roadmap: **classify first, verify before reporting, improve on
|
||||
|
||||
Core `dependencies = []`; HarnessX and grammars stay optional extras; every loop step degrades to a recorded reason without them. No row deleted; unscorable rows keep a reason. Labels stay maintainer-owned: the classifier proposes, disagreements queue for review, and a class change on a reviewed pair is a reviewed edit. Model-authored skills, tools and prompt edits are retained only after held-out validation under the acceptance rule and are journaled with attribution; nothing a round proposes reaches a scan without its class holdout and the cross-class sweep. Split enforced by construction. Verifier code and gold labels outside every editable surface. Prompt length capped per class. Gates (`gate`, `map_gate`, `pair_gate`) stay byte-identical. Redaction on every prompt and trajectory. Per-round cost cap with revert.
|
||||
|
||||
## Open Questions for the maintainer
|
||||
## What this project is for (maintainer, 2026-09-06)
|
||||
|
||||
1. Taxonomy spine: CWE-699 families aligned to our verifiers (recommended, ~10 families) or the mechanism vocabulary with CWE as an attribute?
|
||||
2. Regression budget: the measured per-class seed-noise floor (recommended) or a fixed number?
|
||||
3. Which model runs the detectors and which the meta-agent? Research supports a cheap meta-agent and spending on the detector.
|
||||
4. Round-zero verifiers: which families get a sandbox recipe first, and do families without a verifier report at `suspicion` only until they have one?
|
||||
5. Does `authorization-obligations` task 5 (the lever) become one family's evolve round, or stay a spec task?
|
||||
The target is to catch the taxonomy families below **reliably** on real-world application code, including absence bugs in vibe-coded services (an authorization check that was never written), with an emphasis on dangerous, low-hanging fruit. It is not a bug-finder for JavaScript interpreters, WebGL renderers or browser engines, and it is not a memory-safety fuzzer: memory bugs in C and C++ are a different game played by specialized systems (AIxCC-class CRSs, Big Sleep) that this project learns from and does not replicate or compete with. The balance to hold is between two failure modes, being useless on actual code and being blind to forgotten checks in generated code; both are in scope, engine and renderer bugs are not. The vfc C slice therefore stays a measured, non-gating slice with one coarse memory-safety family and no crash oracle of our own.
|
||||
|
||||
## Decisions (maintainer, 2026-09-06)
|
||||
|
||||
1. **Taxonomy spine.** Ten families keyed to danger and to a verifier we can run, with CWE and the mechanism vocabulary as attributes, never the routing key: injection into a query or command (SQL, command, code, template); file path from user input; deserialization of untrusted data; access control (IDOR, missing guard, identity from request body; the obligations checker is its verifier); output encoding (XSS, template to HTML); server-side request forgery; prototype pollution and unsafe object merge; permissive configuration and secrets (CORS, debug, cookie flags, weak crypto, hardcoded credentials); memory safety as one coarse family for the vfc slice only; unknown. Multi-label routing.
|
||||
2. **Regression budget.** Measured, not chosen: round zero runs the cloned hunter K = 5 times per family at temperature 0 over the vendored corpus; the per-family negative-flip rate between seeds is the budget. A round that exceeds it in any family is rejected regardless of its target gain.
|
||||
3. **Models.** Detectors and meta-agent on `deepseek-v4-flash` through DeepSeek's own OpenAI-compatible endpoint (`https://api.deepseek.com`); `deepseek-v4-pro` only as the second independent judge above suspicion and for classifier disagreements queued for review; embeddings stay on OpenRouter. The chat endpoint and the embedding endpoint are configured separately (today one base-URL variable serves both). The 1M context lets the detector read whole files instead of hotspot excerpts.
|
||||
4. **Round-zero verifiers.** Snippet templates with an oracle per family, on the existing per-language sandbox recipes: canary row in an in-memory database for injection, canary file for command execution and traversal, canary object for deserialization. Access control gets the obligations checker as static corroboration. Output encoding, SSRF, prototype pollution and configuration report at `suspicion` only until they have an oracle. A family without a verifier never reports above suspicion in a scan but is still scored class-aware on the corpus: the corpus is the verifier for learning, the sandbox is the verifier for reporting.
|
||||
5. **Obligations lever.** `authorization-obligations` task 5 is retired as a spec task and becomes the access-control family's first evolve round; obligation shapes are that family's memory entries under the split rule.
|
||||
6. **Order.** Round zero on the web families over vibe-py and agent-vfc, where real-world code, dangerous fruit and a working hunter coincide; the vfc C slice later, as a measured slice only.
|
||||
|
||||
## Evidence
|
||||
|
||||
|
||||
Reference in New Issue
Block a user