mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-18 20:49:28 +00:00
docs: plain-English pass over the active specs
Ran the humanizer rules over 79 prose files in the active specs (brief, design, requirements, research and narrative docs). Removed em dashes, significance inflation, forced rule-of-three, bold-label list items and filler. Skipped tasks.md checklists and the retired specs (harnessx-*, openrouter-sast-harness, unseen-repo-evaluation). Every number, path, command, hash and identifier was preserved: checked each file against HEAD for drift in numbers, inline code and code fences, zero drift across all 79 files. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
Marius Ciepluch
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
b08f372022
commit
39a1873e5b
@@ -24,8 +24,8 @@ Build a name-resolved call graph from the `FunctionIR` records tree-sitter alrea
|
||||
|
||||
## Scope
|
||||
|
||||
- **In**: call graph, summaries, path records, reachability attachment, rank and map integration, hunter on paths, hunter-to-candidate path, fact oracle with cache, `facts` lever with validator, fix-point ranking and report section, measurement on split-sink, vibe-py, vfc slices, offline tests.
|
||||
- **Out**: guard dominance semantics (`guard-dominance-regime` consumes path records), CST walker fixes (`overlay-ir-completeness`), a whole-program points-to or symbolic engine, concolic execution, executing target code outside the sandbox, new dispositions, LLM as parser, any merge gate change.
|
||||
- In: call graph, summaries, path records, reachability attachment, rank and map integration, hunter on paths, hunter-to-candidate path, fact oracle with cache, `facts` lever with validator, fix-point ranking and report section, measurement on split-sink, vibe-py, vfc slices, offline tests.
|
||||
- Out: guard dominance semantics (`guard-dominance-regime` consumes path records), CST walker fixes (`overlay-ir-completeness`), a whole-program points-to or symbolic engine, concolic execution, executing target code outside the sandbox, new dispositions, LLM as parser, any merge gate change.
|
||||
|
||||
## Boundary Candidates
|
||||
|
||||
@@ -42,14 +42,14 @@ Build a name-resolved call graph from the `FunctionIR` records tree-sitter alrea
|
||||
|
||||
## Upstream / Downstream
|
||||
|
||||
- **Upstream**: `FileIR`/`FunctionIR`, `analyze_entry_points`, `analyze_ir`, `load_facts`, `Hotspot`, `run_tool_hunter`, `select_candidates`, `run_improvement`, pair holdout metrics per profile.
|
||||
- **Downstream**: `guard-dominance-regime` attaches guards to path records; later web-authorization detection is a path whose hops include no auth guard; reports and MCP read fix points.
|
||||
- Upstream: `FileIR`/`FunctionIR`, `analyze_entry_points`, `analyze_ir`, `load_facts`, `Hotspot`, `run_tool_hunter`, `select_candidates`, `run_improvement`, pair holdout metrics per profile.
|
||||
- Downstream: `guard-dominance-regime` attaches guards to path records; later web-authorization detection is a path whose hops include no auth guard; reports and MCP read fix points.
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
|
||||
- **Extends**: `three-stage-scan` (hunter tools, candidate selection, worth-fixing input), `propose-adjudicate-prove` (facts consumed unchanged; one new fact source), `harnessx-self-improving-rulesets` (one new lever under the same gate).
|
||||
- **Depends**: `pair-corpus-honesty` for per-profile holdout metrics and the hunter scorer.
|
||||
- **Adjacent**: `overlay-ir-completeness` improves the records this spec walks; neither blocks the other.
|
||||
- Extends: `three-stage-scan` (hunter tools, candidate selection, worth-fixing input), `propose-adjudicate-prove` (facts consumed unchanged; one new fact source), `harnessx-self-improving-rulesets` (one new lever under the same gate).
|
||||
- Depends: `pair-corpus-honesty` for per-profile holdout metrics and the hunter scorer.
|
||||
- Adjacent: `overlay-ir-completeness` improves the records this spec walks; neither blocks the other.
|
||||
|
||||
## Constraints
|
||||
|
||||
|
||||
@@ -146,10 +146,10 @@ tests/
|
||||
|
||||
### Modified Files
|
||||
|
||||
- `semantic/cst.py`, `semantic/ir.py` — collect import statements into `FileIR.imports`; no other IR change.
|
||||
- `mapping.py` — new function; `EntryPointRecord` unchanged.
|
||||
- `stage_processors.py` — MAP stage runs graph, summaries, paths before overlay; overlay unchanged.
|
||||
- `map_gate.py` — path assertion per language family.
|
||||
- `semantic/cst.py`, `semantic/ir.py`, collect import statements into `FileIR.imports`; no other IR change.
|
||||
- `mapping.py`, new function; `EntryPointRecord` unchanged.
|
||||
- `stage_processors.py`, MAP stage runs graph, summaries, paths before overlay; overlay unchanged.
|
||||
- `map_gate.py`, path assertion per language family.
|
||||
|
||||
## System Flows
|
||||
|
||||
|
||||
@@ -8,9 +8,9 @@ OpenUltraSAST cannot yet steer effort where risk is reduced because it has no mo
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: call graph, taint summaries, path records, reachability attachment to findings, rank, complexity map and worth-fixing; hunter path context; hunter findings as sandbox candidates; fact oracle and candidate-fact cache; `facts` improve lever; fix-point ranking and report section; offline tests and measurement on existing slices.
|
||||
- **Out of scope**: guard semantics; CST walker fixes; whole-program alias analysis; symbolic or concolic execution; new dispositions or evidence rungs; LLM as parser; any merge gate change; profile-specific fact tables beyond the gate defined by `pair-corpus-honesty`.
|
||||
- **Adjacent expectations**: `FileIR` shape is frozen except additive fields; `load_facts` merges candidate facts below hand-written facts; the improve loop's existing accept gate and per-profile holdout clause apply unchanged to the new lever; sandbox safety checks apply to hunter snippets as today.
|
||||
- In scope: call graph, taint summaries, path records, reachability attachment to findings, rank, complexity map and worth-fixing; hunter path context; hunter findings as sandbox candidates; fact oracle and candidate-fact cache; `facts` improve lever; fix-point ranking and report section; offline tests and measurement on existing slices.
|
||||
- Out of scope: guard semantics; CST walker fixes; whole-program alias analysis; symbolic or concolic execution; new dispositions or evidence rungs; LLM as parser; any merge gate change; profile-specific fact tables beyond the gate defined by `pair-corpus-honesty`.
|
||||
- Adjacent expectations: `FileIR` shape is frozen except additive fields; `load_facts` merges candidate facts below hand-written facts; the improve loop's existing accept gate and per-profile holdout clause apply unchanged to the new lever; sandbox safety checks apply to hunter snippets as today.
|
||||
|
||||
## Requirements
|
||||
|
||||
@@ -79,7 +79,7 @@ OpenUltraSAST cannot yet steer effort where risk is reduced because it has no mo
|
||||
|
||||
### Requirement 6: Facts as an improve-loop lever
|
||||
|
||||
**Objective:** As a maintainer, I want the improve loop to be able to accept or reject facts, so that the evolutionary machinery can move detection and not only rule status.
|
||||
**Objective:** As a maintainer, I want the improve loop to be able to accept or reject facts, so that the evolutionary machinery can move both detection and rule status.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
|
||||
@@ -2,9 +2,9 @@
|
||||
|
||||
## Summary
|
||||
|
||||
- **Feature**: `reachability-flow-model`
|
||||
- **Discovery Scope**: Complex Integration across semantic, mapping, rank, complexity, hunter, regress, improve
|
||||
- **Key Findings**:
|
||||
- Feature: `reachability-flow-model`
|
||||
- Discovery Scope: Complex Integration across semantic, mapping, rank, complexity, hunter, regress, improve
|
||||
- Key Findings:
|
||||
- `FunctionIR` already carries every call site per function for Python, JS, TS, C, and Java when the extra is present. A name-resolved call graph is a few hundred lines over data that exists.
|
||||
- Reachability today is "line inside an entry-point handler" (`mapping.attach_reachability_hints`); `rank` bumps an integer from tags and hints; `worth_fixing` needs `reachable` or `inferred-file-surface`. No call edge exists anywhere.
|
||||
- Sandbox candidates come only from hotspots filtered to overlay-promoted inventory ids (`cli.py:404`, `prove_filter.filter_promoted_hotspots`); a hunter snippet is used only for an already promoted hotspot (`candidate.py:199`). The LLM plane cannot create a candidate.
|
||||
@@ -18,38 +18,38 @@
|
||||
|
||||
### Where a call graph can come from
|
||||
|
||||
- **Context**: Avoid a new parser or a whole-program engine.
|
||||
- **Sources Consulted**: `semantic/ir.py`, `semantic/cst.py`, `mapping.py`, `preprocess.py`, `hunter_tools.find_refs`.
|
||||
- **Findings**: `FileIR.functions[*].calls[*].name` is the callee text. Python stdlib parse yields the same records. Imports are not in the IR; the walker can add an `imports` tuple per file cheaply. `find_refs` is a word-boundary regex over files and is the only cross-file lookup today.
|
||||
- **Implications**: `semantic/callgraph.py` consumes `FileIR` plus an additive `imports` field. Resolution order: same-file definition, imported module definition, any definition with that name (ambiguous edges). Method calls resolve by trailing name. C resolves by bare name across the tree.
|
||||
- Context: Avoid a new parser or a whole-program engine.
|
||||
- Sources Consulted: `semantic/ir.py`, `semantic/cst.py`, `mapping.py`, `preprocess.py`, `hunter_tools.find_refs`.
|
||||
- Findings: `FileIR.functions[*].calls[*].name` is the callee text. Python stdlib parse yields the same records. Imports are not in the IR; the walker can add an `imports` tuple per file cheaply. `find_refs` is a word-boundary regex over files and is the only cross-file lookup today.
|
||||
- Implications: `semantic/callgraph.py` consumes `FileIR` plus an additive `imports` field. Resolution order: same-file definition, imported module definition, any definition with that name (ambiguous edges). Method calls resolve by trailing name. C resolves by bare name across the tree.
|
||||
|
||||
### Summaries versus whole-program taint
|
||||
|
||||
- **Context**: `propose-adjudicate-prove` deferred inter-file taint as too large and risky for false demotes.
|
||||
- **Sources Consulted**: `semantic/taint.py` (`_analyze_function` seeds parameters as tainted, tracks names, emits `TaintPath` and `Demotion`), overlay invariants.
|
||||
- **Findings**: Because parameters are already seeded tainted, a per-function run already answers "which parameter reaches which sink." Adding "which parameter reaches the return" is a small extension. Composition is then a graph walk with a depth bound. Nothing needs alias analysis to produce a useful path; anything unresolved stays `flow_incomplete`.
|
||||
- **Implications**: Summaries are a pure function of `FunctionIR` plus facts, cacheable by file hash. Composition never demotes; it only adds paths or marks incompleteness.
|
||||
- Context: `propose-adjudicate-prove` deferred inter-file taint as too large and risky for false demotes.
|
||||
- Sources Consulted: `semantic/taint.py` (`_analyze_function` seeds parameters as tainted, tracks names, emits `TaintPath` and `Demotion`), overlay invariants.
|
||||
- Findings: Because parameters are already seeded tainted, a per-function run already answers "which parameter reaches which sink." Adding "which parameter reaches the return" is a small extension. Composition is then a graph walk with a depth bound. Nothing needs alias analysis to produce a useful path; anything unresolved stays `flow_incomplete`.
|
||||
- Implications: Summaries are a pure function of `FunctionIR` plus facts, cacheable by file hash. Composition never demotes; it only adds paths or marks incompleteness.
|
||||
|
||||
### Hunter to candidate
|
||||
|
||||
- **Context**: The LLM plane is the recall lever the field data supports, but it is walled off from prove.
|
||||
- **Sources Consulted**: `tool_hunter.py`, `regress/candidate.py`, `three-stage-scan` design (hunter snippets run only in the sandbox after safety check), `verification.py`.
|
||||
- **Findings**: The safety check and sandbox are origin-agnostic. The wall is `filter_promoted_hotspots` and the candidate selector's reliance on inventory ids. A hunter finding with a locatable sink line can be given a synthetic hotspot the same way `_hotspot_from_forced_finding` already does for severity-5 inventory.
|
||||
- **Implications**: Add eligibility for hunter findings that name a sink line resolvable to a path record or an overlay sink. Evidence stays suspicion until sandbox trigger or path agreement.
|
||||
- Context: The LLM plane is the recall lever the field data supports, but it is walled off from prove.
|
||||
- Sources Consulted: `tool_hunter.py`, `regress/candidate.py`, `three-stage-scan` design (hunter snippets run only in the sandbox after safety check), `verification.py`.
|
||||
- Findings: The safety check and sandbox are origin-agnostic. The wall is `filter_promoted_hotspots` and the candidate selector's reliance on inventory ids. A hunter finding with a locatable sink line can be given a synthetic hotspot the same way `_hotspot_from_forced_finding` already does for severity-5 inventory.
|
||||
- Implications: Add eligibility for hunter findings that name a sink line resolvable to a path record or an overlay sink. Evidence stays suspicion until sandbox trigger or path agreement.
|
||||
|
||||
### Facts from a model, safely
|
||||
|
||||
- **Context**: The engine lacks callee knowledge: nullable returns (`tls1_lookup_sigalg`), bounded arguments, transformer set/get (`ConfigParser`), macro out-params (`n2s`).
|
||||
- **Sources Consulted**: `semantic/facts.py` loader, `improve/validator.py`, `improve/journal.py`, `calibration.py` ledger patterns, `redaction.py`.
|
||||
- **Findings**: Facts are versioned TOML loaded by `load_facts`. The loader can merge a second directory at lower precedence. The improve loop has a strict validator and byte-for-byte revert. Redaction exists for prompts.
|
||||
- **Implications**: Model answers are candidate facts in `.openultrasast/calibration/candidate_facts.toml`; accepted facts in `accepted_facts.toml`; loader precedence hand-written > accepted > none. The `facts` lever moves rows between candidate and accepted under the same gate. No pattern text is ever model-authored; question kinds are closed.
|
||||
- Context: The engine lacks callee knowledge: nullable returns (`tls1_lookup_sigalg`), bounded arguments, transformer set/get (`ConfigParser`), macro out-params (`n2s`).
|
||||
- Sources Consulted: `semantic/facts.py` loader, `improve/validator.py`, `improve/journal.py`, `calibration.py` ledger patterns, `redaction.py`.
|
||||
- Findings: Facts are versioned TOML loaded by `load_facts`. The loader can merge a second directory at lower precedence. The improve loop has a strict validator and byte-for-byte revert. Redaction exists for prompts.
|
||||
- Implications: Model answers are candidate facts in `.openultrasast/calibration/candidate_facts.toml`; accepted facts in `accepted_facts.toml`; loader precedence hand-written > accepted > none. The `facts` lever moves rows between candidate and accepted under the same gate. No pattern text is ever model-authored; question kinds are closed.
|
||||
|
||||
### Fix points
|
||||
|
||||
- **Context**: The principle is risk reduced per fix, not findings per file.
|
||||
- **Sources Consulted**: `complexity/map.py`, `complexity/ledger.py`, `regress/verdict.py`, `scoring/project_score.py`.
|
||||
- **Findings**: The path graph is small (functions, not statements). Dominators over a DAG of entry-to-sink paths are cheap. Severity is available per CWE from policy.
|
||||
- **Implications**: `risk/fixpoints.py` computes dominator sets per sink path and aggregates risk removed. It is a report and artifact, not a gate; it cannot hide severity-5 reachable findings.
|
||||
- Context: The principle is risk reduced per fix, not findings per file.
|
||||
- Sources Consulted: `complexity/map.py`, `complexity/ledger.py`, `regress/verdict.py`, `scoring/project_score.py`.
|
||||
- Findings: The path graph is small (functions, not statements). Dominators over a DAG of entry-to-sink paths are cheap. Severity is available per CWE from policy.
|
||||
- Implications: `risk/fixpoints.py` computes dominator sets per sink path and aggregates risk removed. It is a report and artifact, not a gate; it cannot hide severity-5 reachable findings.
|
||||
|
||||
## Architecture Pattern Evaluation
|
||||
|
||||
@@ -65,49 +65,49 @@
|
||||
|
||||
### Decision: Over-approximate edges, never infer safety
|
||||
|
||||
- **Context**: Missing edges silently hide risk; false edges cost hunter budget.
|
||||
- **Selected Approach**: Ambiguous names produce edges to every definition, flagged. Unresolved calls produce an unresolved edge. Distance is infinite only within the depth bound and is reported as `unknown`.
|
||||
- **Rationale**: Consistent with the fail-closed rule in the overlay.
|
||||
- **Trade-offs**: More hunter work on large trees; bounded by existing hotspot caps.
|
||||
- Context: Missing edges silently hide risk; false edges cost hunter budget.
|
||||
- Selected Approach: Ambiguous names produce edges to every definition, flagged. Unresolved calls produce an unresolved edge. Distance is infinite only within the depth bound and is reported as `unknown`.
|
||||
- Rationale: Consistent with the fail-closed rule in the overlay.
|
||||
- Trade-offs: More hunter work on large trees; bounded by existing hotspot caps.
|
||||
|
||||
### Decision: Summaries are pure and cached
|
||||
|
||||
- **Context**: Cost on large repos.
|
||||
- **Selected Approach**: Summary keyed by file hash plus facts version; recomputed only when either changes.
|
||||
- **Trade-offs**: Cache invalidation on facts version bump; acceptable.
|
||||
- Context: Cost on large repos.
|
||||
- Selected Approach: Summary keyed by file hash plus facts version; recomputed only when either changes.
|
||||
- Trade-offs: Cache invalidation on facts version bump; acceptable.
|
||||
|
||||
### Decision: Evidence rises on checks, not on origin
|
||||
|
||||
- **Context**: Regex stamped corroboration, hunter stamped suspicion.
|
||||
- **Selected Approach**: This spec does not change existing stamps. It adds the checks that raise hunter evidence: composed path agreement or sandbox trigger. A follow-up may lower regex-only hits to inventory; recorded as a revalidation trigger for `propose-adjudicate-prove`.
|
||||
- **Rationale**: Keep this spec additive; changing existing stamps changes gate baselines.
|
||||
- Context: Regex stamped corroboration, hunter stamped suspicion.
|
||||
- Selected Approach: This spec does not change existing stamps. It adds the checks that raise hunter evidence: composed path agreement or sandbox trigger. A follow-up may lower regex-only hits to inventory; recorded as a revalidation trigger for `propose-adjudicate-prove`.
|
||||
- Rationale: Keep this spec additive; changing existing stamps changes gate baselines.
|
||||
|
||||
### Decision: Closed question kinds, gated acceptance
|
||||
|
||||
- **Context**: Auto-facts were excluded for good reason: unbounded model-authored patterns.
|
||||
- **Selected Approach**: Five question kinds, structured answers, candidate cache with provenance, acceptance only via the `facts` lever under the per-profile holdout gate.
|
||||
- **Trade-offs**: Slower than trusting answers directly; the gate is the point.
|
||||
- Context: Auto-facts were excluded for good reason: unbounded model-authored patterns.
|
||||
- Selected Approach: Five question kinds, structured answers, candidate cache with provenance, acceptance only via the `facts` lever under the per-profile holdout gate.
|
||||
- Trade-offs: Slower than trusting answers directly; the gate is the point.
|
||||
|
||||
### Decision: Fix points are dominators, reported not gated
|
||||
|
||||
- **Context**: Do not add a seventh evidence rung or a new CI condition.
|
||||
- **Selected Approach**: Compute and report; the severity-5 reachable invariant from `three-stage-scan` stays.
|
||||
- Context: Do not add a seventh evidence rung or a new CI condition.
|
||||
- Selected Approach: Compute and report; the severity-5 reachable invariant from `three-stage-scan` stays.
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
- Dynamic dispatch and callbacks in JS and Python produce unresolved edges — report unresolved counts; hunter `find_refs` remains available as a fallback.
|
||||
- Graph blow-up on large trees — depth and fan-out bounds in config; hotspot caps unchanged.
|
||||
- Model answers wrong — never used unless accepted by the gate; provenance recorded; hand facts win.
|
||||
- Hunter candidates increase sandbox load — existing `max_candidates` cap; hunter candidates ranked after overlay promotions unless a path agrees.
|
||||
- Fix-point estimates misread as guarantees — label as estimates; show witness paths.
|
||||
- Dynamic dispatch and callbacks in JS and Python produce unresolved edges. Report unresolved counts; hunter `find_refs` remains available as a fallback.
|
||||
- Graph blow-up on large trees, depth and fan-out bounds in config; hotspot caps unchanged.
|
||||
- Model answers can be wrong. They are never used unless accepted by the gate; provenance recorded; hand facts win.
|
||||
- Hunter candidates increase sandbox load. The existing `max_candidates` cap applies; hunter candidates ranked after overlay promotions unless a path agrees.
|
||||
- Fix-point estimates misread as guarantees. Label as estimates; show witness paths.
|
||||
|
||||
## References
|
||||
|
||||
- Clearwing sourcehunt callgraph, taint, ranker (reference architecture; not a dependency)
|
||||
- [RealVuln](https://arxiv.org/abs/2604.13764) — rule-based versus LLM scanners on real Python web code
|
||||
- [CVE-Bench](https://arxiv.org/abs/2503.17332) — agent exploitation ceiling on real web CVEs
|
||||
- `.kiro/specs/three-stage-scan/design.md` — hunter tools, candidate selection, worth-fixing
|
||||
- `.kiro/specs/propose-adjudicate-prove/design.md` — facts as data, dispositions, evidence honesty
|
||||
- `.kiro/specs/harnessx-self-improving-rulesets/design.md` — bounded levers, accept gate, revert
|
||||
- `.kiro/specs/pair-corpus-honesty/` — per-profile holdout gate, hunter scorer
|
||||
- [RealVuln](https://arxiv.org/abs/2604.13764), rule-based versus LLM scanners on real Python web code
|
||||
- [CVE-Bench](https://arxiv.org/abs/2503.17332), agent exploitation ceiling on real web CVEs
|
||||
- `.kiro/specs/three-stage-scan/design.md`, hunter tools, candidate selection, worth-fixing
|
||||
- `.kiro/specs/propose-adjudicate-prove/design.md`, facts as data, dispositions, evidence honesty
|
||||
- `.kiro/specs/harnessx-self-improving-rulesets/design.md`, bounded levers, accept gate, revert
|
||||
- `.kiro/specs/pair-corpus-honesty/`, per-profile holdout gate, hunter scorer
|
||||
- Memory note `overlay-gaps-diagnosis-2026-09`
|
||||
|
||||
Reference in New Issue
Block a user