mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-18 12:39:28 +00:00
Marius CiepluchandClaude Opus 5.5
34da916030
`ousast plane harvest` (plane/harvest.py) writes two Runs over a label snapshot:
verify a/b + agree on the verify-family units (both sides of each labelled pair as
an opaque files-only Workspace, candidates outside the hunt's root, no declared
sites; v1 and dev-php pins with repo-facts first) and model roles on the candidate
files (pair excerpts grouped, one Task per population pin). Budgets sum under the
ceiling by construction. New roles task template; the generator re-pins it.
Spent by attribution: verify $2.54 (1961 calls, 150 units, 182 hunts per pass),
roles $1.40 (1034 calls, 799 files, 0 unclassified chunks); DeepSeek balance
19.63 -> 18.23. Engine on the pair corpus (benchmarks/learn/engine_pairs.py,
frozen e6ed785, one 3g container) is PARTIAL: 384 of 686 sides at commit, still
running; unfinished sides stay missing. benchmarks/learn/build_features.py built
1712 static+plane records into the memory store; counts-only record under
benchmarks/measurements/2026-10-01-decision-engine-harvest/.
source/roles parts are `none`, not `failed`, for a language without a declaration
pattern (C). ops/ax/README: Substrate golden-actor disk leak and the inline
Workspace limit (~20 KB).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
67 lines
3.8 KiB
Markdown
67 lines
3.8 KiB
Markdown
# Implementation Plan
|
|
|
|
Order follows the design's commit sequence. Every task is a commit gated on the full suite's own exit code,
|
|
`ruff check`, `ruff format --check` and `mypy`. No reported number changes until task 9 adopts a model; without an
|
|
adopted model every path is today's. Tasks 5, 7 and 10 spend model money or host hours: each starts only after
|
|
the maintainer confirms its budget.
|
|
|
|
- [x] 1. Schema and features
|
|
- `learn/schema.py` (closed, typed allow-list; no identities; missing != zero), `learn/features.py` (host
|
|
builder), new memory `KINDS`, `plane/tasks/features.py` and the `remember` input; tests enforcing the
|
|
allow-list and the exclusion of label-carrying fields.
|
|
- _Requirements: 1.1, 1.2, 1.3, 7.2_
|
|
|
|
- [x] 2. Framework tags
|
|
- `ruleset/frameworks.toml`; tag and split every framework/library entry listed in design section 2; `priors=`
|
|
in the semantic and quick loaders (default `all`, so today's scan is unchanged -- the golden benchmark test
|
|
stays byte-identical); a lint test that no untagged framework entry remains.
|
|
- _Requirements: 8.2, 8.4_
|
|
|
|
- [x] 3. Role inference
|
|
- `learn/roles.py` (deterministic wrapper inference), `plane/tasks/roles.py` (model roles, the `model_sinks.py`
|
|
approach), the vocabulary overlay for the engine; fixture tests with no model calls.
|
|
- _Requirements: 8.1_
|
|
|
|
- [x] 4. Labels
|
|
- `learn/sources.toml` (fail-closed allowed sources, v3 excluded unread), `learn/labels.py`, `ousast learn
|
|
labels`, repository-group normalisation and CVE merge, the `assumed_benign` source mined from non-security
|
|
commits (separate, never merged); a first label snapshot committed as counts only.
|
|
- _Requirements: 2.1, 2.2, 2.3, 7.1_
|
|
|
|
- [x] 5. Harvest (needs budget go-ahead)
|
|
- A plane Run of verify a/b on the verify-family pairs and development cases (~$6, ceiling $10), model roles on
|
|
the candidate files (ceiling $10), engine features for the pair corpus on the host (frozen source, one
|
|
container at a time, background).
|
|
- Done 2026-10-01 (`benchmarks/measurements/2026-10-01-decision-engine-harvest/record.json`): verify $2.54 and roles
|
|
$1.40 by attribution. The engine part was PARTIAL at commit (384 of 686 pair sides recorded, the job still
|
|
running): unfinished sides are `missing` in the feature records; rebuild them with
|
|
`benchmarks/learn/build_features.py` once `benchmarks/learn/engine_pairs.py` has finished.
|
|
- _Requirements: 1.1_
|
|
|
|
- [ ] 6. Train, calibrate, evaluate
|
|
- `learn/{folds,train,calibrate,evaluate}.py` (repository-grouped, leave-one-source-out, leave-one-framework-out,
|
|
nested tuning), the model blob API in `plane/memory.py`, `ousast learn train|evaluate|curve`, the first
|
|
evaluation report with priors off under `benchmarks/measurements/<date>-decision-engine-v1/`.
|
|
- _Requirements: 3.1-3.5, 5.1, 5.2, 7.3, 7.4, 8.3_
|
|
|
|
- [ ] 7. Experiments (exp-001 needs budget go-ahead)
|
|
- `learn/experiments.py`, `register`, `experiment-outcome`, Run annotation checks, `OUSAST_VERIFY_CALLERS`;
|
|
exp-001 (known-callers line on/off, ~$7.7 expected, ceiling $15) registered, run and recorded -- an
|
|
inconclusive result is pre-registered as likely.
|
|
- _Requirements: 4.1-4.4_
|
|
|
|
- [ ] 8. Integration
|
|
- The `decide` task, the `scan` decision stage and `dropped_by_engine.json`, `pre-push` admission with the engine
|
|
(structural admission reasons unchanged), report fields, the fallback line, the opt-in adaptation layer.
|
|
- _Requirements: 6.1-6.3, 7.3, 7.5_
|
|
|
|
- [ ] 9. Adopt
|
|
- `ousast learn adopt`, packaged models for the families that pass, the priors-on/off experiment per framework
|
|
with a recorded decision.
|
|
- _Requirements: 3.2, 8.2_
|
|
|
|
- [ ] 10. The one-time v3 check
|
|
- Commit `prediction-v3.json` first; then v3 is run under protocol v3 and `result-v3-engine.json` is committed.
|
|
Nothing between the two commits may touch the engine, features, priors or thresholds.
|
|
- _Requirements: 5.3_
|