mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-19 21:19:28 +00:00

Files

T

Marius CiepluchandClaude Opus 5.5 b68fe27c74 feat(plane): the improvement loop as a plane Run (harnessx-removal task 4)

`ousast plane workspaces --validation-set ... --loop` appends the loop to a Run:
per case `alerts` (plane/tasks/alerts.py: the quick-mode scan on the vulnerable
and the fixed pin, a fixed-pin Workspace per case, alerts attributed to their
enclosing function by the repo-facts patterns, `in_fix_range` from the fix's
old-side and new-side ranges, failing on an unread tree) feeding `remember`,
then `memory-snapshot` -> `loop-measure` -> `loop-propose` -> `loop-improve`
(plane/tasks/loop.py). Every new task binds no Model and has budget
{usd: 0, calls: 0}. The snapshot is the store's rows after the train-on-test
guard, written on the host by memory.seed (a sandboxed run of it fails loudly);
propose runs M1/M2 against this repository at the commit under test; improve
runs one unchanged evolve.run_round and delivers gate.json, journal.json, the
sidecar, rule_policy.json only when accepted, and proposal_outcome rows that
the host ingests. Adoption stays a maintainer commit.

`remember` is now a Run entry per case (the image contains it), with the facts
row keyed by the image the facts were computed on (repo-facts records it in
its summary; unknown -> the row is omitted, never guessed). Alert rows carry
their own pin; improve/memory.py counts a repeated alert once.

Runner image rebuilt (+0.39 MB: only the new modules; the quick scan needs no
extra and the rulesets were already in the package), pushed, plane/ re-pinned
and regenerated (validation-46: 105 entries). Fixture test runs the whole loop
graph through the fake ax, whose new `exec` mode runs the real task modules.
Live: one loop Run over three validation-46 cases on the local ax, recorded in
benchmarks/measurements/2026-09-30-harnessx-removal-loop/ (all done, $0 new
spend, 0 proposals, gate verdict no_proposals).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo

2026-09-30 11:59:50 +02:00

3.8 KiB

Raw Blame History

Implementation Plan

Order follows the design's commit sequence (Requirement 5): nothing is deleted before its replacement has landed and passed its fixture test. Every task is a commit gated on the full suite's own exit code, ruff check, ruff format --check and mypy; the reconciler stays under 500 lines.

1. Baseline without the extra
- Equality script (design §3) with its find_spec("harnessx") and freeze checks; re-sync .venv without the harnessx extra first.
- Record benchmarks/measurements/<date>-harnessx-removal-baseline/: full suite, ousast pairs --json, the python-vulnerable quick benchmark, one standard-mode scan, ousast improve --dry-run; the exclusion list written next to it.
- Requirements: 3.1, 5.1
2. Memory store and the remember task
- plane/memory.py: the MemoryStore interface with FileStore and MinioStore (object metadata/tags, S3 Select JSON queries probed with a local fallback, versioned provenance, lifecycle, presign stubs; OUSAST_MEMORY selects; MinIO settings from .env; minio SDK as an optional extra); JSON rows per repository + pin, facts by content hash, FileStore ingest refuses below 1 GiB free, plane/tasks/remember.py (model-free), reconciler.mark_done under the run lock, seed and ingest wiring in cli._plane, generator annotations; fact reuse when repository, pin, candidate set and runner image match.
- Fixture tests: rows from a recorded Run, reuse on an unchanged pin, recompute on a changed one, disk refusal.
- Requirements: 6.1, 6.4
3. Proposals from memory
- improve/memory.py with rules M1 (demote) and M2 (promote) and the advisory disputed-family signal; run_round(proposals=...); ousast improve --memory (opt-in; default output byte-identical); provenance per proposal in the sidecar and journal; the train-on-test guard (qualification population, gated benchmark cases, holdout pairs).
- Tests: each rule on crafted rows, the guard drops excluded rows, provenance round-trips, default improve unchanged.
- Requirements: 6.2, 6.4
4. The loop as a plane Run
- Tasks alerts, measure, propose, improve (no Model, zero budget); --loop Run generation; fixed-pin Workspaces; the gate's verdict in <run>/loop-improve/gate.json; adoption stays a maintainer commit.
- Fixture tests with the fake ax; one live loop Run on the local ax recorded (no model spend beyond the verify passes it reuses from memory).
- Requirements: 6.3
5. Retire the HarnessX scan paths
- cli.py, fusion.py, verify_judge.py, stage_processors.py changes; RetiredConfigError for [models] verifier, [fusion] panel_model/decider_model; one warning for a [harnessx] section (the only tagged retirement text allowed in src, enforced by a test); tests renamed to the behaviour they assert.
- Requirements: 1.1, 1.2, 1.3, 3.3
6. Delete HarnessX code and packaging
- Delete harness_ext.py, hunter_harness.py, verify_judge.py's judge path, the HarnessX-only tests; remove the extra, the mypy override and the uv.lock entry; rename the conftest fixture; audit rows and manifest; the reference search test (Req 2.4).
- Requirements: 2.1, 2.2, 2.3, 2.4
7. Equality proof
- Rerun the baseline script; commit the comparison record next to the baseline; any difference outside the exclusion list blocks the merge.
- Requirements: 3.2
8. Documentation and specs
- README ax-plane section pointing at ops/ax/README.md; docs/examples.md §6 and docs/threat-model.md (egress policy, budgets); dated retirement notes in the four specs and .kiro/steering/overview.md; release note naming the silent loss of the second hunter for users who set [models] hunter.
- Requirements: 4.1, 4.2, 4.3

3.8 KiB Raw Blame History

Implementation Plan

3.8 KiB

Raw Blame History