marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
e6ed7854bb8ef17d6a4c39971ea253b4e385af23
Open_Ultra_SAST/.kiro/specs/learned-decision-engine/tasks.md
T
Marius CiepluchandClaude Opus 5.5 e6ed7854bb feat(learn): labels from ground truth with provenance (decision-engine task 4) 
learn/sources.toml is the fail-closed source list: every read goes through one guard
that refuses an unlisted path or an [[excluded]] file name before opening it, and a
source whose evaluation record is missing cannot label. Population v3 is named only
as excluded, by file name, and is never opened (tested with a v3-named fixture and
open/read_text tracked).

learn/labels.py and `ousast learn labels` apply the design's rules: v2 declared
sites and v1 fix-range functions positive, the same function on the fixed pin
negative only when the fix changed it, introduce/repair delta units, benign control
pushes as conditional negatives; pairs vulnerable/fixed (unscorable and title tier
out, title as a sensitivity arm); dev-php recipe known functions; recorded
adjudications with their reasons. v2 rows are labels marked design_informed, as the
design says. Repository groups: normalised owner/name, merged by shared CVE/GHSA,
shared fix commits and [[alias]] rows.

assumed_benign (maintainer decision): ordinary commits of the population
repositories, >= 90 days from any fix, no later security commit touching their
files, weight 0.5, conditional, reported separately and never merged with verified
negatives. The security filter is multilingual (CJK, Korean, Russian, European and
other languages) plus CVE/GHSA/CWE references and dependency bumps; widened after
reading accepted pushes (validation, passwords, guards).

First snapshot, counts only (benchmarks/measurements/2026-09-30-decision-engine-
labels/labels.json): verified pin labels in positive/negative repository groups
injection 44/41, untrusted_destination 35/34, deserialization 20/20 (per-family),
output_encoding 15/15, path 15/14, access_control 12/12 (pooled), config_secrets 5/5,
memory 5/5 (insufficient). assumed_benign: 969 pushes in 27 groups from 36,000
commits scanned.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-09-30 19:54:26 +02:00

3.4 KiB
Raw Blame History

Implementation Plan

Order follows the design's commit sequence. Every task is a commit gated on the full suite's own exit code, ruff check, ruff format --check and mypy. No reported number changes until task 9 adopts a model; without an adopted model every path is today's. Tasks 5, 7 and 10 spend model money or host hours: each starts only after the maintainer confirms its budget.

  • 1. Schema and features

    • learn/schema.py (closed, typed allow-list; no identities; missing != zero), learn/features.py (host builder), new memory KINDS, plane/tasks/features.py and the remember input; tests enforcing the allow-list and the exclusion of label-carrying fields.
    • Requirements: 1.1, 1.2, 1.3, 7.2

  • 2. Framework tags

    • ruleset/frameworks.toml; tag and split every framework/library entry listed in design section 2; priors= in the semantic and quick loaders (default all, so today's scan is unchanged -- the golden benchmark test stays byte-identical); a lint test that no untagged framework entry remains.
    • Requirements: 8.2, 8.4

  • 3. Role inference

    • learn/roles.py (deterministic wrapper inference), plane/tasks/roles.py (model roles, the model_sinks.py approach), the vocabulary overlay for the engine; fixture tests with no model calls.
    • Requirements: 8.1

  • 4. Labels

    • learn/sources.toml (fail-closed allowed sources, v3 excluded unread), learn/labels.py, ousast learn labels, repository-group normalisation and CVE merge, the assumed_benign source mined from non-security commits (separate, never merged); a first label snapshot committed as counts only.
    • Requirements: 2.1, 2.2, 2.3, 7.1

  • 5. Harvest (needs budget go-ahead)

    • A plane Run of verify a/b on the verify-family pairs and development cases (~$6, ceiling $10), model roles on the candidate files (ceiling $10), engine features for the pair corpus on the host (frozen source, one container at a time, background).
    • Requirements: 1.1

  • 6. Train, calibrate, evaluate

    • learn/{folds,train,calibrate,evaluate}.py (repository-grouped, leave-one-source-out, leave-one-framework-out, nested tuning), the model blob API in plane/memory.py, ousast learn train|evaluate|curve, the first evaluation report with priors off under benchmarks/measurements/<date>-decision-engine-v1/.
    • Requirements: 3.1-3.5, 5.1, 5.2, 7.3, 7.4, 8.3

  • 7. Experiments (exp-001 needs budget go-ahead)

    • learn/experiments.py, register, experiment-outcome, Run annotation checks, OUSAST_VERIFY_CALLERS; exp-001 (known-callers line on/off, ~$7.7 expected, ceiling $15) registered, run and recorded -- an inconclusive result is pre-registered as likely.
    • Requirements: 4.1-4.4

  • 8. Integration

    • The decide task, the scan decision stage and dropped_by_engine.json, pre-push admission with the engine (structural admission reasons unchanged), report fields, the fallback line, the opt-in adaptation layer.
    • Requirements: 6.1-6.3, 7.3, 7.5

  • 9. Adopt

    • ousast learn adopt, packaged models for the families that pass, the priors-on/off experiment per framework with a recorded decision.
    • Requirements: 3.2, 8.2

  • 10. The one-time v3 check

    • Commit prediction-v3.json first; then v3 is run under protocol v3 and result-v3-engine.json is committed. Nothing between the two commits may touch the engine, features, priors or thresholds.
    • Requirements: 5.3
Reference in New Issue View Git Blame Copy Permalink