mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
Marius CiepluchandClaude Fable 5.1
fbc3220bbe
learn/experiments.py: YAML manifests under plane/experiments/<id>.yaml, digest-checked at register, run and analyse time; `experiment`, `arm_outcome` and `experiment_result` rows; paired arms in a seeded repository order with a per-unit coin; arm A replay-only at $0; canary replicates under a fresh cache salt; repository-cluster paired bootstrap, exact McNemar, two O'Brien-Fleming looks, adopt/reject/equivalent/ inconclusive. `ousast learn experiment register|units|run|analyse`. Program: `ProgramSpec.sampling = retrieval_ensemble` (k samples at temperature 0 over k disjoint retrieved example sets; default unchanged); `Caller.salt`; `retrieve(exclude=)`. verify: OUSAST_VERIFY_CALLERS=off. Manifests exp-001-known-callers (plane arms, units pending) and exp-002-retrieval-ensemble (slice-2 incumbents, ceiling $4) -- not run. 18 scripted tests; module audit regenerated (151 modules). Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
35 lines
2.5 KiB
YAML
35 lines
2.5 KiB
YAML
# Experiment exp-001 (learned-decision-engine design section 6, Requirement 4.4): the known-callers prompt line.
|
|
# Registered before it runs; `ousast learn experiment register` records this file's sha256, and a changed file is a new
|
|
# experiment. Pre-registered expectation: with about 55 positive units the paired MDE is about 0.15, the observed gap
|
|
# 0.10 needs about 115 -- an inconclusive result is the likely and honest outcome; a follow-up needs new units.
|
|
id: exp-001-known-callers
|
|
hypothesis: >
|
|
Removing the known-callers line from the verify prompt changes the rate at which verify flags a true site in any
|
|
pass (plane-increment-2 recorded 17 of 20 declared sites flagged with the line vs 19 without, confounded).
|
|
kind: plane # arms are plane Run configurations; the plane's experiment-outcome task writes the arm_outcome rows
|
|
arms:
|
|
A:
|
|
label: verify as today (known-callers line on; CALLERS_CAP 8)
|
|
task_env: {}
|
|
B:
|
|
label: verify with the known-callers line off
|
|
task_env: {OUSAST_VERIFY_CALLERS: "off"} # everything else identical: model, PER_HUNT, MAX_STEPS, facts, runner image
|
|
unit: candidate
|
|
pairing: paired # every unit runs in both arms
|
|
cluster: repository # the resampling unit of the bootstrap
|
|
order: {randomise: per_case, seed: 1}
|
|
metric:
|
|
primary: "flag_rate[label=1]" # flagged in any pass, over the positive units (the 17-vs-19 metric)
|
|
secondary: ["flag_rate[label=0]", "usd_per_candidate"] # false flags on fixed-pin negatives; cost
|
|
test: {primary: cluster_bootstrap_paired_diff, secondary: exact_mcnemar, alpha: 0.05, sides: 2, resamples: 10000}
|
|
mde: 0.15
|
|
stopping: {looks: [0.5, 1.0], boundary: obrien_fleming} # |z| >= 2.797 at half the repositories, 1.977 at the end
|
|
budget_usd: {total: 15, per_arm: 7.5} # ~$7.7 expected: 552 hunts x $0.014
|
|
families: [injection, access_control, path, untrusted_destination] # the verify families
|
|
# The development corpus, not v2 (v2 generated the hypothesis): population v1 (11 cases), pmpro, MediaWiki, WP Statistics,
|
|
# and the 27 full-repository pointer pairs of the verify families -- 41 repositories. Candidates are generated model-free
|
|
# (fix-changed functions plus the top engine- and quick-alert functions, up to 8 per pin, on both pins) and frozen here;
|
|
# only units whose arm-A callers line is non-empty are treated. The file does not exist yet: registration records the
|
|
# units as `pending`, and the experiment cannot run until it is frozen, committed and registered again.
|
|
units_file: plane/experiments/exp-001-known-callers.units.jsonl
|