mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-18 12:39:28 +00:00
docs: plain-English pass over the active specs
Ran the humanizer rules over 79 prose files in the active specs (brief, design, requirements, research and narrative docs). Removed em dashes, significance inflation, forced rule-of-three, bold-label list items and filler. Skipped tasks.md checklists and the retired specs (harnessx-*, openrouter-sast-harness, unseen-repo-evaluation). Every number, path, command, hash and identifier was preserved: checked each file against HEAD for drift in numbers, inline code and code fences, zero drift across all 79 files. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
b08f372022
commit
39a1873e5b
@@ -2,7 +2,7 @@
|
||||
|
||||
## Problem
|
||||
|
||||
`learning-harness` built the measurement layer honestly — a closed family taxonomy, a class-aware pair
|
||||
`learning-harness` built an honest measurement layer with a closed family taxonomy, a class-aware pair
|
||||
scorer, per-family detector configurations, a split enforced by construction, verifier-gated reporting,
|
||||
and a journal with attribution. What it did **not** build is a search. `run_learning_round` is a
|
||||
hand-rolled hill-climber: one proposal, one minibatch, one holdout check, a threshold rule, accept or
|
||||
@@ -28,26 +28,26 @@ Round zero on vibe-py, `deepseek-v4-flash`, K = 5, $0.69, 23/30 pair-correct:
|
||||
|
||||
Four defects follow from the loop's shape, not from its bugs:
|
||||
|
||||
1. **The acceptance rule cannot discriminate.** `decide` compares raw deltas: any regression on train or
|
||||
1. The acceptance rule cannot discriminate. `decide` compares raw deltas: any regression on train or
|
||||
holdout rejects, a tie rejects. The measured floor says 11 of 20 injection pairs disagree with
|
||||
*themselves* across five runs, so a delta over a minibatch of 8 is mostly noise. The spec built
|
||||
`sign_test` and `reliable_change` for exactly this and the rule calls neither.
|
||||
2. **The holdout is an input to the decision.** `decide` reads `holdout_delta`. Req 5.4 keeps the
|
||||
2. The holdout is an input to the decision. `decide` reads `holdout_delta`. Req 5.4 keeps the
|
||||
*proposer* blind to the holdout, and then the *selector* reads it every round. Repeated rounds
|
||||
selecting on holdout performance is search-time overfitting of the test set — the same leak class
|
||||
selecting on holdout performance is search-time overfitting of the test set, the same leak class
|
||||
this project has already measured once, one level up.
|
||||
3. **Noise is fought with repetition instead of diversity.** K = 5 spends five rollouts per pair per
|
||||
3. Noise is fought with repetition instead of diversity. K = 5 spends five rollouts per pair per
|
||||
side estimating variance. The published alternative spends that budget on more candidates and lets a
|
||||
frontier keep whatever wins somewhere.
|
||||
4. **The proposer is told a score, not what went wrong.** `FailureFacts` carries outcomes, run strings
|
||||
4. The proposer is told a score, not what went wrong. `FailureFacts` carries outcomes, run strings
|
||||
and counts. The single largest lever in the published work is textual feedback from the *evaluation
|
||||
trace* — and our evaluator produces exactly the right material (which finding landed on which line,
|
||||
trace*, and our evaluator produces exactly the right material (which finding landed on which line,
|
||||
whether it fell inside the labeled span, whether the fixed side leaked, which family was claimed)
|
||||
and then throws all of it away.
|
||||
|
||||
## What others built
|
||||
|
||||
**GEPA — Reflective Prompt Evolution (Agrawal et al., arXiv:2507.19457, ICLR 2026 oral).** Algorithm 1,
|
||||
**GEPA, Reflective Prompt Evolution (Agrawal et al., arXiv:2507.19457, ICLR 2026 oral).** Algorithm 1,
|
||||
read from the paper rather than from a summary:
|
||||
|
||||
- Split `D_train` into `D_feedback` and `D_pareto`. Initialise the candidate pool `P = [Φ]` and score
|
||||
@@ -56,7 +56,7 @@ read from the paper rather than from a summary:
|
||||
minibatch drawn from `D_feedback`; collect scores, traces and **textual feedback** via a feedback
|
||||
function `μ_f`; have a reflection LM rewrite that module's prompt; re-run the mutated system on the
|
||||
*same* minibatch.
|
||||
- **Accept if and only if the minibatch average improved.** Only then is the candidate scored on all of
|
||||
- Accept if and only if the minibatch average improved. Only then is the candidate scored on all of
|
||||
`D_pareto` and added to the pool with its parent recorded.
|
||||
- Return the candidate with the best average on `D_pareto`.
|
||||
|
||||
@@ -65,7 +65,7 @@ candidates that achieve it; drop strictly dominated ones; sample a survivor with
|
||||
proportional to the number of instances it leads on.
|
||||
|
||||
Three things about that are directly relevant to us. The acceptance gate is **cheap and noisy on
|
||||
purpose** — a single minibatch comparison — because the frontier, not the gate, is what protects
|
||||
purpose** (a single minibatch comparison) because the frontier, not the gate, is what protects
|
||||
against a bad accept: anything that wins on one instance is retained. Selection is **stochastic and
|
||||
diversity-preserving**, which is what stops the search collapsing onto one lineage. And the expensive
|
||||
full evaluation is spent only on candidates that already passed the cheap gate.
|
||||
@@ -76,20 +76,20 @@ GRPO's 24,000. Sample efficiency is the claim that matters for us, because our r
|
||||
|
||||
**DSPy's optimizer interface.** `MIPROv2` proposes instruction and demonstration candidates and searches
|
||||
them with Bayesian optimisation (Optuna TPE), scoring on minibatches and running a full validation pass
|
||||
on the incumbent every `minibatch_full_eval_steps` — an explicit two-fidelity design for a noisy metric.
|
||||
on the incumbent every `minibatch_full_eval_steps`, an explicit two-fidelity design for a noisy metric.
|
||||
`dspy.GEPA` takes `metric`, `reflection_lm`, a budget (`auto` / `max_full_evals` / `max_metric_calls`),
|
||||
`reflection_minibatch_size`, `candidate_selection_strategy` (`pareto` | `current_best`) and
|
||||
`track_stats`, and its metric returns `Prediction(score, feedback)` rather than a float.
|
||||
|
||||
**SIMBA — Stochastic Introspective Mini-Batch Ascent (`dspy.SIMBA`).** Two of its ideas fit our corpus
|
||||
**SIMBA, Stochastic Introspective Mini-Batch Ascent (`dspy.SIMBA`).** Two of its ideas fit our corpus
|
||||
better than GEPA's do. It runs `num_candidates` (6 by default) program variants per example, buckets the
|
||||
trajectories, and ranks examples by **max-to-min score gap** — deliberately spending the budget on the
|
||||
trajectories, and ranks examples by **max-to-min score gap**, deliberately spending the budget on the
|
||||
instances where variants disagree most. On our corpus that is not a nuisance, it is the training signal:
|
||||
the 11 injection pairs that disagree with themselves are precisely the pairs where a rule change can flip
|
||||
an outcome, and uniform minibatch sampling would spend most of its budget on the 9 that never move. It
|
||||
also alternates two strategies rather than round-robining blindly — `append_a_demo` (a successful
|
||||
also alternates two strategies rather than round-robining blindly, `append_a_demo` (a successful
|
||||
trajectory becomes a demonstration) and `append_a_rule` (the model introspects on the bucket and writes a
|
||||
rule) — which is exactly the split our levers already have between `counterexamples`/`hard_negatives` and
|
||||
rule), which is exactly the split our levers already have between `counterexamples`/`hard_negatives` and
|
||||
`checklist`/`prompt`. Candidates are sampled by softmax over historical scores at temperature 0.2, a
|
||||
cheaper third option beside GEPA's `pareto` and `current_best`.
|
||||
|
||||
@@ -97,16 +97,16 @@ cheaper third option beside GEPA's `pareto` and `current_best`.
|
||||
produce. It shows the reflector the current instruction, then "inputs, outputs and feedback" for the
|
||||
minibatch, and asks it to infer the task, extract "all niche and domain specific factual information ...
|
||||
as a lot of it may not be available to the assistant in the future", and name any generalizable strategy.
|
||||
That is a specification for `μ_f`, not just a prompt: feedback has to carry the domain facts a detector
|
||||
cannot re-derive — which sink, which span, which side leaked — rather than a verdict.
|
||||
That is a prompt and a specification for `μ_f`: feedback has to carry the domain facts a detector
|
||||
cannot re-derive (which sink, which span, which side leaked) rather than a verdict.
|
||||
|
||||
**GEPA's merge** is gated strictly: two candidates are crossed only when they share a common ancestor,
|
||||
optimised *disjoint* sets of prompts, are both Pareto-optimal, and both beat the ancestor's aggregate.
|
||||
Worth having once we have per-lever candidates, and worth not having before then.
|
||||
|
||||
**The standalone `gepa` library** (MIT) exposes `gepa.optimize(seed_candidate, trainset, valset,
|
||||
task_lm, reflection_lm, max_metric_calls)` over a `GEPAAdapter` protocol of three methods —
|
||||
`evaluate`, `make_reflective_dataset`, `propose_new_texts` — with a seed candidate that is just a dict
|
||||
task_lm, reflection_lm, max_metric_calls)` over a `GEPAAdapter` protocol of three methods
|
||||
(`evaluate`, `make_reflective_dataset`, `propose_new_texts`), with a seed candidate that is just a dict
|
||||
of component name to text. That adapter shape is the interface we want whether or not we take the
|
||||
dependency: it is the smallest surface that separates *what a detector is* from *how it is searched*.
|
||||
|
||||
@@ -114,26 +114,26 @@ dependency: it is the smallest surface that separates *what a detector is* from
|
||||
|
||||
Restate the detector as an optimisable program and put a published optimiser behind it.
|
||||
|
||||
- **A detector is a candidate**: a dict of named text components (`prompt`, `checklist`,
|
||||
- A detector is a candidate: a dict of named text components (`prompt`, `checklist`,
|
||||
`hard_negatives`, `counterexamples`) plus its frozen non-text settings (tools, budgets). That is
|
||||
already what `FamilyConfig` holds; it becomes the seed candidate.
|
||||
- **The metric returns a score and a reason.** `score_pair_family` gains a feedback string built from
|
||||
- The metric returns a score and a reason. `score_pair_family` gains a feedback string built from
|
||||
what the evaluation already knows. This is the highest-value change in the spec and it is independent
|
||||
of which optimiser sits on top.
|
||||
- **`D_pareto` is carved out of the train split.** The holdout leaves the acceptance rule entirely and
|
||||
- `D_pareto` is carved out of the train split. The holdout leaves the acceptance rule entirely and
|
||||
is scored once, for reporting, when the search is over. This is a requirements change to Req 9.5 and
|
||||
the most important honesty consequence of the redesign.
|
||||
- **An adapter protocol** (`evaluate`, `make_reflective_dataset`, `propose_new_texts`) with two backends:
|
||||
- An adapter protocol (`evaluate`, `make_reflective_dataset`, `propose_new_texts`) with two backends:
|
||||
our own implementation of Algorithms 1 and 2 by default, so the core install keeps
|
||||
`dependencies = []`, and `gepa` behind an optional extra so its improvements arrive for free.
|
||||
- **The Pareto pool replaces the accept/revert tree.** `Archive` already records per-pair winners; it
|
||||
- The Pareto pool replaces the accept/revert tree. `Archive` already records per-pair winners; it
|
||||
becomes the frontier, and the journal's parent records become the ancestry GEPA needs.
|
||||
|
||||
## Where the noise actually is, and why the architecture may be wrong
|
||||
|
||||
The maintainer's objection, and I think it is correct: the harness treats the detector as a black box
|
||||
emitting a noisy scalar, and everything follows from that — K runs, majority vote, a measured floor, a
|
||||
regression budget, a sign test. That is the stance you take when you cannot see inside the thing you are
|
||||
emitting a noisy scalar, and everything follows from that (K runs, majority vote, a measured floor, a
|
||||
regression budget, a sign test). That is the stance you take when you cannot see inside the thing you are
|
||||
measuring. We can see inside it.
|
||||
|
||||
The variance is not in the judgement. `temperature: 0` is sent and thinking is disabled, and 11 of 20
|
||||
@@ -147,8 +147,8 @@ than the symptom.
|
||||
|
||||
**Constraints as predicates, enforced at generation time** (`dspy.Assert`/`Suggest`, arXiv:2312.13382).
|
||||
A constraint is a boolean function of the output; on failure the past output *and the error* are injected
|
||||
into a retry. We already have the predicates — is the reported line inside a parsed function, does the
|
||||
named sink occur at that line, is the claimed family in the taxonomy, did the run cite a tool result —
|
||||
into a retry. We already have the predicates (is the reported line inside a parsed function, does the
|
||||
named sink occur at that line, is the claimed family in the taxonomy, did the run cite a tool result),
|
||||
and we apply every one of them **after the fact, at scoring time, to decide whether to count a finding**.
|
||||
Applying them at generation time to decide whether to *accept* one is free, needs no dependency, and is
|
||||
the single highest-value change available. The paper reports near-total intrinsic constraint satisfaction
|
||||
@@ -156,10 +156,10 @@ and 5–15% downstream gains.
|
||||
|
||||
**Typed decomposition instead of one free-running loop.** Our detector is a single ReAct loop that decides
|
||||
which files to read, what to grep, when to stop and what to report; every one of those is a variance
|
||||
source. DSPy's thesis is that each step gets a signature — a typed input/output contract — and a little
|
||||
source. DSPy's thesis is that each step gets a signature (a typed input/output contract), and a little
|
||||
control flow composes them. Ours would be roughly: locate candidate sink sites; trace whether an untrusted
|
||||
value reaches each; check for a guard between source and sink; judge. **Several of those steps need no
|
||||
model at all** — the CST, the entry-point mapper, `flows` and `obligations` already answer them
|
||||
model at all**. The CST, the entry-point mapper, `flows` and `obligations` already answer them
|
||||
deterministically, and we are currently paying a model to re-derive facts we hold.
|
||||
|
||||
**Best-of-N against a checkable reward, instead of a majority vote** (`dspy.Refine`, `BestOfN`). Both
|
||||
@@ -170,7 +170,7 @@ have checkable rewards for free. The same budget, used as evidence rather than a
|
||||
**Why this matters more for our models, not less.** GEPA and SIMBA are demonstrated with a strong
|
||||
reflection model over a weaker task model, and their gains come from evolving instructions. A cheap model
|
||||
fails on an under-specified task and does well on a narrow typed one. So the cheaper the detector model,
|
||||
the more of the return sits in decomposition and constraints and the less in prompt evolution — which is
|
||||
the more of the return sits in decomposition and constraints and the less in prompt evolution, which is
|
||||
the opposite of the order this spec originally proposed.
|
||||
|
||||
### The experiment, run (2026-09-06, `benchmarks/measurements/2026-09-06-trajectory-divergence.json`)
|
||||
@@ -178,7 +178,7 @@ the opposite of the order this spec originally proposed.
|
||||
Twelve injection pairs, both sides, K = 5, temperature 0, thinking disabled, recording the tool-call
|
||||
sequence of every run. $0.23.
|
||||
|
||||
**The intended statistic could not be computed. Across 120 runs there was not one repeated trajectory** —
|
||||
**The intended statistic could not be computed. Across 120 runs there was not one repeated trajectory**. There were
|
||||
4.67 distinct trajectories per vulnerable side and 4.92 per fixed side, out of 5, and 78 distinct tool-*name*
|
||||
sequences ignoring arguments entirely. The detector is not sampling one measurement five times. It is running
|
||||
five different investigations and voting on the results.
|
||||
@@ -193,21 +193,21 @@ Asked as a gradient instead, the relationship is monotone and clean:
|
||||
|
||||
Three more facts from the same run:
|
||||
|
||||
- **The variance is a search problem, and the search is `grep_repo`.** 555 grep calls against 181 `read_file`,
|
||||
- The variance is a search problem, and the search is `grep_repo`. 555 grep calls against 181 `read_file`,
|
||||
71 `entry_points`, 43 `find_refs`, 13 `read_definition`. The model spends its budget guessing patterns to
|
||||
locate a sink — while the CST, the entry-point mapper, `flows` and `obligations` already know where the
|
||||
locate a sink, while the CST, the entry-point mapper, `flows` and `obligations` already know where the
|
||||
sinks are.
|
||||
- **All of the instability is on the vulnerable side.** Every one of the twelve pairs produced exactly one
|
||||
- All of the instability is on the vulnerable side. Every one of the twelve pairs produced exactly one
|
||||
distinct finding set on the fixed side. The detector is reliably silent on the fix and unreliably vocal on
|
||||
the bug, so the noise is entirely in detection and none of it in false positives.
|
||||
- **Findings converge far more than trajectories do.** 78 distinct trajectories, but one to three distinct
|
||||
- Findings converge far more than trajectories do. 78 distinct trajectories, but one to three distinct
|
||||
finding sets per pair. Many different investigations reach the same answer; six of twelve pairs are stable
|
||||
despite never once repeating a trajectory.
|
||||
|
||||
**What this does to the noise architecture.** The floor is not measuring the detector's uncertainty about a
|
||||
question. It is measuring how often an unconstrained search happens to reach the sink. A majority vote over
|
||||
five runs is therefore not variance reduction — it is an unweighted ensemble of five arbitrary search
|
||||
procedures — and a regression budget derived from it is a budget for the search's luck. K runs, the floor,
|
||||
five runs is therefore an unweighted ensemble of five arbitrary search
|
||||
procedures rather than variance reduction, and a regression budget derived from it is a budget for the search's luck. K runs, the floor,
|
||||
the budget and the sign test are all statistical treatment for a structural problem, and the 0% and 4.2%
|
||||
rows of that table say what happens when the structure is fixed instead.
|
||||
|
||||
@@ -224,45 +224,45 @@ answer "is our own instability structural?" in twenty minutes is the thing that
|
||||
|
||||
**Was answering the wrong question.**
|
||||
|
||||
- **Req 3.5, K ≥ 3 runs per pair.** The premise is variance reduction over a repeated measurement. There is
|
||||
- Req 3.5, K ≥ 3 runs per pair. The premise is variance reduction over a repeated measurement. There is
|
||||
no repeated measurement: 120 runs, 120 trajectories. K runs is an *ensemble of distinct search
|
||||
procedures*, and a majority vote over it is an unweighted ensemble, not an estimate. It should either be
|
||||
renamed and published as an ensemble — which changes what every family number means — or dropped once the
|
||||
renamed and published as an ensemble, which changes what every family number means, or dropped once the
|
||||
search is constrained.
|
||||
- **Req 8.2, the noise floor as the regression budget.** The floor measures how often an unconstrained
|
||||
- Req 8.2, the noise floor as the regression budget. The floor measures how often an unconstrained
|
||||
search happens to reach the sink. Deriving a regression budget from it means the worse the search wanders,
|
||||
the more collateral damage a round is licensed to do. That is backwards, and it is why injection's budget
|
||||
is 11 of 20.
|
||||
- **Req 9.5, the acceptance rule.** Already flagged for reading the holdout every round. Now also: it
|
||||
- Req 9.5, the acceptance rule. Already flagged for reading the holdout every round. Now also: it
|
||||
compares deltas between two ensemble votes, and calls the difference an improvement.
|
||||
- **The sign test and `reliable_change`.** Added this session in good faith; they test whether an ensemble's
|
||||
- The sign test and `reliable_change`. Added this session in good faith; they test whether an ensemble's
|
||||
vote moved, which is not the question either.
|
||||
|
||||
**What the data adds that neither spec anticipated.**
|
||||
|
||||
- **The instability is entirely one-sided.** All twelve pairs produced exactly one finding set on the fixed
|
||||
- The instability is entirely one-sided. All twelve pairs produced exactly one finding set on the fixed
|
||||
side. Specificity is solid and recall is the noisy half, so the leak ceiling and the false-positive
|
||||
machinery are sound and it is the recall numbers that carry the caveat.
|
||||
- **`grep_repo` is the variance engine** — 555 calls against 181 `read_file` and 71 `entry_points`. The
|
||||
- `grep_repo` is the variance engine. It accounts for 555 calls against 181 `read_file` and 71 `entry_points`. The
|
||||
single highest-leverage change in either spec is not a prompt, a lever or an optimiser: **stop asking the
|
||||
model to locate the sink.** The CST, the entry-point mapper, `flows` and `obligations` already know. Give
|
||||
the detector the candidate sites and ask it the question only a model can answer — does untrusted input
|
||||
the detector the candidate sites and ask it the question only a model can answer. Does untrusted input
|
||||
reach this one, and is there a guard.
|
||||
- **The loop is partly self-correcting**, so this should be a large win rather than a marginal one: 78
|
||||
- The loop is partly self-correcting, so this should be a large win rather than a marginal one: 78
|
||||
distinct trajectories collapse to one to three finding sets per pair, and half the pairs are already
|
||||
stable despite never repeating an investigation.
|
||||
|
||||
**Limits of the experiment, stated so the conclusion is not over-read.** Twelve pairs, one family, one
|
||||
slice, one model. The high-similarity bucket holds only two run pairs, so its 0% is decoration; the
|
||||
defensible number is the mid bucket's 4.2% over 24 run pairs against 30.9% over 94. And the direction of
|
||||
causation is inferred, not established — similar trajectories may be a symptom of easy pairs rather than a
|
||||
causation is inferred, not established. Similar trajectories may be a symptom of easy pairs rather than a
|
||||
cause of agreement. The way to settle that is the constrained detector itself: fix the investigation, and
|
||||
if the floor does not collapse, this reading was wrong.
|
||||
|
||||
## The constraint that outranks the algorithm
|
||||
|
||||
Both algorithms assume a dataset. GEPA splits `D_train` into `D_feedback` and `D_pareto`; SIMBA's default
|
||||
batch size is 32. **Our injection family has 12 train teachers** — fewer than one SIMBA batch, and after a
|
||||
batch size is 32. **Our injection family has 12 train teachers** (fewer than one SIMBA batch), and after a
|
||||
GEPA split roughly six instances on each side. Every other family has one to three.
|
||||
|
||||
So the honest statement of what this spec can and cannot buy: GEPA and SIMBA are sample-efficient in
|
||||
@@ -279,42 +279,42 @@ selection, the budget accounting, and the artifacts that make a run reproducible
|
||||
|
||||
Out, and deliberately: the taxonomy, the classifier, the pair scorer's *outcome* rules, the verifier
|
||||
boundary, the write roots, redaction, the three gates. Those are the measurement, and a search whose
|
||||
objective is not trustworthy is worse than no search. Nothing here relaxes them — in particular the
|
||||
objective is not trustworthy is worse than no search. Nothing here relaxes them. In particular, the
|
||||
proposer stays blind to the holdout, verifier code and gold labels stay outside every write root, and
|
||||
no pattern or policy is written by a model.
|
||||
|
||||
## Decisions for the maintainer
|
||||
|
||||
0. **Is the optimiser even the next thing to build?** Proposed no, and this reverses the order above.
|
||||
0. Is the optimiser even the next thing to build? Proposed no, and this reverses the order above.
|
||||
Constraints at generation time, typed decomposition and best-of-N attack the variance at its source; an
|
||||
optimiser only tunes text around a program whose freedom is the problem. Proposed sequence: measure where
|
||||
the variance is, constrain generation, decompose the loop, then optimise — and at a trainset of twelve the
|
||||
the variance is, constrain generation, decompose the loop, then optimise, and at a trainset of twelve the
|
||||
optimiser that fits is `BootstrapFewShot` (collect traces that scored well, install them as demonstrations
|
||||
in a lever we already have) rather than an evolutionary search with nothing to search over.
|
||||
|
||||
0b. **Adopt DSPy itself?** Proposed no. Our detector is not a DSPy program — it is an agentic tool loop over
|
||||
0b. Adopt DSPy itself? Proposed no. Our detector is not a DSPy program. It is an agentic tool loop over
|
||||
a repository with curated tools, redaction on every prompt, path clamping, a "no findings unless a tool
|
||||
was called" rule and a final-answer turn, several of which are security properties rather than
|
||||
conveniences. Handing that loop to `dspy.ReAct` and the calls to litellm would put them behind someone
|
||||
else's abstraction, and DSPy could only ever be an optional extra under `dependencies = []`, which means
|
||||
two code paths for the thing that *is* the product. What DSPy would genuinely buy — signatures, adapters,
|
||||
multi-module composition — we barely need at one module per family. The optimisers are available without
|
||||
two code paths for the thing that *is* the product. What DSPy would buy (signatures, adapters,
|
||||
multi-module composition), we barely need at one module per family. The optimisers are available without
|
||||
the framework: `gepa` ships standalone under MIT, and SIMBA's two ideas are a hundred lines of logic.
|
||||
Proposed instead: take the interface shape and the algorithms, not the framework.
|
||||
|
||||
1. **Reimplement Algorithms 1 and 2, take `gepa` as an extra, or both?** Both is proposed: ours by
|
||||
1. Reimplement Algorithms 1 and 2, take `gepa` as an extra, or both? Both is proposed: ours by
|
||||
default (zero-dependency, and the security properties are ours to own), `gepa` behind an extra for
|
||||
comparison. The cost is one adapter and one conformance test that both backends must pass.
|
||||
2. **Drop the holdout from the acceptance rule?** Proposed yes, and it amends approved Req 9.5. The
|
||||
2. Drop the holdout from the acceptance rule? Proposed yes, and it amends approved Req 9.5. The
|
||||
holdout becomes what its name says.
|
||||
3. **Sample instances by disagreement (SIMBA) or uniformly (GEPA)?** Proposed SIMBA's ranking. Our measured
|
||||
3. Sample instances by disagreement (SIMBA) or uniformly (GEPA)? Proposed SIMBA's ranking. Our measured
|
||||
floor makes it the better fit, and it turns the noise floor from an obstacle into an instance sampler.
|
||||
|
||||
4. **K = 1 inside the search, K ≥ 3 only for published numbers?** Proposed yes. That is roughly a 5×
|
||||
4. K = 1 inside the search, K ≥ 3 only for published numbers? Proposed yes. That is roughly a 5×
|
||||
budget reclaim spent on more candidates, which is where the sample efficiency comes from.
|
||||
5. **Does the reflection model differ from the detector model?** The published work uses a stronger
|
||||
5. Does the reflection model differ from the detector model? The published work uses a stronger
|
||||
reflector than the task model. We have `deepseek-v4-pro` configured as the judge; proposed to reuse it.
|
||||
6. **Is `access_control` still the family we care most about?** It has one train teacher. Until agent-vfc
|
||||
6. Is `access_control` still the family we care most about? It has one train teacher. Until agent-vfc
|
||||
rows are reviewed above tier `title`, the optimiser cannot work on it, whatever its algorithm.
|
||||
|
||||
## Open questions
|
||||
@@ -322,7 +322,7 @@ no pattern or policy is written by a model.
|
||||
- Does `gepa` run against an OpenAI-shaped base URL that is not OpenAI (our DeepSeek endpoint), and what
|
||||
does it pull in? Both must be checked before the extra is offered, under the no-undeclared-dependency rule.
|
||||
- Our "module" granularity: is a module a family, or a lever within a family? Round-robin over levers is
|
||||
the closer analogue, but the levers are not independent — a checklist edit and a prompt edit interact.
|
||||
the closer analogue, but the levers are not independent. A checklist edit and a prompt edit interact.
|
||||
- The feedback function's failure mode is a reflection model that writes plausible prose about a trace it
|
||||
misread. What check makes a feedback string answerable rather than persuasive?
|
||||
- GEPA's acceptance is a single noisy comparison. Our floor says a single comparison is close to a coin
|
||||
|
||||
@@ -29,10 +29,10 @@ already reaches, while the **semantic IR caps it at 96.6%** with a median of ten
|
||||
|
||||
### This Spec Owns
|
||||
|
||||
- `learning/candidates.py` — what a candidate is, how the IR enumerates one per family shape, and the ceiling report.
|
||||
- `learning/judgments.py` — the typed answer, and the toolless call that produces it.
|
||||
- `learning/predicates.py` — the checks that constrain a claim at generation time and score it afterwards, defined once.
|
||||
- `learning/slices.py` — deciding versus development slices, the overfitting gap, and the cross-slice rejection rule.
|
||||
- `learning/candidates.py`, what a candidate is, how the IR enumerates one per family shape, and the ceiling report.
|
||||
- `learning/judgments.py`, the typed answer, and the toolless call that produces it.
|
||||
- `learning/predicates.py`, the checks that constrain a claim at generation time and score it afterwards, defined once.
|
||||
- `learning/slices.py`, deciding versus development slices, the overfitting gap, and the cross-slice rejection rule.
|
||||
- `run_constrained_detector` in `learning/detectors.py`, beside the existing `run_family_detector`.
|
||||
- `ousast learning candidates`, and a detector selector on `score` and `baseline`.
|
||||
|
||||
@@ -113,16 +113,16 @@ src/openultrasast/learning/
|
||||
|
||||
### Modified Files
|
||||
|
||||
- `src/openultrasast/learning/detectors.py` — `run_constrained_detector(root, region, config, *, client, model,
|
||||
- `src/openultrasast/learning/detectors.py`, `run_constrained_detector(root, region, config, *, client, model,
|
||||
taxonomy, ranges)`; `FamilyConfig` gains nothing, because a constrained detector is configured by the same
|
||||
prompt/checklist/limits and reads no tools.
|
||||
- `src/openultrasast/learning/scoring.py` — `_hits`, `_others` and `_fabricated` delegate to `predicates`; the
|
||||
- `src/openultrasast/learning/scoring.py`, `_hits`, `_others` and `_fabricated` delegate to `predicates`; the
|
||||
family metrics gain nothing. A test asserts the recorded baseline numbers are reproduced exactly.
|
||||
- `src/openultrasast/learning/publish.py` — the overfitting gap and the per-slice table (Req 3.1, 3.4).
|
||||
- `src/openultrasast/config.py` — `[learning] detector = "hunter" | "constrained"`, default `hunter` until the
|
||||
- `src/openultrasast/learning/publish.py`, the overfitting gap and the per-slice table (Req 3.1, 3.4).
|
||||
- `src/openultrasast/config.py`, `[learning] detector = "hunter" | "constrained"`, default `hunter` until the
|
||||
comparison says otherwise.
|
||||
- `src/openultrasast/cli.py` — `learning candidates`; `--detector` on `score` and `baseline`.
|
||||
- `pyproject.toml` — unchanged.
|
||||
- `src/openultrasast/cli.py`, `learning candidates`; `--detector` on `score` and `baseline`.
|
||||
- `pyproject.toml`, unchanged.
|
||||
|
||||
## System Flows
|
||||
|
||||
@@ -282,13 +282,13 @@ def teachable(slice_name: str, cases) -> bool # False while every row of a
|
||||
|
||||
## Migration and Phases
|
||||
|
||||
1. **Candidates and the ceiling, offline.** `candidates.py`, `learning candidates`, the per-slice ceiling artifact
|
||||
1. Candidates and the ceiling, offline. `candidates.py`, `learning candidates`, the per-slice ceiling artifact
|
||||
including the real-world slices (Req 3.5). No model call, so the phase is free and its result can still cancel
|
||||
the design — if the ceiling on a deciding slice is below that slice's current detection rate, stop here.
|
||||
2. **Predicates extracted.** `predicates.py`, `scoring.py` delegating, the baseline-reproduction test.
|
||||
3. **Judgment and the constrained detector.** `judgments.py`, `run_constrained_detector`, the determinism test.
|
||||
4. **Slices and reporting.** `slices.py`, the overfitting gap, `publish` per slice.
|
||||
5. **The comparison.** Constrained against unconstrained on every slice, the re-measured trajectory count and noise
|
||||
the design. If the ceiling on a deciding slice is below that slice's current detection rate, stop here.
|
||||
2. Predicates extracted. `predicates.py`, `scoring.py` delegating, the baseline-reproduction test.
|
||||
3. Judgment and the constrained detector. `judgments.py`, `run_constrained_detector`, the determinism test.
|
||||
4. Slices and reporting. `slices.py`, the overfitting gap, `publish` per slice.
|
||||
5. The comparison. Constrained against unconstrained on every slice, the re-measured trajectory count and noise
|
||||
floor, and the named `learning-harness` amendments if the floor collapses.
|
||||
|
||||
Phase 1 is deliberately able to kill the design before any money is spent on it.
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
## Introduction
|
||||
|
||||
The detector's instability is structural, not statistical. Measured on 2026-09-06 over twelve injection pairs at
|
||||
temperature 0 with thinking disabled, **120 runs produced 120 distinct tool-call trajectories** — not one repeat —
|
||||
temperature 0 with thinking disabled, **120 runs produced 120 distinct tool-call trajectories**, not one repeat,
|
||||
and outcome disagreement tracks trajectory divergence (30.9% at low similarity against 4.2% at mid). The model
|
||||
spends its budget searching: 555 `grep_repo` calls against 71 `entry_points`, guessing where sinks are while the
|
||||
semantic IR already knows. So `learning-harness`'s K runs are an ensemble of distinct search procedures rather than
|
||||
@@ -20,22 +20,22 @@ ruleset caps recall at 12.1%, below the 64.9% the unconstrained loop already rea
|
||||
at 96.6% with a median of ten sites per labeled function.
|
||||
|
||||
Baseline to beat, `deepseek-v4-flash`, K = 5, 57 scorable web pairs: **detection 64.9%, leak 12.3%, pair-correct
|
||||
52.6%** — and 83.3% detection on hand-written vibe-py against 44.4% on agent-written agent-vfc, which is the gap
|
||||
52.6%**, and 83.3% detection on hand-written vibe-py against 44.4% on agent-written agent-vfc, which is the gap
|
||||
this feature must close rather than widen. Context and measurements: `brief.md`,
|
||||
`benchmarks/measurements/2026-09-06-trajectory-divergence.json`, `2026-09-06-candidate-ceiling.json`.
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: candidate enumeration from the semantic IR, per family shape; the typed, toolless judgment step;
|
||||
- In scope: candidate enumeration from the semantic IR, per family shape; the typed, toolless judgment step;
|
||||
constraint predicates enforced at generation time with a bounded retry; the evidence that decides whether the
|
||||
K-run and noise-floor machinery still earns its place; per-slice reporting that keeps real-world code in front.
|
||||
- **Out of scope**: the family taxonomy, the classifier, the pair scorer's outcome rules, the split, the verifier
|
||||
boundary and write roots, redaction, the three gates, and the publication machinery — all of which stand and are
|
||||
- Out of scope: the family taxonomy, the classifier, the pair scorer's outcome rules, the split, the verifier
|
||||
boundary and write roots, redaction, the three gates, and the publication machinery, all of which stand and are
|
||||
what made this feature's measurements possible. Also out: prompt optimisation, reflective evolution and any
|
||||
optimiser (deferred to a later spec, on this feature's evidence); finding bugs in JavaScript interpreters, WebGL
|
||||
renderers or browser engines; memory-safety fuzzing for C and C++; fine-tuning model weights; adopting DSPy as a
|
||||
dependency.
|
||||
- **Adjacent expectations**: `learning-harness` keeps owning the taxonomy, the scorer, the split and publication;
|
||||
- Adjacent expectations: `learning-harness` keeps owning the taxonomy, the scorer, the split and publication;
|
||||
its Req 3.5 (K runs), 8.2 (noise floor) and 9.5 (acceptance) are amended by this feature only on evidence it
|
||||
produces, never on argument. The pair corpus keeps owning labels, tiers and splits. The unconstrained tool hunter
|
||||
stays as the generalist and as the baseline this feature is measured against.
|
||||
@@ -49,8 +49,8 @@ model stops searching and the limit of what it can possibly find is a number rat
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The Constrained Detector shall enumerate candidate sites for a region from the semantic IR — call sites and
|
||||
bindings — and shall not use the pattern ruleset as its candidate generator.
|
||||
1. The Constrained Detector shall enumerate candidate sites for a region from the semantic IR (call sites and
|
||||
bindings), and shall not use the pattern ruleset as its candidate generator.
|
||||
2. The Constrained Detector shall shape candidates by family: a call site for injection, path, deserialization,
|
||||
untrusted destination and prototype; an operation on a protected resource with the guards in scope for access
|
||||
control; a configuration assignment or literal for permissive configuration and secrets.
|
||||
@@ -90,13 +90,13 @@ teaching examples, so that the tool is not tuned for cases nobody has.
|
||||
|
||||
1. The Constrained Detector shall report every number per slice and shall never publish an aggregate across slices
|
||||
without the per-slice numbers beside it.
|
||||
2. The Constrained Detector shall designate the slices harvested from real projects — agent-vfc, vfc-js, vfc and
|
||||
github — as the deciding slices, and the hand-written vibe-py slice as a development slice that cannot on its
|
||||
2. The Constrained Detector shall designate the slices harvested from real projects (agent-vfc, vfc-js, vfc and
|
||||
github) as the deciding slices, and the hand-written vibe-py slice as a development slice that cannot on its
|
||||
own justify accepting a change.
|
||||
3. If a change improves the development slice and regresses any deciding slice, then the Constrained Detector shall
|
||||
reject it, and the rejection shall name the slice that regressed.
|
||||
4. The Constrained Detector shall publish the **overfitting gap** — development-slice detection minus deciding-slice
|
||||
detection — beside every headline number, and shall state that the baseline gap is 83.3% against 44.4%.
|
||||
4. The Constrained Detector shall publish the **overfitting gap** (development-slice detection minus deciding-slice
|
||||
detection) beside every headline number, and shall state that the baseline gap is 83.3% against 44.4%.
|
||||
5. The Constrained Detector shall measure and publish the candidate ceiling separately for the real-world slices,
|
||||
because the functions there are larger than the teaching corpus's and the enumerator's behaviour on them is not
|
||||
implied by its behaviour on vibe-py.
|
||||
|
||||
Reference in New Issue
Block a user