marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 11:49:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(detector-optimizer): brief for a published optimizer behind a DSPy-shaped interface

Browse Source 
The learning harness built the measurement honestly and then hill-climbed on
it by hand. Four review rounds' worth of defects in that loop were defects of
its shape, not its code.

Read from GEPA's Algorithm 1 and 2 rather than from a summary, three things
bear directly on ours: the acceptance gate is deliberately cheap and noisy,
because the Pareto frontier -- not the gate -- is what protects against a bad
accept; selection is stochastic and diversity-preserving, which is what stops
the search collapsing onto one lineage; and the expensive full evaluation is
spent only on candidates that already passed the cheap gate.

Two consequences for us are honesty findings, not performance ones. Our
acceptance rule reads the holdout every round, so repeated rounds overfit the
test set -- the same leak class this project already measured once, one level
up. And the proposer is handed a score where the published work hands it
textual feedback from the evaluation trace, which our evaluator produces and
then discards.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-06 15:16:29 +02:00
co-authored by Claude Opus 5
parent b57b1a35d8
commit 78f20df167
2 changed files with 175 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.kiro/specs/detector-optimizer/brief.md
+149
View File
@@ -0,0 +1,149 @@
# Brief: the detector optimizer
## Problem
`learning-harness` built the measurement layer honestly — a closed family taxonomy, a class-aware pair
scorer, per-family detector configurations, a split enforced by construction, verifier-gated reporting,
and a journal with attribution. What it did **not** build is a search. `run_learning_round` is a
hand-rolled hill-climber: one proposal, one minibatch, one holdout check, a threshold rule, accept or
revert. Every defect four review rounds found in it was a defect of that improvised loop rather than of
the measurement underneath it.
The maintainer's judgement, and the reason for this spec: the remaining work is not a sequence of
patches to that loop. It is adopting an interface and an algorithm that already exist, are published,
and are better than what we would arrive at by fixing ours one finding at a time.
## What the current loop gets wrong, measured
Round zero on vibe-py, `deepseek-v4-flash`, K = 5, $0.69, 23/30 pair-correct:
| family | scorable | recall | silence | Youden | noise floor |
|---|---|---|---|---|---|
| injection | 20 | 0.75 | 0.90 | +0.65 | **11** |
| access_control | 3 | 1.00 | 1.00 | +1.00 | 3 |
| config_secrets | 3 | 1.00 | 1.00 | +1.00 | 1 |
| deserialization | 2 | 1.00 | 1.00 | +1.00 | 0 |
| output_encoding | 1 | 1.00 | 1.00 | +1.00 | 1 |
| path | 1 | 1.00 | 1.00 | +1.00 | 1 |
Four defects follow from the loop's shape, not from its bugs:
1. **The acceptance rule cannot discriminate.** `decide` compares raw deltas: any regression on train or
holdout rejects, a tie rejects. The measured floor says 11 of 20 injection pairs disagree with
*themselves* across five runs, so a delta over a minibatch of 8 is mostly noise. The spec built
`sign_test` and `reliable_change` for exactly this and the rule calls neither.
2. **The holdout is an input to the decision.** `decide` reads `holdout_delta`. Req 5.4 keeps the
*proposer* blind to the holdout, and then the *selector* reads it every round. Repeated rounds
selecting on holdout performance is search-time overfitting of the test set — the same leak class
this project has already measured once, one level up.
3. **Noise is fought with repetition instead of diversity.** K = 5 spends five rollouts per pair per
side estimating variance. The published alternative spends that budget on more candidates and lets a
frontier keep whatever wins somewhere.
4. **The proposer is told a score, not what went wrong.** `FailureFacts` carries outcomes, run strings
and counts. The single largest lever in the published work is textual feedback from the *evaluation
trace* — and our evaluator produces exactly the right material (which finding landed on which line,
whether it fell inside the labeled span, whether the fixed side leaked, which family was claimed)
and then throws all of it away.
## What others built
**GEPA — Reflective Prompt Evolution (Agrawal et al., arXiv:2507.19457, ICLR 2026 oral).** Algorithm 1,
read from the paper rather than from a summary:
- Split `D_train` into `D_feedback` and `D_pareto`. Initialise the candidate pool `P = [Φ]` and score
the seed on every instance of `D_pareto`, giving a score **matrix** `S[candidate][instance]`.
- Each iteration: select a candidate from the Pareto frontier; select a module round-robin; run it on a
minibatch drawn from `D_feedback`; collect scores, traces and **textual feedback** via a feedback
function `μ_f`; have a reflection LM rewrite that module's prompt; re-run the mutated system on the
*same* minibatch.
- **Accept if and only if the minibatch average improved.** Only then is the candidate scored on all of
`D_pareto` and added to the pool with its parent recorded.
- Return the candidate with the best average on `D_pareto`.
Algorithm 2, the selection: for each instance take the best score any candidate achieved; keep the
candidates that achieve it; drop strictly dominated ones; sample a survivor with probability
proportional to the number of instances it leads on.
Three things about that are directly relevant to us. The acceptance gate is **cheap and noisy on
purpose** — a single minibatch comparison — because the frontier, not the gate, is what protects
against a bad accept: anything that wins on one instance is retained. Selection is **stochastic and
diversity-preserving**, which is what stops the search collapsing onto one lineage. And the expensive
full evaluation is spent only on candidates that already passed the cheap gate.
Reported: 93% on MATH with a DSPy `ChainOfThought` program against 67% unoptimised, +10% over MIPROv2
and +20% over GRPO with **35× fewer rollouts**; on IFBench, optimal prompts after 678 rollouts against
GRPO's 24,000. Sample efficiency is the claim that matters for us, because our rollouts cost money.
**DSPy's optimizer interface.** `MIPROv2` proposes instruction and demonstration candidates and searches
them with Bayesian optimisation (Optuna TPE), scoring on minibatches and running a full validation pass
on the incumbent every `minibatch_full_eval_steps` — an explicit two-fidelity design for a noisy metric.
`dspy.GEPA` takes `metric`, `reflection_lm`, a budget (`auto` / `max_full_evals` / `max_metric_calls`),
`reflection_minibatch_size`, `candidate_selection_strategy` (`pareto` | `current_best`) and
`track_stats`, and its metric returns `Prediction(score, feedback)` rather than a float.
**The standalone `gepa` library** (MIT) exposes `gepa.optimize(seed_candidate, trainset, valset,
task_lm, reflection_lm, max_metric_calls)` over a `GEPAAdapter` protocol of three methods —
`evaluate`, `make_reflective_dataset`, `propose_new_texts` — with a seed candidate that is just a dict
of component name to text. That adapter shape is the interface we want whether or not we take the
dependency: it is the smallest surface that separates *what a detector is* from *how it is searched*.
## What this spec proposes
Restate the detector as an optimisable program and put a published optimiser behind it.
- **A detector is a candidate**: a dict of named text components (`prompt`, `checklist`,
`hard_negatives`, `counterexamples`) plus its frozen non-text settings (tools, budgets). That is
already what `FamilyConfig` holds; it becomes the seed candidate.
- **The metric returns a score and a reason.** `score_pair_family` gains a feedback string built from
what the evaluation already knows. This is the highest-value change in the spec and it is independent
of which optimiser sits on top.
- **`D_pareto` is carved out of the train split.** The holdout leaves the acceptance rule entirely and
is scored once, for reporting, when the search is over. This is a requirements change to Req 9.5 and
the most important honesty consequence of the redesign.
- **An adapter protocol** (`evaluate`, `make_reflective_dataset`, `propose_new_texts`) with two backends:
our own implementation of Algorithms 1 and 2 by default, so the core install keeps
`dependencies = []`, and `gepa` behind an optional extra so its improvements arrive for free.
- **The Pareto pool replaces the accept/revert tree.** `Archive` already records per-pair winners; it
becomes the frontier, and the journal's parent records become the ancestry GEPA needs.
## Scope
In: the search over a family's text components, the feedback function, the candidate pool and its
selection, the budget accounting, and the artifacts that make a run reproducible.
Out, and deliberately: the taxonomy, the classifier, the pair scorer's *outcome* rules, the verifier
boundary, the write roots, redaction, the three gates. Those are the measurement, and a search whose
objective is not trustworthy is worse than no search. Nothing here relaxes them — in particular the
proposer stays blind to the holdout, verifier code and gold labels stay outside every write root, and
no pattern or policy is written by a model.
## Decisions for the maintainer
1. **Reimplement Algorithms 1 and 2, take `gepa` as an extra, or both?** Both is proposed: ours by
default (zero-dependency, and the security properties are ours to own), `gepa` behind an extra for
comparison. The cost is one adapter and one conformance test that both backends must pass.
2. **Drop the holdout from the acceptance rule?** Proposed yes, and it amends approved Req 9.5. The
holdout becomes what its name says.
3. **K = 1 inside the search, K ≥ 3 only for published numbers?** Proposed yes. That is roughly a 5×
budget reclaim spent on more candidates, which is where the sample efficiency comes from.
4. **Does the reflection model differ from the detector model?** The published work uses a stronger
reflector than the task model. We have `deepseek-v4-pro` configured as the judge; proposed to reuse it.
5. **Is `access_control` still the family we care most about?** It has one train teacher. Until agent-vfc
rows are reviewed above tier `title`, the optimiser cannot work on it, whatever its algorithm.
## Open questions
- Does `gepa` run against an OpenAI-shaped base URL that is not OpenAI (our DeepSeek endpoint), and what
does it pull in? Both must be checked before the extra is offered, under the no-undeclared-dependency rule.
- Our "module" granularity: is a module a family, or a lever within a family? Round-robin over levers is
the closer analogue, but the levers are not independent — a checklist edit and a prompt edit interact.
- The feedback function's failure mode is a reflection model that writes plausible prose about a trace it
misread. What check makes a feedback string answerable rather than persuasive?
- GEPA's acceptance is a single noisy comparison. Our floor says a single comparison is close to a coin
flip on this corpus. Does the frontier really absorb that at n ≈ 20 instances, or is our corpus simply
too small for the algorithm's assumptions?
Sources: [GEPA (arXiv:2507.19457)](https://arxiv.org/abs/2507.19457) ·
[DSPy GEPA overview](https://dspy.ai/api/optimizers/GEPA/overview/) ·
[DSPy MIPROv2](https://dspy.ai/api/optimizers/MIPROv2/) ·
[gepa-ai/gepa](https://github.com/gepa-ai/gepa)
.kiro/specs/detector-optimizer/spec.json
+26
View File
@@ -0,0 +1,26 @@
{
"feature_name": "detector-optimizer",
"created_at": "2026-09-06T14:30:00Z",
"updated_at": "2026-09-06T14:30:00Z",
"language": "en",
"phase": "brief",
"approvals": {
"requirements": {
"generated": false,
"approved": false
},
"design": {
"generated": false,
"approved": false
},
"tasks": {
"generated": false,
"approved": false
}
},
"ready_for_implementation": false,
"depends_on": [
"learning-harness"
],
"approval_note": "2026-09-06 brief drafted at the maintainer's direction ('I'd rather spend time to design the exact DSPy style interface and evolutionary / statistical prompt optimization than fixing piece by piece'). Not approved; requirements not generated."
}