marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-18 12:39:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(learning-harness): the published table lost a whole slice

Browse Source 
`publish` keyed baselines by model alone, so the vibe-py run silently
replaced the agent-vfc run and the table showed hand-written teaching code as
if it were the corpus. That is precisely the failure the constrained-detector
requirements exist to prevent, live in a published artifact.

Third time today that a key was fixed in one place and not another --
`compare_baselines` got the model+slice label and `_baselines` did not, the
same way the exception path got checkpointing and the timeout path did not.

The table now carries a slice column and one row per slice, and the artifact
is keyed by what identifies a run. Republished: agent-vfc's six families are
back, including access_control at 0.000 recall on four pairs, which is the
number that was missing.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-06 20:44:56 +02:00
co-authored by Claude Opus 5
parent 506bbe5f5c
commit 84b1ba7a72
4 changed files with 715 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.kiro/steering/roadmap.md
+27 -2
View File
@@ -94,8 +94,33 @@ sast: java-sqli moved MISS→LEAK (coverage counted on both sides; the fixed twi
## Classified detector numbers
Generated by `uv run ousast learning publish`.
Generated by `uv run ousast learning publish` from the artifacts committed beside these numbers.
No baseline has been recorded yet, so there is nothing to publish. Run round zero first; a number without the run that produced it is a claim, not a measurement.
| model | slice | family | scorable | unscorable | recall | silence | Youden | at fixed FPR | noise floor | cost per correct pair |
|---|---|---|---|---|---|---|---|---|---|---|
| deepseek-v4-flash | agent-vfc | access_control | 4 (taxonomy 1) | known_limit:client_side_caller: 1, unresolved_label: 1 | 0.000 | 1.000 | 0.000 | 0.000 | budget 4 | - |
| deepseek-v4-flash | agent-vfc | config_secrets | 6 (taxonomy 1) | none | 0.333 | 0.833 | 0.167 | 0.000 | budget 2 | 0.95 |
| deepseek-v4-flash | agent-vfc | injection | 2 (taxonomy 1) | none | 0.500 | 0.500 | 0.000 | 0.000 | budget 1 | - |
| deepseek-v4-flash | agent-vfc | path | 3 (taxonomy 1) | none | 1.000 | 0.667 | 0.667 | 0.000 | budget 3 | 0.48 |
| deepseek-v4-flash | agent-vfc | prototype | 7 (taxonomy 1) | none | 0.429 | 1.000 | 0.429 | 0.429 | budget 5 | 0.32 |
| deepseek-v4-flash | agent-vfc | untrusted_destination | 5 (taxonomy 1) | none | 0.600 | 0.600 | 0.200 | 0.000 | budget 3 | 0.95 |
| deepseek-v4-flash | vibe-py | access_control | 3 (taxonomy 1) | identical_twin: 1 | 1.000 | 1.000 | 1.000 | 1.000 | budget 3 | 0.23 |
| deepseek-v4-flash | vibe-py | config_secrets | 3 (taxonomy 1) | none | 1.000 | 1.000 | 1.000 | 1.000 | budget 1 | 0.23 |
| deepseek-v4-flash | vibe-py | deserialization | 2 (taxonomy 1) | identical_twin: 1 | 1.000 | 1.000 | 1.000 | 1.000 | budget 0 | 0.34 |
| deepseek-v4-flash | vibe-py | injection | 20 (taxonomy 1) | identical_twin: 2 | 0.750 | 0.900 | 0.650 | 0.750 | budget 11 | 0.05 |
| deepseek-v4-flash | vibe-py | output_encoding | 1 (taxonomy 1) | identical_twin: 1 | 1.000 | 1.000 | 1.000 | 1.000 | budget 1 | 0.69 |
| deepseek-v4-flash | vibe-py | path | 1 (taxonomy 1) | none | 1.000 | 1.000 | 1.000 | 1.000 | budget 1 | 0.69 |
| deepseek-v4-pro | vibe-py | access_control | 3 (taxonomy 1) | identical_twin: 1 | 0.000 | 1.000 | 0.000 | 0.000 | budget 2 | - |
| deepseek-v4-pro | vibe-py | config_secrets | 3 (taxonomy 1) | none | 1.000 | 1.000 | 1.000 | 1.000 | budget 2 | 0.16 |
| deepseek-v4-pro | vibe-py | deserialization | 2 (taxonomy 1) | identical_twin: 1 | 1.000 | 1.000 | 1.000 | 1.000 | budget 1 | 0.24 |
| deepseek-v4-pro | vibe-py | injection | 20 (taxonomy 1) | identical_twin: 2 | 0.350 | 0.900 | 0.250 | 0.350 | budget 14 | 0.09 |
| deepseek-v4-pro | vibe-py | output_encoding | 1 (taxonomy 1) | identical_twin: 1 | 1.000 | 1.000 | 1.000 | 1.000 | budget 0 | 0.47 |
| deepseek-v4-pro | vibe-py | path | 1 (taxonomy 1) | none | 1.000 | 1.000 | 1.000 | 1.000 | budget 0 | 0.47 |
No family in this taxonomy declares a parent, so hierarchical partial credit could not apply to any figure above.
No detector cutoff is configured, so no post-cutoff slice is reported; every figure above is over the whole corpus.
Mechanism lever on the vibe-py holdout: detected 5 of 17 at Youden -0.0588 once only train-split pairs teach, against detected 7 at Youden 0.0588 when every pair taught. The difference is what holdout pairs stopped teaching.
<!-- learning-harness:end -->