mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-19 21:19:28 +00:00
feat(plane): honest coverage in alerts; PHP alerts from the engine on the host
Quick mode has rules only for c/cpp, groovy, java, javascript/typescript and python, so every PHP case's "0 alerts" read as clean. The `alerts` task now records per language of the files read whether quick mode covers it (derived from the bundled ruleset, not a list): coverage quick | none, with file counts per pin, and `uncovered`. `remember` turns that into `coverage` memory rows (a new kind; the Run passes the alerts summary.json as input), loop-measure reports coverage per language, and improve/memory.py's M1/M2 skip a repository+pin whose coverage for the rule's language is only `none` (counted under `coverage_none` in the proposal's excluded). `ousast plane alerts-engine <Run.yaml> --source <frozen export>` (plane/engine_alerts.py) does the alerts task on the host for cases whose files include a language the engine covers but quick mode does not (PHP): exports both pins from the case cache, runs the quick scan and the openultrasast:dev engine container (finding_dump.py, --network none, --memory 3g, one at a time, per-pin deadline), writes engine findings in the alerts.jsonl row shape (rule_id engine:<family>, source engine, in_fix_range from case.json) with a summary (files read, questions, completed, seconds, degradations per pin), and marks the task done via reconciler.mark_done. No result, no files read, or questions == 0 fails loudly and leaves the run untouched. Refuses a git checkout as source. Runner image rebuilt and re-pinned (sha256:ebb40a9a...); validation-46 regenerated (digest only). Live, recorded in benchmarks/measurements/2026-09-30-php-engine-alerts/: froxlor 366 files, 130/130 questions per pin, 6 alerts per pin, 0 in the fix range; wp-gopay 19 files, 101/101 questions, 0 alerts. The loop: 32 run rows (coverage 8, php=engine), 0 proposals, gate no_proposals, $0. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
f4bd984fda
commit
45d02ecf20
@@ -0,0 +1,235 @@
|
||||
# generated by `ousast plane workspaces benchmarks/independent/population-v2.toml --validation-set /tmp/claude-1000/-home-mc-Source-OpenUltraSAST/c09e30b2-39a4-4564-bc31-6ea71ffaa198/scratchpad/phpgap/set/validation.json --candidates ~/ousast-results/independent-v2-modelsinks --triage ~/ousast-results/independent-v2-batched-check/scans --repos ~/.cache/openultrasast/independent --run validation-46 --loop --loop-commit 1ad472c68d759e4ef010bd7a2e23532b1de538a9 --loop-repo https://github.com/norandom/OpenUltraSAST.git --loop-manifest benchmarks/manifests/java-spring-boot-vulnerable.toml`; do not edit, regenerate.
|
||||
# 2 cases, 4 candidates (4 kept by the recorded triage); usd ceiling 0.27.
|
||||
apiVersion: openultrasast.io/v1alpha1
|
||||
kind: Run
|
||||
metadata:
|
||||
name: validation-46
|
||||
annotations:
|
||||
openultrasast.io/population: population-v2
|
||||
openultrasast.io/split: validation
|
||||
spec:
|
||||
tasks:
|
||||
- name: froxlor-admin-ip-second-order-sqli-facts
|
||||
task: repo-facts-froxlor-admin-ip-second-order-sqli
|
||||
outputs:
|
||||
- facts.json
|
||||
budget: &id001
|
||||
usd: 0
|
||||
calls: 0
|
||||
serialize: repo-facts
|
||||
- name: froxlor-admin-ip-second-order-sqli-va
|
||||
task: verify-a-froxlor-admin-ip-second-order-sqli
|
||||
inputs:
|
||||
facts: froxlor-admin-ip-second-order-sqli-facts/facts.json
|
||||
outputs:
|
||||
- units.jsonl
|
||||
- summary.json
|
||||
budget:
|
||||
usd: 0.07
|
||||
calls: 16
|
||||
- name: froxlor-admin-ip-second-order-sqli-vb
|
||||
task: verify-b-froxlor-admin-ip-second-order-sqli
|
||||
inputs:
|
||||
facts: froxlor-admin-ip-second-order-sqli-facts/facts.json
|
||||
outputs:
|
||||
- units.jsonl
|
||||
- summary.json
|
||||
budget:
|
||||
usd: 0.07
|
||||
calls: 16
|
||||
- name: froxlor-admin-ip-second-order-sqli-agree
|
||||
task: agree-froxlor-admin-ip-second-order-sqli
|
||||
inputs:
|
||||
pass_a: froxlor-admin-ip-second-order-sqli-va/units.jsonl
|
||||
pass_b: froxlor-admin-ip-second-order-sqli-vb/units.jsonl
|
||||
outputs:
|
||||
- agreed.json
|
||||
- disputed.json
|
||||
budget: *id001
|
||||
- name: wp-gopay-log-filter-sqli-facts
|
||||
task: repo-facts-wp-gopay-log-filter-sqli
|
||||
outputs:
|
||||
- facts.json
|
||||
budget: &id002
|
||||
usd: 0
|
||||
calls: 0
|
||||
serialize: repo-facts
|
||||
- name: wp-gopay-log-filter-sqli-va
|
||||
task: verify-a-wp-gopay-log-filter-sqli
|
||||
inputs:
|
||||
facts: wp-gopay-log-filter-sqli-facts/facts.json
|
||||
outputs:
|
||||
- units.jsonl
|
||||
- summary.json
|
||||
budget:
|
||||
usd: 0.02
|
||||
calls: 8
|
||||
- name: wp-gopay-log-filter-sqli-vb
|
||||
task: verify-b-wp-gopay-log-filter-sqli
|
||||
inputs:
|
||||
facts: wp-gopay-log-filter-sqli-facts/facts.json
|
||||
outputs:
|
||||
- units.jsonl
|
||||
- summary.json
|
||||
budget:
|
||||
usd: 0.02
|
||||
calls: 8
|
||||
- name: wp-gopay-log-filter-sqli-agree
|
||||
task: agree-wp-gopay-log-filter-sqli
|
||||
inputs:
|
||||
pass_a: wp-gopay-log-filter-sqli-va/units.jsonl
|
||||
pass_b: wp-gopay-log-filter-sqli-vb/units.jsonl
|
||||
outputs:
|
||||
- agreed.json
|
||||
- disputed.json
|
||||
budget: *id002
|
||||
- name: froxlor-admin-ip-second-order-sqli-vc
|
||||
task: verify-c-froxlor-admin-ip-second-order-sqli
|
||||
dependsOn:
|
||||
- froxlor-admin-ip-second-order-sqli-agree
|
||||
inputs:
|
||||
facts: froxlor-admin-ip-second-order-sqli-facts/facts.json
|
||||
only: froxlor-admin-ip-second-order-sqli-agree/disputed.json
|
||||
outputs:
|
||||
- units.jsonl
|
||||
- summary.json
|
||||
budget:
|
||||
usd: 0.07
|
||||
calls: 16
|
||||
- name: froxlor-admin-ip-second-order-sqli-final
|
||||
task: agree-froxlor-admin-ip-second-order-sqli
|
||||
inputs:
|
||||
pass_a: froxlor-admin-ip-second-order-sqli-va/units.jsonl
|
||||
pass_b: froxlor-admin-ip-second-order-sqli-vb/units.jsonl
|
||||
pass_c: froxlor-admin-ip-second-order-sqli-vc/units.jsonl
|
||||
outputs:
|
||||
- agreed.json
|
||||
- disputed.json
|
||||
budget:
|
||||
usd: 0
|
||||
calls: 0
|
||||
- name: wp-gopay-log-filter-sqli-vc
|
||||
task: verify-c-wp-gopay-log-filter-sqli
|
||||
dependsOn:
|
||||
- wp-gopay-log-filter-sqli-agree
|
||||
inputs:
|
||||
facts: wp-gopay-log-filter-sqli-facts/facts.json
|
||||
only: wp-gopay-log-filter-sqli-agree/disputed.json
|
||||
outputs:
|
||||
- units.jsonl
|
||||
- summary.json
|
||||
budget:
|
||||
usd: 0.02
|
||||
calls: 8
|
||||
- name: wp-gopay-log-filter-sqli-final
|
||||
task: agree-wp-gopay-log-filter-sqli
|
||||
inputs:
|
||||
pass_a: wp-gopay-log-filter-sqli-va/units.jsonl
|
||||
pass_b: wp-gopay-log-filter-sqli-vb/units.jsonl
|
||||
pass_c: wp-gopay-log-filter-sqli-vc/units.jsonl
|
||||
outputs:
|
||||
- agreed.json
|
||||
- disputed.json
|
||||
budget:
|
||||
usd: 0
|
||||
calls: 0
|
||||
- name: froxlor-admin-ip-second-order-sqli-alerts
|
||||
task: alerts-froxlor-admin-ip-second-order-sqli
|
||||
outputs:
|
||||
- alerts.jsonl
|
||||
- summary.json
|
||||
budget:
|
||||
usd: 0
|
||||
calls: 0
|
||||
serialize: engine
|
||||
- name: froxlor-admin-ip-second-order-sqli-remember
|
||||
task: remember-froxlor-admin-ip-second-order-sqli
|
||||
inputs:
|
||||
facts: froxlor-admin-ip-second-order-sqli-facts/facts.json
|
||||
facts_summary: froxlor-admin-ip-second-order-sqli-facts/summary.json
|
||||
pass_a: froxlor-admin-ip-second-order-sqli-va/units.jsonl
|
||||
summary_a: froxlor-admin-ip-second-order-sqli-va/summary.json
|
||||
pass_b: froxlor-admin-ip-second-order-sqli-vb/units.jsonl
|
||||
summary_b: froxlor-admin-ip-second-order-sqli-vb/summary.json
|
||||
pass_c: froxlor-admin-ip-second-order-sqli-vc/units.jsonl
|
||||
summary_c: froxlor-admin-ip-second-order-sqli-vc/summary.json
|
||||
agreed: froxlor-admin-ip-second-order-sqli-final/agreed.json
|
||||
alerts: froxlor-admin-ip-second-order-sqli-alerts/alerts.jsonl
|
||||
alerts_summary: froxlor-admin-ip-second-order-sqli-alerts/summary.json
|
||||
outputs:
|
||||
- memory.jsonl
|
||||
budget:
|
||||
usd: 0
|
||||
calls: 0
|
||||
- name: wp-gopay-log-filter-sqli-alerts
|
||||
task: alerts-wp-gopay-log-filter-sqli
|
||||
outputs:
|
||||
- alerts.jsonl
|
||||
- summary.json
|
||||
budget:
|
||||
usd: 0
|
||||
calls: 0
|
||||
serialize: engine
|
||||
- name: wp-gopay-log-filter-sqli-remember
|
||||
task: remember-wp-gopay-log-filter-sqli
|
||||
inputs:
|
||||
facts: wp-gopay-log-filter-sqli-facts/facts.json
|
||||
facts_summary: wp-gopay-log-filter-sqli-facts/summary.json
|
||||
pass_a: wp-gopay-log-filter-sqli-va/units.jsonl
|
||||
summary_a: wp-gopay-log-filter-sqli-va/summary.json
|
||||
pass_b: wp-gopay-log-filter-sqli-vb/units.jsonl
|
||||
summary_b: wp-gopay-log-filter-sqli-vb/summary.json
|
||||
pass_c: wp-gopay-log-filter-sqli-vc/units.jsonl
|
||||
summary_c: wp-gopay-log-filter-sqli-vc/summary.json
|
||||
agreed: wp-gopay-log-filter-sqli-final/agreed.json
|
||||
alerts: wp-gopay-log-filter-sqli-alerts/alerts.jsonl
|
||||
alerts_summary: wp-gopay-log-filter-sqli-alerts/summary.json
|
||||
outputs:
|
||||
- memory.jsonl
|
||||
budget:
|
||||
usd: 0
|
||||
calls: 0
|
||||
- name: memory-snapshot
|
||||
task: memory-snapshot
|
||||
outputs:
|
||||
- rows.jsonl
|
||||
- index.json
|
||||
budget:
|
||||
usd: 0
|
||||
calls: 0
|
||||
- name: loop-measure
|
||||
task: loop-measure
|
||||
inputs:
|
||||
remember_froxlor-admin-ip-second-order-sqli: froxlor-admin-ip-second-order-sqli-remember/memory.jsonl
|
||||
remember_wp-gopay-log-filter-sqli: wp-gopay-log-filter-sqli-remember/memory.jsonl
|
||||
snapshot: memory-snapshot/rows.jsonl
|
||||
outputs:
|
||||
- measure.json
|
||||
- rows.jsonl
|
||||
budget:
|
||||
usd: 0
|
||||
calls: 0
|
||||
- name: loop-propose
|
||||
task: loop-propose
|
||||
inputs:
|
||||
measure: loop-measure/measure.json
|
||||
rows: loop-measure/rows.jsonl
|
||||
index: memory-snapshot/index.json
|
||||
outputs:
|
||||
- proposals.jsonl
|
||||
- memory_proposals.jsonl
|
||||
- signals.json
|
||||
budget:
|
||||
usd: 0
|
||||
calls: 0
|
||||
- name: loop-improve
|
||||
task: loop-improve
|
||||
inputs:
|
||||
proposals: loop-propose/proposals.jsonl
|
||||
outputs:
|
||||
- gate.json
|
||||
- journal.json
|
||||
budget:
|
||||
usd: 0
|
||||
calls: 0
|
||||
serialize: engine
|
||||
Reference in New Issue
Block a user