mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
feat(plane): the improvement loop as a plane Run (harnessx-removal task 4)
`ousast plane workspaces --validation-set ... --loop` appends the loop to a Run:
per case `alerts` (plane/tasks/alerts.py: the quick-mode scan on the vulnerable
and the fixed pin, a fixed-pin Workspace per case, alerts attributed to their
enclosing function by the repo-facts patterns, `in_fix_range` from the fix's
old-side and new-side ranges, failing on an unread tree) feeding `remember`,
then `memory-snapshot` -> `loop-measure` -> `loop-propose` -> `loop-improve`
(plane/tasks/loop.py). Every new task binds no Model and has budget
{usd: 0, calls: 0}. The snapshot is the store's rows after the train-on-test
guard, written on the host by memory.seed (a sandboxed run of it fails loudly);
propose runs M1/M2 against this repository at the commit under test; improve
runs one unchanged evolve.run_round and delivers gate.json, journal.json, the
sidecar, rule_policy.json only when accepted, and proposal_outcome rows that
the host ingests. Adoption stays a maintainer commit.
`remember` is now a Run entry per case (the image contains it), with the facts
row keyed by the image the facts were computed on (repo-facts records it in
its summary; unknown -> the row is omitted, never guessed). Alert rows carry
their own pin; improve/memory.py counts a repeated alert once.
Runner image rebuilt (+0.39 MB: only the new modules; the quick scan needs no
extra and the rulesets were already in the package), pushed, plane/ re-pinned
and regenerated (validation-46: 105 entries). Fixture test runs the whole loop
graph through the fake ax, whose new `exec` mode runs the real task modules.
Live: one loop Run over three validation-46 cases on the local ax, recorded in
benchmarks/measurements/2026-09-30-harnessx-removal-loop/ (all done, $0 new
spend, 0 proposals, gate verdict no_proposals).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
d104166697
commit
b68fe27c74
@@ -939,3 +939,258 @@ spec:
|
||||
budget:
|
||||
usd: 0
|
||||
calls: 0
|
||||
- name: budibase-mongo-template-nosqli-remember
|
||||
task: remember-budibase-mongo-template-nosqli
|
||||
inputs:
|
||||
facts: budibase-mongo-template-nosqli-facts/facts.json
|
||||
facts_summary: budibase-mongo-template-nosqli-facts/summary.json
|
||||
pass_a: budibase-mongo-template-nosqli-va/units.jsonl
|
||||
summary_a: budibase-mongo-template-nosqli-va/summary.json
|
||||
pass_b: budibase-mongo-template-nosqli-vb/units.jsonl
|
||||
summary_b: budibase-mongo-template-nosqli-vb/summary.json
|
||||
pass_c: budibase-mongo-template-nosqli-vc/units.jsonl
|
||||
summary_c: budibase-mongo-template-nosqli-vc/summary.json
|
||||
agreed: budibase-mongo-template-nosqli-final/agreed.json
|
||||
outputs:
|
||||
- memory.jsonl
|
||||
budget:
|
||||
usd: 0
|
||||
calls: 0
|
||||
- name: cct-studio-server-cmdi-remember
|
||||
task: remember-cct-studio-server-cmdi
|
||||
inputs:
|
||||
facts: cct-studio-server-cmdi-facts/facts.json
|
||||
facts_summary: cct-studio-server-cmdi-facts/summary.json
|
||||
pass_a: cct-studio-server-cmdi-va/units.jsonl
|
||||
summary_a: cct-studio-server-cmdi-va/summary.json
|
||||
pass_b: cct-studio-server-cmdi-vb/units.jsonl
|
||||
summary_b: cct-studio-server-cmdi-vb/summary.json
|
||||
pass_c: cct-studio-server-cmdi-vc/units.jsonl
|
||||
summary_c: cct-studio-server-cmdi-vc/summary.json
|
||||
agreed: cct-studio-server-cmdi-final/agreed.json
|
||||
outputs:
|
||||
- memory.jsonl
|
||||
budget:
|
||||
usd: 0
|
||||
calls: 0
|
||||
- name: contextforge-prompt-ssti-remember
|
||||
task: remember-contextforge-prompt-ssti
|
||||
inputs:
|
||||
facts: contextforge-prompt-ssti-facts/facts.json
|
||||
facts_summary: contextforge-prompt-ssti-facts/summary.json
|
||||
pass_a: contextforge-prompt-ssti-va/units.jsonl
|
||||
summary_a: contextforge-prompt-ssti-va/summary.json
|
||||
pass_b: contextforge-prompt-ssti-vb/units.jsonl
|
||||
summary_b: contextforge-prompt-ssti-vb/summary.json
|
||||
pass_c: contextforge-prompt-ssti-vc/units.jsonl
|
||||
summary_c: contextforge-prompt-ssti-vc/summary.json
|
||||
agreed: contextforge-prompt-ssti-final/agreed.json
|
||||
outputs:
|
||||
- memory.jsonl
|
||||
budget:
|
||||
usd: 0
|
||||
calls: 0
|
||||
- name: froxlor-admin-ip-second-order-sqli-remember
|
||||
task: remember-froxlor-admin-ip-second-order-sqli
|
||||
inputs:
|
||||
facts: froxlor-admin-ip-second-order-sqli-facts/facts.json
|
||||
facts_summary: froxlor-admin-ip-second-order-sqli-facts/summary.json
|
||||
pass_a: froxlor-admin-ip-second-order-sqli-va/units.jsonl
|
||||
summary_a: froxlor-admin-ip-second-order-sqli-va/summary.json
|
||||
pass_b: froxlor-admin-ip-second-order-sqli-vb/units.jsonl
|
||||
summary_b: froxlor-admin-ip-second-order-sqli-vb/summary.json
|
||||
pass_c: froxlor-admin-ip-second-order-sqli-vc/units.jsonl
|
||||
summary_c: froxlor-admin-ip-second-order-sqli-vc/summary.json
|
||||
agreed: froxlor-admin-ip-second-order-sqli-final/agreed.json
|
||||
outputs:
|
||||
- memory.jsonl
|
||||
budget:
|
||||
usd: 0
|
||||
calls: 0
|
||||
- name: fuxa-tdengine-escape-sqli-remember
|
||||
task: remember-fuxa-tdengine-escape-sqli
|
||||
inputs:
|
||||
facts: fuxa-tdengine-escape-sqli-facts/facts.json
|
||||
facts_summary: fuxa-tdengine-escape-sqli-facts/summary.json
|
||||
pass_a: fuxa-tdengine-escape-sqli-va/units.jsonl
|
||||
summary_a: fuxa-tdengine-escape-sqli-va/summary.json
|
||||
pass_b: fuxa-tdengine-escape-sqli-vb/units.jsonl
|
||||
summary_b: fuxa-tdengine-escape-sqli-vb/summary.json
|
||||
pass_c: fuxa-tdengine-escape-sqli-vc/units.jsonl
|
||||
summary_c: fuxa-tdengine-escape-sqli-vc/summary.json
|
||||
agreed: fuxa-tdengine-escape-sqli-final/agreed.json
|
||||
outputs:
|
||||
- memory.jsonl
|
||||
budget:
|
||||
usd: 0
|
||||
calls: 0
|
||||
- name: lmdeploy-media-url-ssrf-remember
|
||||
task: remember-lmdeploy-media-url-ssrf
|
||||
inputs:
|
||||
facts: lmdeploy-media-url-ssrf-facts/facts.json
|
||||
facts_summary: lmdeploy-media-url-ssrf-facts/summary.json
|
||||
pass_a: lmdeploy-media-url-ssrf-va/units.jsonl
|
||||
summary_a: lmdeploy-media-url-ssrf-va/summary.json
|
||||
pass_b: lmdeploy-media-url-ssrf-vb/units.jsonl
|
||||
summary_b: lmdeploy-media-url-ssrf-vb/summary.json
|
||||
pass_c: lmdeploy-media-url-ssrf-vc/units.jsonl
|
||||
summary_c: lmdeploy-media-url-ssrf-vc/summary.json
|
||||
agreed: lmdeploy-media-url-ssrf-final/agreed.json
|
||||
outputs:
|
||||
- memory.jsonl
|
||||
budget:
|
||||
usd: 0
|
||||
calls: 0
|
||||
- name: medplum-external-auth-open-redirect-remember
|
||||
task: remember-medplum-external-auth-open-redirect
|
||||
inputs:
|
||||
facts: medplum-external-auth-open-redirect-facts/facts.json
|
||||
facts_summary: medplum-external-auth-open-redirect-facts/summary.json
|
||||
pass_a: medplum-external-auth-open-redirect-va/units.jsonl
|
||||
summary_a: medplum-external-auth-open-redirect-va/summary.json
|
||||
pass_b: medplum-external-auth-open-redirect-vb/units.jsonl
|
||||
summary_b: medplum-external-auth-open-redirect-vb/summary.json
|
||||
pass_c: medplum-external-auth-open-redirect-vc/units.jsonl
|
||||
summary_c: medplum-external-auth-open-redirect-vc/summary.json
|
||||
agreed: medplum-external-auth-open-redirect-final/agreed.json
|
||||
outputs:
|
||||
- memory.jsonl
|
||||
budget:
|
||||
usd: 0
|
||||
calls: 0
|
||||
- name: pgadmin-maintenance-sqli-remember
|
||||
task: remember-pgadmin-maintenance-sqli
|
||||
inputs:
|
||||
facts: pgadmin-maintenance-sqli-facts/facts.json
|
||||
facts_summary: pgadmin-maintenance-sqli-facts/summary.json
|
||||
pass_a: pgadmin-maintenance-sqli-va/units.jsonl
|
||||
summary_a: pgadmin-maintenance-sqli-va/summary.json
|
||||
pass_b: pgadmin-maintenance-sqli-vb/units.jsonl
|
||||
summary_b: pgadmin-maintenance-sqli-vb/summary.json
|
||||
pass_c: pgadmin-maintenance-sqli-vc/units.jsonl
|
||||
summary_c: pgadmin-maintenance-sqli-vc/summary.json
|
||||
agreed: pgadmin-maintenance-sqli-final/agreed.json
|
||||
outputs:
|
||||
- memory.jsonl
|
||||
budget:
|
||||
usd: 0
|
||||
calls: 0
|
||||
- name: qwed-sympy-eval-remember
|
||||
task: remember-qwed-sympy-eval
|
||||
inputs:
|
||||
facts: qwed-sympy-eval-facts/facts.json
|
||||
facts_summary: qwed-sympy-eval-facts/summary.json
|
||||
pass_a: qwed-sympy-eval-va/units.jsonl
|
||||
summary_a: qwed-sympy-eval-va/summary.json
|
||||
pass_b: qwed-sympy-eval-vb/units.jsonl
|
||||
summary_b: qwed-sympy-eval-vb/summary.json
|
||||
pass_c: qwed-sympy-eval-vc/units.jsonl
|
||||
summary_c: qwed-sympy-eval-vc/summary.json
|
||||
agreed: qwed-sympy-eval-final/agreed.json
|
||||
outputs:
|
||||
- memory.jsonl
|
||||
budget:
|
||||
usd: 0
|
||||
calls: 0
|
||||
- name: reactpress-findall-column-sqli-remember
|
||||
task: remember-reactpress-findall-column-sqli
|
||||
inputs:
|
||||
facts: reactpress-findall-column-sqli-facts/facts.json
|
||||
facts_summary: reactpress-findall-column-sqli-facts/summary.json
|
||||
pass_a: reactpress-findall-column-sqli-va/units.jsonl
|
||||
summary_a: reactpress-findall-column-sqli-va/summary.json
|
||||
pass_b: reactpress-findall-column-sqli-vb/units.jsonl
|
||||
summary_b: reactpress-findall-column-sqli-vb/summary.json
|
||||
pass_c: reactpress-findall-column-sqli-vc/units.jsonl
|
||||
summary_c: reactpress-findall-column-sqli-vc/summary.json
|
||||
agreed: reactpress-findall-column-sqli-final/agreed.json
|
||||
outputs:
|
||||
- memory.jsonl
|
||||
budget:
|
||||
usd: 0
|
||||
calls: 0
|
||||
- name: unleash-addon-webhook-ssrf-remember
|
||||
task: remember-unleash-addon-webhook-ssrf
|
||||
inputs:
|
||||
facts: unleash-addon-webhook-ssrf-facts/facts.json
|
||||
facts_summary: unleash-addon-webhook-ssrf-facts/summary.json
|
||||
pass_a: unleash-addon-webhook-ssrf-va/units.jsonl
|
||||
summary_a: unleash-addon-webhook-ssrf-va/summary.json
|
||||
pass_b: unleash-addon-webhook-ssrf-vb/units.jsonl
|
||||
summary_b: unleash-addon-webhook-ssrf-vb/summary.json
|
||||
pass_c: unleash-addon-webhook-ssrf-vc/units.jsonl
|
||||
summary_c: unleash-addon-webhook-ssrf-vc/summary.json
|
||||
agreed: unleash-addon-webhook-ssrf-final/agreed.json
|
||||
outputs:
|
||||
- memory.jsonl
|
||||
budget:
|
||||
usd: 0
|
||||
calls: 0
|
||||
- name: winml-cli-serve-cors-remember
|
||||
task: remember-winml-cli-serve-cors
|
||||
inputs:
|
||||
facts: winml-cli-serve-cors-facts/facts.json
|
||||
facts_summary: winml-cli-serve-cors-facts/summary.json
|
||||
pass_a: winml-cli-serve-cors-va/units.jsonl
|
||||
summary_a: winml-cli-serve-cors-va/summary.json
|
||||
pass_b: winml-cli-serve-cors-vb/units.jsonl
|
||||
summary_b: winml-cli-serve-cors-vb/summary.json
|
||||
pass_c: winml-cli-serve-cors-vc/units.jsonl
|
||||
summary_c: winml-cli-serve-cors-vc/summary.json
|
||||
agreed: winml-cli-serve-cors-final/agreed.json
|
||||
outputs:
|
||||
- memory.jsonl
|
||||
budget:
|
||||
usd: 0
|
||||
calls: 0
|
||||
- name: wp-directorist-avatar-ssrf-remember
|
||||
task: remember-wp-directorist-avatar-ssrf
|
||||
inputs:
|
||||
facts: wp-directorist-avatar-ssrf-facts/facts.json
|
||||
facts_summary: wp-directorist-avatar-ssrf-facts/summary.json
|
||||
pass_a: wp-directorist-avatar-ssrf-va/units.jsonl
|
||||
summary_a: wp-directorist-avatar-ssrf-va/summary.json
|
||||
pass_b: wp-directorist-avatar-ssrf-vb/units.jsonl
|
||||
summary_b: wp-directorist-avatar-ssrf-vb/summary.json
|
||||
pass_c: wp-directorist-avatar-ssrf-vc/units.jsonl
|
||||
summary_c: wp-directorist-avatar-ssrf-vc/summary.json
|
||||
agreed: wp-directorist-avatar-ssrf-final/agreed.json
|
||||
outputs:
|
||||
- memory.jsonl
|
||||
budget:
|
||||
usd: 0
|
||||
calls: 0
|
||||
- name: wp-gopay-log-filter-sqli-remember
|
||||
task: remember-wp-gopay-log-filter-sqli
|
||||
inputs:
|
||||
facts: wp-gopay-log-filter-sqli-facts/facts.json
|
||||
facts_summary: wp-gopay-log-filter-sqli-facts/summary.json
|
||||
pass_a: wp-gopay-log-filter-sqli-va/units.jsonl
|
||||
summary_a: wp-gopay-log-filter-sqli-va/summary.json
|
||||
pass_b: wp-gopay-log-filter-sqli-vb/units.jsonl
|
||||
summary_b: wp-gopay-log-filter-sqli-vb/summary.json
|
||||
pass_c: wp-gopay-log-filter-sqli-vc/units.jsonl
|
||||
summary_c: wp-gopay-log-filter-sqli-vc/summary.json
|
||||
agreed: wp-gopay-log-filter-sqli-final/agreed.json
|
||||
outputs:
|
||||
- memory.jsonl
|
||||
budget:
|
||||
usd: 0
|
||||
calls: 0
|
||||
- name: wp-groundhogg-confirm-redirect-remember
|
||||
task: remember-wp-groundhogg-confirm-redirect
|
||||
inputs:
|
||||
facts: wp-groundhogg-confirm-redirect-facts/facts.json
|
||||
facts_summary: wp-groundhogg-confirm-redirect-facts/summary.json
|
||||
pass_a: wp-groundhogg-confirm-redirect-va/units.jsonl
|
||||
summary_a: wp-groundhogg-confirm-redirect-va/summary.json
|
||||
pass_b: wp-groundhogg-confirm-redirect-vb/units.jsonl
|
||||
summary_b: wp-groundhogg-confirm-redirect-vb/summary.json
|
||||
pass_c: wp-groundhogg-confirm-redirect-vc/units.jsonl
|
||||
summary_c: wp-groundhogg-confirm-redirect-vc/summary.json
|
||||
agreed: wp-groundhogg-confirm-redirect-final/agreed.json
|
||||
outputs:
|
||||
- memory.jsonl
|
||||
budget:
|
||||
usd: 0
|
||||
calls: 0
|
||||
|
||||
@@ -7,7 +7,7 @@ metadata:
|
||||
name: agree
|
||||
atespace: default
|
||||
spec:
|
||||
image: "localhost:5001/ousast-runner@sha256:fb9a743027518976659fd11505e58cc3fc84fb42e07627196e13b76ac01d0617"
|
||||
image: "localhost:5001/ousast-runner@sha256:7fbd523f127f1e15eddc48b06e73acf75fa70f741d36e0749e74ca883797f6c9"
|
||||
command: ["agree"]
|
||||
resources:
|
||||
requests:
|
||||
|
||||
@@ -0,0 +1,18 @@
|
||||
# alerts (harnessx-removal Req 6.3, design section 6): quick-mode rules (enabled and shadow) on a case's vulnerable
|
||||
# and fixed pins, each alert attributed to its enclosing function and marked inside or outside the fix. No model.
|
||||
# Heavy like repo-facts (a full-tree scan per pin): the Run serializes it. The image is digest-pinned.
|
||||
apiVersion: ax.io/v1alpha1
|
||||
kind: Task
|
||||
metadata:
|
||||
name: alerts
|
||||
atespace: default
|
||||
spec:
|
||||
image: "localhost:5001/ousast-runner@sha256:7fbd523f127f1e15eddc48b06e73acf75fa70f741d36e0749e74ca883797f6c9"
|
||||
command: ["alerts"]
|
||||
resources:
|
||||
requests:
|
||||
cpu: "250m"
|
||||
memory: "512Mi"
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: "1280Mi"
|
||||
@@ -0,0 +1,18 @@
|
||||
# loop (harnessx-removal Req 6.3, design section 6): the improvement loop's singletons -- the generator appends the
|
||||
# step (snapshot, measure, propose, improve) to the command. No model, budget {usd: 0, calls: 0}; improve replays the
|
||||
# benchmark gate unchanged and delivers its verdict as gate.json. The image is digest-pinned.
|
||||
apiVersion: ax.io/v1alpha1
|
||||
kind: Task
|
||||
metadata:
|
||||
name: loop
|
||||
atespace: default
|
||||
spec:
|
||||
image: "localhost:5001/ousast-runner@sha256:7fbd523f127f1e15eddc48b06e73acf75fa70f741d36e0749e74ca883797f6c9"
|
||||
command: ["loop"]
|
||||
resources:
|
||||
requests:
|
||||
cpu: "250m"
|
||||
memory: "512Mi"
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: "1280Mi"
|
||||
@@ -0,0 +1,18 @@
|
||||
# remember (harnessx-removal Req 6.1, design section 4): a case's delivered artifacts as memory rows. No model.
|
||||
# The template of the per-case Tasks (bound to the case's vulnerable Workspace for the repository and pin the rows
|
||||
# carry); the host ingests the delivered memory.jsonl after the Run. The image is digest-pinned.
|
||||
apiVersion: ax.io/v1alpha1
|
||||
kind: Task
|
||||
metadata:
|
||||
name: remember
|
||||
atespace: default
|
||||
spec:
|
||||
image: "localhost:5001/ousast-runner@sha256:7fbd523f127f1e15eddc48b06e73acf75fa70f741d36e0749e74ca883797f6c9"
|
||||
command: ["remember"]
|
||||
resources:
|
||||
requests:
|
||||
cpu: "100m"
|
||||
memory: "256Mi"
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: "512Mi"
|
||||
@@ -8,7 +8,7 @@ metadata:
|
||||
name: repo-facts
|
||||
atespace: default
|
||||
spec:
|
||||
image: "localhost:5001/ousast-runner@sha256:fb9a743027518976659fd11505e58cc3fc84fb42e07627196e13b76ac01d0617"
|
||||
image: "localhost:5001/ousast-runner@sha256:7fbd523f127f1e15eddc48b06e73acf75fa70f741d36e0749e74ca883797f6c9"
|
||||
command: ["repo-facts"]
|
||||
resources:
|
||||
requests:
|
||||
|
||||
+465
-90
File diff suppressed because it is too large
Load Diff
@@ -10,7 +10,7 @@ metadata:
|
||||
annotations:
|
||||
openultrasast.io/model: deepseek-flash
|
||||
spec:
|
||||
image: "localhost:5001/ousast-runner@sha256:fb9a743027518976659fd11505e58cc3fc84fb42e07627196e13b76ac01d0617"
|
||||
image: "localhost:5001/ousast-runner@sha256:7fbd523f127f1e15eddc48b06e73acf75fa70f741d36e0749e74ca883797f6c9"
|
||||
command: ["verify"]
|
||||
resources:
|
||||
requests:
|
||||
|
||||
Reference in New Issue
Block a user