marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(plane): memory store (file and MinIO) and the remember task (harnessx-removal task 2)

Browse Source 
plane/memory.py: the MemoryStore interface with FileStore (index.jsonl, facts by
content hash, repos/<host>__<owner>__<name>/<pin>.jsonl, idempotent ingest,
deterministic row ids, refusal below 1 GiB free naming the path) and MinioStore
(object metadata and tags, S3 Select pushdown probed once per process with a
logged local fallback, object version ids on every row read, versioning and
runs/ lifecycle helper, presigned URLs). OUSAST_MEMORY selects the backend;
MINIO_* settings come from .env; the minio SDK is the optional extra `minio`.

plane/tasks/remember.py: model-free rows (facts, verdict, unit_cost, alert)
from a case's artifacts; remember_run computes them on the host for a recorded
run without writing into it. reconciler.mark_done (477 lines total), seed
before and ingest after `ousast plane run`, `ousast plane remember <run>`.
The generator annotates each repo-facts Task with openultrasast.io/memory-key
(repo, pin, candidates digest, image digest) and the Run with its population
and split; plane/ regenerated.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-09-30 11:10:28 +02:00
co-authored by Claude Opus 5.5
parent d64ab2817e
commit 52d28b5121
31 changed files with 2121 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/test_plane_increment.py
+10 -1
View File
@@ -19,6 +19,8 @@ from openultrasast.plane import reconciler
from openultrasast.plane.budget import prices_from
from openultrasast.plane.generate import TEMPLATES, fix_ranges, increment, read_cases, render, repin_templates
from openultrasast.plane.manifests import load_manifests
from openultrasast.plane.memory import MEMORY_KEY_ANNOTATION, POPULATION_ANNOTATION, SPLIT_ANNOTATION, parse_memory_key
from openultrasast.plane.tasks.repo_facts import _digest
PLANE = Path("plane")
RUN = PLANE / "runs" / "validation-46.yaml"
@@ -160,7 +162,8 @@ def test_committed_manifests_match_the_generator() -> None:
POPULATION,
REPOS,
)
for relative, text in render(cases, templates, "validation-46", _committed_command()).items():
rendered = render(cases, templates, "validation-46", _committed_command(), population=POPULATION.stem, split="validation")
for relative, text in rendered.items():
assert (PLANE / relative).read_text() == text, relative
@@ -216,6 +219,12 @@ def test_generator_is_deterministic_and_follows_the_reference(tmp_path: Path) ->
assert inputs["triage.json"]["candidates"] == [["a.py", "helper", 20], ["a.py", "run", 9]]
assert run.task("alpha-va").budget is not None and run.task("alpha-va").budget.usd == 0.38 # 1.5 * 0.5 / 2, cent up
assert yaml.safe_load(first[0].read_text())["metadata"]["annotations"] == {"openultrasast.io/git-commits": f"repo={PIN}"}
# memory (harnessx-removal §4): the facts Task names its reuse key, the Run its population and split
key = parse_memory_key(manifests.tasks["repo-facts-alpha"].metadata.annotations[MEMORY_KEY_ANNOTATION])
image = manifests.tasks["repo-facts-alpha"].image or ""
assert key == {"repo": "example.com/alpha", "pin": PIN, "candidates": _digest(["run"]), "image": image.rpartition("@")[2]}
assert all(MEMORY_KEY_ANNOTATION not in t.metadata.annotations for n, t in manifests.tasks.items() if not n.startswith("repo-facts-"))
assert dict(run.metadata.annotations) == {POPULATION_ANNOTATION: "population", SPLIT_ANNOTATION: "set"}
def test_generator_refuses_a_set_the_recorded_scans_did_not_ask(tmp_path: Path) -> None: