mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
fix(learn): withhold label-tracking signals from the v1 classifier; leak audit
Maintainer decision "Neutralise both for v1". 1. Phantom candidates. A pair side's candidates are now only the functions that side declares (plane/harvest.py `declares`, the excerpt builder's function_bounds); a labelled function absent from a side is listed under `absent_on_side` and never asked, and a side declaring none of its verify family's functions is no verify unit. The recorded harvest units held 58 phantom verify candidates (53 fixed, 5 vulnerable sides). build_features.py skips label rows whose side does not declare the function and drops their earlier rows (MemoryStore.drop_rows); rebuilt the pair records from the recorded signals, no model call: 126 label rows absent_on_side (117 fixed, 9 vulnerable) -- access_control 13, config_secrets 4, deserialization 20, injection 28, memory 4, output_encoding 16, path 15, untrusted_destination 26 -- so 252 feature rows dropped (1712 -> 1460); no other row changed. 2. Engine withheld. Input profiles in learn/schema.py (`v1` withholds the engine instrument, `full` withholds nothing); ProgramSpec.inputs defaults to v1 and compile.toml records it. Neither the engine's state nor its values reach the candidate, examples, demonstrations or the instruction proposer's error summary, and the Gower retrieval distance leaves the engine block out. Stored records are unchanged. design.md Risks records the decision and that the engine returns only through a pre-registered A/B on assumed-benign and benign-control negatives. 3. `ousast learn audit-leaks` (learn/leaks.py): per feature and per instrument state, separation = (pairs the signal alone puts on the right side - on the wrong side) / pairs, over all pairs and per family; flagged at >= 0.20 over >= 10 pairs. Counts under benchmarks/measurements/2026-10-01-decision-engine-leak-audit/ (before and after the rebuild). After: facts.function_lines (0.80 untrusted_destination, 0.75 config_secrets, 0.26 injection; higher on fixed), verify.turns_mean 0.67 config_secrets, eng.* 0.35-0.40 output_encoding, verify.flag_a/votes/ site_offset and agree.final 0.20-0.30 on 10-12 pair families. Before the rebuild the phantom fixed sides drove state.source/state.roles/ms.flagged to 1.0 and state.engine to 0.95 on deserialization. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
4c96a59f8f
commit
9e3c83cfdc
@@ -1163,6 +1163,15 @@ start only after the maintainer confirms their budget against the balance read j
|
||||
near-duplicate drop at cosine >= 0.98 in evaluation, and `test_boundary`. Demonstrations from the compile split
|
||||
are, by construction, from repositories never evaluated. The seed for plane Runs filters on the host, where the
|
||||
labels are; the sandbox never receives an example of its own case's group.
|
||||
- **Label-tracking signals (maintainer decision 2026-10-01: "Neutralise both for v1").** The harvest found two
|
||||
signals that follow the pair label instead of the code: verify candidates on fixed sides named functions the side
|
||||
no longer declares (now never produced: a side's candidates are only the functions it declares, the rest counted
|
||||
`absent_on_side`), and the engine's "read the file, asked no question" state, which a fixed side reaches by losing
|
||||
its sink. The v1 program withholds the engine instrument entirely -- state and values -- from prompts,
|
||||
demonstrations, the instruction proposer and the retrieval distance (input profile `v1`, `ProgramSpec.inputs`);
|
||||
the stored record keeps it. The engine returns only through a pre-registered A/B evaluated on assumed-benign and
|
||||
benign-control negatives, where it cannot track a fix. `ousast learn audit-leaks` reports how well every signal
|
||||
alone separates the sides of a pair and flags any above 0.20 (counts under `benchmarks/measurements/`).
|
||||
- **Earlier leakage paths still apply.** Hand rules and framework priors written after seeing a training repository
|
||||
(the `$wpdb` rule from PMPro, the hook source from WP Statistics, `taint.sc:911-949`): priors off by default,
|
||||
leave-one-framework-out, best-effort `taught_by = [repo]`. The mechanism exporter's train-on-test leak
|
||||
|
||||
@@ -12,6 +12,10 @@ from:
|
||||
``validation-46`` Run; model roles from ``harvest-roles``;
|
||||
- callers and entry-point distance on a repository (``not_applicable`` on a pair excerpt: no repository exists).
|
||||
|
||||
A pair side yields a record only for a labelled function that side declares (the excerpt builder's matcher,
|
||||
:func:`openultrasast.plane.harvest.declares`); a name taken from the other side is counted as ``absent_on_side``, its
|
||||
recorded verify verdicts are left unused, and its earlier row is dropped from the store (``dropped_stale_rows``).
|
||||
|
||||
An instrument that has not run for a unit yet (the engine job still running, a Run task not done) is recorded as
|
||||
``failed`` with version ``missing:not-run`` -- null features, never zeros -- and counted as *missing*. Rows go to the
|
||||
memory store (``OUSAST_MEMORY``, default ``~/ousast-results/plane/memory``) as kind ``features``; a rerun replaces
|
||||
@@ -60,6 +64,7 @@ from openultrasast.learn.features import (
|
||||
from openultrasast.learn.roles import RoleSet, from_model_roles, infer_for_checkout
|
||||
from openultrasast.pairs import DEFAULT_CATALOG, _materialize_side, _targets, load_pair_catalog
|
||||
from openultrasast.plane import memory
|
||||
from openultrasast.plane.harvest import declares
|
||||
from openultrasast.plane.tasks.alerts import covers
|
||||
from openultrasast.plane.tasks.alerts import engine_languages as _engine_languages
|
||||
from openultrasast.plane.tasks.alerts import quick_languages as _quick_languages
|
||||
@@ -268,6 +273,8 @@ def main() -> int:
|
||||
tally: Counter[tuple[str, ...]] = Counter()
|
||||
coverage: dict[str, Counter[str]] = defaultdict(Counter)
|
||||
skipped: Counter[str] = Counter()
|
||||
absent: Counter[str] = Counter() # family -> labelled pair candidates their side does not declare
|
||||
dropped = 0
|
||||
args.scratch.mkdir(parents=True, exist_ok=True)
|
||||
chosen = [(k, r) for k, r in sorted(targets.items()) if k[0] in args.kinds.split(",")]
|
||||
for number, ((kind, ref, role_or_pin), rows) in enumerate(chosen):
|
||||
@@ -327,6 +334,10 @@ def main() -> int:
|
||||
continue
|
||||
seen.add((candidate, family))
|
||||
path, _, function = candidate.partition("::")
|
||||
if kind == "pairs" and not declares(index.lines(path), path, function):
|
||||
skipped["absent_on_side"] += 1 # a name from the other side: no candidate here, its verdicts unused
|
||||
absent[family] += 1
|
||||
continue
|
||||
cand_line = next((c[2] for n in ([owner] if kind == "pairs" else by_pin[(repo, pin)]) for c in units[n]["candidates"]
|
||||
if f"{c[0]}::{c[1]}" == candidate), None) # fmt: skip
|
||||
parts = static_parts(root, index, quick, engine, engine_state, engine_result, inferred, rules, quick_version, path,
|
||||
@@ -347,8 +358,14 @@ def main() -> int:
|
||||
state = "missing" if info.get("version") == "missing:not-run" else info["state"]
|
||||
coverage[f"{family}:{inst}"][state] += 1
|
||||
store.ingest_rows("harvest", f"features:{owner}", out_rows)
|
||||
print(json.dumps({"target": owner, "rows": len(out_rows)}), flush=True)
|
||||
built = {r["id"] for r in out_rows}
|
||||
stored = store.rows(repo=memory.repo_key(repo), pin=pin, kind="features", where={"run": "harvest", "task": owner})
|
||||
stale = [r.row["id"] for r in stored if r.row["id"] not in built]
|
||||
dropped += store.drop_rows(memory.repo_key(repo), pin, stale)
|
||||
print(json.dumps({"target": owner, "rows": len(out_rows), "dropped": len(stale)}), flush=True)
|
||||
counts = {
|
||||
"absent_on_side": dict(sorted(absent.items())),
|
||||
"dropped_stale_rows": dropped,
|
||||
"records": {f"{p}:{f}": n for (p, f), n in sorted(tally.items())},
|
||||
"instrument_states_plane": {k: dict(sorted(v.items())) for k, v in sorted(coverage.items())},
|
||||
"skipped_label_rows": dict(skipped),
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"modules": 154,
|
||||
"modules": 155,
|
||||
"counts": {
|
||||
"load_bearing": 147,
|
||||
"load_bearing": 148,
|
||||
"standalone_capability": 7
|
||||
},
|
||||
"orphaned": [],
|
||||
@@ -363,6 +363,7 @@
|
||||
"learn.features",
|
||||
"learn.folds",
|
||||
"learn.labels",
|
||||
"learn.leaks",
|
||||
"learn.program",
|
||||
"learn.retrieve",
|
||||
"learn.roles",
|
||||
@@ -435,7 +436,8 @@
|
||||
"importers": [
|
||||
"learn.embeddings",
|
||||
"learn.examples",
|
||||
"learn.labels"
|
||||
"learn.labels",
|
||||
"plane.harvest"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -475,6 +477,14 @@
|
||||
"plane.harvest"
|
||||
]
|
||||
},
|
||||
{
|
||||
"module": "learn.leaks",
|
||||
"classification": "load_bearing",
|
||||
"reason": "reachable from gate, map_gate, pair_gate or the CLI",
|
||||
"importers": [
|
||||
"learn.cli"
|
||||
]
|
||||
},
|
||||
{
|
||||
"module": "learn.program",
|
||||
"classification": "load_bearing",
|
||||
@@ -512,6 +522,7 @@
|
||||
"learn.compile",
|
||||
"learn.evaluate",
|
||||
"learn.features",
|
||||
"learn.leaks",
|
||||
"learn.program",
|
||||
"learn.retrieve",
|
||||
"learn.roles",
|
||||
|
||||
+6600
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,236 @@
|
||||
{
|
||||
"audit": {
|
||||
"after_rebuild": {
|
||||
"flagged": [
|
||||
{
|
||||
"correct": 8,
|
||||
"direction": "higher on fixed",
|
||||
"pairs": 10,
|
||||
"scope": "untrusted_destination",
|
||||
"separation": 0.8,
|
||||
"signal": "facts.function_lines",
|
||||
"wrong": 0
|
||||
},
|
||||
{
|
||||
"correct": 8,
|
||||
"direction": "higher on fixed",
|
||||
"pairs": 12,
|
||||
"scope": "config_secrets",
|
||||
"separation": 0.6667,
|
||||
"signal": "verify.turns_mean",
|
||||
"wrong": 0
|
||||
},
|
||||
{
|
||||
"correct": 18,
|
||||
"direction": "higher on vulnerable",
|
||||
"pairs": 40,
|
||||
"scope": "output_encoding",
|
||||
"separation": 0.4,
|
||||
"signal": "eng.witness_steps_min",
|
||||
"wrong": 2
|
||||
},
|
||||
{
|
||||
"correct": 15,
|
||||
"direction": "higher on vulnerable",
|
||||
"pairs": 40,
|
||||
"scope": "output_encoding",
|
||||
"separation": 0.35,
|
||||
"signal": "eng.findings",
|
||||
"wrong": 1
|
||||
},
|
||||
{
|
||||
"correct": 15,
|
||||
"direction": null,
|
||||
"pairs": 40,
|
||||
"scope": "output_encoding",
|
||||
"separation": 0.35,
|
||||
"signal": "eng.rung_max",
|
||||
"wrong": 1
|
||||
},
|
||||
{
|
||||
"correct": 15,
|
||||
"direction": null,
|
||||
"pairs": 40,
|
||||
"scope": "output_encoding",
|
||||
"separation": 0.35,
|
||||
"signal": "eng.sink_kind",
|
||||
"wrong": 1
|
||||
},
|
||||
{
|
||||
"correct": 15,
|
||||
"direction": null,
|
||||
"pairs": 40,
|
||||
"scope": "output_encoding",
|
||||
"separation": 0.35,
|
||||
"signal": "eng.sink_origin",
|
||||
"wrong": 1
|
||||
},
|
||||
{
|
||||
"correct": 15,
|
||||
"direction": null,
|
||||
"pairs": 40,
|
||||
"scope": "output_encoding",
|
||||
"separation": 0.35,
|
||||
"signal": "eng.source_kind",
|
||||
"wrong": 1
|
||||
},
|
||||
{
|
||||
"correct": 15,
|
||||
"direction": null,
|
||||
"pairs": 40,
|
||||
"scope": "output_encoding",
|
||||
"separation": 0.35,
|
||||
"signal": "eng.source_origin",
|
||||
"wrong": 1
|
||||
},
|
||||
{
|
||||
"correct": 3,
|
||||
"direction": "higher on vulnerable",
|
||||
"pairs": 10,
|
||||
"scope": "untrusted_destination",
|
||||
"separation": 0.3,
|
||||
"signal": "verify.flag_a",
|
||||
"wrong": 0
|
||||
},
|
||||
{
|
||||
"correct": 3,
|
||||
"direction": null,
|
||||
"pairs": 12,
|
||||
"scope": "config_secrets",
|
||||
"separation": 0.25,
|
||||
"signal": "agree.final",
|
||||
"wrong": 0
|
||||
},
|
||||
{
|
||||
"correct": 3,
|
||||
"direction": "higher on vulnerable",
|
||||
"pairs": 12,
|
||||
"scope": "config_secrets",
|
||||
"separation": 0.25,
|
||||
"signal": "verify.site_offset",
|
||||
"wrong": 0
|
||||
},
|
||||
{
|
||||
"correct": 3,
|
||||
"direction": "higher on vulnerable",
|
||||
"pairs": 12,
|
||||
"scope": "config_secrets",
|
||||
"separation": 0.25,
|
||||
"signal": "verify.votes",
|
||||
"wrong": 0
|
||||
},
|
||||
{
|
||||
"correct": 2,
|
||||
"direction": "higher on vulnerable",
|
||||
"pairs": 10,
|
||||
"scope": "untrusted_destination",
|
||||
"separation": 0.2,
|
||||
"signal": "eng.completion",
|
||||
"wrong": 0
|
||||
}
|
||||
],
|
||||
"pairs": {
|
||||
"all": 279,
|
||||
"config_secrets": 12,
|
||||
"injection": 43,
|
||||
"memory": 171,
|
||||
"output_encoding": 40,
|
||||
"path": 1,
|
||||
"unknown": 2,
|
||||
"untrusted_destination": 10
|
||||
}
|
||||
},
|
||||
"before_rebuild": {
|
||||
"flagged_signals": [
|
||||
"agree.final",
|
||||
"eng.completion",
|
||||
"eng.degraded",
|
||||
"eng.findings",
|
||||
"eng.rung_max",
|
||||
"eng.sink_kind",
|
||||
"eng.sink_origin",
|
||||
"eng.source_kind",
|
||||
"eng.source_origin",
|
||||
"eng.witness_steps_min",
|
||||
"facts.function_lines",
|
||||
"ms.flagged",
|
||||
"ms.operation",
|
||||
"qr.enabled_hits",
|
||||
"qr.max_precision_estimate",
|
||||
"qr.mechanism.source_reaches_sink",
|
||||
"qr.prior_hits",
|
||||
"roles.sink_confidence",
|
||||
"roles.source_in_function",
|
||||
"state.engine",
|
||||
"state.roles",
|
||||
"state.source",
|
||||
"verify.flag_a",
|
||||
"verify.flag_b",
|
||||
"verify.site_offset",
|
||||
"verify.turns_mean",
|
||||
"verify.votes"
|
||||
],
|
||||
"pairs": {
|
||||
"access_control": 13,
|
||||
"all": 396,
|
||||
"config_secrets": 16,
|
||||
"deserialization": 20,
|
||||
"injection": 66,
|
||||
"memory": 173,
|
||||
"output_encoding": 56,
|
||||
"path": 14,
|
||||
"unknown": 2,
|
||||
"untrusted_destination": 36
|
||||
}
|
||||
},
|
||||
"command": "ousast learn audit-leaks --profile plane --units <harvest units.json> (threshold 0.20, min 10 pairs)",
|
||||
"metric": "separation = (pairs the signal alone puts on the right side - on the wrong side) / pairs; see learn/leaks.py"
|
||||
},
|
||||
"created": "2026-10-01",
|
||||
"label_snapshot_sha256_prefix": "18337014d8a5c8df",
|
||||
"note": "Maintainer decision 'Neutralise both for v1' (learned-decision-engine). COUNTS ONLY. (1) Pair-side candidates come only from functions the side declares (plane/harvest.py `declares`, the excerpt builder's function_bounds); the harvest feature records were rebuilt with benchmarks/learn/build_features.py --kinds pairs from the recorded signals (no model call): label rows whose side does not declare the function are skipped (absent_on_side) and their earlier rows dropped from the store. (2) The v1 program withholds the engine instrument from prompts and the retrieval distance. (3) `ousast learn audit-leaks` on the plane profile of the memory store, before and after the rebuild.",
|
||||
"rebuild": {
|
||||
"absent_on_side_by_side": {
|
||||
"fixed": 117,
|
||||
"vulnerable": 9
|
||||
},
|
||||
"absent_on_side_label_rows_by_family": {
|
||||
"access_control": 13,
|
||||
"config_secrets": 4,
|
||||
"deserialization": 20,
|
||||
"injection": 28,
|
||||
"memory": 4,
|
||||
"output_encoding": 16,
|
||||
"path": 15,
|
||||
"untrusted_destination": 26
|
||||
},
|
||||
"feature_rows_after": 1460,
|
||||
"feature_rows_before": 1712,
|
||||
"feature_rows_changed_otherwise": 0,
|
||||
"feature_rows_dropped": 252,
|
||||
"phantom_verify_candidates_in_recorded_units": {
|
||||
"fixed": 53,
|
||||
"vulnerable": 5
|
||||
},
|
||||
"records_after_rebuild_pairs": {
|
||||
"plane:access_control": 13,
|
||||
"plane:config_secrets": 4,
|
||||
"plane:deserialization": 20,
|
||||
"plane:injection": 42,
|
||||
"plane:memory": 342,
|
||||
"plane:output_encoding": 96,
|
||||
"plane:path": 13,
|
||||
"plane:unknown": 4,
|
||||
"plane:untrusted_destination": 26,
|
||||
"static:access_control": 13,
|
||||
"static:config_secrets": 4,
|
||||
"static:deserialization": 20,
|
||||
"static:injection": 42,
|
||||
"static:memory": 342,
|
||||
"static:output_encoding": 96,
|
||||
"static:path": 13,
|
||||
"static:unknown": 4,
|
||||
"static:untrusted_destination": 26
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1418,6 +1418,9 @@ def _plane_harvest(args: argparse.Namespace) -> int:
|
||||
) # fmt: skip
|
||||
for relative in sorted(k for k in files if k.startswith("runs/")):
|
||||
print(f"{args.plane / relative}: {files[relative].splitlines()[1].lstrip('# ')}")
|
||||
index = json.loads(files["units.json"])
|
||||
absent = sum(len(u.get("absent_on_side") or ()) for u in index.values())
|
||||
print(f"absent_on_side: {absent} labelled candidates not declared on their side (not asked there)")
|
||||
return 0
|
||||
|
||||
|
||||
|
||||
@@ -53,8 +53,19 @@ def add_commands(learn_sub: Any) -> None:
|
||||
if name == "curve":
|
||||
sub.add_argument("--subset", type=int, default=200)
|
||||
|
||||
leaks = learn_sub.add_parser("audit-leaks", help="how well each signal alone separates the sides of a pair (no model)")
|
||||
leaks.add_argument("--memory", help="the store (default OUSAST_MEMORY, then results_root()/memory)")
|
||||
leaks.add_argument("--units", type=Path, help="the harvest units.json (unit -> case and side) for feature rows without a side")
|
||||
leaks.add_argument("--profile", choices=("static", "plane"), default="plane")
|
||||
leaks.add_argument("--kind", default="features", help="the row kind to audit (features, or example)")
|
||||
leaks.add_argument("--threshold", type=float, help="flag a separation at least this (default 0.20)")
|
||||
leaks.add_argument("--min-pairs", type=int, help="over at least this many pairs (default 10)")
|
||||
leaks.add_argument("--out", type=Path, help="write the counts-only report here")
|
||||
|
||||
|
||||
def run(args: argparse.Namespace) -> int:
|
||||
if args.learn_command == "audit-leaks":
|
||||
return _audit_leaks(args)
|
||||
if args.learn_command == "memory" and args.memory_command == "build":
|
||||
return _memory_build(args)
|
||||
if args.learn_command == "memory" and args.memory_command == "embed":
|
||||
@@ -225,4 +236,31 @@ def _memory_build(args: argparse.Namespace) -> int:
|
||||
return 0
|
||||
|
||||
|
||||
def _audit_leaks(args: argparse.Namespace) -> int:
|
||||
"""Exit 2 when the store holds no pair to audit (an unread store is not a clean one)."""
|
||||
from ..plane.memory import open_store
|
||||
from .leaks import MIN_PAIRS, THRESHOLD, audit, pair_rows, sides_from_units
|
||||
|
||||
store = open_store(args.memory)
|
||||
rows = [r.row for r in store.rows(kind=args.kind)]
|
||||
units = json.loads(args.units.read_text(encoding="utf-8")) if args.units is not None else {}
|
||||
pairs, skipped = pair_rows(rows, sides_from_units(units), args.profile)
|
||||
if not pairs:
|
||||
print(f"learn audit-leaks: {len(rows)} {args.kind} rows in {store.describe()}, no pair to audit {skipped}", file=sys.stderr)
|
||||
return 2
|
||||
threshold = THRESHOLD if args.threshold is None else args.threshold
|
||||
report = audit(pairs, args.profile, threshold=threshold, min_pairs=MIN_PAIRS if args.min_pairs is None else args.min_pairs)
|
||||
report = {"rows_read": len(rows), "kind": args.kind, "skipped": skipped, **report}
|
||||
text = json.dumps(report, indent=1, sort_keys=True) + "\n"
|
||||
if args.out is not None:
|
||||
args.out.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.out.write_text(text, encoding="utf-8")
|
||||
print(json.dumps({"rows_read": len(rows), "pairs": report["pairs"], "flagged_signals": report["flagged_signals"]}, sort_keys=True))
|
||||
for cell in report["flagged"]:
|
||||
print(
|
||||
f" {cell['separation']:.3f} {cell['signal']} [{cell['scope']}] {cell['correct']}-{cell['wrong']} of {cell['pairs']} pairs"
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
__all__ = ["add_commands", "run"]
|
||||
|
||||
@@ -52,7 +52,7 @@ from .program import (
|
||||
ProgramSpec,
|
||||
label_word,
|
||||
)
|
||||
from .schema import SCHEMA_VERSION, feature_set_digest
|
||||
from .schema import BY_NAME, DEFAULT_INPUTS, SCHEMA_VERSION, feature_set_digest, withheld
|
||||
|
||||
DEFAULT_SPEC = Path(__file__).resolve().parent / "compile.toml"
|
||||
KIND = "llm-program"
|
||||
@@ -152,8 +152,13 @@ class Bootstrap:
|
||||
self.rejected[why] = self.rejected.get(why, 0) + 1
|
||||
|
||||
|
||||
def _signal_summary(example: Example) -> dict[str, Any]:
|
||||
ran = {k: v for k, v in example.x.items() if v not in (None, 0, False) and not k.startswith("lang.")}
|
||||
def _signal_summary(example: Example, inputs: str = DEFAULT_INPUTS) -> dict[str, Any]:
|
||||
"""The non-zero signals of a mistake for the instruction proposer, without the instruments ``inputs`` withholds."""
|
||||
hidden = withheld(inputs)
|
||||
ran = {
|
||||
k: v for k, v in example.x.items()
|
||||
if v not in (None, 0, False) and not k.startswith("lang.") and not (k in BY_NAME and BY_NAME[k].instrument in hidden)
|
||||
} # fmt: skip
|
||||
return dict(sorted(ran.items())[:12])
|
||||
|
||||
|
||||
@@ -179,7 +184,7 @@ def bootstrap(
|
||||
right = decision.verdict == label_word(example.label)
|
||||
if not right:
|
||||
out.errors.append({"family": example.family, "label": label_word(example.label), "verdict": decision.verdict,
|
||||
"confidence": answer.confidence, "signals": _signal_summary(example)}) # fmt: skip
|
||||
"confidence": answer.confidence, "signals": _signal_summary(example, program.spec.inputs)}) # fmt: skip
|
||||
out.reject("wrong or unsure")
|
||||
continue
|
||||
if answer.parse_failed or answer.confidence < min_confidence:
|
||||
@@ -370,6 +375,7 @@ def compile_program(
|
||||
base = ProgramSpec(
|
||||
profile=profile, model=str(classify.get("model", "deepseek-flash")), k=int(classify.get("k", 1)), k_ret=int(retrieval.get("k", 6)),
|
||||
lam=float(retrieval.get("lam", 0.5)), n1=int(retrieval.get("n1", 40)), priors=str(retrieval.get("priors", "off")),
|
||||
inputs=str(classify.get("inputs", DEFAULT_INPUTS)),
|
||||
max_output_tokens=int(classify.get("max_output_tokens", 300)),
|
||||
) # fmt: skip
|
||||
from .evaluate import candidate_from_example
|
||||
@@ -434,6 +440,8 @@ def compile_program(
|
||||
"temperature": [0.0, 0.7],
|
||||
"confidence": "vote_mean",
|
||||
"max_output_tokens": base.max_output_tokens,
|
||||
"inputs": base.inputs,
|
||||
"withheld": sorted(withheld(base.inputs)),
|
||||
},
|
||||
"calibration": None,
|
||||
"operating_points": None,
|
||||
@@ -491,6 +499,7 @@ def spec_of(artifact: Mapping[str, Any]) -> ProgramSpec:
|
||||
lam=float(retrieval["lam"]),
|
||||
n1=int(retrieval["n1"]),
|
||||
priors=str(retrieval.get("priors", "off")),
|
||||
inputs=str(classify.get("inputs", DEFAULT_INPUTS)),
|
||||
max_output_tokens=int(classify.get("max_output_tokens", 300)),
|
||||
)
|
||||
|
||||
|
||||
@@ -31,6 +31,7 @@ val_candidates = 60
|
||||
model = "deepseek-flash"
|
||||
k = 1 # k and lam are A/B variants (design section 6), not searched here
|
||||
max_output_tokens = 300
|
||||
inputs = "v1" # the input profile: the engine instrument is withheld from prompts and retrieval (2026-10-01)
|
||||
|
||||
[retrieval]
|
||||
n1 = 40
|
||||
|
||||
@@ -0,0 +1,151 @@
|
||||
"""The leakage audit: how well each signal *alone* tells the sides of a pair apart (``ousast learn audit-leaks``).
|
||||
|
||||
No model. A pair is one candidate (``path::function`` and family) on its vulnerable side and on its fixed side -- a
|
||||
pair-corpus case, or a population case's vulnerable and fixed pins. Within a pair the code differs by the fix, so a
|
||||
signal that differs between the sides *and whose difference names the side* is either real evidence about the fix or
|
||||
an instrument artifact that tracks the label (the engine's "read the file, asked no question" state on a fixed side
|
||||
that lost its sink). The audit cannot tell which; it ranks every signal so a person can.
|
||||
|
||||
For each feature of the profile, and for each instrument's state (``state.<instrument>``), over the ``N`` pairs:
|
||||
|
||||
- an ordered feature (``int``, ``float``, ``bool``; ``null`` ranks below every value) predicts the side by one global
|
||||
direction: ``correct`` = pairs where the vulnerable side is higher (or lower, whichever is more), ``wrong`` = the
|
||||
other direction;
|
||||
- an unordered one (``enum``, an instrument state) by the majority side of each unordered pair of values;
|
||||
- **separation** = ``(correct - wrong) / N``: the share of all pairs the signal alone puts on the right side, net of
|
||||
the pairs it puts on the wrong one. 0 is no information (or no difference), 1 is every pair.
|
||||
|
||||
A signal is **flagged** when its separation is at least :data:`THRESHOLD` (0.20) over at least :data:`MIN_PAIRS`
|
||||
(10) pairs, over all pairs or within one family. The output is counts only: feature, instrument and family names.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections import Counter
|
||||
from collections.abc import Iterable, Mapping, Sequence
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
from .schema import FeatureSpec, features_for, instruments_for
|
||||
|
||||
THRESHOLD = 0.20
|
||||
MIN_PAIRS = 10
|
||||
ALL = "all"
|
||||
Record = Mapping[str, Any]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Pair:
|
||||
family: str
|
||||
vulnerable: Record # a feature record: x and instruments
|
||||
fixed: Record
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Signal:
|
||||
"""A feature (``spec``) or an instrument's state (``instrument``)."""
|
||||
|
||||
name: str
|
||||
instrument: str
|
||||
ordered: bool
|
||||
spec: FeatureSpec | None = None
|
||||
|
||||
def value(self, record: Record) -> Any:
|
||||
if self.spec is None:
|
||||
return str((record["instruments"].get(self.instrument) or {}).get("state", "none"))
|
||||
return record["x"].get(self.name)
|
||||
|
||||
|
||||
def signals(profile: str) -> list[Signal]:
|
||||
out = [Signal(f"state.{name}", name, False) for name in instruments_for(profile) if name != "language"]
|
||||
out += [Signal(s.name, s.instrument, s.type != "enum", s) for s in features_for(profile)]
|
||||
return out
|
||||
|
||||
|
||||
def _rank(value: Any) -> float:
|
||||
return -1.0 if value is None else float(value)
|
||||
|
||||
|
||||
def separation(signal: Signal, pairs: Sequence[Pair]) -> dict[str, Any]:
|
||||
"""``pairs``, ``differ``, ``correct``, ``wrong``, ``separation`` and (ordered) the ``direction`` of one signal."""
|
||||
n = len(pairs)
|
||||
values = [(signal.value(p.vulnerable), signal.value(p.fixed)) for p in pairs]
|
||||
out: dict[str, Any] = {"pairs": n}
|
||||
if signal.ordered:
|
||||
higher = sum(_rank(v) > _rank(f) for v, f in values)
|
||||
lower = sum(_rank(v) < _rank(f) for v, f in values)
|
||||
correct, wrong = max(higher, lower), min(higher, lower)
|
||||
if correct:
|
||||
out["direction"] = "higher on vulnerable" if higher >= lower else "higher on fixed"
|
||||
else:
|
||||
counts: Counter[tuple[str, str]] = Counter((repr(v), repr(f)) for v, f in values if v != f)
|
||||
correct = wrong = 0
|
||||
for a, b in {tuple(sorted(k)) for k in counts}:
|
||||
one, other = counts[(a, b)], counts[(b, a)]
|
||||
correct, wrong = correct + max(one, other), wrong + min(one, other)
|
||||
out.update({"differ": correct + wrong, "correct": correct, "wrong": wrong, "separation": round((correct - wrong) / n, 4) if n else 0.0})
|
||||
return out
|
||||
|
||||
|
||||
def audit(pairs: Sequence[Pair], profile: str, *, threshold: float = THRESHOLD, min_pairs: int = MIN_PAIRS) -> dict[str, Any]:
|
||||
"""Every signal's separation over all pairs and per family, and the flagged (signal, scope) cells, strongest first."""
|
||||
families = sorted({p.family for p in pairs})
|
||||
scopes = {ALL: list(pairs), **{f: [p for p in pairs if p.family == f] for f in families}}
|
||||
table: dict[str, dict[str, Any]] = {}
|
||||
flagged: list[dict[str, Any]] = []
|
||||
for signal in signals(profile):
|
||||
row = {scope: separation(signal, members) for scope, members in scopes.items() if members}
|
||||
table[signal.name] = row
|
||||
for scope, cell in row.items():
|
||||
if cell["pairs"] >= min_pairs and cell["separation"] >= threshold:
|
||||
flagged.append({"signal": signal.name, "instrument": signal.instrument, "scope": scope, **cell})
|
||||
flagged.sort(key=lambda c: (-c["separation"], c["signal"], c["scope"]))
|
||||
return {
|
||||
"profile": profile,
|
||||
"threshold": threshold,
|
||||
"min_pairs": min_pairs,
|
||||
"pairs": {scope: len(members) for scope, members in scopes.items()},
|
||||
"flagged": flagged,
|
||||
"flagged_signals": sorted({c["signal"] for c in flagged}),
|
||||
"signals": table,
|
||||
}
|
||||
|
||||
|
||||
def pair_rows(rows: Iterable[Record], side_of: Mapping[str, tuple[str, str]], profile: str) -> tuple[list[Pair], dict[str, int]]:
|
||||
"""Pairs from stored feature rows of ``profile``. ``side_of`` maps a row's ``task`` (the harvest unit) to
|
||||
``(case reference, side)``; a row carrying ``pin_role`` and ``source_ref`` needs no entry. A candidate pairs when it
|
||||
has exactly one vulnerable and one fixed record in the same case; the rest are counted."""
|
||||
groups: dict[tuple[str, str, str], dict[str, list[Record]]] = {}
|
||||
skipped: Counter[str] = Counter()
|
||||
for row in rows:
|
||||
if row.get("profile") != profile:
|
||||
continue
|
||||
if row.get("pin_role") and row.get("source_ref"):
|
||||
ref, side = str(row["source_ref"]), str(row["pin_role"])
|
||||
elif str(row.get("task")) in side_of:
|
||||
ref, side = side_of[str(row["task"])]
|
||||
else:
|
||||
skipped["no side"] += 1
|
||||
continue
|
||||
if side not in ("vulnerable", "fixed"):
|
||||
skipped[f"side {side}"] += 1
|
||||
continue
|
||||
groups.setdefault((ref, str(row["candidate"]), str(row["family"])), {}).setdefault(side, []).append(row)
|
||||
pairs: list[Pair] = []
|
||||
for (_, _, family), sides in sorted(groups.items()):
|
||||
vulnerable, fixed = sides.get("vulnerable", []), sides.get("fixed", [])
|
||||
if len(vulnerable) == 1 and len(fixed) == 1:
|
||||
pairs.append(Pair(family, vulnerable[0], fixed[0]))
|
||||
elif not vulnerable or not fixed:
|
||||
skipped["one side only"] += 1
|
||||
else:
|
||||
skipped["ambiguous"] += 1
|
||||
return pairs, dict(sorted(skipped.items()))
|
||||
|
||||
|
||||
def sides_from_units(units: Mapping[str, Mapping[str, Any]]) -> dict[str, tuple[str, str]]:
|
||||
"""Harvest ``units.json`` -> unit name: (case reference, side)."""
|
||||
return {name: (str(unit["ref"]), str(unit["side"])) for name, unit in units.items()}
|
||||
|
||||
|
||||
__all__ = ["ALL", "MIN_PAIRS", "THRESHOLD", "Pair", "Signal", "audit", "pair_rows", "separation", "sides_from_units", "signals"]
|
||||
@@ -13,7 +13,9 @@ sentences citing at least one line of the code. The **modules** are ``Retrieve -
|
||||
``not_vulnerable``, 0.5 for ``unsure``); the verdict is the majority (ties are ``unsure``), and a majority
|
||||
``unsure`` can never BLOCK.
|
||||
|
||||
Signals reach the prompt only through :func:`.schema.validate_x` (no ``EXCLUDED_FIELDS``, no free text); a missing
|
||||
Signals reach the prompt only through :func:`.schema.validate_x` (no ``EXCLUDED_FIELDS``, no free text) and the input
|
||||
profile (``ProgramSpec.inputs``, default ``v1``: the engine instrument is withheld from prompts and from the retrieval
|
||||
distance, see :data:`.schema.INPUT_PROFILES`); a missing
|
||||
instrument is shown in words. Every model call goes through :class:`Caller`: a response cache keyed by
|
||||
``sha256(model, Model parameters digest, messages, temperature, sample, json_object)`` stored as
|
||||
``responses/<sha>.json``, so a rerun on the same inputs replays at $0 and reproduces byte for byte. Before a call the
|
||||
@@ -35,7 +37,7 @@ from ..plane.memory import MemoryStore
|
||||
from .examples import Example
|
||||
from .folds import Fold
|
||||
from .retrieve import Retrieval, Target, assert_boundary, demonstrations_for, retrieve
|
||||
from .schema import features_for, instruments_for, validate_x
|
||||
from .schema import DEFAULT_INPUTS, features_for, instruments_for, validate_x, withheld
|
||||
|
||||
VERDICTS = ("vulnerable", "not_vulnerable", "unsure")
|
||||
UNSURE = "unsure"
|
||||
@@ -70,7 +72,12 @@ class Signature:
|
||||
SIGNATURE = Signature(
|
||||
(
|
||||
Field("code", "in", "excerpt", "numbered lines, <= 80 lines and 4,000 characters; a delta adds the diff (<= 40 lines)"),
|
||||
Field("signals", "in", "record", "the allow-listed features as name: value lines; a missing instrument in words"),
|
||||
Field(
|
||||
"signals",
|
||||
"in",
|
||||
"record",
|
||||
"the allow-listed features the input profile shows, as name: value lines; a missing instrument in words",
|
||||
),
|
||||
Field("roles", "in", "roles", "inferred roles of the function: role, operation, line, origin"),
|
||||
Field("family", "in", "family", "the candidate's family, or unknown"),
|
||||
Field("verdict", "out", "enum", "vulnerable | not_vulnerable | unsure"),
|
||||
@@ -95,13 +102,16 @@ BASELINE_INSTRUCTION = (
|
||||
# --- rendering ---------------------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def render_signals(x: Mapping[str, Any], instruments: Mapping[str, Mapping[str, Any]], profile: str) -> str:
|
||||
def render_signals(x: Mapping[str, Any], instruments: Mapping[str, Mapping[str, Any]], profile: str, inputs: str = DEFAULT_INPUTS) -> str:
|
||||
"""``name: value`` per feature of an instrument that ran; ``<instrument>: no coverage | failed | not applicable``
|
||||
otherwise. The record is validated first: an unknown, label or identity field never reaches a prompt."""
|
||||
otherwise. The record is validated first: an unknown, label or identity field never reaches a prompt. An
|
||||
instrument the input profile withholds (:data:`.schema.INPUT_PROFILES`; v1: the engine) is not rendered at all --
|
||||
neither its state nor its values -- though the stored record keeps it."""
|
||||
validate_x(x, instruments, profile)
|
||||
hidden = withheld(inputs)
|
||||
lines: list[str] = []
|
||||
for name in instruments_for(profile):
|
||||
if name == "language":
|
||||
if name == "language" or name in hidden:
|
||||
continue
|
||||
state = str(instruments[name]["state"])
|
||||
if state != "ran":
|
||||
@@ -129,8 +139,9 @@ def _case(
|
||||
family: str,
|
||||
profile: str,
|
||||
language: str,
|
||||
inputs: str = DEFAULT_INPUTS,
|
||||
) -> str:
|
||||
signals = render_signals(x, instruments, profile)
|
||||
signals = render_signals(x, instruments, profile, inputs)
|
||||
return f"Family: {family}\nLanguage: {language}\nCode:\n{code.rstrip()}\nSignals:\n{signals}\nRoles:\n{render_roles(roles)}"
|
||||
|
||||
|
||||
@@ -181,6 +192,7 @@ class ProgramSpec:
|
||||
lam: float = 0.5
|
||||
n1: int = 40
|
||||
priors: str = "off"
|
||||
inputs: str = DEFAULT_INPUTS # the input profile: which instruments the classifier never sees (v1: the engine)
|
||||
max_output_tokens: int = MAX_OUTPUT_TOKENS
|
||||
|
||||
|
||||
@@ -228,7 +240,7 @@ def render_prefix(spec: ProgramSpec, demos: Sequence[Demo], examples: Mapping[st
|
||||
"cited_lines": list(demo.cited_lines)} # fmt: skip
|
||||
parts.append(
|
||||
f"Demonstration {number}:\n"
|
||||
+ _case(code, example.x, example.instruments, example.roles, example.family, spec.profile, example.language)
|
||||
+ _case(code, example.x, example.instruments, example.roles, example.family, spec.profile, example.language, spec.inputs)
|
||||
+ f"\nAnswer: {json.dumps(answer, sort_keys=True)}"
|
||||
)
|
||||
return "\n\n".join(parts)
|
||||
@@ -249,10 +261,12 @@ def render(
|
||||
code = excerpt_text(example.excerpt_sha) or ""
|
||||
blocks.append(
|
||||
f"Example {number} (label: {label_word(example.label)}):\n"
|
||||
+ _case(code, example.x, example.instruments, example.roles, example.family, spec.profile, example.language)
|
||||
+ _case(code, example.x, example.instruments, example.roles, example.family, spec.profile, example.language, spec.inputs)
|
||||
)
|
||||
t = candidate.target
|
||||
blocks.append("Candidate:\n" + _case(candidate.code, t.x, t.instruments, candidate.roles, t.family, spec.profile, candidate.language))
|
||||
blocks.append(
|
||||
"Candidate:\n" + _case(candidate.code, t.x, t.instruments, candidate.roles, t.family, spec.profile, candidate.language, spec.inputs)
|
||||
)
|
||||
messages: list[dict[str, object]] = [
|
||||
{"role": "system", "content": render_prefix(spec, demos, examples, excerpt_text)},
|
||||
{"role": "user", "content": "\n\n".join(blocks)},
|
||||
@@ -454,7 +468,7 @@ class Program:
|
||||
def prepare(self, candidate: Candidate, fold: Fold, *, seed: int = 0) -> tuple[Prompt, Retrieval]:
|
||||
target = candidate.target
|
||||
got = retrieve(target, self.memory, fold, vectors=self.vectors, n1=self.spec.n1, k=self.spec.k_ret, lam=self.spec.lam, seed=seed,
|
||||
priors=self.spec.priors) # fmt: skip
|
||||
priors=self.spec.priors, inputs=self.spec.inputs) # fmt: skip
|
||||
usable = [Demo.from_dict(d) for d in demonstrations_for([d.as_dict() for d in self.spec.demos], self.index, target, fold)]
|
||||
prompt = render(self.spec, candidate, got.examples, usable, self.index, self.excerpt_text)
|
||||
assert_boundary([*prompt.example_ids, *prompt.demo_ids], self.index, target, fold, vectors=self.vectors)
|
||||
|
||||
@@ -8,7 +8,8 @@ profile, computed per instrument block so that missing is explicit, never impute
|
||||
both is ``0``, in one is ``1``;
|
||||
- different states (``ran`` vs ``none``/``failed``/``not_applicable``, or ``failed`` vs ``none``): ``1``; the
|
||||
same non-``ran`` state: ``0``;
|
||||
- ``D`` is the uniform mean over the profile's blocks; prior features are dropped with ``priors = "off"``.
|
||||
- ``D`` is the uniform mean over the profile's blocks; prior features are dropped with ``priors = "off"``, and the blocks
|
||||
of instruments the input profile withholds (``inputs``, default ``v1``: the engine) are left out entirely.
|
||||
|
||||
Ties break by ``sha256(example id + seed)``; the first ``n1 = 40`` go to stage 2, a cosine re-rank of the code
|
||||
embeddings, ``r = lam (1 - D) + (1 - lam) cos``. Without the target's vector the whole retrieval is
|
||||
@@ -34,7 +35,7 @@ from .embeddings import cosine
|
||||
from .examples import Example
|
||||
from .folds import Fold
|
||||
from .labels import repo_name
|
||||
from .schema import UNBOUNDED_FLOATS, FeatureSpec, features_for, instruments_for
|
||||
from .schema import DEFAULT_INPUTS, UNBOUNDED_FLOATS, FeatureSpec, features_for, instruments_for, withheld
|
||||
|
||||
N1 = 40
|
||||
K_RET = 6
|
||||
@@ -91,9 +92,12 @@ def gower(
|
||||
profile: str,
|
||||
*,
|
||||
priors: str = "off",
|
||||
inputs: str = DEFAULT_INPUTS,
|
||||
) -> float:
|
||||
"""The per-instrument Gower distance of two records of ``profile`` (module docstring), in [0, 1]."""
|
||||
total, blocks = 0.0, _blocks(profile, priors)
|
||||
"""The per-instrument Gower distance of two records of ``profile`` (module docstring), in [0, 1]. The blocks of
|
||||
instruments the input profile withholds (v1: the engine) are not compared, so neighbours cannot carry them."""
|
||||
hidden = withheld(inputs)
|
||||
total, blocks = 0.0, [(name, specs) for name, specs in _blocks(profile, priors) if name not in hidden]
|
||||
for name, specs in blocks:
|
||||
sa, sb = a_instruments.get(name, {}).get("state", "none"), b_instruments.get(name, {}).get("state", "none")
|
||||
if sa != sb:
|
||||
@@ -192,11 +196,12 @@ def retrieve(
|
||||
lam: float = LAM,
|
||||
seed: int = 0,
|
||||
priors: str = "off",
|
||||
inputs: str = DEFAULT_INPUTS,
|
||||
) -> Retrieval:
|
||||
"""The ``k`` balanced examples for ``target`` from the eligible part of ``pool`` (module docstring)."""
|
||||
candidates = [e for e in pool if e.profile == target.profile and eligible(e, target, fold)]
|
||||
scored = sorted(
|
||||
((gower(target.x, target.instruments, e.x, e.instruments, target.profile, priors=priors), e) for e in candidates),
|
||||
((gower(target.x, target.instruments, e.x, e.instruments, target.profile, priors=priors, inputs=inputs), e) for e in candidates),
|
||||
key=lambda pair: (pair[0], _tie(pair[1].id, seed)),
|
||||
)[:n1]
|
||||
mode = "signals+embeddings" if target.vector is not None and vectors else "signals_only"
|
||||
|
||||
@@ -94,6 +94,15 @@ EXCLUDED_FIELDS = frozenset(
|
||||
) # fmt: skip
|
||||
|
||||
|
||||
# What the classifier may see of a record (design, Risks: label-tracking signals, 2026-10-01). A record is stored whole;
|
||||
# an input profile names the instruments whose state and values are withheld from the prompt, the demonstrations, the
|
||||
# instruction proposer and the retrieval distance. On the pairs the engine's "read the file, asked no question" state
|
||||
# tracks the label (a fixed side loses its sink), so v1 withholds the engine entirely; it returns only through a
|
||||
# pre-registered A/B on negatives where it cannot track a fix.
|
||||
INPUT_PROFILES: Mapping[str, tuple[str, ...]] = {"v1": ("engine",), "full": ()}
|
||||
DEFAULT_INPUTS = "v1"
|
||||
|
||||
|
||||
class FeatureRecordError(ValueError):
|
||||
"""A record names a feature outside the allow-list, a value outside its type, or a label/identity field."""
|
||||
|
||||
@@ -172,6 +181,13 @@ def instruments_for(profile: str) -> tuple[str, ...]:
|
||||
return tuple(name for name in INSTRUMENTS if name in used)
|
||||
|
||||
|
||||
def withheld(inputs: str) -> frozenset[str]:
|
||||
"""The instruments an input profile withholds from the classifier (:data:`INPUT_PROFILES`)."""
|
||||
if inputs not in INPUT_PROFILES:
|
||||
raise FeatureRecordError(f"unknown input profile {inputs!r} (one of {', '.join(INPUT_PROFILES)})")
|
||||
return frozenset(INPUT_PROFILES[inputs])
|
||||
|
||||
|
||||
def feature_set_digest() -> str:
|
||||
"""sha256 of the canonical feature list: any change to a name, type, cap, vocabulary or profile changes it."""
|
||||
canonical = json.dumps([asdict(spec) for spec in FEATURES], sort_keys=True, separators=(",", ":"))
|
||||
@@ -251,8 +267,9 @@ def names(specs: Iterable[FeatureSpec] = FEATURES) -> tuple[str, ...]:
|
||||
|
||||
|
||||
__all__ = [
|
||||
"BY_NAME", "EXCLUDED_FIELDS", "FEATURES", "FINALS", "INSTRUMENTS", "LANGUAGES", "MECHANISMS", "NOVELTY", "OPERATIONS",
|
||||
"BY_NAME", "DEFAULT_INPUTS", "EXCLUDED_FIELDS", "FEATURES", "FINALS", "INPUT_PROFILES", "INSTRUMENTS", "LANGUAGES", "MECHANISMS",
|
||||
"NOVELTY", "OPERATIONS",
|
||||
"ORIGINS", "PROFILES", "RULE_TAG_MECHANISM", "RUNGS", "SCHEMA_VERSION", "SOURCE_KINDS", "STATES", "UNITS",
|
||||
"FeatureRecordError", "FeatureSpec", "feature_set_digest", "features_for", "instruments_for", "names",
|
||||
"validate_record", "validate_x",
|
||||
"validate_record", "validate_x", "withheld",
|
||||
] # fmt: skip
|
||||
|
||||
@@ -14,8 +14,11 @@ root: the excerpts and candidates are inputs, never committed):
|
||||
Units carry no label. A pair side is a files-only Workspace holding the excerpt (and the documents that register
|
||||
it) under ``repo/`` and the candidates under ``inputs/`` -- outside the hunt's root, so the tools cannot read them;
|
||||
its name is opaque (``h-<sha>``): the pair's name says its CWE and its side. ``agree`` gets no declared sites, so
|
||||
no ``site_match`` is computed. ``units.json`` maps each unit name to its source, reference, side, repository, pin
|
||||
and candidates for the host-side feature build; it names functions and stays local.
|
||||
no ``site_match`` is computed. A side's candidates are only the functions that side declares (the excerpt builder's
|
||||
matcher, :func:`declares`): a labelled function absent from a side -- a fixed side whose fix removed or renamed it --
|
||||
is never asked there and is listed under ``absent_on_side`` instead. ``units.json`` maps each unit name to its
|
||||
source, reference, side, repository, pin and candidates for the host-side feature build; it names functions and stays
|
||||
local.
|
||||
|
||||
Every Run entry has a budget; the sum over each Run is held under ``ceiling`` (USD) by construction: verify gets
|
||||
:data:`USD_PER_HUNT` per hunt of an excerpt or a repository (scaled down when the hunts would exceed the ceiling),
|
||||
@@ -44,7 +47,7 @@ from .tasks.repo_facts import DECLARATION, GLOBAL, _declared_name
|
||||
from .tasks.roles import CHUNK_LINES
|
||||
from .tasks.verify import OPERATIONS, units_of
|
||||
|
||||
__all__ = ["Unit", "declaration_line", "pair_units", "population_units", "render_harvest", "write_harvest"]
|
||||
__all__ = ["Unit", "declaration_line", "declares", "pair_units", "population_units", "render_harvest", "write_harvest"]
|
||||
|
||||
Candidate = tuple[str, str, int]
|
||||
REPO = "repo"
|
||||
@@ -79,14 +82,18 @@ class Unit:
|
||||
files: tuple[tuple[str, str], ...] = field(default=())
|
||||
verify: tuple[Candidate, ...] = field(default=()) # the candidates of ``family`` (all of them when empty)
|
||||
sizes: tuple[tuple[str, int, int], ...] = field(default=()) # (path, characters, lines) of a pin's candidate files
|
||||
absent_on_side: tuple[str, ...] = field(default=()) # labelled ``path::function`` names this side does not declare
|
||||
|
||||
@property
|
||||
def paths(self) -> list[str]:
|
||||
return sorted({c[0] for c in self.candidates})
|
||||
|
||||
def index_row(self) -> dict[str, Any]:
|
||||
return {"source": self.source, "ref": self.ref, "side": self.side, "repo": self.repo, "pin": self.pin,
|
||||
"family": self.family, "candidates": [list(c) for c in self.candidates]} # fmt: skip
|
||||
row: dict[str, Any] = {"source": self.source, "ref": self.ref, "side": self.side, "repo": self.repo, "pin": self.pin,
|
||||
"family": self.family, "candidates": [list(c) for c in self.candidates]} # fmt: skip
|
||||
if self.absent_on_side:
|
||||
row["absent_on_side"] = list(self.absent_on_side)
|
||||
return row
|
||||
|
||||
|
||||
def _opaque(*parts: str) -> str:
|
||||
@@ -106,6 +113,14 @@ def declaration_line(lines: Sequence[str] | None, path: str, function: str) -> i
|
||||
return next((i + 1 for i, line in enumerate(lines) if _declared_name(pattern, line) == function), 1)
|
||||
|
||||
|
||||
def declares(lines: Sequence[str] | None, path: str, function: str) -> bool:
|
||||
"""Whether a side's file declares ``function``, by the excerpt builder's matcher
|
||||
(:func:`..learn.excerpt.function_bounds`): a candidate is asked only where an excerpt of it exists."""
|
||||
from ..learn.excerpt import function_bounds
|
||||
|
||||
return bool(lines) and function_bounds(lines or (), detect_language(Path(path)) or "", function) is not None
|
||||
|
||||
|
||||
def _read_labels(labels: Path) -> list[dict[str, Any]]:
|
||||
return [json.loads(line) for line in labels.read_text(encoding="utf-8").splitlines() if line.strip()]
|
||||
|
||||
@@ -135,13 +150,18 @@ def pair_units(labels: Path, catalog: Path) -> list[Unit]:
|
||||
lines = {path: text.splitlines() for path, text in texts.items()}
|
||||
|
||||
def located(names: Iterable[str], lines: Mapping[str, list[str]] = lines) -> tuple[Candidate, ...]:
|
||||
return tuple(sorted({(p, fn, declaration_line(lines.get(p), p, fn)) for c in names for p, _, fn in [c.partition("::")]}))
|
||||
"""The candidates this side declares: a name taken from the other side is never asked here."""
|
||||
split = [(p, fn) for c in names for p, _, fn in [c.partition("::")]]
|
||||
return tuple(sorted({(p, fn, declaration_line(lines.get(p), p, fn)) for p, fn in split if declares(lines.get(p), p, fn)}))
|
||||
|
||||
labelled = set().union(*families.values())
|
||||
absent = tuple(sorted(c for c in labelled if not declares(lines.get(c.partition("::")[0]), *c.partition("::")[::2])))
|
||||
asked = located(families[verify[0]]) if verify else () # none declared here: no verify unit on this side
|
||||
role = "vulnerable" if side == "vuln" else "fixed"
|
||||
out.append(Unit(
|
||||
_opaque("pairs", name, side), "pairs", name, role, case.repo or f"pairs/{case.slice}/{name}",
|
||||
blob_sha1(texts[case.relpath]), verify[0] if verify else "", located(set().union(*families.values())),
|
||||
tuple(sorted(texts.items())), located(families[verify[0]]) if verify else (),
|
||||
blob_sha1(texts[case.relpath]), verify[0] if asked else "", located(labelled),
|
||||
tuple(sorted(texts.items())), asked, absent_on_side=absent,
|
||||
)) # fmt: skip
|
||||
return out
|
||||
|
||||
|
||||
@@ -325,6 +325,24 @@ class MemoryStore(ABC):
|
||||
ordered = [merged[i] for i in sorted(merged)]
|
||||
self._put(key, "".join(_line(r) + "\n" for r in ordered).encode("utf-8"), _labels(ordered))
|
||||
|
||||
def drop_rows(self, repo: str, pin: str, ids: Iterable[str]) -> int:
|
||||
"""Remove the rows ``ids`` from one repository + pin object (a rebuild that no longer produces them); returns
|
||||
how many were removed. The object is deleted when it ends up empty."""
|
||||
key = f"repos/{repo_dir(repo)}/{pin}.jsonl"
|
||||
found = self._get(key)
|
||||
unwanted = set(ids)
|
||||
if found is None or not unwanted:
|
||||
return 0
|
||||
rows = _parse_lines(found[0], key)
|
||||
kept = [r for r in rows if str(r["id"]) not in unwanted]
|
||||
if len(kept) == len(rows):
|
||||
return 0
|
||||
if kept:
|
||||
self._put(key, "".join(_line(r) + "\n" for r in kept).encode("utf-8"), _labels(kept))
|
||||
else:
|
||||
self._delete(key)
|
||||
return len(rows) - len(kept)
|
||||
|
||||
def ingest_rows(self, run: str, task: str, rows: Sequence[Mapping[str, Any]], facts: bytes | None = None) -> IngestResult:
|
||||
"""One (run, task)'s rows and facts; a repeat of the same rows is skipped by the index."""
|
||||
checked = [validate_row(dict(r), f"{run}/{task} row {i + 1}") for i, r in enumerate(rows)]
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
"""The leakage audit (``ousast learn audit-leaks``): a signal that alone names the side of a pair is flagged, one that
|
||||
does not is not; rows pair by case, candidate and family; an empty store is an error, never a clean audit."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from learn_fixtures import x_record
|
||||
|
||||
from openultrasast.cli import main
|
||||
from openultrasast.learn.leaks import Pair, Signal, audit, pair_rows, separation
|
||||
from openultrasast.learn.schema import BY_NAME
|
||||
from openultrasast.plane.memory import FileStore
|
||||
|
||||
|
||||
def _pairs(n: int = 20) -> list[Pair]:
|
||||
"""``n`` pairs: the engine state tracks the side (ran on the vulnerable side, failed on the fixed one) in all but
|
||||
two; quick hits differ in half the pairs, in alternating directions (no information)."""
|
||||
out = []
|
||||
for i in range(n):
|
||||
leaks = i >= 2
|
||||
vulnerable = x_record(hits=(i % 2) * 2, engine="ran", findings=1)
|
||||
fixed = x_record(hits=((i + 1) % 2) * 2, engine="failed" if leaks else "ran", findings=1)
|
||||
out.append(Pair("injection", vulnerable, fixed))
|
||||
return out
|
||||
|
||||
|
||||
def test_a_leaking_signal_is_flagged_and_a_clean_one_is_not() -> None:
|
||||
report = audit(_pairs(), "static")
|
||||
assert report["pairs"] == {"all": 20, "injection": 20}
|
||||
engine = report["signals"]["state.engine"]["all"]
|
||||
assert engine == {"pairs": 20, "differ": 18, "correct": 18, "wrong": 0, "separation": 0.9}
|
||||
assert "state.engine" in report["flagged_signals"] and "eng.findings" in report["flagged_signals"]
|
||||
hits = report["signals"]["qr.enabled_hits"]["all"]
|
||||
assert hits["differ"] == 20 and hits["correct"] == hits["wrong"] == 10 and hits["separation"] == 0.0
|
||||
assert "qr.enabled_hits" not in report["flagged_signals"] and "state.quick" not in report["flagged_signals"]
|
||||
assert report["flagged"][0]["separation"] == 0.9 and all(c["separation"] >= 0.2 for c in report["flagged"])
|
||||
few = audit(_pairs()[:9], "static") # under min_pairs: never flagged, however clean the split
|
||||
assert few["flagged"] == [] and few["signals"]["state.engine"]["all"]["separation"] > 0.7
|
||||
|
||||
|
||||
def test_an_ordered_signal_has_one_direction() -> None:
|
||||
spec = BY_NAME["qr.enabled_hits"]
|
||||
signal = Signal(spec.name, spec.instrument, True, spec)
|
||||
higher = [Pair("f", x_record(hits=3), x_record(hits=1)) for _ in range(6)] + [Pair("f", x_record(hits=0), x_record(hits=2))] * 2
|
||||
cell = separation(signal, higher)
|
||||
assert cell["direction"] == "higher on vulnerable" and (cell["correct"], cell["wrong"]) == (6, 2) and cell["separation"] == 0.5
|
||||
|
||||
|
||||
def _row(task: str, family: str = "injection", candidate: str = "a.py::f", **kw: object) -> dict:
|
||||
rec = x_record(**kw) # type: ignore[arg-type]
|
||||
pin = ("0" if task.endswith("v") else "1") * 40
|
||||
return {"id": f"{task}:{candidate}", "kind": "features", "repo": "github.com/o/r", "pin": pin, "run": "harvest", "task": task,
|
||||
"population": "pairs", "split": "pairs", "image": "none", **rec} # fmt: skip
|
||||
|
||||
|
||||
def test_rows_pair_by_case_candidate_and_family() -> None:
|
||||
sides = {"u1v": ("case-1", "vulnerable"), "u1f": ("case-1", "fixed"), "u2v": ("case-2", "vulnerable")}
|
||||
rows = [_row("u1v"), _row("u1f", engine="failed"), _row("u2v"), _row("u9")]
|
||||
pairs, skipped = pair_rows(rows, sides, "static")
|
||||
assert len(pairs) == 1 and pairs[0].vulnerable["instruments"]["engine"]["state"] == "ran"
|
||||
assert pairs[0].fixed["instruments"]["engine"]["state"] == "failed"
|
||||
assert skipped == {"no side": 1, "one side only": 1}
|
||||
assert pair_rows(rows, sides, "plane")[0] == [] # another profile's rows are not read
|
||||
|
||||
|
||||
def test_cli_audits_the_store_and_refuses_an_empty_one(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
|
||||
store = FileStore(tmp_path / "memory")
|
||||
units = tmp_path / "units.json"
|
||||
sides = {}
|
||||
rows = []
|
||||
for i in range(12):
|
||||
sides[f"c{i}v"], sides[f"c{i}f"] = {"ref": f"case-{i}", "side": "vulnerable"}, {"ref": f"case-{i}", "side": "fixed"}
|
||||
rows += [_row(f"c{i}v", findings=1), _row(f"c{i}f", engine="failed")]
|
||||
store.put_rows(rows)
|
||||
units.write_text(json.dumps(sides))
|
||||
out = tmp_path / "audit.json"
|
||||
args = ["learn", "audit-leaks", "--memory", f"file://{tmp_path / 'memory'}", "--units", str(units), "--profile", "static"]
|
||||
assert main([*args, "--out", str(out)]) == 0
|
||||
report = json.loads(out.read_text())
|
||||
assert report["rows_read"] == 24 and report["pairs"]["all"] == 12 and "state.engine" in report["flagged_signals"]
|
||||
assert "a.py" not in out.read_text() # counts only: no candidate reaches the report
|
||||
empty = ["learn", "audit-leaks", "--memory", f"file://{tmp_path / 'empty'}", "--units", str(units)]
|
||||
assert main(empty) == 2 and "no pair to audit" in capsys.readouterr().err
|
||||
|
||||
|
||||
def test_drop_rows_removes_only_the_named_rows(tmp_path: Path) -> None:
|
||||
store = FileStore(tmp_path / "memory")
|
||||
store.put_rows([_row("u1v"), _row("u1v", candidate="a.py::g")])
|
||||
assert store.drop_rows("github.com/o/r", "0" * 40, ["u1v:a.py::g", "absent"]) == 1
|
||||
assert [r.row["id"] for r in store.rows(kind="features")] == ["u1v:a.py::f"]
|
||||
assert store.drop_rows("github.com/o/r", "0" * 40, ["u1v:a.py::f"]) == 1 and store.rows(kind="features") == []
|
||||
@@ -75,11 +75,42 @@ def test_prompts_hold_no_excluded_field_and_no_candidate_label() -> None:
|
||||
|
||||
|
||||
def test_missing_instruments_are_said_in_words() -> None:
|
||||
prompt, _ = program().prepare(candidate(engine="none"), FOLD)
|
||||
prompt, _ = program(inputs="full").prepare(candidate(engine="none"), FOLD)
|
||||
tail = str(prompt.messages[-1]["content"]).rsplit("Candidate:\n", 1)[1]
|
||||
assert "engine: no coverage" in tail and "delta: not applicable" in tail and "eng.findings" not in tail
|
||||
|
||||
|
||||
ENGINE_FIELD = re.compile(r"^(engine:|eng\.)", re.MULTILINE)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("engine", ["ran", "failed", "none"])
|
||||
def test_v1_prompts_contain_no_engine_field(engine: str) -> None:
|
||||
"""v1 (the default input profile) withholds the engine -- state and values -- from every rendered message: the
|
||||
candidate, the retrieved examples and the demonstrations, as the scripted client receives them."""
|
||||
assert ProgramSpec().inputs == "v1"
|
||||
demo_example = EXAMPLES[0]
|
||||
demo = Demo(demo_example.id, demo_example.excerpt_sha, 0, "injection", "not_vulnerable", 0.8, "Line 11 escapes q.", (11,))
|
||||
chat = ScriptedChat()
|
||||
caller = Caller(MeteredClient(chat, prices=PRICES), "deepseek-flash", PRICES)
|
||||
program(demos=(demo,), k=2).decide(candidate(engine=engine, findings=2 if engine == "ran" else 0), FOLD, caller)
|
||||
assert len(chat.calls) == 2
|
||||
for call in chat.calls:
|
||||
text = "\n".join(str(m["content"]) for m in call["messages"])
|
||||
assert "Demonstration 1:" in text and text.count("(label: ") == 6 and "Candidate:" in text
|
||||
assert not ENGINE_FIELD.search(text), ENGINE_FIELD.search(text)
|
||||
assert "qr.enabled_hits:" in text and "delta: not applicable" in text # the other instruments still show
|
||||
full, _ = program(demos=(demo,), inputs="full").prepare(candidate(engine=engine), FOLD)
|
||||
assert ENGINE_FIELD.search("\n".join(str(m["content"]) for m in full.messages))
|
||||
|
||||
|
||||
def test_v1_proposer_summary_contains_no_engine_field() -> None:
|
||||
from openultrasast.learn.compile import _signal_summary
|
||||
|
||||
leaky = replace(EXAMPLES[0], **{k: v for k, v in x_record(hits=2, findings=3).items() if k in ("x", "instruments")})
|
||||
assert not any(k.startswith("eng.") for k in _signal_summary(leaky))
|
||||
assert any(k.startswith("eng.") for k in _signal_summary(leaky, "full"))
|
||||
|
||||
|
||||
def test_the_prefix_is_byte_identical_across_candidates() -> None:
|
||||
demo_example = EXAMPLES[0]
|
||||
demo = Demo(demo_example.id, demo_example.excerpt_sha, 0, "injection", "not_vulnerable", 0.8, "Line 11 escapes q.", (11,))
|
||||
|
||||
@@ -30,6 +30,8 @@ QUICK_ALL = [s for s in features_for("static") if s.instrument == "quick"]
|
||||
|
||||
|
||||
def _d(a: dict, b: dict, **kw: str) -> float:
|
||||
"""The full distance (every block, the engine included): the definition. v1 withholds the engine (below)."""
|
||||
kw.setdefault("inputs", "full")
|
||||
return gower(a["x"], a["instruments"], b["x"], b["instruments"], "static", **kw)
|
||||
|
||||
|
||||
@@ -53,6 +55,22 @@ def test_prior_features_are_dropped_with_priors_off() -> None:
|
||||
assert _d(plain, prior, priors="on") == pytest.approx(5 / 10 / len(QUICK_ALL) / BLOCKS)
|
||||
|
||||
|
||||
def test_v1_distance_contains_no_engine_field() -> None:
|
||||
"""v1 (the default) withholds the engine: neither its state nor its values move the distance, so neighbours cannot
|
||||
carry it; every other block still counts, over one block fewer."""
|
||||
ran, failed, none = x_record(findings=3), x_record(engine="failed"), x_record(engine="none")
|
||||
for a, b in ((ran, failed), (ran, none), (failed, none), (x_record(findings=0), ran)):
|
||||
assert gower(a["x"], a["instruments"], b["x"], b["instruments"], "static") == 0.0
|
||||
assert _d(a, b) > 0.0
|
||||
hits = x_record(hits=10, engine="failed")
|
||||
expected = (10 / 20 + 10 / 10) / len(QUICK_NO_PRIOR) / (BLOCKS - 1)
|
||||
assert gower(ran["x"], ran["instruments"], hits["x"], hits["instruments"], "static") == pytest.approx(expected)
|
||||
same, other = example("same-engine", group="g1", label=0, engine="failed"), example("other-engine", group="g2", label=1, findings=4)
|
||||
v1 = [gower(failed["x"], failed["instruments"], e.x, e.instruments, "static") for e in (same, other)]
|
||||
full = [gower(failed["x"], failed["instruments"], e.x, e.instruments, "static", inputs="full") for e in (same, other)]
|
||||
assert v1[0] == v1[1] and full[0] < full[1] # the engine state ranked them; in v1 it cannot
|
||||
|
||||
|
||||
def _target(**kw: object) -> Target:
|
||||
rec = x_record(**kw) # type: ignore[arg-type]
|
||||
return Target("target/repo", "injection", "static", rec["x"], rec["instruments"])
|
||||
|
||||
@@ -5,7 +5,9 @@ from __future__ import annotations
|
||||
import json
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
import yaml
|
||||
|
||||
from openultrasast.plane import reconciler
|
||||
@@ -98,3 +100,28 @@ def test_blob_sha1_is_gits_and_declaration_lines() -> None:
|
||||
lines = text.splitlines()
|
||||
assert declaration_line(lines, "m.py", "b") == 4
|
||||
assert declaration_line(lines, "m.py", "missing") == 1 and declaration_line(None, "m.py", "a") == 1
|
||||
|
||||
|
||||
def test_pair_sides_ask_only_the_functions_they_declare(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""A fixed side whose fix removed a labelled function never gets it as a candidate (no phantom verify candidate);
|
||||
it is listed under ``absent_on_side``. A side declaring none of its verify family's functions is no verify unit."""
|
||||
from openultrasast import pairs
|
||||
from openultrasast.plane.harvest import pair_units
|
||||
|
||||
vuln, fixed = tmp_path / "vuln.py", tmp_path / "fixed.py"
|
||||
vuln.write_text("import os\n\ndef run(cmd):\n os.system(cmd)\n\ndef keep(x):\n return x\n")
|
||||
fixed.write_text("import shlex\n\ndef keep(x):\n return shlex.quote(x)\n")
|
||||
case = SimpleNamespace(name="p-cwe-078", vuln_file=vuln, fixed_file=fixed, relpath="app.py", context_files=(), repo="", slice="s")
|
||||
monkeypatch.setattr(pairs, "load_pair_catalog", lambda _catalog: [case])
|
||||
rows = [{"source": "pairs", "unit": "pin", "source_ref": "p-cwe-078", "family": "injection", "candidate": f"app.py::{fn}"}
|
||||
for fn in ("run", "keep")] # fmt: skip
|
||||
labels = tmp_path / "labels.jsonl"
|
||||
labels.write_text("".join(json.dumps(r) + "\n" for r in rows))
|
||||
by_side = {u.side: u for u in pair_units(labels, tmp_path / "catalog.toml")}
|
||||
assert [c[1] for c in by_side["vulnerable"].candidates] == ["keep", "run"] and by_side["vulnerable"].absent_on_side == ()
|
||||
assert [c[1] for c in by_side["fixed"].candidates] == ["keep"] and [c[1] for c in by_side["fixed"].verify] == ["keep"]
|
||||
assert by_side["fixed"].absent_on_side == ("app.py::run",)
|
||||
assert by_side["fixed"].index_row()["absent_on_side"] == ["app.py::run"]
|
||||
fixed.write_text("import shlex\n\ndef other(x):\n return shlex.quote(x)\n")
|
||||
gone = next(u for u in pair_units(labels, tmp_path / "catalog.toml") if u.side == "fixed")
|
||||
assert gone.candidates == () and gone.verify == () and gone.family == "" and len(gone.absent_on_side) == 2
|
||||
|
||||
Reference in New Issue
Block a user