marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-17 03:59:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(model-grounded-detection): group 1, subtract the noise architecture

Browse Source 
99 source modules -> 81. 832 tests -> 646. The whole learning/ package (16
modules, ~4,100 lines) and six semantic/ modules removed, with the three gates
byte-identical to 2026-09-06-gate-baseline.txt at every step.

The sequence is what made it safe: vocabularies ported first (1.1), importers
detached second (1.2-1.3), deletion third (1.4). Every import break during the
deletion was therefore a detachment miss in the importer and was fixed there --
no module was restored to satisfy one. Both misses were the same shape and were
fixed as a class: operations.py and shapes.py each imported trailing_name from
the deleted variants.py, so it moved to semantic/ir.py, its natural home, and
both importers changed in one edit.

Ported before deleting: the closed guard vocabulary (verbatim, nothing else
holds it) into model/specs.py, which derives TaintSpec and DominanceSpec from
the retained ruleset facts rather than copying them. Re-homed: families ->
model/taxonomy, candidates -> model/candidates, endpoint -> model/endpoint,
plus Region, labeled_family and labeled_spans. labeled_family no longer falls
back to the LLM classifier -- an unlabeled pair is honestly unknown.

Four deviations, each recorded in the spec notes with its reason. improve/
evolve.py and validator.py were stripped, not deleted: each mixes the mechanism
lever (gone) with the deterministic rule-status loop `ousast improve` still
runs, which has no LLM in it; a test now asserts the lever's absence by symbol
so it cannot creep back. The learning CLI group became `model candidates`,
because deleting it outright would have orphaned the retained enumerator by
removing its only entry point. thinking moved from [learning] to [models].

A gap named rather than papered over: no retained sink fact routes by CWE to
path, output_encoding, untrusted_destination or prototype -- the same hole the
entailment ceiling diagnosed. The test asserts the seeded set and the gap set
exactly, so task 4.2 must update it when it authors those sinks.

The audit found exactly one genuine orphan: vectorstore.py, a bakeoff harness
with no production caller and no CLI, alive only through its own test. The five
named subsystems each carry an explicit decision. Manifest committed, with a
test asserting the tree still matches it.

Group 2 (the Joern go/no-go) has not started.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-08 12:15:54 +02:00
co-authored by Claude Opus 5
parent fc4dab97fb
commit 9ed0e45c5f
86 changed files with 1710 additions and 12158 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/openultrasast/cli.py
+46 -612
View File
@@ -3,7 +3,7 @@ from __future__ import annotations
import argparse
import json
import tempfile
from collections.abc import Callable, Sequence
from collections.abc import Sequence
from dataclasses import dataclass, replace
from pathlib import Path
from time import perf_counter
@@ -31,7 +31,6 @@ from .calibration import (
)
from .complexity import map as complexity_map
from .complexity.ledger import persist_verdicts
from .complexity.map import Hotspot
from .config import ObligationsConfig, load_config, load_dotenv
from .findings import StaticFinding, quick_scan_findings, write_findings
from .fusion import FusionDecision, fuse_findings_dispatch
@@ -42,16 +41,13 @@ from .hunter import run_hunter_pool, write_hunter_trajectories
from .hunter_harness import HxScanOrchestrator
from .improve import RoundOutcome, run_improvement
from .index import build_code_chunks
from .learning.families import FamilyTaxonomy
from .mapping import analyze_entry_points, attach_reachability_hints, ingest_sarif, write_entry_points, write_static_hints
from .pair_gate import print_pair_metrics
from .pairs import (
DEFAULT_CATALOG,
SLICE_NAMES,
PairCase,
evaluate_catalog,
load_pair_catalog,
make_hunter_scan,
result_payload,
select_profile,
select_slice,
@@ -61,24 +57,19 @@ from .pairs import (
from .policy import assert_rules_resolve, load_policy
from .preprocess import FileTarget, preprocess_repository, write_preprocess_artifact
from .provenance import fingerprint
from .provider.openrouter import OpenRouterEmbeddingClient, OpenRouterError
from .rank import rank_obligations, rank_targets, write_rankings
from .regress import TRIGGERABLE, CandidateVerdict, run_regression, write_verdicts
from .regress.candidate import hotspot_from_finding
from .reports import scan_exit_code, write_manifest, write_markdown_report, write_sarif_report
from .ruleset import DEFAULT_RULESET_DIR, load_ruleset
from .run import ScanRun, create_scan_run
from .sandbox import resolve_sandbox_probe, resolve_sandbox_runner
from .scoring import build_score_artifact
from .semantic import (
MechanismStore,
OverlayRecord,
adjudicate,
append_mechanism,
filter_promoted_hotspots,
finding_from_coverage,
load_facts,
order_promotions,
write_overlay,
)
from .semantic.facts import FactLoadError, SemanticFacts
@@ -101,7 +92,7 @@ class ScanOutcome:
if TYPE_CHECKING:
from .learning.proposer import Proposer
pass
def main(argv: list[str] | None = None) -> int:
@@ -154,22 +145,6 @@ def main(argv: list[str] | None = None) -> int:
"--profile-tolerance", type=float, default=0.0, help="allowed per-profile drop in pair_correct/Youden before rejecting"
)
improve.add_argument("--min-holdout-pairs", type=int, default=5, help="profiles with fewer holdout pairs are reported, not gated")
improve.add_argument(
"--mechanism-candidates",
type=Path,
default=None,
help=(
"exporter candidates the mechanisms lever may admit (default: <target>/.openultrasast/calibration/"
"mechanism-candidates.jsonl, then ./.openultrasast/calibration/mechanism-candidates.jsonl)"
),
)
improve.add_argument(
"--mechanism-store",
type=Path,
default=None,
help="scan-time mechanisms.jsonl the lever writes (default: <target>/.openultrasast/calibration/mechanisms.jsonl)",
)
improve.add_argument("--no-mechanisms", action="store_true", help="disable the mechanisms lever for this run")
pairs = subparsers.add_parser(
"pairs",
@@ -181,72 +156,18 @@ def main(argv: list[str] | None = None) -> int:
"--profile", choices=("all", "human", "agent", "mixed", "synthetic"), default="all", help="filter pairs by provenance profile"
)
pairs.add_argument("--split", choices=("all", "train", "holdout"), default="all", help="filter pairs by declared split")
pairs.add_argument("--hunter", action="store_true", help="also score the LLM tool hunter on overlay slices (needs a hunter model)")
pairs.add_argument("--hunter-model", default=None, help="model id for --hunter (default: [models].hunter from openultrasast.toml)")
pairs.add_argument("--pointers", action="store_true", help="allow network for non-vendored pointer pairs this run (nightly; CI never)")
pairs.add_argument(
"--loo", action="store_true", help="leave-one-out: score each pair against a store seeded from the other trusted pairs"
)
pairs.add_argument("--loo-out", type=Path, default=None, help="where to write loo.json (default: reports/loo.json)")
pairs.add_argument("--json", action="store_true", help="print the pair scoreboard as JSON")
pairs.add_argument(
"--k-runs", type=int, default=None, help="runs per pair on the hunter path; never fewer than three, and three by default"
)
subparsers.add_parser("mcp", help="run the narrow MCP server over stdio for OpenCode integration")
mechanisms = subparsers.add_parser("mechanisms", help="maintainer: mechanism memory seeded from trusted pairs")
mechanisms_sub = mechanisms.add_subparsers(dest="mechanisms_command", required=True)
export = mechanisms_sub.add_parser("export", help="derive mechanism records from seeded/reviewed pairs (offline)")
export.add_argument("--slice", choices=SLICE_NAMES, default="all")
export.add_argument("--catalog", type=Path, default=DEFAULT_CATALOG)
export.add_argument(
"--store",
type=Path,
default=Path(CALIBRATION_DIR) / "mechanism-candidates.jsonl",
help="append-only candidates file the improve lever admits from (the scan reads mechanisms.jsonl next to it; never written here)",
)
export.add_argument("--json", action="store_true")
learning = subparsers.add_parser("learning", help="maintainer: the classified detector set and its rounds")
learning_sub = learning.add_subparsers(dest="learning_command", required=True)
for name, help_text in (
("classify", "classify every labeled pair and report the classifier against itself"),
("score", "score a catalog by family, with the denominator every number was computed over"),
("baseline", "round zero: clone one detector into every family and measure each against itself"),
("candidates", "offline: what a candidate-driven detector could find, per family and per slice"),
("round", "one evolve round: propose one change for one family and let the evidence decide"),
("publish", "regenerate every published number from the artifacts that produced it"),
):
command = learning_sub.add_parser(name, help=help_text)
command.add_argument("--out", type=Path, default=Path(".openultrasast/learning"), help="where artifacts are read and written")
if name != "publish":
command.add_argument("--catalog", type=Path, default=DEFAULT_CATALOG)
command.add_argument("--slice", choices=SLICE_NAMES, default="all")
command.add_argument("--json", action="store_true")
if name == "baseline":
command.add_argument(
"--resume",
action="store_true",
help="keep the pairs a previous run of this baseline already measured and paid for",
)
if name == "candidates":
command.add_argument("--generator", choices=("ir", "ruleset"), default="ir", help="which pass enumerates the candidate sites")
if name in {"baseline", "round", "score"}:
command.add_argument("--k-runs", type=int, default=None, help="runs per pair; never fewer than three")
command.add_argument("--model", default=None, help="detector model; artifacts are keyed by it")
command.add_argument("--config", type=Path, default=Path("openultrasast.toml"))
if name == "round":
command.add_argument("--family", required=True)
command.add_argument("--cost-cap-usd", type=float, default=None)
command.add_argument(
"--proposals",
type=Path,
default=None,
help="a JSON list of proposals to try in order; without it the round asks the meta-agent",
)
if name == "publish":
command.add_argument("--measurements", type=Path, default=Path("benchmarks/measurements"))
command.add_argument("--roadmap", type=Path, default=Path(".kiro/steering/roadmap.md"))
model_cmd = subparsers.add_parser("model", help="maintainer: the model layer that arbitrates detector claims")
model_sub = model_cmd.add_subparsers(dest="model_command", required=True)
candidates = model_sub.add_parser("candidates", help="offline: what the candidate enumerator can reach, per family and per slice")
candidates.add_argument("--out", type=Path, default=Path(".openultrasast/model"), help="where artifacts are written")
candidates.add_argument("--catalog", type=Path, default=DEFAULT_CATALOG)
candidates.add_argument("--slice", choices=SLICE_NAMES, default="all")
candidates.add_argument("--generator", choices=("ir", "ruleset"), default="ir", help="which pass enumerates the candidate sites")
candidates.add_argument("--json", action="store_true")
args = parser.parse_args(argv)
if args.command == "scan":
@@ -268,9 +189,6 @@ def main(argv: list[str] | None = None) -> int:
pair_catalog=None if args.no_pair_gate else args.pair_catalog,
profile_tolerance=args.profile_tolerance,
min_holdout_pairs=args.min_holdout_pairs,
mechanism_candidates=args.mechanism_candidates,
mechanism_store=args.mechanism_store,
mechanisms_enabled=not args.no_mechanisms,
)
if args.command == "pairs":
return _pairs(
@@ -279,17 +197,10 @@ def main(argv: list[str] | None = None) -> int:
json_out=args.json,
profile=args.profile,
split=args.split,
hunter=args.hunter,
hunter_model=args.hunter_model,
pointers=args.pointers,
loo=args.loo,
loo_out=args.loo_out,
k_runs=args.k_runs,
)
if args.command == "mechanisms":
return _mechanisms_export(args.catalog, args.slice, args.store, json_out=args.json)
if args.command == "learning":
return _learning(args)
if args.command == "model":
return _model_candidates(args)
if args.command == "mcp":
from .mcp import serve # lazy: keeps the import cycle (mcp -> cli) one-directional
@@ -478,12 +389,7 @@ def _run_scan(path: Path, config_path: Path, mode: str, fail_on: str) -> ScanOut
built_map = None
overlay_records: list[OverlayRecord] = []
wrote_overlay = False
variants_payload: dict[str, object] | None = None
variant_findings: list[StaticFinding] = []
mechanisms_cited: dict[str, dict[str, object]] = {}
obligations_payload: dict[str, object] | None = None
learning_payload: dict[str, object] | None = None
learning_cited: dict[str, dict[str, object]] = {}
obligations_cited: dict[str, dict[str, object]] = {}
if Stage.MAP in plan.requested:
built_map = runtime.run_stage(
@@ -517,19 +423,6 @@ def _run_scan(path: Path, config_path: Path, mode: str, fail_on: str) -> ScanOut
if extra:
findings = findings + extra
write_findings(findings, findings_path)
if config.variants.enabled:
variant_findings, overlay_records, variants_payload, mechanisms_cited = _search_variants(
root=run.target,
targets=targets,
overlay_records=overlay_records,
facts=facts_or_error,
max_mechanisms=config.variants.max_mechanisms,
runtime=runtime,
)
if variant_findings:
findings = findings + variant_findings
write_findings(findings, findings_path)
write_overlay(overlay_records, overlay_path)
if config.obligations.enabled:
obligation_findings, obligations_payload, obligations_cited = _check_obligations(
root=run.target,
@@ -543,18 +436,6 @@ def _run_scan(path: Path, config_path: Path, mode: str, fail_on: str) -> ScanOut
if obligation_findings:
findings = rank_obligations(findings + obligation_findings)
write_findings(findings, findings_path)
if config.learning.enabled:
family_findings, learning_payload, learning_cited = _run_family_detectors(
root=run.target,
hotspots=built_map.hotspots if built_map is not None else (),
cases=None,
settings=config.learning,
config=config,
runtime=runtime,
)
if family_findings:
findings = findings + family_findings
write_findings(findings, findings_path)
plan = record_completed(plan, Stage.MAP)
if not hunter_model:
runtime.state["degradations"].append(skip_as_degradation(Stage.MAP, "hunter_model_unavailable"))
@@ -588,33 +469,11 @@ def _run_scan(path: Path, config_path: Path, mode: str, fail_on: str) -> ScanOut
sandbox = resolve_sandbox_runner()
hotspots = built_map.hotspots if built_map is not None else ()
hotspots = filter_promoted_hotspots(hotspots, overlay_records, findings)
store = MechanismStore(run.target / CALIBRATION_DIR / "mechanisms.jsonl")
embed_client = None
embed_model = config.embeddings.model
if embed_model:
try:
embed_client = OpenRouterEmbeddingClient.from_env()
except OpenRouterError:
embed_client = None
hotspots, budget_degradation = order_promotions(
hotspots,
overlay_records,
findings,
store=store,
client=embed_client,
model=embed_model,
)
if budget_degradation:
runtime.state["degradations"].append({"stage": "prove_budget", "reason": budget_degradation})
promoted = [
finding
for finding in findings
if finding.finding_id in {record.proposal_id for record in overlay_records if record.disposition == "promote"}
]
# Variant findings are candidates after the promotions (Req 3.4): same safety check, same cap, lowest rank.
variant_ids = {finding.finding_id for finding in variant_findings}
promoted = promoted + [finding for finding in findings if finding.finding_id in variant_ids]
hotspots = hotspots + tuple(_hotspot_from_variant(finding) for finding in findings if finding.finding_id in variant_ids)
cwe_by_rule_id = {rule.rule_id: rule.cwe for rule in ruleset}
rule_cwe_by_finding = {finding.finding_id: cwe_by_rule_id.get(finding.finding_id.split(":", 1)[0], "") for finding in promoted}
languages_by_path = {target.path: target.language for target in targets}
@@ -640,7 +499,6 @@ def _run_scan(path: Path, config_path: Path, mode: str, fail_on: str) -> ScanOut
if proven_ids and any(finding.finding_id.startswith("obligation:") for finding in findings):
findings = rank_obligations(findings, proven_ids=proven_ids)
write_findings(findings, findings_path)
_append_proven_mechanisms(store, records, overlay_records, findings, run.target)
verdict_records = records
wrote_verdicts = True
plan = record_completed(plan, Stage.REGRESS)
@@ -655,10 +513,8 @@ def _run_scan(path: Path, config_path: Path, mode: str, fail_on: str) -> ScanOut
complexity_map=built_map,
verdicts=verdict_records if wrote_verdicts else None,
overlay=overlay_records if wrote_overlay else None,
mechanisms=mechanisms_cited or None,
obligations=obligations_cited or None,
obligations_summary=obligations_payload or None,
learning=learning_cited or None,
),
)
runtime.run_stage(
@@ -668,9 +524,7 @@ def _run_scan(path: Path, config_path: Path, mode: str, fail_on: str) -> ScanOut
verifications,
sarif_path,
overlay=overlay_records if wrote_overlay else None,
mechanisms=mechanisms_cited or None,
obligations=obligations_cited or None,
learning=learning_cited or None,
),
)
runtime.run_stage(
@@ -699,9 +553,7 @@ def _run_scan(path: Path, config_path: Path, mode: str, fail_on: str) -> ScanOut
complexity=complexity_payload,
worth_fixing=worth_fixing_payload,
provenance=provenance.to_dict(),
variants=variants_payload,
obligations=obligations_payload,
learning=learning_payload,
),
)
runtime.finish(status="succeeded")
@@ -716,35 +568,6 @@ def _run_scan(path: Path, config_path: Path, mode: str, fail_on: str) -> ScanOut
)
def _append_proven_mechanisms(
store: MechanismStore,
verdicts: tuple[CandidateVerdict, ...],
overlay_records: list[OverlayRecord],
findings: list[StaticFinding],
target: Path,
) -> None:
overlay_by_id = {record.proposal_id: record for record in overlay_records}
finding_by_id = {finding.finding_id: finding for finding in findings}
for verdict in verdicts:
if verdict.verdict != TRIGGERABLE:
continue
for finding_id in verdict.inventory_finding_ids:
record = overlay_by_id.get(finding_id)
if record is None or record.disposition != "promote":
continue
finding = finding_by_id.get(finding_id)
append_mechanism(
store,
summary=record.reason,
cwe=record.cwe,
language=record.language or verdict.language,
tags=finding.tags if finding is not None else (),
what_made_it_exploitable=record.reason,
source_finding_id=finding_id,
source_repo=str(target),
)
def _attach_tool_hunter(
*,
root: Path,
@@ -900,9 +723,6 @@ def _improve(
pair_catalog: Path | None = None,
profile_tolerance: float = 0.0,
min_holdout_pairs: int = 5,
mechanism_candidates: Path | None = None,
mechanism_store: Path | None = None,
mechanisms_enabled: bool = True,
) -> int:
if not manifest_path.exists() or not manifest_path.is_file():
raise SystemExit(f"benchmark manifest is not a file: {manifest_path}")
@@ -913,19 +733,6 @@ def _improve(
pair_cases = select_split(load_pair_catalog(pair_catalog), "holdout")
manifest = load_benchmark_manifest(manifest_path)
target = resolve_benchmark_source(manifest_path, manifest)
# corpus-seeded-mechanisms Req 5: the lever runs when a candidates file exists (exporter output) and the pair gate is on.
# Candidates: an explicit path, else the target's calibration dir, else the working directory's (where `mechanisms export`
# writes by default). The store always lives where a scan of this target reads it unless overridden.
calibration = target / CALIBRATION_DIR
candidates_path: Path | None = mechanism_candidates
if candidates_path is None:
for candidate in (calibration / "mechanism-candidates.jsonl", Path(CALIBRATION_DIR) / "mechanism-candidates.jsonl"):
if candidate.is_file():
candidates_path = candidate
break
if not mechanisms_enabled or not pair_cases or candidates_path is None or not candidates_path.is_file():
candidates_path = None
store_path = mechanism_store if mechanism_store is not None else calibration / "mechanisms.jsonl"
policy = load_policy()
# The loop writes its accepted ledger where `scan`/`benchmark` read it, so a
@@ -935,11 +742,6 @@ def _improve(
if dry_run:
ledger_path = Path(scratch) / "rule_policy.json"
journal_path = Path(scratch) / "improve_journal.json"
if candidates_path is not None: # a dry run must not mutate the scan-time store either
scratch_store = Path(scratch) / "mechanisms.jsonl"
if store_path.is_file():
scratch_store.write_bytes(store_path.read_bytes())
store_path = scratch_store
else:
ledger_path = ledger or default_ledger
journal_path = journal or ledger_path.with_name("improve_journal.json")
@@ -957,8 +759,6 @@ def _improve(
pair_cases=pair_cases,
profile_tolerance=profile_tolerance,
min_holdout_pairs=min_holdout_pairs,
mechanism_candidates=candidates_path,
mechanism_store=store_path if candidates_path is not None else None,
)
_print_improve_outcomes(outcomes, manifest_path, target, ledger_path, dry_run=dry_run)
return 0
@@ -971,30 +771,13 @@ def _pairs(
json_out: bool,
profile: str = "all",
split: str = "all",
hunter: bool = False,
hunter_model: str | None = None,
pointers: bool = False,
loo: bool = False,
loo_out: Path | None = None,
k_runs: int | None = None,
) -> int:
if not catalog.exists() or not catalog.is_file():
raise SystemExit(f"pair catalog is not a file: {catalog}")
cases = select_split(select_profile(select_slice(load_pair_catalog(catalog), slice_name), profile), split)
if loo:
return _pairs_loo(cases, loo_out or Path("reports") / "loo.json", json_out=json_out)
scan = None
if hunter:
config_path = Path("openultrasast.toml")
config = load_config(config_path if config_path.exists() else None)
model = hunter_model or config.models.hunter or ""
client = tool_hunter.resolve_hunter_client()
if model and client is not None:
from .learning.families import load_families
scan = make_hunter_scan(client, model, taxonomy=load_families())
try:
result = evaluate_catalog(cases, hunter=scan, pointers=True if pointers else None, k_runs=k_runs)
result = evaluate_catalog(cases, pointers=True if pointers else None)
except ValueError as exc:
raise SystemExit(f"--k-runs: {exc}") from exc
if json_out:
@@ -1005,50 +788,6 @@ def _pairs(
return 0
def _search_variants(
*,
root: Path,
targets: list[FileTarget],
overlay_records: list[OverlayRecord],
facts: SemanticFacts | FactLoadError,
max_mechanisms: int,
runtime: HarnessRuntime,
) -> tuple[list[StaticFinding], list[OverlayRecord], dict[str, object], dict[str, dict[str, object]]]:
"""Third MAP proposer (corpus-seeded-mechanisms Req 3): shapes from the mechanism store, merged with overlay flows.
Also returns the mechanisms the findings cite (summary, guard, pairs, cwe) for the reports (Req 6.1).
"""
from .semantic.mechanisms import corpus_mechanisms
from .semantic.variant_search import hits_to_findings, search_tree
store = MechanismStore(root / CALIBRATION_DIR / "mechanisms.jsonl")
empty: dict[str, object] = {"mechanisms_searched": 0, "files_searched": 0, "findings": 0, "merged_into_overlay": 0}
if isinstance(facts, FactLoadError):
return [], overlay_records, empty, {}
result = runtime.run_stage("variants", lambda: search_tree(root, targets, store, facts, max_mechanisms=max_mechanisms))
for degradation in result.degradations:
runtime.state["degradations"].append(degradation)
records_by_id = {record.id: record for record in corpus_mechanisms(store.load())}
summaries = {record.id: (record.summary, record.pairs, record.cwe) for record in records_by_id.values()}
findings, records = hits_to_findings(result.hits, overlay_records, summaries)
merged = sum(1 for before, after in zip(overlay_records, records, strict=True) if before.mechanism_id is None and after.mechanism_id)
payload: dict[str, object] = {
"mechanisms_searched": result.mechanisms_searched,
"files_searched": result.files_searched,
"findings": len(findings),
"merged_into_overlay": merged,
}
cited_ids = {tag.split(":", 1)[1] for finding in findings for tag in finding.tags if tag.startswith("mechanism:")} | {
str(record.mechanism_id) for record in records if record.mechanism_id
}
cited: dict[str, dict[str, object]] = {
mechanism_id: {"summary": record.summary, "guard": record.guard, "pairs": list(record.pairs), "cwe": record.cwe}
for mechanism_id, record in records_by_id.items()
if mechanism_id in cited_ids
}
return findings, records, payload, cited
def _check_obligations(
*,
root: Path,
@@ -1065,7 +804,7 @@ def _check_obligations(
enter the sandbox candidate set.
"""
from .semantic.ir import parse_file
from .semantic.obligations import load_obligation_facts, obligation_mechanisms
from .semantic.obligations import load_obligation_facts
from .semantic.obligations.check import check_obligations, findings_to_static
from .semantic.obligations.dominance import OrderDominance
from .semantic.obligations.policy import PolicyError, load_declared_policy
@@ -1089,7 +828,6 @@ def _check_obligations(
policy = load_declared_policy(policy_path)
except PolicyError as exc:
runtime.state["degradations"].append({"stage": "obligations", "reason": "policy_invalid", "detail": str(exc)[:200]})
store = MechanismStore(root / CALIBRATION_DIR / "mechanisms.jsonl")
result = runtime.run_stage(
"obligations",
lambda: check_obligations(
@@ -1100,7 +838,7 @@ def _check_obligations(
policy=policy,
paths=(),
dominance=OrderDominance(texts=texts),
store_shapes=obligation_mechanisms(store.load()),
store_shapes=(), # known-fix hints came from the deleted mechanism store
min_siblings=settings.min_siblings,
),
)
@@ -1138,292 +876,6 @@ def _check_obligations(
return statics, payload, cited
def _run_family_detectors(
*,
root: Path,
hotspots: Sequence[object],
cases: object,
settings: object,
config: object,
runtime: HarnessRuntime,
) -> tuple[list[StaticFinding], dict[str, object], dict[str, dict[str, object]]]:
"""Fifth MAP proposer (learning-harness Req 6.3, 7.1): one detector per admitted family, plus the generalist.
Every claim passes through its family's verifier and carries what produced it. Nothing rises above
suspicion here: the verifier writes a tag, and the evidence ladder is another boundary's to move.
"""
del cases
from .learning.classify import classify_region
from .learning.detectors import DetectorConfigError, Region, load_family_configs, run_region_detectors
from .learning.endpoint import resolve_chat_endpoint, resolve_models
from .learning.families import FamiliesError, load_families
from .learning.verifiers import apply_verification, verifier_for
empty: dict[str, dict[str, object]] = {}
configs_dir = root / str(getattr(settings, "configs_dir", ".openultrasast/learning/configs"))
try:
taxonomy = load_families(Path(p) if (p := getattr(settings, "families_path", None)) else None)
configs = load_family_configs(configs_dir, taxonomy)
except (FamiliesError, DetectorConfigError) as exc:
runtime.state["degradations"].append({"stage": "learning", "reason": "learning_configs_unavailable", "detail": str(exc)[:200]})
return [], {}, empty
if not configs:
runtime.state["degradations"].append(
{"stage": "learning", "reason": "learning_configs_unavailable", "detail": f"no family configurations under {configs_dir}"}
)
return [], {}, empty
resolved = resolve_chat_endpoint(config) # type: ignore[arg-type]
if resolved is None:
runtime.state["degradations"].append({"stage": "learning", "reason": "learning_endpoint_unavailable"})
return [], {}, empty
client, endpoint = resolved
model = resolve_models(config)[0] # type: ignore[arg-type]
findings: list[StaticFinding] = []
per_family: dict[str, int] = {}
per_tier: dict[str, int] = {}
abstained = 0
seen: set[str] = set()
for hotspot in hotspots:
path = str(getattr(hotspot, "path", ""))
function = getattr(hotspot, "function_name", None)
key = f"{path}::{function}"
if not path or key in seen:
continue
seen.add(key)
answer = classify_region(root, path, function=function, taxonomy=taxonomy, client=None)
per_tier[answer.tier] = per_tier.get(answer.tier, 0) + 1
abstained += 0 if answer.families else 1
region = Region(path=path, function=str(function) if function else None, families=answer.families)
def _run(region: Region = region) -> list[StaticFinding]:
return run_region_detectors(root, region, configs, client=client, model=model)
produced = runtime.run_stage("learning", _run)
for finding in produced:
family_id = next((tag.split(":", 1)[1] for tag in finding.tags if tag.startswith("family:")), "unknown")
verification = verifier_for(taxonomy.by_id(family_id)).verify(finding, root=root, language="python")
findings.append(apply_verification(finding, verification))
per_family[family_id] = per_family.get(family_id, 0) + 1
payload: dict[str, object] = {
"taxonomy_version": taxonomy.version,
"families": per_family,
# What the classifier did on *this* scan. Scoring it needs labels a maintainer stands behind, which a scan
# of someone else's repository does not have, so the block reports the routing rather than an accuracy.
"classifier": {
"regions": len(seen),
"per_tier": dict(sorted(per_tier.items())),
"abstained": abstained,
# Req 2.6/3.7: say out loud that no family declares a parent at this taxonomy version, so the
# partial credit for a parent-child relation never applied to any number here.
"hierarchical_credit": taxonomy.hierarchical,
},
"round": None,
"endpoint": endpoint.provider,
"regions": len(seen),
}
cited: dict[str, dict[str, object]] = {
finding.finding_id: {
"family": next((tag.split(":", 1)[1] for tag in finding.tags if tag.startswith("family:")), ""),
"detector": next((tag.split(":", 1)[1] for tag in finding.tags if tag.startswith("detector:")), ""),
"verifier": next((tag.split(":", 1)[1] for tag in finding.tags if tag.startswith("verifier:")), ""),
}
for finding in findings
}
return findings, payload, cited
def _learning(args: argparse.Namespace) -> int:
"""The maintainer's entry points into the classified detector set. Everything degrades with a reason."""
from .learning.classify import measure_classifier, write_review_queue
from .learning.families import load_families
from .learning.publish import publish
out: Path = args.out
taxonomy = load_families()
if args.learning_command == "publish":
report = publish(learning_dir=out, measurements_dir=args.measurements, roadmap=args.roadmap)
print(f"learning publish: {len(report.families)} families over {len(report.models)} model(s) -> {args.roadmap}")
for artifact in report.artifacts:
print(f" artifact {artifact}")
return 0
cases = select_vendored(select_slice(load_pair_catalog(args.catalog), args.slice))
if args.learning_command == "candidates":
# No model is called anywhere in this command. The ceiling is a property of the corpus and the enumerator,
# so it can end this feature before a cent is spent on it (constrained-detector task 1.3).
from .learning.candidates import ceiling
coverage = ceiling(cases, taxonomy=taxonomy, generator=args.generator)
out.mkdir(parents=True, exist_ok=True)
payload = coverage.to_dict()
(out / "candidate-ceiling.json").write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
if args.json:
print(json.dumps(payload, indent=2, sort_keys=True))
else:
for family, block in sorted(coverage.per_family.items()):
print(
f"learning candidates {family}: {block['with_candidate']}/{block['pairs']} "
f"= {_as_float(block['ceiling']):.1%} ceiling, sites {block['sites']}"
)
for slice_name, block in sorted(coverage.per_slice.items()):
print(f" slice {slice_name}: {_as_float(block['ceiling']):.1%} over {block['pairs']} pairs")
return 0
if args.learning_command == "classify":
measured = measure_classifier(cases, taxonomy, client=None)
out.mkdir(parents=True, exist_ok=True)
payload = measured.to_dict()
(out / "classifier.json").write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
written = write_review_queue(out / "review-queue.jsonl", measured.queue)
if args.json:
print(json.dumps(payload, indent=2, sort_keys=True))
else:
print(f"learning classify: classified {measured.classified}, unknown {measured.unknown}, queued {written}")
return 0
return _learning_run(args, cases, taxonomy)
def _learning_run(args: argparse.Namespace, cases: Sequence[PairCase], taxonomy: FamilyTaxonomy) -> int:
"""score, baseline and round: the three that need a detector, and degrade with a reason without one."""
from .learning.detectors import load_family_configs, write_default_configs
from .learning.endpoint import resolve_chat_endpoint, resolve_models
from .learning.journal import Archive, LearningJournal
from .learning.rounds import DEFAULT_SLICES, MIN_K_RUNS, load_noise_floors, run_baseline, run_learning_round
from .learning.scoring import aggregate
from .tool_hunter import _SYSTEM_PROMPT
out: Path = args.out
config = load_config(args.config if args.config.exists() else None)
k_runs = args.k_runs or max(config.learning.k_runs, MIN_K_RUNS)
model = args.model or resolve_models(config)[0]
configs_dir = out / "configs"
if not configs_dir.exists():
write_default_configs(configs_dir, taxonomy, prompt=_SYSTEM_PROMPT, version="0") # type: ignore[arg-type]
resolved = resolve_chat_endpoint(config)
if resolved is None:
print("learning: learning_endpoint_unavailable; scoring what can be scored without a detector")
client = resolved[0] if resolved else None
# The client meters itself; a run that never reads the meter publishes a cost of zero and can never trip its
# own cap (Req 8.3, 9.8).
meter = getattr(client, "cost_usd", None)
spent: Callable[[], float] | None = meter if callable(meter) else None
configs = load_family_configs(configs_dir, taxonomy) # type: ignore[arg-type]
def factory(family_config: object) -> Callable[[Path], list[StaticFinding]]:
from .learning.detectors import Region, run_family_detector
def scan(root: Path) -> list[StaticFinding]:
if client is None:
return []
region = Region(path=_first_source(root), function=None)
return run_family_detector(root, region, family_config, client=client, model=model) # type: ignore[arg-type]
return scan
if args.learning_command == "score":
from .learning.rounds import score_case
scores = [
score_case(case, factory(configs[_label(case, taxonomy)]), taxonomy=taxonomy, runs=k_runs, family=_label(case, taxonomy))
for case in cases
if _label(case, taxonomy) in configs
]
cutoff = config.learning.cutoff_date or ""
families = {name: block.to_dict() for name, block in aggregate(scores, taxonomy=taxonomy, cutoff=cutoff).items()}
if args.json:
print(json.dumps({"families": families}, indent=2, sort_keys=True))
else:
for name, block in sorted(families.items()):
raw = block.get("post_cutoff")
recent: dict[str, object] = raw if isinstance(raw, dict) else {}
tail = (
f" post-cutoff({cutoff})={recent.get('scorable', 0)} undated={block['undated']}"
if cutoff
else " post-cutoff=none configured"
)
print(
f"learning score {name}: scorable={block['scorable']} recall={block['recall']:.3f} youden={block['youden']:.3f}{tail}"
)
return 0
if args.learning_command == "baseline":
report = run_baseline(
cases,
taxonomy=taxonomy,
configs_dir=configs_dir,
scan_factory=factory,
model=model,
out_dir=out,
k_runs=k_runs,
cutoff=config.learning.cutoff_date or "",
# `--slice` already chose the rows; without this the baseline's own default filters them out again and
# a run over the memory-safety slice reports nothing at all (Req 8.4).
slices=(args.slice,) if args.slice != "all" else DEFAULT_SLICES,
spent_usd=spent,
thinking=config.learning.thinking,
resume=args.resume,
)
print(f"learning baseline cost: ${report.cost_usd:.4f}")
print(f"learning baseline {model}: {len(report.floors)} families, k={report.k_runs} -> {out / 'baseline'}")
return 0
journal = LearningJournal(out / "journal.jsonl")
record = run_learning_round(
cases,
family=args.family,
taxonomy=taxonomy,
configs_dir=configs_dir,
proposer=_proposer(args, configs_dir, model, config.harnessx.provider),
scan_factory=factory,
model=model,
journal=journal,
archive=Archive(out / "archive.jsonl"),
floors=load_noise_floors(out, model, (args.slice,) if args.slice != "all" else DEFAULT_SLICES, config.learning.thinking),
out_dir=out,
cost_cap_usd=args.cost_cap_usd or config.learning.round_cost_cap_usd,
minibatch=config.learning.minibatch,
k_runs=k_runs,
spent_usd=spent,
# Every other family the corpus actually carries, so a change that reaches beyond its own directory is
# measured rather than assumed impossible (Req 9.5).
sweep_families=tuple(sorted({_label(case, taxonomy) for case in cases} & set(configs) - {args.family})),
)
print(f"learning round {record.round} {record.family}: {record.outcome} ({record.reason}) cost ${record.cost_usd:.4f}")
return 0
def _as_float(value: object) -> float:
return float(value) if isinstance(value, int | float) else 0.0
def _label(case: PairCase, taxonomy: object) -> str:
"""The family that routes this pair: the maintainer's label when there is one, else the classifier's answer.
Reading only the declared label would send every unlabeled row to the generalist, whose findings earn no credit
against any real family, so a scoreboard built that way reports structural zeros (Req 2.1, 8.2)."""
from .learning.rounds import _family_of
return _family_of(case, taxonomy) # type: ignore[arg-type]
def _proposer(args: argparse.Namespace, configs_dir: Path, model: str, provider: str = "anthropic") -> Proposer:
"""`--proposals FILE` runs a recorded sequence offline; without it the round asks the meta-agent."""
from .learning.proposer import HarnessXProposer, Proposal, ScriptedProposer
path = getattr(args, "proposals", None)
if path is None:
return HarnessXProposer(configs_dir=configs_dir, model=model, provider=provider)
rows = json.loads(Path(path).read_text(encoding="utf-8"))
return ScriptedProposer(
[
Proposal(
hypothesis=str(row["hypothesis"]),
lever=row["lever"],
change=dict(row["change"]),
predicted_affected=tuple(row.get("predicted_affected", ())),
predicted_at_risk=tuple(row.get("predicted_at_risk", ())),
)
for row in rows
]
)
def _first_source(root: Path) -> str:
from .preprocess import preprocess_repository
@@ -1431,60 +883,12 @@ def _first_source(root: Path) -> str:
return targets[0].path if targets else "."
def _hotspot_from_variant(finding: StaticFinding) -> Hotspot:
return hotspot_from_finding(finding, rationale="variant of a known mechanism (suspicion; sandbox may raise)")
def _pairs_loo(cases: Sequence[PairCase], out: Path, *, json_out: bool) -> int:
"""Req 4: the corpus's own detection rate; written as an artifact, never a gate."""
from .semantic.loo import evaluate_loo
result = evaluate_loo(select_vendored(cases))
payload = result.to_dict()
out.parent.mkdir(parents=True, exist_ok=True)
out.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
if json_out:
print(json.dumps(payload, indent=2, sort_keys=True))
return 0
for slice_name, metrics in result.per_slice.items():
print(
f"loo {slice_name}: pairs={metrics['pairs']} detected={metrics['detected']} silent={metrics['silent']} "
f"youden={metrics['youden']:+.3f} teaching={metrics['teaching']}"
)
print(f"artifact={out}")
return 0
def _mechanisms_export(catalog: Path, slice_name: str, store_path: Path, *, json_out: bool) -> int:
from .semantic.mechanisms import MechanismStore
from .semantic.seed import export_mechanisms
if not catalog.is_file():
raise SystemExit(f"pair catalog is not a file: {catalog}")
cases = select_vendored(select_slice(load_pair_catalog(catalog), slice_name))
report = export_mechanisms(cases, MechanismStore(store_path))
payload = {"slice": slice_name, "store": str(store_path), "pairs": len(cases), **report.to_dict()}
if json_out:
print(json.dumps(payload, indent=2, sort_keys=True))
return 0
print(f"mechanisms export slice={slice_name} pairs={len(cases)} seeded={report.seeded} records={report.records} -> {store_path}")
for pair, reason in report.skipped:
print(f" skip {pair}: {reason}")
return 0
def _print_improve_outcomes(outcomes: list[RoundOutcome], manifest_path: Path, target: Path, ledger_path: Path, *, dry_run: bool) -> None:
print(f"manifest={manifest_path}")
print(f"target={target}")
accepted = [o for o in outcomes if o.accepted]
for outcome in outcomes:
edits = (
", ".join(
[f"{e.rule_id}:{e.from_status}->{e.to_status}" for e in outcome.edits]
+ [f"{m.action} {m.mechanism_id}" for m in outcome.mechanism_edits]
)
or "-"
)
edits = ", ".join(f"{e.rule_id}:{e.from_status}->{e.to_status}" for e in outcome.edits) or "-"
print(
f"round {outcome.round}: {outcome.reason} | "
f"recall {outcome.recall_before:.2%}->{outcome.recall_after:.2%} "
@@ -1506,6 +910,36 @@ def _load_scan_findings(run_dir: Path) -> list[StaticFinding]:
return load_findings(run_dir / "findings.json")
def _model_candidates(args: argparse.Namespace) -> int:
"""What the candidate enumerator can reach, per family and per slice. No model is called anywhere here.
The ceiling is a property of the corpus and the enumerator, so it can end a design before a cent is spent
on it — which is what it did for the pattern-ruleset generator (12.1% against the IR's 96.6%).
"""
from .model.candidates import ceiling
from .model.taxonomy import load_families
cases = select_vendored(select_slice(load_pair_catalog(args.catalog), args.slice))
coverage = ceiling(cases, taxonomy=load_families(), generator=args.generator)
out: Path = args.out
out.mkdir(parents=True, exist_ok=True)
payload = coverage.to_dict()
(out / "candidate-ceiling.json").write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
if args.json:
print(json.dumps(payload, indent=2, sort_keys=True))
else:
for family, block in sorted(coverage.per_family.items()):
ceiling_value = float(block["ceiling"]) if isinstance(block["ceiling"], int | float) else 0.0
print(
f"model candidates {family}: {block['with_candidate']}/{block['pairs']} "
f"= {ceiling_value:.1%} ceiling, sites {block['sites']}"
)
for slice_name, block in sorted(coverage.per_slice.items()):
ceiling_value = float(block["ceiling"]) if isinstance(block["ceiling"], int | float) else 0.0
print(f" slice {slice_name}: {ceiling_value:.1%} over {block['pairs']} pairs")
return 0
def _index(path: Path, config_path: Path, chunk_lines: int) -> int:
if not path.exists() or not path.is_dir():
raise SystemExit(f"index path is not a directory: {path}")
src/openultrasast/config.py
+6 -38
View File
@@ -36,6 +36,11 @@ class ModelConfig:
judge: str | None = None # second, independent judgement before anything is published as proven
chat_base_url: str | None = None # chat endpoint; independent of the embedding endpoint
chat_api_key_env: str | None = None # environment variable holding that endpoint's key
# Whether the chat model reasons before answering. Off by default because that is what the committed
# baseline was measured under: the provider silently ignores `temperature` while thinking, so thinking was
# disabled to make `temperature: 0` mean something. It did not — the measured run-to-run disagreement came
# from the agentic path, not the decoder — which is why the model layer arbitrates instead of averaging.
thinking: bool = False
@dataclass(frozen=True)
@@ -138,26 +143,6 @@ class VariantsConfig:
max_mechanisms: int = 500
@dataclass(frozen=True)
class LearningConfig:
# learning-harness: the classified detector set and its rounds.
enabled: bool = False # off until round zero has written the family configurations this reads
families_path: str | None = None # taxonomy file; the shipped ruleset when unset
configs_dir: str = ".openultrasast/learning/configs"
k_runs: int = 5 # runs per pair; a single run misses most real changes
round_cost_cap_usd: float = 10.0
minibatch: int = 8 # train pairs in a round's first stage
detector_model: str | None = None
judge_model: str | None = None
cutoff_date: str | None = None # rows fixed after this date are reported as a separate slice
# Whether the detector model reasons before answering. Off by default because that is what round zero was
# measured under, not because it is known to be better: the provider silently ignores `temperature` while
# thinking, and thinking was disabled to make `temperature: 0` mean something. It did not — the measured
# disagreement between runs comes from the agentic path, not the decoder — so which mode wins is a question
# for a second baseline rather than for a default (Req 8.3).
thinking: bool = False
@dataclass(frozen=True)
class ObligationsConfig:
# authorization-obligations: the obligation checker in MAP (standard/deep only).
@@ -183,7 +168,6 @@ class ResolvedConfig:
regress: RegressConfig = RegressConfig()
variants: VariantsConfig = VariantsConfig()
obligations: ObligationsConfig = ObligationsConfig()
learning: LearningConfig = LearningConfig()
runs_dir: str = ".openultrasast/runs"
@@ -210,7 +194,6 @@ def load_config(config_path: Path | None = None) -> ResolvedConfig:
regress=_load_regress(data.get("regress", {})),
variants=_load_variants(data.get("variants", {})),
obligations=_load_obligations(data.get("obligations", {})),
learning=_load_learning(data.get("learning", {})),
runs_dir=os.environ.get("OPENULTRASAST_RUNS_DIR", ".openultrasast/runs"),
)
@@ -238,6 +221,7 @@ def _load_models(value: object) -> ModelConfig:
judge=_string(data.get("judge")),
chat_base_url=_string(data.get("chat_base_url")),
chat_api_key_env=_string(data.get("chat_api_key_env")),
thinking=bool(data.get("thinking", False)),
)
@@ -385,22 +369,6 @@ def _load_variants(value: object) -> VariantsConfig:
)
def _load_learning(value: object) -> LearningConfig:
data = _section(value)
return LearningConfig(
enabled=bool(data.get("enabled", LearningConfig.enabled)),
families_path=_string(data.get("families_path")),
configs_dir=_string(data.get("configs_dir")) or LearningConfig.configs_dir,
k_runs=_int_value(data.get("k_runs"), LearningConfig.k_runs),
round_cost_cap_usd=_float_value(data.get("round_cost_cap_usd"), LearningConfig.round_cost_cap_usd),
minibatch=_int_value(data.get("minibatch"), LearningConfig.minibatch),
detector_model=_string(data.get("detector_model")),
judge_model=_string(data.get("judge_model")),
cutoff_date=_string(data.get("cutoff_date")),
thinking=bool(data.get("thinking", False)),
)
def _load_obligations(value: object) -> ObligationsConfig:
data = _section(value)
return ObligationsConfig(
src/openultrasast/improve/evolve.py
+5 -224
View File
@@ -26,9 +26,8 @@ from ..preprocess import preprocess_repository
from ..rank import rank_targets
from ..ruleset import DEFAULT_RULESET_DIR, PatternRule, load_ruleset, read_rule_ledger, write_rule_ledger
from ..scoring import build_score_artifact
from ..semantic.facts import SemanticFacts
from .journal import append_round, load_journal, next_round_index, reverted_edit_keys
from .validator import EvolveValidator, MechanismEdit, RuleStatusEdit, StrictValidationError, apply_mechanism_edits, edits_to_ledger
from .validator import EvolveValidator, RuleStatusEdit, StrictValidationError, edits_to_ledger
@dataclass(frozen=True)
@@ -46,9 +45,6 @@ class RoundOutcome:
matched_before: int = 0
matched_after: int = 0
profile_regressions: list[str] = field(default_factory=list)
mechanism_edits: list[MechanismEdit] = field(default_factory=list)
mechanisms_before: dict[str, dict[str, float]] = field(default_factory=dict)
mechanisms_after: dict[str, dict[str, float]] = field(default_factory=dict)
profiles_under_minimum: list[str] = field(default_factory=list)
per_profile_before: dict[str, dict[str, float]] = field(default_factory=dict)
per_profile_after: dict[str, dict[str, float]] = field(default_factory=dict)
@@ -89,121 +85,6 @@ def evaluate_profiles(
}
def evaluate_mechanism_profiles(pair_cases: Sequence[object], store_path: Path) -> dict[str, dict[str, float]]:
"""Per-profile pair metrics of variant search with the given scan-time store over the holdout pairs (vendored, gating tiers).
Split-blind for the same reason as `evaluate_profiles`: the store is the thing the teacher rule governs, and this
function only reads it (learning-harness Req 5.1)."""
from ..pairs import GATING_TIERS, PairCase, select_tier, select_vendored
from ..semantic.loo import score_pair_with_store
from ..semantic.mechanisms import MechanismStore
cases = list(select_tier(select_vendored([case for case in pair_cases if isinstance(case, PairCase)]), GATING_TIERS))
store = MechanismStore(store_path)
by_profile: dict[str, list[tuple[bool, bool]]] = {}
for case in cases:
outcome = score_pair_with_store(case, store, _facts())
by_profile.setdefault(case.provenance, []).append((outcome.detected, outcome.silent))
result: dict[str, dict[str, float]] = {}
for profile, items in sorted(by_profile.items()):
pairs = len(items)
correct = sum(1 for detected, silent in items if detected and silent)
recall = sum(1 for detected, _ in items if detected) / pairs
silence = sum(1 for _, silent in items if silent) / pairs
result[profile] = {
"pairs": float(pairs),
"pair_correct": float(correct),
"pair_pass_rate": correct / pairs,
"youden": recall + silence - 1.0,
}
return result
def propose_mechanism_edits(
pair_cases: Sequence[object],
candidates: object,
store_path: Path,
degradations: list[dict[str, object]] | None = None,
) -> list[MechanismEdit]:
"""Req 5.3: admit a candidate that recovers a currently missed holdout pair without leaking; retract an admitted leaker."""
from ..pairs import GATING_TIERS, PairCase, select_tier, select_vendored
from ..semantic.loo import score_pair_with_store
from ..semantic.mechanisms import MechanismStore, corpus_mechanisms
cases_all = [case for case in pair_cases if isinstance(case, PairCase)]
cases = list(select_tier(select_vendored(cases_all), GATING_TIERS))
if not cases:
return []
facts = _facts()
scan = MechanismStore(store_path)
admitted = {record.id: record for record in corpus_mechanisms(scan.load())}
current = {case.name: score_pair_with_store(case, scan, facts) for case in cases}
edits: list[MechanismEdit] = []
leakers = sorted({mechanism_id for outcome in current.values() for mechanism_id in outcome.leaked_by if mechanism_id in admitted})
for mechanism_id in leakers:
leaked = sorted(name for name, outcome in current.items() if mechanism_id in outcome.leaked_by)
edits.append(
MechanismEdit(
action="retract", mechanism_id=mechanism_id, source="loo", rationale=f"leaks on the fixed side of {', '.join(leaked)}"
)
)
missed = [case for case in cases if not current[case.name].detected]
from ..learning.split import refuse_if_holdout
for record in corpus_mechanisms(candidates.load()): # type: ignore[attr-defined]
if record.id in admitted or not missed:
continue
refusal = refuse_if_holdout(record.pairs, cases_all)
if refusal is not None:
# learning-harness Req 5.2: a shape a holdout pair taught can never recover that pair, and the run
# says which pairs it was rather than showing a shorter list with no reason.
if degradations is not None:
degradations.append(refusal.degradation())
continue
with tempfile.TemporaryDirectory(prefix="ousast-lever-") as scratch:
trial = MechanismStore(Path(scratch) / "trial.jsonl")
trial.append(record)
recovered = [case.name for case in missed if score_pair_with_store(case, trial, facts).detected]
leaks = [case.name for case in cases if not score_pair_with_store(case, trial, facts).silent]
if recovered and not leaks:
edits.append(
MechanismEdit(
action="admit",
mechanism_id=record.id,
source="loo",
rationale=f"recovers holdout {', '.join(recovered)}; taught by {', '.join(record.pairs)}",
)
)
return edits
def _facts() -> SemanticFacts:
from ..semantic.facts import FactLoadError, load_facts
try:
return load_facts()
except FactLoadError:
return SemanticFacts(version="", sources=(), sinks=(), sanitizers=())
def _leaking_admissions(edits: Sequence[MechanismEdit], pair_cases: Sequence[object], store_path: Path) -> list[str]:
"""Admitted record ids that hit the fixed side of any holdout pair with the store as it stands after the edits."""
from ..pairs import GATING_TIERS, PairCase, select_tier, select_vendored
from ..semantic.loo import score_pair_with_store
from ..semantic.mechanisms import MechanismStore
admitted = {edit.mechanism_id for edit in edits if edit.action == "admit"}
if not admitted:
return []
cases = select_tier(select_vendored([case for case in pair_cases if isinstance(case, PairCase)]), GATING_TIERS)
store = MechanismStore(store_path)
facts = _facts()
leaking: set[str] = set()
for case in cases:
leaking |= set(score_pair_with_store(case, store, facts).leaked_by) & admitted
return sorted(leaking)
def profile_regressions(
before: dict[str, dict[str, float]],
after: dict[str, dict[str, float]],
@@ -278,9 +159,6 @@ def run_round(
pair_cases: Sequence[object] | None = None,
profile_tolerance: float = 0.0,
min_holdout_pairs: int = 5,
mechanism_candidates: Path | None = None,
mechanism_store: Path | None = None,
scripted_mechanism_edits: Sequence[MechanismEdit] | None = None,
) -> RoundOutcome:
policy = policy if policy is not None else load_policy()
rounds = load_journal(journal_path)
@@ -295,46 +173,18 @@ def run_round(
ruleset_by_id = {rule.rule_id: rule for rule in before_rules}
edits = propose_status_edits(before.result.metrics.per_rule, ruleset_by_id, current_ledger, blocked)
# corpus-seeded-mechanisms Req 5: the `mechanisms` lever proposes from the holdout pairs, gated like every other edit.
mechanism_edits: list[MechanismEdit] = []
refusals: list[dict[str, object]] = []
candidates_store = None
if mechanism_candidates is not None and mechanism_store is not None and pair_cases:
from ..semantic.mechanisms import MechanismStore
candidates_store = MechanismStore(mechanism_candidates)
mechanism_edits = (
list(scripted_mechanism_edits)
if scripted_mechanism_edits is not None
else propose_mechanism_edits(pair_cases, candidates_store, mechanism_store, degradations=refusals)
)
blocked_mechanisms = {edit.key() for edit in mechanism_edits} & blocked
mechanism_edits = [edit for edit in mechanism_edits if edit.key() not in blocked_mechanisms]
if not edits and not mechanism_edits:
if not edits:
return _outcome(round_index, accepted=False, reason="no_proposals", edits=[], before=before, after=before, degradations=refusals)
validator = EvolveValidator()
try:
for edit in edits:
validator.validate(edit, ruleset_by_id, policy)
if mechanism_edits and candidates_store is not None and mechanism_store is not None:
from ..semantic.mechanisms import MechanismStore
by_id = {record.id: record for record in candidates_store.load()}
admitted_by_id = {record.id: record for record in MechanismStore(mechanism_store).load()}
for mechanism_edit in mechanism_edits:
validator.validate_mechanism(mechanism_edit, by_id, admitted=admitted_by_id)
except StrictValidationError as exc:
_record(journal_path, round_index, edits, "rejected", before, before, str(exc), mechanism_edits=mechanism_edits)
_record(journal_path, round_index, edits, "rejected", before, before, str(exc))
return _outcome(
round_index,
accepted=False,
reason=f"validation_failed: {exc}",
edits=edits,
before=before,
after=before,
mechanism_edits=mechanism_edits,
degradations=refusals,
round_index, accepted=False, reason=f"validation_failed: {exc}", edits=edits, before=before, after=before, degradations=refusals
)
candidate = edits_to_ledger(edits, current_ledger)
@@ -365,39 +215,10 @@ def run_round(
regressed, small = profile_regressions(profiles_before, profiles_after, tolerance=profile_tolerance, min_pairs=min_holdout_pairs)
if regressed:
accepted = False
mechanisms_before: dict[str, dict[str, float]] = {}
mechanisms_after: dict[str, dict[str, float]] = {}
mechanism_reason = ""
if accepted and mechanism_edits and candidates_store is not None and mechanism_store is not None and pair_cases:
# The lever changes what variant search finds, so its gate is the same per-profile clause over the holdout pairs
# scored with the store before and after; an admission that leaks or lowers any profile is reverted byte for byte.
mechanisms_before = evaluate_mechanism_profiles(pair_cases, mechanism_store)
snapshot = apply_mechanism_edits(mechanism_edits, candidates_store, mechanism_store)
mechanisms_after = evaluate_mechanism_profiles(pair_cases, mechanism_store)
mech_regressed, mech_small = profile_regressions(
mechanisms_before, mechanisms_after, tolerance=profile_tolerance, min_pairs=min_holdout_pairs
)
leaking = _leaking_admissions(mechanism_edits, pair_cases, mechanism_store)
improved = any(
mechanisms_after[p]["pair_correct"] > mechanisms_before.get(p, {}).get("pair_correct", 0.0) for p in mechanisms_after
) or any(edit.action == "retract" for edit in mechanism_edits)
if mech_regressed or leaking or not improved:
snapshot.restore()
accepted = False
regressed = regressed + mech_regressed
if leaking:
mechanism_reason = f"mechanism_leak:{leaking[0]}" # an admission that fires on a holdout fixed side (Req 5.3)
elif mech_regressed:
mechanism_reason = f"profile_regression:{mech_regressed[0]}"
else:
mechanism_reason = "mechanism_no_gain"
small = sorted(set(small) | set(mech_small))
if accepted:
if edits:
write_rule_ledger(ledger_path, candidate)
reason = "accepted"
elif mechanism_reason:
reason = mechanism_reason
elif regressed:
reason = f"profile_regression:{regressed[0]}" # every regressed profile is listed in profile_regressions
else:
@@ -412,20 +233,9 @@ def run_round(
reason,
regressed,
small,
mechanism_edits=mechanism_edits,
mechanisms_before=mechanisms_before,
mechanisms_after=mechanisms_after,
candidates=candidates_store,
)
outcome = _outcome(
round_index,
accepted=accepted,
reason=reason,
edits=edits,
before=before,
after=after,
mechanism_edits=mechanism_edits,
degradations=refusals,
round_index, accepted=accepted, reason=reason, edits=edits, before=before, after=after, degradations=refusals
)
return RoundOutcome(
**{
@@ -434,8 +244,6 @@ def run_round(
"profiles_under_minimum": small,
"per_profile_before": profiles_before,
"per_profile_after": profiles_after,
"mechanisms_before": mechanisms_before,
"mechanisms_after": mechanisms_after,
}
)
@@ -454,8 +262,6 @@ def run_improvement(
pair_cases: Sequence[object] | None = None,
profile_tolerance: float = 0.0,
min_holdout_pairs: int = 5,
mechanism_candidates: Path | None = None,
mechanism_store: Path | None = None,
) -> list[RoundOutcome]:
"""Run improvement rounds until convergence (no new proposals) or ``max_rounds``."""
outcomes: list[RoundOutcome] = []
@@ -472,8 +278,6 @@ def run_improvement(
pair_cases=pair_cases,
profile_tolerance=profile_tolerance,
min_holdout_pairs=min_holdout_pairs,
mechanism_candidates=mechanism_candidates,
mechanism_store=mechanism_store,
)
outcomes.append(outcome)
if outcome.reason == "no_proposals":
@@ -542,7 +346,6 @@ def _outcome(
edits: list[RuleStatusEdit],
before: _Eval,
after: _Eval,
mechanism_edits: list[MechanismEdit] | None = None,
degradations: list[dict[str, object]] | None = None,
) -> RoundOutcome:
return RoundOutcome(
@@ -550,7 +353,6 @@ def _outcome(
accepted=accepted,
reason=reason,
edits=edits,
mechanism_edits=list(mechanism_edits or []),
degradations=list(degradations or []),
recall_before=before.recall,
recall_after=after.recall,
@@ -573,14 +375,7 @@ def _record(
reason: str = "",
profile_regressions: list[str] | None = None,
profiles_under_minimum: list[str] | None = None,
mechanism_edits: Sequence[MechanismEdit] = (),
mechanisms_before: dict[str, dict[str, float]] | None = None,
mechanisms_after: dict[str, dict[str, float]] | None = None,
candidates: object | None = None,
) -> None:
pairs_by_id: dict[str, list[str]] = {}
if candidates is not None:
pairs_by_id = {record.id: list(record.pairs) for record in candidates.load()} # type: ignore[attr-defined]
append_round(
journal_path,
{
@@ -592,21 +387,7 @@ def _record(
"edits": [
{"key": e.key(), "lever": e.lever, "rule_id": e.rule_id, "from": e.from_status, "to": e.to_status, "rationale": e.rationale}
for e in edits
]
+ [
{
"key": m.key(),
"lever": m.lever,
"action": m.action,
"mechanism_id": m.mechanism_id,
"source": m.source,
"pairs": pairs_by_id.get(m.mechanism_id, []),
"rationale": m.rationale,
}
for m in mechanism_edits
],
"mechanisms_before": mechanisms_before or {},
"mechanisms_after": mechanisms_after or {},
"recall_before": round(before.recall, 4),
"recall_after": round(after.recall, 4),
"fp_before": round(before.fp_rate, 4),
src/openultrasast/improve/validator.py
+2 -100
View File
@@ -8,11 +8,9 @@ impossible; the validator additionally enforces status/bounds/staging/resolution
from __future__ import annotations
import json
import re
from collections.abc import Mapping, Sequence
from dataclasses import asdict, dataclass
from pathlib import Path
from collections.abc import Mapping
from dataclasses import dataclass
from ..policy import CwePolicy
from ..ruleset import PatternRule
@@ -62,40 +60,6 @@ class PolicyConstantEdit:
return f"policy:{self.name}:{self.from_value}->{self.to_value}"
@dataclass(frozen=True)
class MechanismEdit:
"""A `mechanisms`-lever edit (corpus-seeded-mechanisms Req 5): admit or retract one exporter record by id.
The loop never writes a shape; it may only move a record the exporter derived from a trusted pair into or out of
the scan-time store. Free text is confined to ``rationale``.
"""
action: str # admit | retract
mechanism_id: str
source: str = "loo" # loo | export
rationale: str = ""
lever: str = "mechanisms"
def key(self) -> str:
return f"mechanisms:{self.action}:{self.mechanism_id}"
@dataclass(frozen=True)
class StoreSnapshot:
"""Bytes of the scan-time mechanism store before a round; ``restore`` is the byte-for-byte revert."""
path: Path
content: bytes | None # None when the file did not exist
def restore(self) -> None:
if self.content is None:
if self.path.exists():
self.path.unlink()
return
self.path.parent.mkdir(parents=True, exist_ok=True)
self.path.write_bytes(self.content)
@dataclass(frozen=True)
class EvolveBounds:
k_range: tuple[float, float] = (20.0, 200.0)
@@ -113,49 +77,9 @@ class EvolveValidator:
self._validate_rule(edit, ruleset_by_id, policy)
elif isinstance(edit, PolicyConstantEdit):
self._validate_policy(edit)
elif isinstance(edit, MechanismEdit):
raise StrictValidationError("mechanism edits need the exporter candidate set: use validate_mechanism", kind="mechanism_change")
else:
raise StrictValidationError(f"unknown edit type {type(edit).__name__}", kind="rule_change")
def validate_mechanism(
self, edit: MechanismEdit, candidates: Mapping[str, object], admitted: Mapping[str, object] | None = None
) -> None:
"""Req 5.2: only records the exporter derived from trusted pairs, with a closed guard and identifier-only shape text.
A retraction may also name a record that is admitted in the scan store but no longer in the candidate set.
"""
from ..semantic.variants import GUARD_KINDS, SOURCE_KINDS
if edit.action not in MECHANISM_ACTIONS:
raise StrictValidationError(f"mechanism edit action {edit.action!r} is not admit or retract", kind="mechanism_change")
record = candidates.get(edit.mechanism_id)
if record is None and edit.action == "retract" and admitted is not None:
record = admitted.get(edit.mechanism_id)
if record is None:
raise StrictValidationError(
f"unknown mechanism {edit.mechanism_id!r}: not in the exporter candidate set", kind="mechanism_change"
)
if getattr(record, "origin", "") != "corpus":
raise StrictValidationError(
f"{edit.mechanism_id}: origin {getattr(record, 'origin', '')!r} is not corpus (only exporter records)",
kind="mechanism_change",
)
shape = getattr(record, "shape", None)
if not isinstance(shape, dict):
raise StrictValidationError(f"{edit.mechanism_id}: record carries no shape", kind="mechanism_change")
if str(shape.get("guard")) not in GUARD_KINDS or str(getattr(record, "guard", "")) not in GUARD_KINDS:
raise StrictValidationError(
f"{edit.mechanism_id}: guard {shape.get('guard')!r} is not a closed guard kind", kind="mechanism_change"
)
for field_name in ("sink_name", "language", "mechanism"):
value = str(shape.get(field_name, ""))
if not _IDENTIFIER.match(value.replace("-", "_")):
raise StrictValidationError(f"{edit.mechanism_id}: {field_name} {value!r} is not an identifier", kind="mechanism_change")
kinds = shape.get("source_kinds") or []
if not isinstance(kinds, list) or any(str(kind) not in SOURCE_KINDS for kind in kinds):
raise StrictValidationError(f"{edit.mechanism_id}: source kinds {kinds!r} are not closed kinds", kind="mechanism_change")
def _validate_rule(self, edit: RuleStatusEdit, ruleset_by_id: Mapping[str, PatternRule], policy: Mapping[str, CwePolicy]) -> None:
if edit.to_status not in VALID_STATUS:
raise StrictValidationError(f"invalid status {edit.to_status!r} for {edit.rule_id}")
@@ -178,25 +102,6 @@ class EvolveValidator:
raise StrictValidationError(f"MIN_SCORE={edit.to_value} out of bounds {self._bounds.min_score_range}", kind="policy_change")
def apply_mechanism_edits(edits: Sequence[MechanismEdit], candidates: object, scan_store: Path) -> StoreSnapshot:
"""Apply admit/retract edits to the scan-time store; returns the snapshot whose ``restore`` reverts byte for byte.
Admission appends the exporter record unchanged; retraction appends a tombstone row (``retracted = true``) so the
log stays append-only and ``MechanismStore.load`` folds the record away.
"""
snapshot = StoreSnapshot(path=scan_store, content=scan_store.read_bytes() if scan_store.exists() else None)
by_id = {record.id: record for record in candidates.load()} # type: ignore[attr-defined]
scan_store.parent.mkdir(parents=True, exist_ok=True)
with scan_store.open("a") as handle:
for edit in edits:
record = by_id[edit.mechanism_id]
payload = asdict(record)
if edit.action == "retract":
payload["retracted"] = True
handle.write(json.dumps(payload, sort_keys=True) + "\n")
return snapshot
def edits_to_ledger(edits: list[RuleStatusEdit], base: Mapping[str, dict[str, object]]) -> dict[str, dict[str, object]]:
"""Overlay rule-status edits onto a copy of the base ledger."""
ledger: dict[str, dict[str, object]] = {key: dict(value) for key, value in base.items()}
@@ -211,9 +116,6 @@ __all__ = [
"EvolveBounds",
"EvolveValidator",
"MECHANISM_ACTIONS",
"MechanismEdit",
"StoreSnapshot",
"apply_mechanism_edits",
"PolicyConstantEdit",
"RuleStatusEdit",
"StrictValidationError",
src/openultrasast/learning/__init__.py
-13
View File
@@ -1,13 +0,0 @@
"""The learning harness: classified detectors improved one family at a time (learning-harness spec).
Layers left to right, and nothing imports rightward: ``families`` -> ``classify`` -> ``scoring`` ->
``split`` -> ``detectors`` and ``verifiers`` -> ``rounds`` -> ``publish``. ``pairs`` and ``benchmark``
never import this package; the corpus stores a family label as an opaque string and the classifier
validates it.
"""
from __future__ import annotations
from .families import FAMILY_IDS, VERIFIER_KINDS, FamiliesError, Family, FamilyTaxonomy, load_families
__all__ = ["FAMILY_IDS", "VERIFIER_KINDS", "FamiliesError", "Family", "FamilyTaxonomy", "load_families"]
src/openultrasast/learning/acceptance.py
-146
View File
@@ -1,146 +0,0 @@
"""When a round is kept, and how it is undone (learning-harness, Req 6.2, 9.5).
A round is accepted only when the family it targets does not go backwards on either the train or the
held-out pairs, at least one of them moves forward, and no other family flips more pairs than that
family showed against itself in round zero. A tie is a rejection: accepting one would let the loop
wander on noise and call it progress.
A family with no measured floor gets no budget. Unmeasured is not the same as unlimited, and the
alternative is a family that never ran in round zero quietly absorbing every regression.
Undoing is a directory restored byte for byte, files the round added removed, files it deleted put back.
A revert that only rewrites what it can see is not a revert.
"""
from __future__ import annotations
import hashlib
from collections.abc import Mapping
from dataclasses import dataclass
from pathlib import Path
from .rounds import NoiseFloor
@dataclass(frozen=True)
class Verdict:
accepted: bool
reason: str
offenders: tuple[str, ...] = ()
def to_dict(self) -> dict[str, object]:
return {"accepted": self.accepted, "reason": self.reason, "offenders": list(self.offenders)}
def decide(
*,
target: str,
train_delta: float,
holdout_delta: float,
sweep: Mapping[str, int],
floors: Mapping[str, NoiseFloor],
) -> Verdict:
"""The acceptance rule, in one place, so no caller can soften it."""
if train_delta < 0:
return Verdict(accepted=False, reason="train_regressed")
if holdout_delta < 0:
return Verdict(accepted=False, reason="holdout_regressed")
if train_delta == 0 and holdout_delta == 0:
return Verdict(accepted=False, reason="tie")
offenders = tuple(
family for family, flips in sorted(sweep.items()) if family != target and _gates(family, floors) and flips > _budget(family, floors)
)
if offenders:
return Verdict(accepted=False, reason="collateral_regression", offenders=offenders)
return Verdict(accepted=True, reason="improved")
@dataclass(frozen=True)
class Reliability:
"""Whether a stage's movement could be told from a coin flip."""
better: int
worse: int
p_value: float
reliable: bool
def to_dict(self) -> dict[str, object]:
return {"better": self.better, "worse": self.worse, "p_value": self.p_value, "reliable": self.reliable}
def reliability(*, better: int, worse: int, alpha: float = 0.05) -> Reliability:
"""The two-sided sign test over the pairs that changed, for the record rather than for the decision.
`decide` compares raw deltas, as Req 9.5 specifies. On a detector whose measured floor says half its pairs
disagree with themselves between runs, a raw delta is mostly noise — which is what makes a deterministic
decoder look necessary. It is not: the alternative is an acceptance rule that models the score as noisy, the
way reflective-evolution and Bayesian prompt optimizers do, and keeps per-instance winners on a frontier
instead of demanding a single global improvement. Changing the rule is a maintainer's call, so every round
records this beside its decision and the journal can be read to answer the question with data.
"""
from .scoring import sign_test
p_value = sign_test(better, worse)
return Reliability(better=better, worse=worse, p_value=p_value, reliable=bool(p_value < alpha and better != worse))
def _gates(family: str, floors: Mapping[str, NoiseFloor]) -> bool:
floor = floors.get(family)
return floor.gates if floor is not None else True
def _budget(family: str, floors: Mapping[str, NoiseFloor]) -> int:
floor = floors.get(family)
return floor.budget_flips if floor is not None else 0
def within_budget(spent_usd: float, cap_usd: float) -> bool:
"""A round may spend up to its cap; crossing it reverts the round rather than finishing it."""
return spent_usd <= cap_usd
@dataclass(frozen=True)
class DirectorySnapshot:
"""Every file of one directory as it was, so a rejected round leaves nothing behind."""
directory: Path
files: tuple[tuple[str, bytes], ...]
@staticmethod
def of(directory: Path) -> DirectorySnapshot:
entries = [(path.relative_to(directory).as_posix(), path.read_bytes()) for path in sorted(directory.rglob("*")) if path.is_file()]
return DirectorySnapshot(directory=directory, files=tuple(entries))
def restore(self) -> None:
kept = {name for name, _ in self.files}
for path in sorted(self.directory.rglob("*"), reverse=True):
if path.is_file() and path.relative_to(self.directory).as_posix() not in kept:
path.unlink()
for name, content in self.files:
target = self.directory / name
target.parent.mkdir(parents=True, exist_ok=True)
target.write_bytes(content)
def digest_configs(configs_dir: Path) -> dict[str, str]:
"""One digest per family directory, so an untouched family can be shown to be untouched."""
digests: dict[str, str] = {}
for directory in sorted(path for path in configs_dir.iterdir() if path.is_dir()) if configs_dir.is_dir() else []:
digest = hashlib.sha256()
for path in sorted(directory.rglob("*")):
if path.is_file():
digest.update(path.relative_to(directory).as_posix().encode())
digest.update(path.read_bytes())
digests[directory.name] = digest.hexdigest()
return digests
def bump_version(directory: Path, version: str) -> None:
"""Write the accepted version into this family's settings, and only this family's."""
settings = directory / "family.toml"
lines = settings.read_text(encoding="utf-8").splitlines()
rewritten = [f'version = "{version}"' if line.strip().startswith("version") else line for line in lines]
settings.write_text("\n".join(rewritten) + "\n", encoding="utf-8")
__all__ = ["DirectorySnapshot", "Reliability", "Verdict", "bump_version", "decide", "digest_configs", "reliability", "within_budget"]
src/openultrasast/learning/canaries.py
-227
View File
@@ -1,227 +0,0 @@
"""Canary snippets and the second judge (learning-harness, Req 7.2, 7.5).
A canary is how a family says "the vulnerability fired": a snippet plants a marker only the
vulnerability can reveal, and the token appearing in the sandbox output is the oracle. A marker file an
injected command writes, a file under the scratch mount that only a traversal reaches, an object that
only an unsafe load reconstructs. The token is derived from the family, file and function, so one
family's oracle can never answer for another.
The injection canary confirms untrusted input reaching an interpreter: command, code and template
execution. A query-language injection whose payload never reaches a shell is not confirmed by it and
stays a suspicion, which is the honest answer rather than a false negative dressed as a verdict.
Nothing reaches `proven` on a machine's say-so alone: a second, independent judgement reads the snippet
and what the sandbox printed and either confirms or drops the claim back to corroboration. Templates
and oracle code live in the package, outside every directory a round may write.
"""
from __future__ import annotations
import hashlib
import json
from dataclasses import dataclass
from pathlib import Path
from ..config import SandboxConfig
from ..findings import StaticFinding
from ..regress.candidate import DEFAULT_IMAGES, RegressionRunner
from ..regress.verdict import TRIGGERABLE
from ..sandbox.runner import SCRATCH_MOUNT, WORKSPACE_MOUNT, SandboxRunner
from ..tool_hunter import ChatClient
from .verifiers import Verification, VerifierKind, token_oracle
_TOKEN_PREFIX = "ousast-canary-"
_JUDGE_SYSTEM = (
"You check one security claim against what a sandbox actually printed. "
'Answer with JSON only: {"confirms": true|false, "reason": "<one sentence>"}. '
"Confirm only when the printed output can be explained by the claimed vulnerability and by nothing else."
)
def canary_token(family: str, path: str, function: str) -> str:
"""The token the oracle looks for. Unique to the experiment, so one canary can never answer for another."""
digest = hashlib.sha256(f"{family}|{path}|{function}".encode()).hexdigest()[:12]
return f"{_TOKEN_PREFIX}{digest}"
def payload_token(family: str, path: str, function: str) -> str:
"""The value carried *inside* an injected payload, deliberately unrelated to the oracle token.
A handler that merely echoes its input would otherwise print the oracle token and satisfy the oracle
without the vulnerability firing at all. The snippet prints the oracle token only after seeing this
value somewhere the payload alone could not have put it.
"""
digest = hashlib.sha256(f"payload|{family}|{path}|{function}".encode()).hexdigest()[:12]
return f"ousast-payload-{digest}"
def render_canary(
family: str, *, language: str, path: str, function: str, workspace: str = WORKSPACE_MOUNT, scratch: str = SCRATCH_MOUNT
) -> tuple[str, str] | None:
"""The snippet that plants this family's canary and the token that proves it fired, or None.
``workspace`` and ``scratch`` default to the sandbox mounts; a test points them at real directories so
the snippet can be run for what it actually does rather than for what a fake runner says it does.
"""
template = _TEMPLATES.get((family, language))
if template is None:
return None
token = canary_token(family, path, function)
module = path.replace("\\", "/")
snippet = template.format(
token=token,
payload=payload_token(family, path, function),
workspace=workspace,
scratch=scratch,
module=module,
function=function,
)
return snippet, token
@dataclass(frozen=True)
class CanaryVerifier:
"""Runs a family's canary in the sandbox and lets only its own token count as proof."""
runner: SandboxRunner
limits: SandboxConfig
images: dict[str, str] | None = None
kind: VerifierKind = "canary"
def verify(self, claim: StaticFinding, *, root: Path, language: str) -> Verification:
family = next((tag.split(":", 1)[1] for tag in claim.tags if tag.startswith("family:")), "")
rendered = render_canary(family, language=language, path=claim.path, function=claim.function_name or "")
if rendered is None:
return Verification(rung="suspicion", reason="no_template")
snippet, token = rendered
images = self.images or dict(DEFAULT_IMAGES)
verdict = RegressionRunner(self.runner).run_recipe(
language, snippet, images.get(language, "ousast-missing-image"), self.limits, repo_root=root, oracle=token_oracle(token)
)
if verdict.verdict == TRIGGERABLE:
return Verification(rung="proven", reason="oracle_fired", oracle_output=token)
return Verification(rung="suspicion", reason="oracle_quiet" if verdict.reason == "oracle_quiet" else verdict.reason)
def confirm_proven(
verification: Verification, claim: StaticFinding, *, client: ChatClient | None, model: str, snippet: str
) -> Verification:
"""Req 7.5: a claim is published as proven only when a second, independent judgement agrees."""
if verification.rung != "proven":
return verification
if client is None or not model:
return Verification(rung="static_corroboration", reason="judge_unavailable", oracle_output=verification.oracle_output)
question = (
f"Claim: {claim.title}. {claim.rationale[:400]}\n"
f"Location: {claim.path} line {claim.line}, function {claim.function_name}.\n\n"
f"Snippet that was run:\n{snippet}\n\nWhat the sandbox printed:\n{verification.oracle_output}\n\n"
"Does that output demonstrate the claim? Answer as JSON."
)
messages: list[dict[str, object]] = [{"role": "system", "content": _JUDGE_SYSTEM}, {"role": "user", "content": question}]
try:
response = client.complete(model=model, messages=messages, tools=[], json_object=True) # type: ignore[call-arg]
except TypeError:
response = client.complete(model=model, messages=messages, tools=[])
except Exception: # noqa: BLE001 — a judge that cannot answer must not promote a claim
return Verification(rung="static_corroboration", reason="judge_unavailable", oracle_output=verification.oracle_output)
if _confirms(response.content):
return verification
return Verification(rung="static_corroboration", reason="judge_declined", oracle_output=verification.oracle_output)
def _confirms(content: str | None) -> bool:
if not isinstance(content, str) or "{" not in content:
return False
try:
payload = json.loads(content[content.index("{") : content.rindex("}") + 1])
except (ValueError, json.JSONDecodeError):
return False
return isinstance(payload, dict) and payload.get("confirms") is True
# Templates are data, not prose: each plants a marker only its own vulnerability can reveal, imports the
# candidate from the read-only source mount, and prints the token on that path alone. A snippet that
# cannot reach the function prints nothing, which is a suspicion, not a proof.
_PYTHON_INJECTION = """# canary: untrusted input reaches an interpreter, which writes a marker only it could write.
# The payload carries a different value from the oracle token, so a handler that merely echoes its input
# cannot satisfy the oracle; only something that executed the payload can put PLANTED into the marker.
import sys
sys.path.insert(0, {workspace!r})
CANARY = {token!r}
PLANTED = {payload!r}
MARKER = {scratch!r} + "/canary-injection.txt"
payload = "x; python3 -c \\"open(%r,'w').write(%r)\\"" % (MARKER, PLANTED)
try:
import importlib.util
spec = importlib.util.spec_from_file_location("candidate", {workspace!r} + "/" + {module!r})
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
handler = getattr(module, {function!r}, None)
if callable(handler):
try:
handler(payload)
except Exception as inner:
print("handler raised:", type(inner).__name__)
import os
if os.path.exists(MARKER):
with open(MARKER, encoding="utf-8") as handle:
if PLANTED in handle.read():
print("CANARY", CANARY)
except Exception as error:
print("canary could not reach the function:", type(error).__name__)
"""
_PYTHON_PATH = """# canary: a traversal reads a file the handler was never meant to open
import sys
sys.path.insert(0, {workspace!r})
CANARY = {token!r}
PLANTED = {payload!r}
MARKER = {scratch!r} + "/canary-path.txt"
with open(MARKER, "w", encoding="utf-8") as handle:
handle.write(CANARY)
try:
import importlib.util
spec = importlib.util.spec_from_file_location("candidate", {workspace!r} + "/" + {module!r})
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
handler = getattr(module, {function!r}, None)
result = handler(MARKER) if callable(handler) else None
if CANARY in str(result):
print("CANARY", CANARY)
except Exception as error:
print("canary could not reach the function:", type(error).__name__)
"""
_PYTHON_DESERIALIZATION = """# canary: an unsafe load reconstructs an object that announces itself
import pickle, sys
sys.path.insert(0, {workspace!r})
CANARY = {token!r}
PLANTED = {payload!r}
class Marker:
def __reduce__(self):
return (print, ("CANARY", CANARY))
blob = pickle.dumps(Marker())
try:
import importlib.util
spec = importlib.util.spec_from_file_location("candidate", {workspace!r} + "/" + {module!r})
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
handler = getattr(module, {function!r}, None)
if callable(handler):
handler(blob)
except Exception as error:
print("canary could not reach the function:", type(error).__name__)
"""
_TEMPLATES: dict[tuple[str, str], str] = {
("injection", "python"): _PYTHON_INJECTION,
("path", "python"): _PYTHON_PATH,
("deserialization", "python"): _PYTHON_DESERIALIZATION,
}
__all__ = ["CanaryVerifier", "canary_token", "confirm_proven", "payload_token", "render_canary"]
src/openultrasast/learning/classify.py
-399
View File
@@ -1,399 +0,0 @@
"""The auto-classifier: which families a pair, a finding or a region belongs to (learning-harness, Req 2).
Three tiers, cheapest first. Tier one decides from what the repository already knows: a declared family
label, an obligation kind, a mechanism id, the rule a finding came from, or the labeled weakness. Tier
two asks the model one closed question in JSON mode, with thinking disabled so temperature applies, and
abstains when the answer is off-vocabulary or the reply's mean token log-probability is below the
threshold. Tier three is abstention. Every answer records which tier decided it, and answers are
multi-label because one function can owe more than one family.
The classifier proposes; it never writes a label. It also owns validation of the labels the corpus
carries, because the corpus stores a family as an opaque string and this module owns the taxonomy.
"""
from __future__ import annotations
import json
from collections.abc import Sequence
from dataclasses import dataclass, field
from pathlib import Path
from typing import Literal
from ..findings import StaticFinding
from ..tool_hunter import ChatClient
from .families import FamilyTaxonomy
Tier = Literal["static", "model", "none"]
MIN_CONFIDENCE = -1.0 # mean token log-probability; below this the model is guessing, so we abstain
_SYSTEM = (
"You classify a piece of code or a labeled bug into vulnerability families for a security tool. "
'Answer with JSON only, of the form {"families": ["<id>", ...]}, using ids from the list you are given. '
"Use an empty list when none of them fits. Do not explain and do not invent an id."
)
class FamilyLabelError(ValueError):
"""Raised when a catalog row carries a family id outside the taxonomy."""
@dataclass(frozen=True)
class Classification:
families: tuple[str, ...] # empty means unknown
tier: Tier
confidence: float | None = None # mean token log-probability when the model answered
def classify_pair(
case: object, taxonomy: FamilyTaxonomy, *, client: ChatClient | None, model: str = "", use_declared: bool = True
) -> Classification:
"""The families a labeled pair belongs to: its declared label, else its mechanism, else its weakness.
``use_declared`` is False when the classifier is being measured against those same labels, so that
agreement is not the classifier reading back the answer it is scored on.
"""
families = _static_families_of_pair(case, taxonomy, use_declared=use_declared)
if families:
return Classification(families=families, tier="static")
return _ask(client, model, taxonomy, _pair_question(case))
def classify_finding(finding: StaticFinding, taxonomy: FamilyTaxonomy, *, client: ChatClient | None, model: str = "") -> Classification:
"""The families a finding belongs to: its own family tag, else what produced it, else the model."""
families = _static_families_of_finding(finding, taxonomy)
if families:
return Classification(families=families, tier="static")
return _ask(client, model, taxonomy, _finding_question(finding))
def classify_region(
root: Path, path: str, *, function: str | None, taxonomy: FamilyTaxonomy, client: ChatClient | None, model: str = ""
) -> Classification:
"""The families a code region belongs to, from what the static tools already see in it."""
families = _static_families_of_region(root, path, function, taxonomy)
if families:
return Classification(families=families, tier="static")
return _ask(client, model, taxonomy, _region_question(root, path, function))
def validate_family_labels(cases: Sequence[object], taxonomy: FamilyTaxonomy) -> None:
"""Every family a catalog row declares must be in the taxonomy. The corpus stores labels; this module owns them."""
known = {family.id for family in taxonomy.families}
for case in cases:
for row in getattr(case, "expected", ()) or ():
label = getattr(row, "family", None)
if label and label not in known:
name = getattr(case, "name", "?")
raise FamilyLabelError(f"pair {name}: unknown family {label!r}; expected one of {', '.join(sorted(known))}")
def _static_families_of_pair(case: object, taxonomy: FamilyTaxonomy, *, use_declared: bool = True) -> tuple[str, ...]:
declared: list[str] = []
derived: list[str] = []
for row in getattr(case, "expected", ()) or ():
label = getattr(row, "family", None)
if use_declared and label and any(family.id == label for family in taxonomy.families):
declared.append(str(label))
continue
obligation = getattr(row, "obligation", None)
if obligation:
derived.append("access_control")
mechanism = getattr(row, "mechanism", None)
by_mechanism = taxonomy.family_of_mechanism(str(mechanism)) if mechanism else None
if by_mechanism is not None and by_mechanism.id != "unknown":
derived.append(by_mechanism.id)
continue
by_cwe = taxonomy.family_of_cwe(str(getattr(row, "cwe", "") or ""))
if by_cwe is not None and by_cwe.id != "unknown":
derived.append(by_cwe.id)
return _ordered(declared or derived, taxonomy)
def _static_families_of_finding(finding: StaticFinding, taxonomy: FamilyTaxonomy) -> tuple[str, ...]:
known = {family.id for family in taxonomy.families}
tagged = [tag.split(":", 1)[1] for tag in finding.tags if tag.startswith("family:") and tag.split(":", 1)[1] in known]
if tagged:
return _ordered(tagged, taxonomy)
found: list[str] = []
for tag in finding.tags:
if tag.startswith("obligation:") or tag.startswith("discharger:"):
found.append("access_control")
elif tag.startswith("mechanism:"):
family = taxonomy.family_of_mechanism(tag.split(":", 1)[1])
if family is not None and family.id != "unknown":
found.append(family.id)
if finding.finding_id.startswith("obligation:"):
found.append("access_control")
rule_family = taxonomy.family_of_cwe(_rule_cwe(finding.finding_id.split(":", 1)[0]))
if rule_family is not None and rule_family.id != "unknown":
found.append(rule_family.id)
return _ordered(found, taxonomy)
def _static_families_of_region(root: Path, path: str, function: str | None, taxonomy: FamilyTaxonomy) -> tuple[str, ...]:
from ..hunter_tools import PathEscapesRepo, flows, obligations
found: list[str] = []
if function:
try:
if obligations(root, path=path, function=function):
found.append("access_control")
for record in flows(root, path=path, function=function):
family = taxonomy.family_of_cwe(str(record.get("cwe") or ""))
if family is not None and family.id != "unknown":
found.append(family.id)
except (PathEscapesRepo, OSError):
return ()
return _ordered(found, taxonomy)
def _ask(client: ChatClient | None, model: str, taxonomy: FamilyTaxonomy, question: str) -> Classification:
if client is None:
return Classification(families=(), tier="none")
catalog = "\n".join(f"- {family.id}: {family.description}" for family in taxonomy.families)
messages: list[dict[str, object]] = [
{"role": "system", "content": _SYSTEM},
{"role": "user", "content": f"Families:\n{catalog}\n\n{question}\n\nAnswer as JSON."},
]
try:
try:
response = client.complete(model=model, messages=messages, tools=[], json_object=True, logprobs=True) # type: ignore[call-arg]
except TypeError: # a client without the adapter's extras
response = client.complete(model=model, messages=messages, tools=[])
except Exception: # noqa: BLE001 — the classifier is advisory; any failed call abstains, including in the fallback
return Classification(families=(), tier="none")
confidence = response.mean_logprob
if confidence is not None and confidence < MIN_CONFIDENCE:
return Classification(families=(), tier="none", confidence=confidence)
families = _ordered(_parse(response.content), taxonomy)
if not families:
return Classification(families=(), tier="none", confidence=confidence)
return Classification(families=families, tier="model", confidence=confidence)
def _parse(content: str | None) -> list[str]:
if not isinstance(content, str) or not content.strip():
return []
try:
payload = json.loads(content[content.index("{") : content.rindex("}") + 1])
except (ValueError, json.JSONDecodeError):
return []
families = payload.get("families") if isinstance(payload, dict) else None
if not isinstance(families, list):
return []
return [str(item) for item in families if isinstance(item, str)]
def _ordered(names: Sequence[str], taxonomy: FamilyTaxonomy) -> tuple[str, ...]:
"""Taxonomy order, de-duplicated, unknown ids dropped, so two runs compare equal."""
wanted = set(names)
return tuple(family.id for family in taxonomy.families if family.id in wanted and family.id != "unknown")
def _rule_cwe(rule_id: str) -> str:
from ..ruleset import DEFAULT_RULESET_DIR, load_ruleset
for rule in load_ruleset(DEFAULT_RULESET_DIR):
if rule.rule_id == rule_id:
return rule.cwe
return ""
def _pair_question(case: object) -> str:
row = next(iter(getattr(case, "expected", ()) or ()), None)
parts = [f"Labeled bug in {getattr(case, 'relpath', '?')}"]
if row is not None:
parts.append(
f"function {getattr(row, 'function', '?')}, weakness {getattr(row, 'cwe', '?')}, mechanism {getattr(row, 'mechanism', '?')}"
)
return "Which families does it belong to? " + "; ".join(parts)
def _finding_question(finding: StaticFinding) -> str:
return f"Which families does this finding belong to? {finding.title}. {finding.rationale[:400]}"
def _region_question(root: Path, path: str, function: str | None) -> str:
from ..hunter_tools import PathEscapesRepo, read_definition
try:
found = read_definition(root, function or "", max_chars=2000) if function else None
except (PathEscapesRepo, OSError):
found = None
body = str(found["text"]) if found else ""
return f"Which families does this code belong to?\n{path}::{function}\n{body}"
@dataclass(frozen=True)
class QueueEntry:
"""One disagreement between a maintainer label and the classifier, for a human to settle."""
pair: str
label: str
classified: tuple[str, ...]
tier: Tier
confidence: float | None = None
def to_dict(self) -> dict[str, object]:
return {
"pair": self.pair,
"label": self.label,
"classified": list(self.classified),
"tier": self.tier,
"confidence": self.confidence,
}
@dataclass(frozen=True)
class ClassifierReport:
"""How the classifier did on its own: coverage over the whole corpus, agreement where a reference exists.
``labeled`` is how many rows carry a maintainer family label. Agreement, the confusion matrix and both
baselines are computed over those rows only, so an unlabeled corpus reports coverage and says the
reference set is empty rather than inventing a score.
"""
taxonomy_version: str
classified: int
labeled: int
agreement: float # credit for the classifier's first answer
oracle_router: float # credit for the best of its answers
random_router: float # a uniform pick over the real families
confusion: dict[str, dict[str, int]] = field(default_factory=dict)
per_tier: dict[str, int] = field(default_factory=dict)
per_family: dict[str, int] = field(default_factory=dict)
unknown: int = 0
queue: tuple[QueueEntry, ...] = ()
hierarchical_credit: bool = False # False: the taxonomy is flat, so no answer could earn partial credit (Req 2.6)
def to_dict(self) -> dict[str, object]:
return {
"taxonomy_version": self.taxonomy_version,
"classified": self.classified,
"labeled": self.labeled,
"agreement": self.agreement,
"oracle_router": self.oracle_router,
"random_router": self.random_router,
"confusion": {label: dict(sorted(answers.items())) for label, answers in sorted(self.confusion.items())},
"per_tier": dict(sorted(self.per_tier.items())),
"per_family": dict(sorted(self.per_family.items())),
"unknown": self.unknown,
"hierarchical_credit": self.hierarchical_credit,
"queue": [entry.to_dict() for entry in self.queue],
}
def measure_classifier(
cases: Sequence[object],
taxonomy: FamilyTaxonomy,
*,
client: ChatClient | None,
model: str = "",
reference_tiers: Sequence[str] = ("seeded", "reviewed"),
) -> ClassifierReport:
"""Classify every case and score the classifier against the labels a maintainer stands behind."""
rows = list(cases)
real = [family.id for family in taxonomy.families if family.id != "unknown"]
per_tier: dict[str, int] = {}
per_family: dict[str, int] = {}
confusion: dict[str, dict[str, int]] = {}
queue: list[QueueEntry] = []
unknown = 0
credits: list[float] = []
oracles: list[float] = []
for case in rows:
answer = classify_pair(case, taxonomy, client=client, model=model, use_declared=False)
per_tier[answer.tier] = per_tier.get(answer.tier, 0) + 1
for family_id in answer.families:
per_family[family_id] = per_family.get(family_id, 0) + 1
if not answer.families:
unknown += 1
label = _reference_label(case, taxonomy, reference_tiers)
if label is None:
continue
first = answer.families[0] if answer.families else "unknown"
credits.append(_credit(taxonomy, label, (first,)))
oracles.append(_credit(taxonomy, label, answer.families or ("unknown",)))
confusion.setdefault(label, {})[first] = confusion.setdefault(label, {}).get(first, 0) + 1
if credits[-1] < 1.0:
queue.append(
QueueEntry(
pair=str(getattr(case, "name", "?")),
label=label,
classified=answer.families,
tier=answer.tier,
confidence=answer.confidence,
)
)
labeled = len(credits)
return ClassifierReport(
taxonomy_version=taxonomy.version,
classified=len(rows),
labeled=labeled,
agreement=sum(credits) / labeled if labeled else 0.0,
oracle_router=sum(oracles) / labeled if labeled else 0.0,
random_router=(1.0 / len(real)) if real and labeled else 0.0,
confusion=confusion,
per_tier=per_tier,
per_family=per_family,
unknown=unknown,
queue=tuple(queue),
hierarchical_credit=taxonomy.hierarchical,
)
def write_review_queue(path: Path, entries: Sequence[QueueEntry]) -> int:
"""Append disagreements a human has not seen yet; a pair already queued is never duplicated."""
path.parent.mkdir(parents=True, exist_ok=True)
seen: set[str] = set()
if path.exists():
for line in path.read_text(encoding="utf-8").splitlines():
if line.strip():
try:
seen.add(str(json.loads(line).get("pair", "")))
except json.JSONDecodeError:
continue
written = 0
with path.open("a", encoding="utf-8") as handle:
for entry in entries:
if entry.pair in seen:
continue
handle.write(json.dumps(entry.to_dict(), sort_keys=True) + "\n")
seen.add(entry.pair)
written += 1
return written
def _reference_label(case: object, taxonomy: FamilyTaxonomy, reference_tiers: Sequence[str]) -> str | None:
"""The family a maintainer stands behind for this pair, or None. Only trusted tiers are a reference."""
if str(getattr(case, "review_tier", "")) not in reference_tiers:
return None
known = {family.id for family in taxonomy.families}
for row in getattr(case, "expected", ()) or ():
label = getattr(row, "family", None)
if label and str(label) in known:
return str(label)
return None
def _credit(taxonomy: FamilyTaxonomy, label: str, answers: Sequence[str]) -> float:
"""1.0 for the same family, half for a parent or child, nothing for a lateral or fabricated answer."""
best = 0.0
for answer in answers:
relation = taxonomy.related(label, answer)
best = max(best, 1.0 if relation == "same" else 0.5 if relation == "parent_child" else 0.0)
return best
__all__ = [
"MIN_CONFIDENCE",
"Classification",
"ClassifierReport",
"FamilyLabelError",
"QueueEntry",
"classify_finding",
"classify_pair",
"classify_region",
"measure_classifier",
"validate_family_labels",
"write_review_queue",
]
src/openultrasast/learning/detectors.py
-211
View File
@@ -1,211 +0,0 @@
"""Per-family detector configurations and the runner that uses them (learning-harness, Req 6.1-6.5).
A detector is a configuration, not code: a system prompt, an evidence checklist, the curated tools it
may call, its step and cost budget, and the hard negatives and counterexamples earlier rounds left it.
Round zero writes one directory per family from the same prompt, so the baseline is controlled, and a
round rewrites exactly one directory and bumps that family's version. Every other family is frozen by
construction, because a round can only touch the directory it targets.
The runner gives a detector the whole file rather than an excerpt around a hotspot, since context loss
is a documented cause of misjudging patched code, and tags each finding with exactly one family and the
configuration version that produced it, so a finding can always be traced back to what made it.
"""
from __future__ import annotations
import json
import tomllib
from collections.abc import Mapping, Sequence
from dataclasses import dataclass
from pathlib import Path
from ..findings import StaticFinding
from ..tool_hunter import HUNTER_TOOLS, ChatClient, run_tool_hunter
from .families import FamilyTaxonomy
MAX_PROMPT_CHARS = 8000 # prompt plus checklist; unbounded prompts are the documented signature of overfitting
GENERALIST = "unknown" # the family every region also runs, so a bug outside the taxonomy is still looked for
CURATED_TOOLS = tuple(str(schema["function"]["name"]) for schema in HUNTER_TOOLS) # type: ignore[index]
_REQUIRED = ("prompt.md", "family.toml")
class DetectorConfigError(ValueError):
"""Raised when a detector configuration is missing, over its cap, or names something outside the closed sets."""
@dataclass(frozen=True)
class FamilyConfig:
family: str
version: str
prompt: str
checklist: str
tools: tuple[str, ...]
max_steps: int
max_cost_usd: float
max_chars: int
hard_negatives: tuple[Mapping[str, object], ...] = ()
counterexamples: tuple[Mapping[str, object], ...] = ()
@property
def detector_id(self) -> str:
return f"{self.family}@{self.version}"
@dataclass(frozen=True)
class Region:
"""A place a detector is pointed at: one file, optionally one function, and the families admitted for it."""
path: str
function: str | None = None
families: tuple[str, ...] = ()
def load_family_configs(configs_dir: Path, taxonomy: FamilyTaxonomy) -> dict[str, FamilyConfig]:
"""Every configuration under ``configs_dir``. A directory that is not a family, or a broken one, fails loud."""
known = {family.id for family in taxonomy.families}
configs: dict[str, FamilyConfig] = {}
for directory in sorted(path for path in configs_dir.iterdir() if path.is_dir()) if configs_dir.is_dir() else []:
if directory.name not in known:
raise DetectorConfigError(f"{directory}: {directory.name!r} is not a family; expected one of {', '.join(sorted(known))}")
configs[directory.name] = _load_one(directory)
return configs
def _load_one(directory: Path) -> FamilyConfig:
for name in _REQUIRED:
if not (directory / name).is_file():
raise DetectorConfigError(f"{directory}: {name} is required")
try:
settings = tomllib.loads((directory / "family.toml").read_text(encoding="utf-8"))
except tomllib.TOMLDecodeError as exc:
raise DetectorConfigError(f"{directory / 'family.toml'}: not valid TOML: {exc}") from exc
version = str(settings.get("version", "")).strip()
if not version:
raise DetectorConfigError(f"{directory}: version is required")
prompt = (directory / "prompt.md").read_text(encoding="utf-8").strip()
checklist = _read_optional(directory / "checklist.md")
if len(prompt) + len(checklist) > MAX_PROMPT_CHARS:
raise DetectorConfigError(
f"{directory}: prompt and checklist length {len(prompt) + len(checklist)} exceeds the cap of {MAX_PROMPT_CHARS}"
)
tools = tuple(str(name) for name in settings.get("tools", []) or ())
if not tools:
raise DetectorConfigError(f"{directory}: tools is required and must not be empty; a detector with no tools reports nothing")
unknown_tools = [name for name in tools if name not in CURATED_TOOLS]
if unknown_tools:
raise DetectorConfigError(f"{directory}: unknown tool(s) {', '.join(unknown_tools)}; expected from {', '.join(CURATED_TOOLS)}")
return FamilyConfig(
family=directory.name,
version=version,
prompt=prompt,
checklist=checklist,
tools=tools,
max_steps=int(settings.get("max_steps", 4)),
max_cost_usd=float(settings.get("max_cost_usd", 0.5)),
max_chars=int(settings.get("max_chars", 4000)),
hard_negatives=_read_jsonl(directory / "hard_negatives.jsonl"),
counterexamples=_read_jsonl(directory / "counterexamples.jsonl"),
)
def run_family_detector(root: Path, region: Region, config: FamilyConfig, *, client: ChatClient, model: str) -> list[StaticFinding]:
"""One family's detector over one region. Findings carry that family and the configuration that found them."""
return run_tool_hunter(
root,
[],
client=client,
model=model,
max_steps=config.max_steps,
system_prompt=config.prompt,
user_prompt=_question(region, config),
context_files=(region.path,),
tools=config.tools,
max_chars=config.max_chars,
tags=(f"family:{config.family}", f"detector:{config.detector_id}"),
)
def run_region_detectors(
root: Path, region: Region, configs: Mapping[str, FamilyConfig], *, client: ChatClient, model: str
) -> list[StaticFinding]:
"""Every family the classifier admitted for this region, plus the generalist (Req 6.3)."""
wanted = [family for family in region.families if family in configs and family != GENERALIST]
if GENERALIST in configs:
wanted.append(GENERALIST)
findings: list[StaticFinding] = []
for family in wanted:
findings.extend(run_family_detector(root, region, configs[family], client=client, model=model))
return findings
def write_default_configs(
configs_dir: Path, taxonomy: FamilyTaxonomy, *, prompt: str, version: str, tools: Sequence[str] | None = None
) -> dict[str, Path]:
"""Round zero: one directory per family from the same prompt, so the baseline is controlled (Req 8.1)."""
written: dict[str, Path] = {}
for family in taxonomy.families:
directory = configs_dir / family.id
directory.mkdir(parents=True, exist_ok=True)
(directory / "prompt.md").write_text(f"{prompt}\n\nFamily: {family.id}. {family.description}\n", encoding="utf-8")
(directory / "checklist.md").write_text("- name the file, the line and why the code is reachable\n", encoding="utf-8")
selected = list(tools) if tools is not None else list(CURATED_TOOLS)
(directory / "family.toml").write_text(
f'version = "{version}"\ntools = {json.dumps(selected)}\nmax_steps = 4\nmax_cost_usd = 0.5\nmax_chars = 4000\n',
encoding="utf-8",
)
for name in ("hard_negatives.jsonl", "counterexamples.jsonl"):
if not (directory / name).exists():
(directory / name).write_text("", encoding="utf-8")
written[family.id] = directory
return written
def _question(region: Region, config: FamilyConfig) -> str:
where = f"{region.path}::{region.function}" if region.function else region.path
parts = [f"Look for {config.family} problems in {where}. The whole file follows."]
if config.checklist:
parts.append(f"Evidence checklist:\n{config.checklist}")
if config.hard_negatives:
parts.append("Known non-issues, do not report these:\n" + "\n".join(_line(item) for item in config.hard_negatives))
if config.counterexamples:
parts.append("Worked examples:\n" + "\n".join(_line(item) for item in config.counterexamples))
parts.append("Reply with a JSON array of objects with path, line, title and rationale, after using a tool.")
return "\n\n".join(parts)
def _line(item: Mapping[str, object]) -> str:
return "- " + "; ".join(f"{key}: {value}" for key, value in sorted(item.items()))
def _read_optional(path: Path) -> str:
return path.read_text(encoding="utf-8").strip() if path.is_file() else ""
def _read_jsonl(path: Path) -> tuple[Mapping[str, object], ...]:
if not path.is_file():
return ()
rows: list[Mapping[str, object]] = []
for line in path.read_text(encoding="utf-8").splitlines():
if not line.strip():
continue
try:
payload = json.loads(line)
except json.JSONDecodeError as exc:
raise DetectorConfigError(f"{path}: not valid JSON lines: {exc}") from exc
if isinstance(payload, dict):
rows.append(payload)
return tuple(rows)
__all__ = [
"CURATED_TOOLS",
"GENERALIST",
"MAX_PROMPT_CHARS",
"DetectorConfigError",
"FamilyConfig",
"Region",
"load_family_configs",
"run_family_detector",
"run_region_detectors",
"write_default_configs",
]
src/openultrasast/learning/difficulty.py
-139
View File
@@ -1,139 +0,0 @@
"""How hard a piece of code makes an absence bug to see (learning-harness, Req 10.4).
A unit fixture that is easier than the corpus is worse than no fixture: the checker passes on it, the number
looks like capability, and the corpus rows it claims to cover keep failing for reasons nobody is looking at. The
three dimensions below are the ones the corpus actually varies on, measured on the vulnerable side:
- ``binding_hops`` — how many assignments sit between the identity source and the constrained use. `filter_by(
user_id=request.user.id)` is nothing to trace; `resp = token_validator(...)` then `User.query.filter_by(
username=resp['sub'])` then `filter_by(user=user, ...)` is two.
- ``registration_distance`` — 0 when the route decorator sits on the handler, 1 when a statement elsewhere in the
same file registers it, 2 when the registration lives in another document entirely (connexion's `operationId`,
an express router). Two is where the entry-point mapper stops being able to name the handler at all.
- ``guard_distance`` — 0 when a decorator carries the guard, 1 when the guard is a call in the body, 2 when the
handler carries no visible guard. A decorator is a pattern; a call in the body has to be understood.
Higher is harder on every axis, so "strictly easier" is: no dimension harder, at least one easier.
"""
from __future__ import annotations
import ast
import re
from collections.abc import Sequence
from dataclasses import dataclass
IDENTITY_SOURCES = ("current_user", "request.user", "session", "g.user", "auth", "token", "jwt", "claims", "sub", "principal")
GUARD_TOKENS = ("login_required", "token_required", "requires_auth", "authenticate", "authorize", "token_validator", "current_user")
_ROUTE_DECORATOR = re.compile(r"\.(route|get|post|put|patch|delete)\b")
@dataclass(frozen=True)
class DifficultyVector:
"""One handler's difficulty. Higher is harder on every axis."""
function: str
binding_hops: int
registration_distance: int
guard_distance: int
def axes(self) -> tuple[int, int, int]:
return (self.binding_hops, self.registration_distance, self.guard_distance)
def to_dict(self) -> dict[str, object]:
return {
"function": self.function,
"binding_hops": self.binding_hops,
"registration_distance": self.registration_distance,
"guard_distance": self.guard_distance,
}
def difficulty_of(source: str, function: str, *, context: Sequence[str] = ()) -> DifficultyVector:
"""The vector for one Python handler. ``context`` is any document that registers it from outside the file."""
try:
tree = ast.parse(source)
except SyntaxError:
return DifficultyVector(function=function, binding_hops=0, registration_distance=2, guard_distance=2)
node = _function(tree, function)
if node is None:
return DifficultyVector(function=function, binding_hops=0, registration_distance=2, guard_distance=2)
decorators = [ast.unparse(item) for item in node.decorator_list] + _class_decorators(tree, node)
return DifficultyVector(
function=function,
binding_hops=_binding_hops(node),
registration_distance=_registration_distance(source, tree, function, decorators, context),
guard_distance=_guard_distance(node, decorators),
)
def hardest(vectors: Sequence[DifficultyVector]) -> tuple[int, int, int]:
"""The per-axis maximum over a set of handlers: what the set can be asked to cope with."""
if not vectors:
return (0, 0, 0)
return tuple(max(row.axes()[index] for row in vectors) for index in range(3)) # type: ignore[return-value]
def strictly_easier(fixture: Sequence[DifficultyVector], corpus: Sequence[DifficultyVector]) -> bool:
"""True when the fixture set reaches no further than the corpus on any axis and falls short on at least one.
Per axis and over the whole set, because a suite covers a bug shape with several handlers between them: one
fixture handler may carry the long binding chain and another the missing guard. What must not happen is a suite
whose hardest case on some axis is easier than the corpus rows its tests claim to cover."""
if not corpus:
return False
limit, reach = hardest(corpus), hardest(fixture)
return all(reach[index] <= limit[index] for index in range(3)) and any(reach[index] < limit[index] for index in range(3))
def _function(tree: ast.AST, name: str) -> ast.FunctionDef | ast.AsyncFunctionDef | None:
for node in ast.walk(tree):
if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) and node.name == name:
return node
return None
def _class_decorators(tree: ast.AST, node: ast.AST) -> list[str]:
for klass in ast.walk(tree):
if isinstance(klass, ast.ClassDef) and any(child is node for child in klass.body):
return [ast.unparse(item) for item in klass.decorator_list]
return []
def _binding_hops(node: ast.FunctionDef | ast.AsyncFunctionDef) -> int:
"""Assignments that carry the identity forward before it constrains anything."""
tainted: set[str] = set()
hops = 0
for statement in ast.walk(node):
if not isinstance(statement, ast.Assign):
continue
text = ast.unparse(statement.value)
if any(token in text for token in IDENTITY_SOURCES) or any(name in text for name in tainted):
hops += 1
for target in statement.targets:
tainted.update(part.id for part in ast.walk(target) if isinstance(part, ast.Name))
return hops
def _registration_distance(source: str, tree: ast.AST, function: str, decorators: Sequence[str], context: Sequence[str]) -> int:
if any(_ROUTE_DECORATOR.search(item) for item in decorators):
return 0
mention = re.compile(rf"(?<![\w.]){re.escape(function)}(?!\w)")
body = _function(tree, function)
span = range(getattr(body, "lineno", 0), getattr(body, "end_lineno", 0) + 1) if body is not None else range(0)
for number, line in enumerate(source.splitlines(), start=1):
if number not in span and mention.search(line) and not line.strip().startswith(("def ", "async def ", "#")):
return 1
if any(re.search(rf"(?<![\w]){re.escape(function)}(?!\w)", item) for item in context):
return 2
return 2 # nothing in reach registers it, which is at least as hard as a registration in another document
def _guard_distance(node: ast.FunctionDef | ast.AsyncFunctionDef, decorators: Sequence[str]) -> int:
if any(token in item.lower() for item in decorators for token in GUARD_TOKENS):
return 0
body = ast.unparse(node).lower()
return 1 if any(token in body for token in GUARD_TOKENS) else 2
__all__ = ["GUARD_TOKENS", "IDENTITY_SOURCES", "DifficultyVector", "difficulty_of", "hardest", "strictly_easier"]
src/openultrasast/learning/journal.py
-237
View File
@@ -1,237 +0,0 @@
"""The round journal, the per-family archive and the rejected buffer (learning-harness, Req 9.3, 9.6, 9.7, 11.3).
The journal is append-only, one JSON line per round, and a round number is never reused, so the record
of what was tried cannot be quietly rewritten by the thing that is trying. Every round writes a line
whether it was accepted or not: a rejection is evidence too, and the rejected hypotheses are what stop
the proposer from offering the same idea twice.
The archive keeps, per family, which configuration version wins which pair, rather than one champion.
Keeping only the best-overall candidate throws away a version that is the only one solving some pair,
which is how a search collapses onto one lineage. A later version takes a pair only by winning it.
Nothing is written without passing redaction, because a trajectory or a hypothesis may quote the code
it was reasoning about.
"""
from __future__ import annotations
import json
from collections.abc import Mapping, Sequence
from dataclasses import dataclass, field
from pathlib import Path
from ..redaction import redact_secrets
from .scoring import PairFamilyScore
Outcome = str # accepted | rejected | reverted_cost | refused_holdout
def _as_int(value: object) -> int:
return int(value) if isinstance(value, int | float | str) and not isinstance(value, bool) else 0
def _as_float(value: object) -> float:
return float(value) if isinstance(value, int | float | str) and not isinstance(value, bool) else 0.0
def _as_strings(value: object) -> tuple[str, ...]:
return tuple(str(item) for item in value) if isinstance(value, list | tuple) else ()
def _as_sweep(value: object) -> dict[str, int]:
return {str(key): _as_int(item) for key, item in value.items()} if isinstance(value, Mapping) else {}
class JournalError(ValueError):
"""Raised when a round would overwrite history."""
@dataclass(frozen=True)
class Attribution:
"""Whether the round moved what it said it would move, and what it moved by accident."""
flipped_predicted: int = 0
flipped_unpredicted: int = 0
precision: float = 0.0
def to_dict(self) -> dict[str, object]:
return {
"flipped_predicted": self.flipped_predicted,
"flipped_unpredicted": self.flipped_unpredicted,
"precision": self.precision,
}
@staticmethod
def from_dict(payload: Mapping[str, object]) -> Attribution:
return Attribution(
flipped_predicted=_as_int(payload.get("flipped_predicted")),
flipped_unpredicted=_as_int(payload.get("flipped_unpredicted")),
precision=_as_float(payload.get("precision")),
)
@dataclass(frozen=True)
class RoundRecord:
round: int
family: str
hypothesis: str
levers: tuple[str, ...]
predicted_affected: tuple[str, ...]
predicted_at_risk: tuple[str, ...]
outcome: Outcome
reason: str = ""
target_train_delta: float = 0.0
target_holdout_delta: float = 0.0
sweep: dict[str, int] = field(default_factory=dict) # family -> negative flips in the cross-family sweep
attribution: Attribution = Attribution()
cost_usd: float = 0.0
taxonomy_version: str = ""
config_version: str = ""
# Pairs that moved each way in each stage, so the record can say whether the decision was distinguishable
# from noise. The decision itself is Req 9.5's raw-delta rule; this is the evidence beside it.
train_better: int = 0
train_worse: int = 0
holdout_better: int = 0
holdout_worse: int = 0
def to_dict(self) -> dict[str, object]:
return {
"round": self.round,
"family": self.family,
"hypothesis": self.hypothesis,
"levers": list(self.levers),
"predicted_affected": list(self.predicted_affected),
"predicted_at_risk": list(self.predicted_at_risk),
"outcome": self.outcome,
"reason": self.reason,
"target_train_delta": self.target_train_delta,
"target_holdout_delta": self.target_holdout_delta,
"sweep": dict(sorted(self.sweep.items())),
"attribution": self.attribution.to_dict(),
"cost_usd": self.cost_usd,
"taxonomy_version": self.taxonomy_version,
"config_version": self.config_version,
**{f"train_{key}": value for key, value in self._reliability(self.train_better, self.train_worse).items()},
**{f"holdout_{key}": value for key, value in self._reliability(self.holdout_better, self.holdout_worse).items()},
}
@staticmethod
def _reliability(better: int, worse: int) -> dict[str, object]:
from .acceptance import reliability
result = reliability(better=better, worse=worse)
return {"better": result.better, "worse": result.worse, "p_value": result.p_value, "reliable": result.reliable}
@staticmethod
def from_dict(payload: Mapping[str, object]) -> RoundRecord:
attribution = payload.get("attribution")
return RoundRecord(
round=_as_int(payload.get("round")),
family=str(payload.get("family", "")),
hypothesis=str(payload.get("hypothesis", "")),
levers=_as_strings(payload.get("levers")),
predicted_affected=_as_strings(payload.get("predicted_affected")),
predicted_at_risk=_as_strings(payload.get("predicted_at_risk")),
outcome=str(payload.get("outcome", "")),
reason=str(payload.get("reason", "")),
target_train_delta=_as_float(payload.get("target_train_delta")),
target_holdout_delta=_as_float(payload.get("target_holdout_delta")),
sweep=_as_sweep(payload.get("sweep")),
attribution=Attribution.from_dict(attribution if isinstance(attribution, Mapping) else {}),
cost_usd=_as_float(payload.get("cost_usd")),
taxonomy_version=str(payload.get("taxonomy_version", "")),
config_version=str(payload.get("config_version", "")),
)
class LearningJournal:
"""Append-only history of rounds. What was tried, what it predicted, and what actually happened."""
def __init__(self, path: Path) -> None:
self.path = path
def append(self, record: RoundRecord) -> None:
existing = {item.round for item in self.rounds()}
if record.round in existing:
raise JournalError(f"round {record.round} is already journaled; history is append-only")
self.path.parent.mkdir(parents=True, exist_ok=True)
line = redact_secrets(json.dumps(record.to_dict(), sort_keys=True))
with self.path.open("a", encoding="utf-8") as handle:
handle.write(line + "\n")
def rounds(self) -> tuple[RoundRecord, ...]:
return tuple(RoundRecord.from_dict(payload) for payload in _read_jsonl(self.path))
def next_round(self) -> int:
rounds = self.rounds()
return max((item.round for item in rounds), default=0) + 1
def rejected_buffer(self, family: str) -> tuple[str, ...]:
"""Hypotheses this family already tried and lost, so the proposer is not handed the same idea twice."""
return tuple(item.hypothesis for item in self.rounds() if item.family == family and item.outcome != "accepted" and item.hypothesis)
class Archive:
"""Per family, which configuration version wins which pair. Never one champion."""
def __init__(self, path: Path) -> None:
self.path = path
def record(self, family: str, config_version: str, score: PairFamilyScore) -> None:
self.path.parent.mkdir(parents=True, exist_ok=True)
payload = {
"family": family,
"config_version": config_version,
"pair": score.pair,
"outcome": score.outcome,
"flips": score.flips,
}
with self.path.open("a", encoding="utf-8") as handle:
handle.write(redact_secrets(json.dumps(payload, sort_keys=True)) + "\n")
def winners(self, family: str) -> dict[str, str]:
"""pair -> the earliest configuration version that scored it correct.
Earliest, not latest: the point of the archive is that a version which solves a pair keeps its
claim to it, so a later version that loses the pair cannot quietly take it off the board.
"""
winners: dict[str, str] = {}
for payload in _read_jsonl(self.path):
if str(payload.get("family")) != family or str(payload.get("outcome")) != "pair_correct":
continue
winners.setdefault(str(payload.get("pair")), str(payload.get("config_version")))
return winners
def _read_jsonl(path: Path) -> list[Mapping[str, object]]:
if not path.is_file():
return []
rows: list[Mapping[str, object]] = []
for line in path.read_text(encoding="utf-8").splitlines():
if not line.strip():
continue
try:
payload = json.loads(line)
except json.JSONDecodeError:
continue
if isinstance(payload, dict):
rows.append(payload)
return rows
def summarize(records: Sequence[RoundRecord]) -> dict[str, object]:
"""What the rounds add up to: how often a prediction landed, and what it cost."""
accepted = [item for item in records if item.outcome == "accepted"]
predicted = sum(item.attribution.flipped_predicted for item in records)
unpredicted = sum(item.attribution.flipped_unpredicted for item in records)
return {
"rounds": len(records),
"accepted": len(accepted),
"flipped_predicted": predicted,
"flipped_unpredicted": unpredicted,
"prediction_precision": predicted / (predicted + unpredicted) if predicted + unpredicted else 0.0,
"cost_usd": round(sum(item.cost_usd for item in records), 4),
}
__all__ = ["Archive", "Attribution", "JournalError", "LearningJournal", "Outcome", "RoundRecord", "summarize"]
src/openultrasast/learning/proposer.py
-318
View File
@@ -1,318 +0,0 @@
"""What a round proposes, and who may propose it (learning-harness, Req 9.1, 9.2, 5.4, 7.6).
A proposer is handed facts, never scores it is about to be judged on: the misses and leaks of one family
on the train split, what each run actually did, and the hypotheses this family already tried and lost.
Self-diagnosis from free narrative confabulates; structured facts are what make a proposal answerable.
A proposal is one hypothesis, one lever, and one file inside one family's directory. The narrowness is
the safety property: a round can only touch what it targets, so every other family is frozen by
construction rather than by promise, and the verifier code and the corpus stay outside every write root.
"""
from __future__ import annotations
import json
import shutil
import tempfile
from collections.abc import Callable, Mapping, Sequence
from dataclasses import dataclass, field
from pathlib import Path
from typing import Literal, Protocol
from ..harness_ext import has_harnessx
from .detectors import MAX_PROMPT_CHARS, FamilyConfig
from .journal import LearningJournal
from .scoring import PairFamilyScore
Lever = Literal["tool", "checklist", "counterexample", "memory", "prompt"]
LEVERS: tuple[Lever, ...] = ("tool", "checklist", "counterexample", "memory", "prompt")
# The only files a round may write, one per lever. Anything else is a proposal reaching beyond its family.
EDITABLE_FILES: dict[Lever, str] = {
"tool": "family.toml",
"checklist": "checklist.md",
"counterexample": "counterexamples.jsonl",
"memory": "hard_negatives.jsonl",
"prompt": "prompt.md",
}
@dataclass(frozen=True)
class PairFact:
"""What one pair did, in terms a proposer can act on."""
pair: str
outcome: str
runs: tuple[str, ...] = ()
other_findings: int = 0
rung: str = "suspicion"
def to_dict(self) -> dict[str, object]:
return {
"pair": self.pair,
"outcome": self.outcome,
"runs": list(self.runs),
"other_findings": self.other_findings,
"rung": self.rung,
}
@dataclass(frozen=True)
class FailureFacts:
family: str
config: FamilyConfig
misses: tuple[PairFact, ...] = ()
leaks: tuple[PairFact, ...] = ()
rejected: tuple[str, ...] = ()
def to_dict(self) -> dict[str, object]:
return {
"family": self.family,
"config_version": self.config.version,
"tools": list(self.config.tools),
"misses": [item.to_dict() for item in self.misses],
"leaks": [item.to_dict() for item in self.leaks],
"rejected": list(self.rejected),
}
@dataclass(frozen=True)
class Proposal:
hypothesis: str
lever: Lever
change: Mapping[str, str] # file name inside the family directory -> its new contents
predicted_affected: tuple[str, ...] = ()
predicted_at_risk: tuple[str, ...] = ()
def to_dict(self) -> dict[str, object]:
return {
"hypothesis": self.hypothesis,
"lever": self.lever,
"change": dict(self.change),
"predicted_affected": list(self.predicted_affected),
"predicted_at_risk": list(self.predicted_at_risk),
}
class Proposer(Protocol):
def propose(self, facts: FailureFacts) -> Proposal | None: ...
def build_failure_facts(
scores: Sequence[PairFamilyScore],
*,
family: str,
config: FamilyConfig,
journal: LearningJournal,
train_pairs: set[str],
) -> FailureFacts:
"""The facts of one family's failures on the train split. A holdout pair never appears here (Req 5.4)."""
mine = [item for item in scores if item.family == family and item.pair in train_pairs and item.outcome != "unscorable"]
return FailureFacts(
family=family,
config=config,
misses=tuple(_fact(item, item.other_findings_vuln) for item in mine if item.outcome in {"both_silent", "reversed"}),
leaks=tuple(_fact(item, item.other_findings_fixed) for item in mine if item.outcome in {"both_flagged", "reversed"}),
rejected=journal.rejected_buffer(family),
)
def refuse_proposal(proposal: Proposal, config: FamilyConfig) -> str | None:
"""None when the proposal is one legal change to its own family; otherwise why it is refused."""
if proposal.lever not in LEVERS:
return f"unknown lever {proposal.lever!r}; expected one of {', '.join(LEVERS)}"
if len(proposal.change) != 1:
return f"a round changes one file, not {len(proposal.change)}"
name, content = next(iter(proposal.change.items()))
if "/" in name or "\\" in name or name.startswith("."):
return f"{name!r} reaches outside the family directory"
if name != EDITABLE_FILES[proposal.lever]:
return f"{name!r} is not the file the {proposal.lever!r} lever writes ({EDITABLE_FILES[proposal.lever]})"
if name == "prompt.md" and len(content) + len(config.checklist) > MAX_PROMPT_CHARS:
return f"the proposed prompt exceeds the cap of {MAX_PROMPT_CHARS} characters"
if name == "checklist.md" and len(content) + len(config.prompt) > MAX_PROMPT_CHARS:
return f"the proposed checklist exceeds the cap of {MAX_PROMPT_CHARS} characters"
return None
def apply_proposal(proposal: Proposal, directory: Path) -> None:
"""Write the one file, inside the one directory. Callers snapshot first so a rejection can revert."""
directory.mkdir(parents=True, exist_ok=True)
for name, content in proposal.change.items():
(directory / name).write_text(content, encoding="utf-8")
def write_roots(configs_dir: Path, family: str) -> tuple[Path, ...]:
"""Everything a round may write: exactly one family's directory, and never the verifiers or the corpus."""
return (configs_dir / family,)
@dataclass
class ScriptedProposer:
"""A fixed sequence of proposals, for tests and for the first offline rounds."""
proposals: list[Proposal] = field(default_factory=list)
used: int = 0
def propose(self, facts: FailureFacts) -> Proposal | None:
del facts
if self.used >= len(self.proposals):
return None
self.used += 1
return self.proposals[self.used - 1]
MetaAgentCall = Callable[..., str]
@dataclass
class HarnessXProposer:
"""The meta-agent, with write access limited to a copy of the family directory it is targeting.
It never edits the live tree. The round snapshots the family directory *after* a proposal exists, so a proposer
that wrote in place would have changed the configuration before there was anything to restore, and a rejection
could not put it back. The agent works in a scratch copy and the proposal is the diff of that copy, which also
means the one-file rule and the lever whitelist are enforced on what the agent actually did rather than on what
it claimed (Req 9.1, 9.2, 5.4).
"""
configs_dir: Path
model: str
# The vendor the meta-agent talks to. Hard-coding anthropic here sent it to a provider with no key configured
# while the detector was talking to DeepSeek, and the round journalled that as "no proposal".
provider: str = "anthropic"
reason: str = ""
agent: MetaAgentCall | None = None # injected in tests; None composes MetaAgent behind harness_ext
def write_roots(self, family: str) -> tuple[Path, ...]:
return write_roots(self.configs_dir, family)
def propose(self, facts: FailureFacts) -> Proposal | None:
call = self.agent
if call is None:
if not has_harnessx():
self.reason = "harnessx_unavailable"
return None
try:
call = _meta_agent_call(self.provider)
except Exception: # noqa: BLE001 — a proposer that cannot start proposes nothing and says so
self.reason = "harnessx_unavailable"
return None
family_dir = self.configs_dir / facts.family
with tempfile.TemporaryDirectory(prefix="ousast-propose-") as scratch:
workspace = Path(scratch) / facts.family
shutil.copytree(family_dir, workspace)
before = _directory_text(workspace)
try:
hypothesis = call(workspace=workspace, facts=facts, model=self.model)
except (TypeError, AttributeError, ImportError):
# "we called the agent wrong" is not "the agent declined", and folding one into the other is how a
# broken composition reads as a quiet round for as long as nobody looks.
raise
except Exception as exc: # noqa: BLE001 — an agent that failed proposed nothing, and the round says why
self.reason = f"meta_agent_failed: {type(exc).__name__}"
return None
after = _directory_text(workspace)
changed = sorted(name for name in set(before) | set(after) if before.get(name) != after.get(name))
if not changed:
self.reason = "no_change_proposed"
return None
if len(changed) > 1:
self.reason = f"a round changes one file, not {len(changed)}: {', '.join(changed)}"
return None
name = changed[0]
lever: Lever | None = next((key for key, filename in EDITABLE_FILES.items() if filename == name), None)
if lever is None or name not in after:
self.reason = f"{name!r} is not a file any lever writes"
return None
self.reason = ""
return Proposal(
hypothesis=str(hypothesis or "").strip() or f"meta-agent edit to {name}",
lever=lever,
change={name: after[name]},
predicted_affected=tuple(item.pair for item in facts.misses),
predicted_at_risk=tuple(item.pair for item in facts.leaks),
)
def _directory_text(directory: Path) -> dict[str, str]:
"""Every readable file directly inside ``directory``, by name. Nested paths are not a lever's business."""
out: dict[str, str] = {}
for path in sorted(directory.iterdir()) if directory.is_dir() else []:
if path.is_file():
try:
out[path.name] = path.read_text(encoding="utf-8")
except (OSError, UnicodeDecodeError):
continue
return out
def _meta_agent_class() -> type:
"""`MetaAgent` behind `harness_ext`. A seam, so the argument contract can be pinned without the extra."""
from ..harness_ext import require_harnessx
require_harnessx()
from harnessx.meta_harness.agent import MetaAgent # type: ignore[import-not-found]
return MetaAgent # type: ignore[no-any-return]
def _model_config(model: str, provider: str = "anthropic") -> object:
"""`MetaAgent(inner_model=...)` takes a `ModelConfig`, never a model name; the name goes to the provider."""
from ..harness_ext import build_provider, require_harnessx
require_harnessx()
from harnessx.core.model_config import ModelConfig # type: ignore[import-not-found]
return ModelConfig(main=build_provider(model, provider))
def _meta_agent_call(provider: str = "anthropic") -> MetaAgentCall:
"""Compose `MetaAgent.evolve` behind `harness_ext`, writing only inside the scratch copy.
`evolve` is a coroutine and returns a HarnessX config path; what this proposer takes from it is the state of the
workspace afterwards, so the return value is only used for the hypothesis line.
"""
from ..harness_ext import require_harnessx
require_harnessx()
def call(*, workspace: Path, facts: FailureFacts, model: str) -> str:
import asyncio
trajectories = workspace.parent / "trajectories"
trajectories.mkdir(parents=True, exist_ok=True)
(trajectories / "facts.json").write_text(facts_prompt(facts), encoding="utf-8")
agent = _meta_agent_class()(inner_model=_model_config(model, provider), allowed_write_roots=(workspace,))
output = workspace.parent / "meta-out"
output.mkdir(parents=True, exist_ok=True)
asyncio.run(agent.evolve(current_config=workspace, trajectories_dir=trajectories, output_dir=output))
return f"meta-agent round for {facts.family}"
return call
def _fact(score: PairFamilyScore, other: int) -> PairFact:
return PairFact(pair=score.pair, outcome=score.outcome, runs=score.runs, other_findings=other, rung=score.rung)
def facts_prompt(facts: FailureFacts) -> str:
"""The facts as the proposer sees them: structured, and free of anything it will be judged on."""
return json.dumps(facts.to_dict(), indent=2, sort_keys=True)
__all__ = [
"EDITABLE_FILES",
"LEVERS",
"FailureFacts",
"HarnessXProposer",
"Lever",
"PairFact",
"Proposal",
"Proposer",
"ScriptedProposer",
"apply_proposal",
"build_failure_facts",
"facts_prompt",
"refuse_proposal",
"write_roots",
]
src/openultrasast/learning/publish.py
-263
View File
@@ -1,263 +0,0 @@
"""Every published number, regenerated from the artifacts that produced it (learning-harness, Req 11.1, 11.2, 4.5, 5.3).
A number in a roadmap that nobody can regenerate is a claim, not a measurement. This writes them all
from one entry point, out of the artifacts committed beside them, and puts the denominator next to each:
how many pairs could be scored, why the rest could not, which taxonomy version scored them, the noise
the family shows against itself, and what the run cost per pair it got right.
Running it twice without new artifacts changes nothing, and the roadmap section is replaced in place
rather than appended, so the file cannot accumulate stale numbers beside fresh ones.
"""
from __future__ import annotations
import json
import re
from collections.abc import Mapping
from dataclasses import dataclass, field
from datetime import UTC, datetime
from pathlib import Path
BEGIN = "<!-- learning-harness:begin -->"
END = "<!-- learning-harness:end -->"
COMMAND = "uv run ousast learning publish"
@dataclass(frozen=True)
class Baseline:
"""One model's round-zero artifact, read back."""
model: str
slices: tuple[str, ...] = ()
metrics: dict[str, Mapping[str, object]] = field(default_factory=dict)
floors: dict[str, Mapping[str, object]] = field(default_factory=dict)
cost_usd: float = 0.0
k_runs: int = 0
@dataclass(frozen=True)
class PublishReport:
command: str
table: str
families: tuple[str, ...] = ()
models: tuple[str, ...] = ()
artifacts: tuple[Path, ...] = field(default_factory=tuple)
def publish(*, learning_dir: Path, measurements_dir: Path, roadmap: Path, date: str | None = None) -> PublishReport:
"""Regenerate the family numbers, write the artifact, and replace the roadmap section in place."""
stamp = date or datetime.now(UTC).strftime("%Y-%m-%d")
models = _baselines(learning_dir)
families = tuple(sorted({family for baseline in models.values() for family in baseline.metrics}))
lever = _lever(measurements_dir)
table = _table(models, families, lever)
artifacts: list[Path] = []
if models:
measurements_dir.mkdir(parents=True, exist_ok=True)
artifact = measurements_dir / f"{stamp}-learning-families.json"
artifact.write_text(
json.dumps(
{
"generated": stamp,
"command": COMMAND,
"models": {name: dict(baseline.metrics) for name, baseline in sorted(models.items())},
"floors": {name: dict(baseline.floors) for name, baseline in sorted(models.items())},
},
indent=2,
sort_keys=True,
)
+ "\n",
encoding="utf-8",
)
artifacts.append(artifact)
_replace_section(roadmap, table)
_supersede(roadmap)
return PublishReport(command=COMMAND, table=table, families=families, models=tuple(sorted(models)), artifacts=tuple(artifacts))
def _label(model: str, slices: tuple[str, ...]) -> str:
"""What identifies a run in the published table: the model, and the slice when it is not the default pair."""
return f"{model} ({', '.join(slices)})" if slices else model
def _baselines(learning_dir: Path) -> dict[str, Baseline]:
root = learning_dir / "baseline"
found: dict[str, Baseline] = {}
for directory in sorted(path for path in root.iterdir() if path.is_dir()) if root.is_dir() else []:
report = directory / "report.json"
if not report.is_file():
continue
payload = json.loads(report.read_text(encoding="utf-8"))
# Keyed by model *and* slice. Keyed by model alone, a second slice silently replaced the first and the
# published table showed one of them as if it were the corpus (Req 3.1, 3.2).
model = str(payload.get("model", directory.name))
slices = tuple(str(item) for item in (payload.get("slices") or ()))
found[_label(model, slices)] = Baseline(
model=model,
slices=slices,
metrics={str(name): block for name, block in (payload.get("metrics") or {}).items() if isinstance(block, Mapping)},
floors={str(name): block for name, block in (payload.get("floors") or {}).items() if isinstance(block, Mapping)},
cost_usd=float(payload.get("cost_usd", 0.0) or 0.0),
k_runs=int(payload.get("k_runs", 0) or 0),
)
return found
def _lever(measurements_dir: Path) -> Mapping[str, object] | None:
path = measurements_dir / "2026-09-06-mechanism-lever-split.json"
if not path.is_file():
return None
payload = json.loads(path.read_text(encoding="utf-8"))
return payload if isinstance(payload, dict) else None
def _table(models: Mapping[str, Baseline], families: tuple[str, ...], lever: Mapping[str, object] | None) -> str:
if not models:
return (
f"Generated by `{COMMAND}`.\n\nNo baseline has been recorded yet, so there is nothing to publish. "
"Run round zero first; a number without the run that produced it is a claim, not a measurement.\n"
)
lines = [
f"Generated by `{COMMAND}` from the artifacts committed beside these numbers.",
"",
(
"| model | slice | family | scorable | unscorable | recall | silence | Youden "
"| at fixed FPR | noise floor | cost per correct pair |"
),
"|---|---|---|---|---|---|---|---|---|---|---|",
]
for _label_key, baseline in sorted(models.items()):
model = baseline.model
slice_name = ", ".join(baseline.slices) or "all"
for family in families:
row = baseline.metrics.get(family)
if row is None:
continue
outcomes = row.get("outcomes")
raw_correct = outcomes.get("pair_correct") if isinstance(outcomes, Mapping) else None
correct = int(raw_correct) if isinstance(raw_correct, int | float) else 0
floor = baseline.floors.get(family)
raw_budget = floor.get("budget_flips") if floor is not None else None
budget = int(raw_budget) if isinstance(raw_budget, int | float) else 0
cost = baseline.cost_usd
lines.append(
f"| {model} | {slice_name} | {family} | {row.get('scorable', 0)} (taxonomy {row.get('taxonomy_version', '')}) "
f"| {_reasons(row)} | {_num(row.get('recall'))} | {_num(row.get('silence'))} | {_num(row.get('youden'))} "
f"| {_num(row.get('fixed_fpr_recall'))} | budget {budget} | {_cost(cost, correct)} |"
)
lines += ["", _credit_note(models), ""]
cutoff = _cutoff_note(models, families)
if cutoff:
lines += [cutoff, ""]
if lever is not None:
lines += [_lever_note(lever), ""]
return "\n".join(lines)
def _cutoff_note(models: Mapping[str, Baseline], families: tuple[str, ...]) -> str:
"""The post-cutoff slice, its cutoff and the undated count (Req 10.5).
A model that may have read the fix cannot be measured on it the same way as one that cannot have, so the rows
fixed after the detector's training cutoff are reported apart. Rows the corpus records no date for are in
neither slice, and their number is here rather than folded silently into the whole."""
rows = [
(model, family, baseline.metrics[family])
for model, baseline in sorted(models.items())
for family in families
if family in baseline.metrics and str(baseline.metrics[family].get("cutoff") or "")
]
if not rows:
return "No detector cutoff is configured, so no post-cutoff slice is reported; every figure above is over the whole corpus."
cutoff = str(rows[0][2].get("cutoff"))
lines = [
f"Post-cutoff slice, fixed after {cutoff} and therefore beyond the detector's training data. "
"A row counts only when its earliest possible date is after the cutoff, so a year-precision date can "
"under-claim this slice but never over-claim it.",
"",
"| model | family | post-cutoff scorable | post-cutoff recall | post-cutoff Youden | undated |",
"|---|---|---|---|---|---|",
]
for model, family, row in rows:
block = row.get("post_cutoff")
block = block if isinstance(block, Mapping) else {}
lines.append(
f"| {model} | {family} | {block.get('scorable', 0)} | {_num(block.get('recall'))} "
f"| {_num(block.get('youden'))} | {row.get('undated', 0)} |"
)
return "\n".join(lines)
def _credit_note(models: Mapping[str, Baseline]) -> str:
hierarchical = any(bool(row.get("hierarchical_credit")) for baseline in models.values() for row in baseline.metrics.values())
if hierarchical:
return "A parent-child answer earned partial credit in these figures."
return "No family in this taxonomy declares a parent, so hierarchical partial credit could not apply to any figure above."
def _lever_note(lever: Mapping[str, object]) -> str:
before = lever.get("before")
after = lever.get("after")
before_map = before if isinstance(before, Mapping) else {}
after_map = after if isinstance(after, Mapping) else {}
pairs = lever.get("holdout_pairs", 0)
return (
f"Mechanism lever on the {lever.get('slice', '')} holdout: detected {after_map.get('detected')} of {pairs} at Youden "
f"{after_map.get('youden')} once only train-split pairs teach, against detected {before_map.get('detected')} at Youden "
f"{before_map.get('youden')} when every pair taught. The difference is what holdout pairs stopped teaching."
)
def _reasons(row: Mapping[str, object]) -> str:
reasons = row.get("unscorable") or {}
if not isinstance(reasons, Mapping) or not reasons:
return "none"
return ", ".join(f"{name}: {count}" for name, count in sorted(reasons.items()))
def _num(value: object) -> str:
return f"{float(value):.3f}" if isinstance(value, int | float) else "-"
def _cost(cost: float, correct: int) -> str:
return f"{cost / correct:.2f}" if cost and correct else "-"
# Claims the split correction supersedes, as they were written before holdout pairs stopped teaching. Req 5.3 says
# the corrected number is published *in place of* the earlier one; a document that carries both asserts two numbers
# for one measurement and the reader has no way to tell which run is current.
_SUPERSEDED = (
re.compile(r"human holdout profile pair_correct \d+ -> \d+ of \d+, Youden [-+]?\d+ -> [-+]?[\d.]+(?:, round \d+ `[^`]+`)?\."),
re.compile(r"The lever recovered \d+ of \d+ holdout pairs at Youden [-+][\d.]+\."),
)
_SUPERSEDED_NOTE = (
"superseded by the learning-harness section below: that number was measured with holdout pairs teaching "
"(see `benchmarks/measurements/2026-09-06-mechanism-lever-split.json`)."
)
def _supersede(roadmap: Path) -> None:
"""Rewrite every superseded lever claim to point at the regenerated section. Idempotent: the pattern is gone
afterwards, so a second run changes nothing."""
if not roadmap.is_file():
return
text = roadmap.read_text(encoding="utf-8")
updated = text
for pattern in _SUPERSEDED:
updated = pattern.sub(_SUPERSEDED_NOTE, updated)
if updated != text:
roadmap.write_text(updated, encoding="utf-8")
def _replace_section(roadmap: Path, table: str) -> None:
body = f"{BEGIN}\n\n## Classified detector numbers\n\n{table.rstrip()}\n\n{END}\n"
text = roadmap.read_text(encoding="utf-8") if roadmap.is_file() else ""
if BEGIN in text and END in text:
head, _, rest = text.partition(BEGIN)
_, _, tail = rest.partition(END)
roadmap.write_text(head + body + tail.lstrip("\n"), encoding="utf-8")
return
roadmap.parent.mkdir(parents=True, exist_ok=True)
roadmap.write_text((text.rstrip("\n") + "\n\n" if text else "") + body, encoding="utf-8")
__all__ = ["BEGIN", "COMMAND", "END", "Baseline", "PublishReport", "publish"]
src/openultrasast/learning/rounds.py
-761
View File
@@ -1,761 +0,0 @@
"""Round zero and the per-family noise floor (learning-harness, Req 8).
Before anything learns, every family gets the same detector and is measured against itself. Running a
pair several times at the same settings does not give the same answer: the chat provider offers no seed,
so the spread between runs is real and unavoidable. That spread is the family's noise floor, and it
becomes the budget a later round must stay inside before a cross-family change counts as a regression.
Choosing a budget instead of measuring one would be a guess dressed as a threshold.
Artifacts are keyed by detector model, so a second model is baselined by changing the model and running
again. Configurations are never touched by a model swap, which is what makes the two numbers comparable
and what makes reconsidering the model cheap.
"""
from __future__ import annotations
import json
import math
import re
import tempfile
from collections.abc import Callable, Mapping, Sequence
from dataclasses import dataclass, field
from pathlib import Path
from ..findings import StaticFinding
from ..pairs import PairCase
from .classify import classify_pair
from .detectors import FamilyConfig, write_default_configs
from .families import FamilyTaxonomy
from .journal import Archive, Attribution, LearningJournal, RoundRecord
from .proposer import Proposal, Proposer
from .scoring import FamilyMetrics, PairFamilyScore, aggregate, score_pair_family
DEFAULT_SLICES = ("vibe-py", "agent-vfc") # the web families, where real-world code and dangerous fruit coincide
NON_GATING_FAMILIES = frozenset({"memory"}) # measured on the vendored slice, never allowed to fail a round
DEFAULT_K_RUNS = 5
Scan = Callable[[Path], list[StaticFinding]]
@dataclass(frozen=True)
class NoiseFloor:
"""How much a family disagrees with itself between runs, and the budget that follows from it."""
family: str
k_runs: int
pairs: int
flaky_pairs: int
negative_flip_rate: float
budget_flips: int
gates: bool = True
def to_dict(self) -> dict[str, object]:
return {
"family": self.family,
"k_runs": self.k_runs,
"pairs": self.pairs,
"flaky_pairs": self.flaky_pairs,
"negative_flip_rate": self.negative_flip_rate,
"budget_flips": self.budget_flips,
"gates": self.gates,
}
@dataclass(frozen=True)
class BaselineReport:
model: str
k_runs: int
taxonomy_version: str
slices: tuple[str, ...]
thinking: bool = False # the condition that produced these numbers, so two modes cannot be read as one
floors: dict[str, NoiseFloor] = field(default_factory=dict)
metrics: dict[str, dict[str, object]] = field(default_factory=dict)
# `<slice>/<family>`: the budgets stay per family, but vibe-py and agent-vfc are different worlds and one
# pooled number hides which of them moved (Req 8.4).
per_slice: dict[str, dict[str, object]] = field(default_factory=dict)
scores: tuple[PairFamilyScore, ...] = ()
cost_usd: float = 0.0
def to_dict(self) -> dict[str, object]:
return {
"model": self.model,
"k_runs": self.k_runs,
"taxonomy_version": self.taxonomy_version,
"slices": list(self.slices),
"thinking": self.thinking,
"cost_usd": self.cost_usd,
"floors": {name: floor.to_dict() for name, floor in sorted(self.floors.items())},
"metrics": {name: dict(block) for name, block in sorted(self.metrics.items())},
"per_slice": {name: dict(block) for name, block in sorted(self.per_slice.items())},
}
def run_baseline(
cases: Sequence[PairCase],
*,
taxonomy: FamilyTaxonomy,
configs_dir: Path,
scan_factory: Callable[[FamilyConfig], Scan],
model: str,
out_dir: Path,
slices: Sequence[str] = DEFAULT_SLICES,
k_runs: int = DEFAULT_K_RUNS,
prompt: str | None = None,
spent_usd: Callable[[], float] | None = None,
cutoff: str = "",
thinking: bool = False,
resume: bool = False,
) -> BaselineReport:
"""Clone one detector into every family, measure each family **with its own configuration**, write the artifacts.
The factory is the point. One shared `scan` would score every family with one detector, and since the generalist
tags its findings `family:unknown` — which earns no credit against any real family, by the taxonomy's own
decision that abstention is not a partial answer — every family but `unknown` would baseline at a structural
zero and the noise floors would all be nil (Req 8.2).
"""
from ..tool_hunter import _SYSTEM_PROMPT
from .detectors import load_family_configs
if k_runs < MIN_K_RUNS:
raise ValueError(f"round zero scores at least three runs per pair; {k_runs} was asked for")
if not configs_dir.exists():
write_default_configs(configs_dir, taxonomy, prompt=prompt or _SYSTEM_PROMPT, version="0")
configs = load_family_configs(configs_dir, taxonomy)
selected = [case for case in cases if case.slice in set(slices)]
# Every pair is written down as it is measured. Round zero over the vfc slice is 1760 detector runs and hours of
# paid calls; accumulating in memory and persisting once at the end means a timeout, a kill or a machine going
# away costs all of it. An exception already cost only one pair; nothing else did.
checkpoint = out_dir / "baseline" / _artifact_key(model, slices, thinking) / "scores.jsonl"
done = _checkpointed(checkpoint) if resume else {}
if not resume and checkpoint.exists():
checkpoint.unlink()
scores = []
for case in selected:
name = _family_of(case, taxonomy)
config = configs.get(name)
if config is None:
continue # a family with no configuration is not measured rather than measured with someone else's
if case.name in done:
scores.append(done[case.name])
continue
try:
score = score_case(case, scan_factory(config), taxonomy=taxonomy, runs=k_runs, family=name)
scores.append(score)
_append_score(checkpoint, score)
except Exception: # noqa: BLE001 — a provider hiccup costs one pair, not the whole run
# Round zero is the most expensive thing this harness does. Raising here threw away everything already
# measured and paid for, so the only honest response to a flaky endpoint was to run it all again.
unreachable = PairFamilyScore(
pair=case.name,
family=name,
slice=case.slice,
fix_date=case.fix_date,
runs=(),
outcome="unscorable",
unscorable_reason="detector_unreachable",
)
scores.append(unreachable)
_append_score(checkpoint, unreachable)
metrics = aggregate(scores, taxonomy=taxonomy, cutoff=cutoff)
floors = {name: _floor(name, block, [item for item in scores if item.family == name], k_runs) for name, block in metrics.items()}
report = BaselineReport(
model=model,
k_runs=k_runs,
taxonomy_version=taxonomy.version,
slices=tuple(slices),
thinking=thinking,
floors=floors,
metrics={name: block.to_dict() for name, block in metrics.items()},
per_slice={name: block.to_dict() for name, block in aggregate(scores, taxonomy=taxonomy, per_slice=True, cutoff=cutoff).items()},
scores=tuple(scores),
cost_usd=spent_usd() if spent_usd is not None else 0.0,
)
_write(out_dir, report)
return report
def _append_score(path: Path, score: PairFamilyScore) -> None:
"""One line per pair, flushed, so an interrupted run keeps what it measured."""
path.parent.mkdir(parents=True, exist_ok=True)
payload = {
"pair": score.pair,
"family": score.family,
"slice": score.slice,
"fix_date": score.fix_date,
"outcome": score.outcome,
"runs": list(score.runs),
"flips": score.flips,
"other_findings_vuln": score.other_findings_vuln,
"other_findings_fixed": score.other_findings_fixed,
"fabricated_vuln": score.fabricated_vuln,
"fabricated_fixed": score.fabricated_fixed,
"rung": score.rung,
"unscorable_reason": score.unscorable_reason,
}
with path.open("a", encoding="utf-8") as handle:
handle.write(json.dumps(payload, sort_keys=True) + "\n")
handle.flush()
def _checkpointed(path: Path) -> dict[str, PairFamilyScore]:
"""What a previous run of this exact baseline already measured and paid for."""
if not path.is_file():
return {}
out: dict[str, PairFamilyScore] = {}
for line in path.read_text(encoding="utf-8").splitlines():
if not line.strip():
continue
try:
row = json.loads(line)
except json.JSONDecodeError:
continue # a line half-written when the run died is not a measurement
out[str(row["pair"])] = PairFamilyScore(
pair=str(row["pair"]),
family=str(row["family"]),
slice=str(row.get("slice", "")),
fix_date=str(row.get("fix_date", "")),
runs=tuple(row.get("runs", ())),
outcome=row["outcome"],
flips=int(row.get("flips", 0)),
other_findings_vuln=int(row.get("other_findings_vuln", 0)),
other_findings_fixed=int(row.get("other_findings_fixed", 0)),
fabricated_vuln=int(row.get("fabricated_vuln", 0)),
fabricated_fixed=int(row.get("fabricated_fixed", 0)),
rung=row.get("rung", "suspicion"),
unscorable_reason=row.get("unscorable_reason"),
)
return out
def score_case(case: PairCase, scan: Scan, *, taxonomy: FamilyTaxonomy, runs: int, family: str | None = None) -> PairFamilyScore:
"""Run one detector over both sides of one pair ``runs`` times and score it for its family.
A pair the corpus already marked unscorable is never run: spending model calls on a row that cannot
be scored buys nothing and would put its noise into a floor it does not belong in.
"""
from ..pairs import _materialize_side, _overlay_scan
name = family or _family_of(case, taxonomy)
if case.unscorable:
return PairFamilyScore(
pair=case.name,
family=name,
runs=(),
outcome="unscorable",
slice=case.slice,
fix_date=case.fix_date,
unscorable_reason=case.unscorable,
)
with tempfile.TemporaryDirectory(prefix="ousast-round-") as scratch:
vuln_root = _materialize_side(Path(scratch) / "vuln", case, side="vuln")
fix_root = _materialize_side(Path(scratch) / "fixed", case, side="fixed")
ranges = _overlay_scan(vuln_root).ranges
fixed_ranges = _overlay_scan(fix_root).ranges
vuln_runs = [scan(vuln_root) for _ in range(max(runs, 1))]
fix_runs = [scan(fix_root) for _ in range(max(runs, 1))]
return score_pair_family(
case, name, vuln_runs, fix_runs, ranges=ranges, taxonomy=taxonomy, parse_ok=bool(ranges), fixed_ranges=fixed_ranges
)
def compare_baselines(out_dir: Path) -> dict[str, dict[str, dict[str, object]]]:
"""Every baseline written under ``out_dir``, model by model, so two detectors can be read side by side."""
root = out_dir / "baseline"
table: dict[str, dict[str, dict[str, object]]] = {}
for directory in sorted(path for path in root.iterdir() if path.is_dir()) if root.is_dir() else []:
report = directory / "report.json"
if not report.is_file():
continue
payload = json.loads(report.read_text(encoding="utf-8"))
model = str(payload.get("model", directory.name))
slices = tuple(str(item) for item in (payload.get("slices") or ()))
conditions = [*(() if not payload.get("thinking") else ("thinking",)), *(() if slices == DEFAULT_SLICES else slices)]
label = f"{model} ({', '.join(conditions)})" if conditions else model
table[label] = {family: dict(block) for family, block in (payload.get("metrics") or {}).items()}
return table
def _floor(family: str, metrics: FamilyMetrics, scores: Sequence[PairFamilyScore], k_runs: int) -> NoiseFloor:
"""A family's own disagreement between runs, as a rate and as a whole number of pairs."""
scorable = [item for item in scores if item.outcome != "unscorable"]
flaky = sum(1 for item in scorable if item.flips)
rate = flaky / len(scorable) if scorable else 0.0
return NoiseFloor(
family=family,
k_runs=k_runs,
pairs=metrics.scorable,
flaky_pairs=flaky,
negative_flip_rate=rate,
budget_flips=math.ceil(rate * metrics.scorable) if metrics.scorable else 0,
gates=family not in NON_GATING_FAMILIES,
)
def _family_of(case: PairCase, taxonomy: FamilyTaxonomy) -> str:
for row in case.expected:
if row.family:
return str(row.family)
answer = classify_pair(case, taxonomy, client=None)
return answer.families[0] if answer.families else "unknown"
def _artifact_key(model: str, slices: Sequence[str], thinking: bool = False) -> str:
"""What identifies a baseline run: the detector model, and the slice selection when it is not the default.
Keyed by model alone, a run over the memory-safety slice silently replaces the run over the web families and
the published table shows one of them as if it were everything (Req 8.4)."""
parts = [_slug(model)]
if thinking:
parts.append("thinking")
if tuple(slices) != DEFAULT_SLICES:
parts.append(_slug("-".join(sorted(slices))))
return "-".join(parts)
def _write(out_dir: Path, report: BaselineReport) -> None:
directory = out_dir / "baseline" / _artifact_key(report.model, report.slices, report.thinking)
directory.mkdir(parents=True, exist_ok=True)
payload = report.to_dict()
(directory / "noise-floors.json").write_text(
json.dumps({"model": report.model, "k_runs": report.k_runs, "floors": payload["floors"]}, indent=2, sort_keys=True) + "\n",
encoding="utf-8",
)
(directory / "report.json").write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n", encoding="utf-8")
def _slug(model: str) -> str:
return re.sub(r"[^A-Za-z0-9._-]+", "-", model).strip("-") or "unnamed"
def load_noise_floors(out_dir: Path, model: str, slices: Sequence[str] = (), thinking: bool = False) -> dict[str, NoiseFloor]:
"""The budgets a later round must stay inside, as round zero measured them for this model."""
path = out_dir / "baseline" / _artifact_key(model, slices or DEFAULT_SLICES, thinking) / "noise-floors.json"
if not path.is_file():
return {}
payload = json.loads(path.read_text(encoding="utf-8"))
floors: dict[str, NoiseFloor] = {}
for name, block in (payload.get("floors") or {}).items():
if isinstance(block, Mapping):
floors[str(name)] = NoiseFloor(
family=str(block.get("family", name)),
k_runs=int(block.get("k_runs", DEFAULT_K_RUNS) or DEFAULT_K_RUNS),
pairs=int(block.get("pairs", 0) or 0),
flaky_pairs=int(block.get("flaky_pairs", 0) or 0),
negative_flip_rate=float(block.get("negative_flip_rate", 0.0) or 0.0),
budget_flips=int(block.get("budget_flips", 0) or 0),
gates=bool(block.get("gates", True)),
)
return floors
MIN_K_RUNS = 3 # Req 3.5: a single run misses most real changes, so no stage may score with fewer
def run_learning_round(
cases: Sequence[PairCase],
*,
family: str,
taxonomy: FamilyTaxonomy,
configs_dir: Path,
proposer: Proposer,
scan_factory: Callable[[FamilyConfig], Scan],
model: str,
journal: LearningJournal,
archive: Archive,
floors: Mapping[str, NoiseFloor],
out_dir: Path,
cost_cap_usd: float = 10.0,
minibatch: int = 8,
k_runs: int = MIN_K_RUNS,
sweep_families: Sequence[str] = (),
spent_usd: Callable[[], float] | None = None,
trajectories: Callable[[], Sequence[Mapping[str, object]]] | None = None,
) -> RoundRecord:
"""One round: propose one change for one family, then let the evidence decide whether it stays.
Staged so that the cheap answer comes first: a train minibatch, then the family's held-out pairs,
then every other family. Each stage scores several runs per pair, because a single run misses most
real changes. The proposer never sees a held-out pair, and the family directory is snapshotted before
the change so any outcome other than acceptance leaves the tree exactly as it was found.
"""
from .acceptance import DirectorySnapshot, bump_version, decide, within_budget
from .detectors import load_family_configs
from .proposer import apply_proposal, build_failure_facts, refuse_proposal
from .split import is_teacher, refuse_if_holdout
if k_runs < MIN_K_RUNS:
raise ValueError(f"a stage scores at least three runs per pair; {k_runs} was asked for")
round_number = journal.next_round() # type: ignore[attr-defined]
directory = out_dir / "rounds" / str(round_number)
if directory.exists():
from .journal import JournalError
raise JournalError(f"round {round_number} already has a directory; history is append-only")
configs = load_family_configs(configs_dir, taxonomy)
config = configs[family]
train = [case for case in cases if is_teacher(case) and _family_of(case, taxonomy) == family]
holdout = [case for case in cases if case.split == "holdout" and _family_of(case, taxonomy) == family]
batch = train[:minibatch]
spent = 0.0
def _score(rows: Sequence[PairCase], scan: Scan, name: str) -> list[PairFamilyScore]:
return [score_case(case, scan, taxonomy=taxonomy, runs=k_runs, family=name) for case in rows]
def _over_cap() -> bool:
"""Req 9.8: cost is read after every stage, so a round stops rather than finishing and then complaining."""
nonlocal spent
spent = spent_usd() if spent_usd is not None else 0.0
return not within_budget(spent, cost_cap_usd)
record = _round_record(round_number, family, None, taxonomy, config)
try:
before_scan = scan_factory(config)
before_train = _score(batch, before_scan, family)
before_hold = _score(holdout, before_scan, family)
except Exception as exc:
# These stages run before anything is applied, so there is nothing to revert — but they have already spent
# money, and a round that leaves no record is a round nobody can account for. Measured: round 5 died on a
# network read during its first stage and vanished.
_finish(
journal,
directory,
record,
outcome="reverted",
reason=f"stage_failed: {type(exc).__name__}",
cost=spent_usd() if spent_usd is not None else 0.0,
trajectories=trajectories,
)
raise
if _over_cap():
return _finish(
journal, directory, record, outcome="reverted_cost", reason="cost_cap_exceeded", cost=spent, trajectories=trajectories
)
facts = build_failure_facts(before_train, family=family, config=config, journal=journal, train_pairs={case.name for case in train})
proposal = proposer.propose(facts) # type: ignore[attr-defined]
record = _round_record(round_number, family, proposal, taxonomy, config)
if proposal is None:
# `no_proposal` covers two different worlds: a proposer that considered the facts and declined, and one
# that could not start at all. Losing that distinction hides a fixable configuration error as a quiet round.
why = str(getattr(proposer, "reason", "") or "")
return _finish(
journal,
directory,
record,
outcome="rejected",
reason=f"no_proposal: {why}" if why else "no_proposal",
cost=spent,
trajectories=trajectories,
)
refusal = refuse_proposal(proposal, config)
if refusal is not None:
return _finish(
journal, directory, record, outcome="rejected", reason=refusal, proposal=proposal, cost=spent, trajectories=trajectories
)
leak = refuse_if_holdout(_pairs_named_by(proposal, cases), cases)
if leak is not None:
# Req 5.2: a change a holdout pair taught, or that names one, is refused by name rather than measured.
return _finish(
journal,
directory,
record,
outcome="refused_holdout",
reason=leak.degradation()["detail"], # type: ignore[arg-type]
proposal=proposal,
cost=spent,
trajectories=trajectories,
)
# The sweep's "before" is measured with the unchanged configurations, so what it later reports is what this
# round flipped and not what was already failing (Req 9.5).
before_sweep = _sweep_scores(cases, taxonomy, configs, scan_factory, sweep_families, family, k_runs)
if _over_cap():
return _finish(
journal,
directory,
record,
outcome="reverted_cost",
reason="cost_cap_exceeded",
proposal=proposal,
cost=spent,
trajectories=trajectories,
)
snapshot = DirectorySnapshot.of(configs_dir / family)
apply_proposal(proposal, configs_dir / family)
version = str(int(config.version) + 1) if config.version.isdigit() else f"{config.version}+1"
try:
after_config = load_family_configs(configs_dir, taxonomy)[family]
after_scan = scan_factory(after_config)
after_train = _score(batch, after_scan, family)
after_hold = _score(holdout, after_scan, family)
for score in after_train + after_hold:
archive.record(family, version, score) # type: ignore[attr-defined]
if _over_cap():
snapshot.restore()
return _finish(
journal,
directory,
record,
outcome="reverted_cost",
reason="cost_cap_exceeded",
proposal=proposal,
cost=spent,
scores=after_train + after_hold,
trajectories=trajectories,
)
after_sweep = _sweep_scores(cases, taxonomy, configs, scan_factory, sweep_families, family, k_runs)
sweep, flipped_unpredicted = _negative_flips(before_sweep, after_sweep)
train_delta = _delta(before_train, after_train)
holdout_delta = _delta(before_hold, after_hold)
flipped = _flipped(before_train + before_hold, after_train + after_hold)
train_moves = _moves(before_train, after_train)
holdout_moves = _moves(before_hold, after_hold)
attribution = Attribution(
flipped_predicted=len(flipped),
flipped_unpredicted=len(flipped_unpredicted),
precision=len(flipped) / (len(flipped) + len(flipped_unpredicted)) if flipped or flipped_unpredicted else 0.0,
)
if _over_cap():
snapshot.restore()
return _finish(
journal,
directory,
record,
outcome="reverted_cost",
reason="cost_cap_exceeded",
proposal=proposal,
cost=spent,
scores=after_train + after_hold,
trajectories=trajectories,
)
verdict = decide(target=family, train_delta=train_delta, holdout_delta=holdout_delta, sweep=sweep, floors=floors) # type: ignore[arg-type]
if not verdict.accepted:
snapshot.restore()
return _finish(
journal,
directory,
record,
outcome="rejected",
reason=verdict.reason,
proposal=proposal,
cost=spent,
train_delta=train_delta,
holdout_delta=holdout_delta,
sweep=sweep,
attribution=attribution,
moves=(train_moves, holdout_moves),
flipped=flipped,
unpredicted=flipped_unpredicted,
scores=after_train + after_hold,
trajectories=trajectories,
)
bump_version(configs_dir / family, version)
return _finish(
journal,
directory,
record,
outcome="accepted",
reason=verdict.reason,
proposal=proposal,
cost=spent,
train_delta=train_delta,
holdout_delta=holdout_delta,
sweep=sweep,
attribution=attribution,
moves=(train_moves, holdout_moves),
flipped=flipped,
unpredicted=flipped_unpredicted,
scores=after_train + after_hold,
version=version,
trajectories=trajectories,
)
except Exception as exc:
# A stage that raises has already spent money and already changed the tree. Both facts belong in the
# journal before the exception leaves, or the next round starts from a state nothing describes.
snapshot.restore()
_finish(
journal,
directory,
record,
outcome="reverted",
reason=f"stage_failed: {type(exc).__name__}",
proposal=proposal,
cost=spent,
trajectories=trajectories,
)
raise
def _pairs_named_by(proposal: Proposal, cases: Sequence[PairCase]) -> tuple[str, ...]:
"""Every corpus pair this proposal names, in its predictions or in the file it writes.
A memory entry that quotes a held-out pair is the loop learning the answer instead of the shape, and it looks
exactly like a legitimate hard negative until the holdout number is read as if it meant something."""
haystack = "\n".join([*proposal.predicted_affected, *proposal.predicted_at_risk, *proposal.change.values()])
# Substring first, regex only for the handful that survive it: the corpus is a few hundred rows and a round
# runs this once, but a per-round sweep of one regex per row is the kind of thing that stops being free.
named = [
name
for case in cases
if (name := str(getattr(case, "name", ""))) and name in haystack and re.search(rf"(?<![\w-]){re.escape(name)}(?![\w-])", haystack)
]
return tuple(sorted(set(named)))
def _sweep_scores(
cases: Sequence[PairCase],
taxonomy: FamilyTaxonomy,
configs: Mapping[str, FamilyConfig],
scan_factory: Callable[[FamilyConfig], Scan],
families: Sequence[str],
target: str,
k_runs: int,
) -> dict[str, dict[str, str]]:
"""Every other family's held-out pairs, family by family, as `{family: {pair: outcome}}`.
Called once before the change and once after it. Scoring both sides is the only way to say which pairs *this*
round broke: on a real corpus most pairs are failing already, and counting failures would reject every round
for damage it did not do (Req 9.5). Unscorable rows are dropped here rather than counted as failures (Req 4.1).
"""
outcomes: dict[str, dict[str, str]] = {}
for name in families:
if name == target or name not in configs:
continue
rows = [case for case in cases if case.split == "holdout" and _family_of(case, taxonomy) == name]
scan = scan_factory(configs[name])
scored = [score_case(case, scan, taxonomy=taxonomy, runs=k_runs, family=name) for case in rows]
outcomes[name] = {item.pair: item.outcome for item in scored if item.outcome != "unscorable"}
return outcomes
def _negative_flips(before: Mapping[str, Mapping[str, str]], after: Mapping[str, Mapping[str, str]]) -> tuple[dict[str, int], list[str]]:
"""Pairs that were right before this round and are not right after it, per family and by name."""
flips: dict[str, int] = {}
broken: list[str] = []
for name, was in before.items():
now = after.get(name, {})
lost = sorted(pair for pair, outcome in was.items() if outcome == "pair_correct" and now.get(pair) != "pair_correct")
flips[name] = len(lost)
broken.extend(lost)
return flips, broken
def _delta(before: Sequence[PairFamilyScore], after: Sequence[PairFamilyScore]) -> float:
"""The change in how many pairs came out right, as a fraction of the pairs that could be scored."""
scorable = [item for item in before if item.outcome != "unscorable"]
if not scorable:
return 0.0
was = sum(1 for item in before if item.outcome == "pair_correct")
now = sum(1 for item in after if item.outcome == "pair_correct")
return (now - was) / len(scorable)
def _moves(before: Sequence[PairFamilyScore], after: Sequence[PairFamilyScore]) -> tuple[int, int]:
"""(pairs that became correct, pairs that stopped being correct) — what the sign test is run over."""
was = {item.pair: item.outcome for item in before if item.outcome != "unscorable"}
better = worse = 0
for item in after:
previous = was.get(item.pair)
if previous is None or item.outcome == "unscorable":
continue
if previous != "pair_correct" and item.outcome == "pair_correct":
better += 1
elif previous == "pair_correct" and item.outcome != "pair_correct":
worse += 1
return better, worse
def _flipped(before: Sequence[PairFamilyScore], after: Sequence[PairFamilyScore]) -> list[str]:
"""Pairs this round turned right, by name and in a stable order."""
was = {item.pair: item.outcome for item in before}
return sorted(item.pair for item in after if item.outcome == "pair_correct" and was.get(item.pair) not in {"pair_correct", None})
def _round_record(
round_number: int, family: str, proposal: Proposal | None, taxonomy: FamilyTaxonomy, config: FamilyConfig
) -> dict[str, object]:
return {
"round": round_number,
"family": family,
"hypothesis": proposal.hypothesis if proposal is not None else "",
"levers": (proposal.lever,) if proposal is not None else (),
"predicted_affected": tuple(proposal.predicted_affected) if proposal is not None else (),
"predicted_at_risk": tuple(proposal.predicted_at_risk) if proposal is not None else (),
"taxonomy_version": taxonomy.version,
"config_version": config.version,
}
def _finish(
journal: LearningJournal,
directory: Path,
base: dict[str, object],
*,
outcome: str,
reason: str,
proposal: Proposal | None = None,
cost: float = 0.0,
train_delta: float = 0.0,
holdout_delta: float = 0.0,
sweep: Mapping[str, int] | None = None,
attribution: Attribution | None = None,
moves: tuple[tuple[int, int], tuple[int, int]] = ((0, 0), (0, 0)),
flipped: Sequence[str] = (),
unpredicted: Sequence[str] = (),
scores: Sequence[PairFamilyScore] = (),
version: str | None = None,
trajectories: Callable[[], Sequence[Mapping[str, object]]] | None = None,
) -> RoundRecord:
"""Write the round directory and journal the outcome. Every outcome is journalled, not just the good ones."""
from ..redaction import redact_secrets
record = RoundRecord(
**base, # type: ignore[arg-type]
outcome=outcome,
reason=reason,
target_train_delta=train_delta,
target_holdout_delta=holdout_delta,
sweep=dict(sweep or {}),
attribution=attribution or Attribution(),
cost_usd=cost,
train_better=moves[0][0],
train_worse=moves[0][1],
holdout_better=moves[1][0],
holdout_worse=moves[1][1],
)
if version is not None:
record = RoundRecord(**{**record.__dict__, "config_version": version})
directory.mkdir(parents=True, exist_ok=True)
if proposal is not None:
(directory / "proposal.json").write_text(
redact_secrets(json.dumps(proposal.to_dict(), indent=2, sort_keys=True)) + "\n",
encoding="utf-8",
)
(directory / "scores.json").write_text(
json.dumps([{"pair": item.pair, "family": item.family, "outcome": item.outcome} for item in scores], indent=2, sort_keys=True)
+ "\n",
encoding="utf-8",
)
(directory / "attribution.json").write_text(
json.dumps({"flipped_predicted": list(flipped), "flipped_unpredicted": list(unpredicted)}, indent=2, sort_keys=True) + "\n",
encoding="utf-8",
)
rows = list(trajectories() if trajectories is not None else ())
(directory / "trajectories.jsonl").write_text(
"".join(redact_secrets(json.dumps(row, sort_keys=True)) + "\n" for row in rows), encoding="utf-8"
)
journal.append(record) # type: ignore[attr-defined]
return record
__all__ = [
"DEFAULT_K_RUNS",
"MIN_K_RUNS",
"run_learning_round",
"DEFAULT_SLICES",
"NON_GATING_FAMILIES",
"BaselineReport",
"NoiseFloor",
"Scan",
"compare_baselines",
"load_noise_floors",
"run_baseline",
"score_case",
]
src/openultrasast/learning/scoring.py
-405
View File
@@ -1,405 +0,0 @@
"""Class-aware, pair-wise scoring (learning-harness, Req 3, 4.1, 4.5).
A pair is scored once per family. Detection means a finding whose family tag relates to the labeled
family, sitting inside the labeled function; the words in a finding are never read, because a text
token was exactly the proxy that made the old hunter number meaningless. Silence means no such finding
on the fixed side: findings of other families and findings outside the labeled function are counted
separately and are never called leaks. A pair that cannot be scored keeps its reason and is never run.
Every evaluation runs a pair K times, because a single run misses most real changes. The recorded
outcome is the majority of the runs, and the runs that disagree with it are the pair's own flakiness.
"""
from __future__ import annotations
import math
from collections import Counter
from collections.abc import Mapping, Sequence
from dataclasses import dataclass, field
from typing import Literal
from ..findings import StaticFinding
from .families import FamilyTaxonomy
FamilyOutcome = Literal["pair_correct", "both_flagged", "both_silent", "reversed", "unscorable"]
# Families whose bug shape is the absence of a check on a reached handler: without the registration that
# makes the handler reachable, the pair says nothing either way, which is a corpus defect with its own name.
CONTEXT_DEPENDENT_FAMILIES = frozenset({"access_control"})
def is_post_cutoff(fix_date: str, cutoff: str) -> bool:
"""True when the *earliest possible* date of ``fix_date`` is after ``cutoff`` (Req 10.5).
The corpus records dates at the precision it has: a full ISO day for a few rows, a year for almost all of them.
A year-precision row counts only when the whole year is after the cutoff, so a coarse date can under-claim the
clean slice but never over-claim it — and over-claiming is the one that would turn contamination into a result.
"""
if not fix_date or len(cutoff) != 10:
return False # a cutoff coarser than a day is the one over-claim path there is, so it is refused outright
earliest = fix_date if len(fix_date) == 10 else f"{fix_date}-01-01"[:10]
return earliest > cutoff
Rung = Literal["suspicion", "static_corroboration", "proven"]
_RUNGS: tuple[Rung, ...] = ("suspicion", "static_corroboration", "proven")
FunctionRanges = Mapping[str, Sequence[tuple[str, int, int]]]
@dataclass(frozen=True)
class PairFamilyScore:
"""What one pair says about one family."""
pair: str
family: str
runs: tuple[FamilyOutcome, ...]
outcome: FamilyOutcome
slice: str = ""
other_findings_vuln: int = 0 # right place, another family: not a detection, never a leak
other_findings_fixed: int = 0
fabricated_vuln: int = 0 # a family id outside the taxonomy: noise, and worth naming
fabricated_fixed: int = 0
flips: int = 0 # runs that disagreed with the majority
rung: Rung = "suspicion"
unscorable_reason: str | None = None
fix_date: str = "" # the date the corpus recorded, at whatever precision it has (Req 10.5)
def score_pair_family(
case: object,
family: str,
vuln_runs: Sequence[Sequence[StaticFinding]],
fixed_runs: Sequence[Sequence[StaticFinding]],
*,
ranges: FunctionRanges,
taxonomy: FamilyTaxonomy,
parse_ok: bool = True,
entry_points: Sequence[str] | None = None,
fixed_ranges: FunctionRanges | None = None,
) -> PairFamilyScore:
"""Score one pair for one family over K runs of the detector on each side.
``fixed_ranges`` is the *fixed* side's parse. A fix that adds a guard above the handler moves its first line,
so judging the fixed side against the vulnerable side's span reads a finding inside the fixed handler as
outside it: a leak recorded as silence, and always in the direction that flatters the number.
"""
name = str(getattr(case, "name", "?"))
slice_name = str(getattr(case, "slice", "") or "")
reason = unscorable_reason(case, parse_ok=parse_ok, ranges=ranges, entry_points=entry_points, family=family)
if reason is not None:
return PairFamilyScore(
pair=name,
family=family,
slice=slice_name,
fix_date=str(getattr(case, "fix_date", "") or ""),
runs=(),
outcome="unscorable",
unscorable_reason=reason,
)
spans = _spans(case, ranges)
if fixed_ranges is None:
fixed_spans = spans
else:
# A trap pair's fixed side holds a different, correctly guarded handler; `fix_function` names it. Without a
# span there, every fixed-side finding reads as outside the labeled function and the pair scores as silence.
fixed_spans = _spans(case, fixed_ranges, function=str(getattr(case, "fix_function", "") or ""))
if not fixed_spans:
return PairFamilyScore(
pair=name,
family=family,
slice=slice_name,
fix_date=str(getattr(case, "fix_date", "") or ""),
runs=(),
outcome="unscorable",
unscorable_reason="unresolved_label:fixed",
)
outcomes: list[FamilyOutcome] = []
other_vuln = other_fixed = fabricated_vuln = fabricated_fixed = 0
rung: Rung = "suspicion"
for index in range(max(len(vuln_runs), len(fixed_runs))):
vuln = list(vuln_runs[index]) if index < len(vuln_runs) else []
fixed = list(fixed_runs[index]) if index < len(fixed_runs) else []
hits_vuln = _hits(vuln, family, spans, taxonomy)
hits_fixed = _hits(fixed, family, fixed_spans, taxonomy)
other_vuln += _others(vuln, family, spans, taxonomy)
other_fixed += _others(fixed, family, fixed_spans, taxonomy)
fabricated_vuln += _fabricated(vuln, taxonomy)
fabricated_fixed += _fabricated(fixed, taxonomy)
rung = _best_rung(rung, hits_vuln)
outcomes.append(_outcome(bool(hits_vuln), bool(hits_fixed)))
if not outcomes:
return PairFamilyScore(
pair=name,
family=family,
slice=slice_name,
fix_date=str(getattr(case, "fix_date", "") or ""),
runs=(),
outcome="unscorable",
unscorable_reason="no_runs",
)
majority = Counter(outcomes).most_common(1)[0][0]
return PairFamilyScore(
pair=name,
family=family,
slice=slice_name,
fix_date=str(getattr(case, "fix_date", "") or ""),
runs=tuple(outcomes),
outcome=majority,
other_findings_vuln=other_vuln,
other_findings_fixed=other_fixed,
fabricated_vuln=fabricated_vuln,
fabricated_fixed=fabricated_fixed,
flips=sum(1 for item in outcomes if item != majority),
rung=rung,
)
def _outcome(detected: bool, leaked: bool) -> FamilyOutcome:
if detected and not leaked:
return "pair_correct"
if detected and leaked:
return "both_flagged"
return "reversed" if leaked else "both_silent"
def unscorable_reason(
case: object,
*,
parse_ok: bool = True,
ranges: FunctionRanges,
entry_points: Sequence[str] | None = None,
family: str = "",
) -> str | None:
"""Why this pair cannot be scored, or None. The reasons have different owners, so they must not be conflated.
A maintainer-declared limit wins over anything computed. An unparsable side is a tooling gap
(``unsupported_language``), a labeled function no parsed range names is a corpus labelling defect
(``unresolved_label``), and a handler that arrived without the registration that makes it reachable is a
harvest defect (``missing_context``), which only matters for families whose bug shape is a missing check.
"""
declared = getattr(case, "unscorable", None)
if declared:
return str(declared)
if not parse_ok:
return "unsupported_language"
if not _spans(case, ranges):
return "unresolved_label"
labeled_family = family or next(
(str(getattr(row, "family", "") or "") for row in getattr(case, "expected", ()) or () if getattr(row, "family", None)), ""
)
if entry_points is not None and labeled_family in CONTEXT_DEPENDENT_FAMILIES:
labeled = {str(getattr(row, "function", "") or "") for row in getattr(case, "expected", ()) or ()}
if not (labeled & set(entry_points)):
return "missing_context"
return None
def _spans(case: object, ranges: FunctionRanges, *, function: str = "") -> tuple[tuple[str, int, int], ...]:
"""The (path, start, end) spans of the labeled functions. ``function`` overrides the label for the fixed side."""
wanted = {function} if function else {str(getattr(row, "function", "") or "") for row in getattr(case, "expected", ()) or ()}
wanted.discard("")
found: list[tuple[str, int, int]] = []
for path, entries in ranges.items():
for name, start, end in entries:
if name in wanted:
found.append((path, start, end))
return tuple(found)
def _inside(finding: StaticFinding, spans: Sequence[tuple[str, int, int]]) -> bool:
return finding.line is not None and any(path == finding.path and start <= finding.line <= end for path, start, end in spans)
def _families_of(finding: StaticFinding) -> tuple[str, ...]:
return tuple(tag.split(":", 1)[1] for tag in finding.tags if tag.startswith("family:"))
def _relation(finding: StaticFinding, family: str, taxonomy: FamilyTaxonomy) -> str:
relations = [taxonomy.related(family, candidate) for candidate in _families_of(finding)]
for wanted in ("same", "parent_child", "lateral"):
if wanted in relations:
return wanted
return "fabricated" if relations else "none"
def _hits(
findings: Sequence[StaticFinding], family: str, spans: Sequence[tuple[str, int, int]], taxonomy: FamilyTaxonomy
) -> list[StaticFinding]:
return [finding for finding in findings if _inside(finding, spans) and _relation(finding, family, taxonomy) in {"same", "parent_child"}]
def _others(findings: Sequence[StaticFinding], family: str, spans: Sequence[tuple[str, int, int]], taxonomy: FamilyTaxonomy) -> int:
return sum(1 for finding in findings if _relation(finding, family, taxonomy) == "lateral" or not _inside(finding, spans))
def _fabricated(findings: Sequence[StaticFinding], taxonomy: FamilyTaxonomy) -> int:
known = {item.id for item in taxonomy.families}
return sum(1 for finding in findings if any(candidate not in known for candidate in _families_of(finding)))
def _best_rung(current: Rung, hits: Sequence[StaticFinding]) -> Rung:
best = current
for finding in hits:
for tag in finding.tags:
if tag.startswith("verifier:"):
candidate = tag.split(":", 1)[1]
if candidate in _RUNGS and _RUNGS.index(candidate) > _RUNGS.index(best): # type: ignore[arg-type]
best = candidate # type: ignore[assignment]
return best
@dataclass(frozen=True)
class FamilyMetrics:
"""One family's numbers, with the denominator they were computed over and why rows fell out of it."""
family: str
taxonomy_version: str
scorable: int
unscorable: dict[str, int] = field(default_factory=dict)
outcomes: dict[str, int] = field(default_factory=dict)
recall: float = 0.0 # flagged the vulnerable side
silence: float = 0.0 # said nothing on the fixed side
youden: float = 0.0
directional_bias: float = 0.0 # positive: flags both sides; negative: silent on both. Youden cannot tell them apart.
leak_rate: float = 0.0
fixed_fpr_recall: float = 0.0 # recall, forfeited entirely when the leak rate exceeds the ceiling
negative_flips: int = 0 # pairs that were correct in the baseline and are not now
positive_flips: int = 0
p_value: float | None = None # two-sided sign test over the discordant pairs
reliable_change: bool = False
hierarchical_credit: bool = False # False: the taxonomy is flat, so no answer could earn partial credit (Req 3.7)
cutoff: str = "" # the detector cutoff these numbers were split on, or "" when none is configured (Req 10.5)
undated: int = 0 # rows the corpus records no date for; they are in neither slice and are counted here
post_cutoff: dict[str, object] = field(default_factory=dict) # the same numbers over rows fixed after the cutoff
def to_dict(self) -> dict[str, object]:
return {
"family": self.family,
"taxonomy_version": self.taxonomy_version,
"scorable": self.scorable,
"unscorable": dict(sorted(self.unscorable.items())),
"outcomes": dict(sorted(self.outcomes.items())),
"recall": self.recall,
"silence": self.silence,
"youden": self.youden,
"directional_bias": self.directional_bias,
"leak_rate": self.leak_rate,
"fixed_fpr_recall": self.fixed_fpr_recall,
"negative_flips": self.negative_flips,
"positive_flips": self.positive_flips,
"p_value": self.p_value,
"reliable_change": self.reliable_change,
"hierarchical_credit": self.hierarchical_credit,
"cutoff": self.cutoff,
"undated": self.undated,
"post_cutoff": dict(self.post_cutoff),
}
def aggregate(
scores: Sequence[PairFamilyScore],
*,
taxonomy: FamilyTaxonomy,
baseline: Sequence[PairFamilyScore] | None = None,
fpr_ceiling: float = 0.1,
per_slice: bool = False,
alpha: float = 0.05,
cutoff: str = "",
) -> dict[str, FamilyMetrics]:
"""Group scores into per-family numbers. Unscorable rows are listed by reason and leave every denominator.
With a ``cutoff`` each family also reports the same numbers over the rows fixed after it, beside the whole
number and never instead of it: a model that may have read the fix cannot be measured on it the same way as one
that cannot have. Rows the corpus records no date for are counted on their own (Req 10.5)."""
grouped: dict[str, list[PairFamilyScore]] = {}
for score in scores:
grouped.setdefault(f"{score.slice}/{score.family}" if per_slice else score.family, []).append(score)
was_correct = {(item.slice if per_slice else "", item.family, item.pair): item.outcome for item in baseline or ()}
out: dict[str, FamilyMetrics] = {}
for key, group in sorted(grouped.items()):
scorable = [item for item in group if item.outcome != "unscorable"]
reasons: dict[str, int] = {}
for item in group:
if item.outcome == "unscorable":
reason = item.unscorable_reason or "unknown"
reasons[reason] = reasons.get(reason, 0) + 1
counts = Counter(item.outcome for item in scorable)
total = len(scorable)
detected = counts["pair_correct"] + counts["both_flagged"]
silent = counts["pair_correct"] + counts["both_silent"]
leaks = counts["both_flagged"] + counts["reversed"]
recall = detected / total if total else 0.0
silence = silent / total if total else 0.0
leak_rate = leaks / total if total else 0.0
negative = positive = 0
for item in scorable:
before = was_correct.get((item.slice if per_slice else "", item.family, item.pair))
if before is None or before == "unscorable":
continue
if before == "pair_correct" and item.outcome != "pair_correct":
negative += 1
elif before != "pair_correct" and item.outcome == "pair_correct":
positive += 1
p_value = sign_test(positive, negative) if baseline is not None else None
recent = [item for item in scorable if is_post_cutoff(item.fix_date, cutoff)] if cutoff else []
out[key] = FamilyMetrics(
family=group[0].family,
taxonomy_version=taxonomy.version,
scorable=total,
unscorable=reasons,
outcomes={name: count for name, count in sorted(counts.items())},
recall=recall,
silence=silence,
youden=(recall + silence - 1.0) if total else 0.0,
directional_bias=((counts["both_flagged"] - counts["both_silent"]) / total) if total else 0.0,
leak_rate=leak_rate,
fixed_fpr_recall=recall if leak_rate <= fpr_ceiling else 0.0,
negative_flips=negative,
positive_flips=positive,
p_value=p_value,
reliable_change=bool(p_value is not None and p_value < alpha and positive != negative),
hierarchical_credit=taxonomy.hierarchical,
cutoff=cutoff,
undated=sum(1 for item in group if not item.fix_date) if cutoff else 0,
post_cutoff=_slice_numbers(recent) if cutoff else {},
)
return out
def _slice_numbers(scores: Sequence[PairFamilyScore]) -> dict[str, object]:
"""Recall, silence and Youden over a subset, with the subset's own denominator beside them."""
total = len(scores)
counts = Counter(item.outcome for item in scores)
recall = (counts["pair_correct"] + counts["both_flagged"]) / total if total else 0.0
silence = (counts["pair_correct"] + counts["both_silent"]) / total if total else 0.0
return {
"scorable": total,
"outcomes": {name: count for name, count in sorted(counts.items())},
"recall": recall,
"silence": silence,
"youden": (recall + silence - 1.0) if total else 0.0,
}
def sign_test(better: int, worse: int) -> float:
"""Two-sided exact sign test over the pairs that changed. 1.0 when nothing changed, so noise never reads as progress."""
total = better + worse
if total == 0:
return 1.0
smaller = min(better, worse)
tail = sum(math.comb(total, index) for index in range(smaller + 1)) / float(2**total)
return float(min(1.0, 2.0 * tail))
__all__ = [
"CONTEXT_DEPENDENT_FAMILIES",
"FamilyMetrics",
"FamilyOutcome",
"FunctionRanges",
"PairFamilyScore",
"Rung",
"aggregate",
"is_post_cutoff",
"score_pair_family",
"sign_test",
"unscorable_reason",
]
src/openultrasast/learning/split.py
-64
View File
@@ -1,64 +0,0 @@
"""One rule for who may teach, and one refusal for anything that touches the holdout (Req 5.1, 5.2).
Every learning path in the tool asks this module the same question, so the split cannot be enforced in
one place and forgotten in another. A teacher is a vendored pair, at a review tier a maintainer stands
behind, on the train split, and scorable. Everything else may be measured; nothing else may teach.
A refusal is data, not an exception: the path records which holdout pairs a candidate touched and moves
on, so a leak is visible in the run rather than silently accepted.
"""
from __future__ import annotations
from collections.abc import Iterable, Sequence
from dataclasses import dataclass
GATING_TIERS = frozenset({"seeded", "reviewed"})
TRAIN = "train"
HOLDOUT = "holdout"
REFUSED = "holdout_pair_refused"
@dataclass(frozen=True)
class Refusal:
"""A candidate that a holdout pair taught, or that touches one."""
pairs: tuple[str, ...]
reason: str = REFUSED
def degradation(self) -> dict[str, object]:
return {
"stage": "learning",
"reason": self.reason,
"pairs": list(self.pairs),
"detail": f"refused: taught by or touching holdout pair(s) {', '.join(self.pairs)}",
}
def teachers(cases: Sequence[object]) -> tuple[object, ...]:
"""The pairs a learning path may learn from. Everything else is measured, never learned from."""
return tuple(case for case in cases if is_teacher(case))
def is_teacher(case: object) -> bool:
if not bool(getattr(case, "vendored", True)):
return False
if str(getattr(case, "review_tier", "")) not in GATING_TIERS:
return False
if str(getattr(case, "split", TRAIN)) != TRAIN:
return False
return getattr(case, "unscorable", None) is None
def holdout_names(cases: Sequence[object]) -> frozenset[str]:
return frozenset(str(getattr(case, "name", "")) for case in cases if str(getattr(case, "split", "")) == HOLDOUT)
def refuse_if_holdout(taught_by: Iterable[str], cases: Sequence[object]) -> Refusal | None:
"""None when nothing in ``taught_by`` is a holdout pair; a refusal naming them when something is."""
held = holdout_names(cases)
touched = tuple(sorted({name for name in taught_by if name in held}))
return Refusal(pairs=touched) if touched else None
__all__ = ["GATING_TIERS", "HOLDOUT", "REFUSED", "TRAIN", "Refusal", "holdout_names", "is_teacher", "refuse_if_holdout", "teachers"]
src/openultrasast/learning/verifiers.py
-140
View File
@@ -1,140 +0,0 @@
"""What may raise a claim above suspicion, keyed by family (learning-harness, Req 7.1, 7.3, 7.4).
A detector's claim is a suspicion until something other than the model agrees with it. What counts as
agreement depends on the family: a canary oracle observed in the sandbox, corroboration from a static
checker, or nothing at all. A family whose verifier does not exist yet reports at suspicion rather than
pretending, and that is the honest default rather than a gap.
The canary oracle is a function over a sandbox result, so the sandbox keeps deciding what happened and
this module only says what would count. Exit codes are not oracles: a crash and a demonstrated
injection are different evidence, and one family's oracle must not silently answer for another.
"""
from __future__ import annotations
from collections.abc import Callable
from dataclasses import dataclass, replace
from pathlib import Path
from typing import Literal, Protocol
from ..findings import StaticFinding
from ..sandbox import SandboxResult
from ..verification import EvidenceLevel
from .families import Family
Rung = Literal["suspicion", "static_corroboration", "proven"]
RUNGS: tuple[Rung, ...] = ("suspicion", "static_corroboration", "proven")
VerifierKind = Literal["canary", "static", "none"]
@dataclass(frozen=True)
class Verification:
"""What a verifier saw, and how high it lets the claim go."""
rung: Rung
reason: str
oracle_output: str = ""
class Verifier(Protocol):
@property
def kind(self) -> VerifierKind: ...
def verify(self, claim: StaticFinding, *, root: Path, language: str) -> Verification: ...
def token_oracle(token: str) -> Callable[[SandboxResult], bool]:
"""The oracle a canary snippet is written against: the token appears only if the vulnerability fired."""
def fired(result: SandboxResult) -> bool:
return token in result.stdout or token in result.stderr
return fired
@dataclass(frozen=True)
class NoVerifier:
"""Nothing can confirm this family yet, so the claim stays where the detector left it."""
kind: VerifierKind = "none"
def verify(self, claim: StaticFinding, *, root: Path, language: str) -> Verification:
del claim, root, language
return Verification(rung="suspicion", reason="no_verifier")
@dataclass(frozen=True)
class StaticVerifier:
"""Another checker looked at the same function and agrees the obligation is undischarged."""
kind: VerifierKind = "static"
def verify(self, claim: StaticFinding, *, root: Path, language: str) -> Verification:
del language
from ..hunter_tools import PathEscapesRepo, obligations
if not claim.function_name:
return Verification(rung="suspicion", reason="no_function")
try:
rows = obligations(root, path=claim.path, function=claim.function_name)
except (PathEscapesRepo, OSError):
return Verification(rung="suspicion", reason="checker_unavailable")
if not rows:
return Verification(rung="suspicion", reason="checker_disagrees")
summary = "; ".join(f"{row.get('operation')} missing {row.get('missing')}" for row in rows)
return Verification(rung="static_corroboration", reason="checker_agrees", oracle_output=summary)
def verifier_for(family: Family, *, canary: Verifier | None = None) -> Verifier:
"""The verifier a family declares. A canary family with no implementation registered stays at suspicion."""
if family.verifier == "static":
return StaticVerifier()
if family.verifier == "canary":
return canary if canary is not None else NoVerifier(kind="canary")
return NoVerifier()
def apply_verification(claim: StaticFinding, verification: Verification) -> StaticFinding:
"""A copy of the claim carrying what the verifier said, as a tag.
The evidence ladder belongs to another boundary and this spec does not change it, so nothing here
writes ``evidence_level``. "proven" is this spec's word, not a rung the ladder defines, and stamping
it onto the field made the project's own verification helpers raise. A caller that wants to move a
claim up the ladder does it through :func:`raise_to`, which can only produce a level the ladder has.
"""
tags = [tag for tag in claim.tags if not tag.startswith("verifier:")] + [f"verifier:{verification.rung}"]
return replace(claim, tags=tags)
def evidence_level_for(rung: Rung) -> EvidenceLevel:
"""The project's evidence level a verifier rung corresponds to. The only bridge between the two vocabularies."""
return _LADDER[rung]
def raise_to(claim: StaticFinding, rung: Rung) -> StaticFinding:
"""A copy of the claim at the ladder level this rung corresponds to. Never a value the ladder lacks."""
return replace(claim, evidence_level=str(evidence_level_for(rung)))
_LADDER: dict[Rung, EvidenceLevel] = {
"suspicion": EvidenceLevel.SUSPICION,
"static_corroboration": EvidenceLevel.STATIC_CORROBORATION,
# A canary that fired in the sandbox demonstrated the vulnerability, which is what this level means here.
"proven": EvidenceLevel.EXPLOIT_DEMONSTRATED,
}
__all__ = [
"RUNGS",
"NoVerifier",
"Rung",
"StaticVerifier",
"Verification",
"Verifier",
"VerifierKind",
"apply_verification",
"evidence_level_for",
"raise_to",
"token_oracle",
"verifier_for",
]
src/openultrasast/model/__init__.py
+7
View File
@@ -0,0 +1,7 @@
"""The model layer: a deterministic program model that arbitrates detector claims.
The taxonomy routes work, the candidate enumerator feeds the `suspicion` band, the specs carry the security
vocabularies, and (from group 2) the CPG-backed taint and dominance checks establish the rungs above it.
"""
from __future__ import annotations
src/openultrasast/model/audit.py
+166
View File
@@ -0,0 +1,166 @@
"""Module audit: what is load-bearing, what stands alone, and what is orphaned (Req 3).
The tool outgrew itself — 99 modules and 832 tests at the point this feature began, a large share of them
serving an architecture its own measurements retired. This classifier exists so outgrowth is visible and
checkable rather than a thing someone notices two specs later.
A module is:
load_bearing reachable from a gate, the scan CLI, or a surviving requirement
standalone_capability its own entry point and users, not on the scan path
orphaned no importer, no entry point, no requirement — it should not be in the tree
Pure stdlib, no imports of the package it audits: it reads the import graph with ``ast`` so a broken module
is still classifiable.
"""
from __future__ import annotations
import ast
from collections.abc import Iterable, Mapping
from dataclasses import dataclass
from pathlib import Path
PACKAGE = "openultrasast"
# Entry points a module can be reachable from. The three gates are the contract; the CLI is the product.
ROOTS = ("gate", "map_gate", "pair_gate", "cli", "__main__")
# Modules that stand alone: their own CLI subcommand or external protocol, deliberately off the scan path.
SPEC_OWNED = {
"model.specs": "model-grounded-detection Req 7.2-7.4: the security vocabularies the CPG queries are parameterised by",
"model.endpoint": "model-grounded-detection Req 8: the judge client, wired by model/judge.py in group 3",
"model.audit": "model-grounded-detection Req 3: this classifier; run by the maintainer, not by the scan",
}
STANDALONE = {
"mcp": "narrow MCP server over stdio; its own `ousast mcp` entry point and OpenCode integration",
"skills": "skill router; consumed by mcp and by the OpenCode integration, not by the scan path",
"fusion": "two-panel adjudication engine; reached from the scan's report stage and used standalone",
"harness_ext": "HarnessX extension surface; loaded by name when the optional harness is present",
"hunter_harness": "harness wiring for the tool hunter; used by the hunter path and by mcp",
"stage_processors": "HarnessX slot-contracted scan stages; pinned by the zero-dependency guard in test_gate",
"slot_contract": "the slot/contract protocol stage_processors implements",
}
@dataclass(frozen=True)
class ModuleAudit:
module: str
classification: str # load_bearing | standalone_capability | orphaned
reason: str
importers: tuple[str, ...]
def module_name(path: Path, root: Path) -> str:
relative = path.relative_to(root).with_suffix("")
parts = [part for part in relative.parts if part != "__init__"]
return ".".join(parts)
def import_graph(root: Path) -> dict[str, set[str]]:
"""``module -> the modules it imports`` for every module under ``root``, resolved to package-relative names."""
graph: dict[str, set[str]] = {}
for path in sorted(root.rglob("*.py")):
if "__pycache__" in path.parts:
continue
name = module_name(path, root)
# For a package's `__init__`, the containing package *is* this module, so `from .x import y` resolves to
# `<name>.x`; for a plain module it resolves against the parent. Getting this backwards made every module
# a package re-exports look unimported.
is_package = path.name == "__init__.py"
imported: set[str] = set()
try:
tree = ast.parse(path.read_text())
except (OSError, SyntaxError):
graph[name] = imported
continue
package_parts = name.split(".") if is_package else name.split(".")[:-1]
package_parts = [part for part in package_parts if part]
for node in ast.walk(tree):
if isinstance(node, ast.ImportFrom):
if node.level:
base = package_parts[: len(package_parts) - node.level + 1]
target = ".".join([*base, node.module]) if node.module else ".".join(base)
else:
target = node.module or ""
if target.startswith(f"{PACKAGE}."):
target = target[len(PACKAGE) + 1 :]
elif target == PACKAGE:
target = ""
else:
continue
if target:
imported.add(target)
elif isinstance(node, ast.Import):
for alias in node.names:
if alias.name.startswith(f"{PACKAGE}."):
imported.add(alias.name[len(PACKAGE) + 1 :])
graph[name] = imported
return graph
def _reachable(graph: Mapping[str, set[str]], roots: Iterable[str]) -> set[str]:
seen: set[str] = set()
stack = [root for root in roots if root in graph]
while stack:
current = stack.pop()
if current in seen:
continue
seen.add(current)
for target in graph.get(current, ()):
# an import of `a.b` also keeps the package `a` alive
for candidate in (target, target.rsplit(".", 1)[0]):
if candidate in graph and candidate not in seen:
stack.append(candidate)
return seen
def audit(root: Path) -> tuple[ModuleAudit, ...]:
"""Classify every module under ``root``."""
graph = import_graph(root)
importers: dict[str, set[str]] = {name: set() for name in graph}
for name, targets in graph.items():
for target in targets:
for candidate in (target, target.rsplit(".", 1)[0]):
if candidate in importers and candidate != name:
importers[candidate].add(name)
live = _reachable(graph, ROOTS)
results: list[ModuleAudit] = []
for name in sorted(graph):
if not name: # the package __init__ itself
continue
who = tuple(sorted(importers[name]))
if name in SPEC_OWNED:
results.append(ModuleAudit(name, "load_bearing", SPEC_OWNED[name], who))
elif name in ROOTS:
results.append(ModuleAudit(name, "load_bearing", "entry point", who))
elif name in STANDALONE:
results.append(ModuleAudit(name, "standalone_capability", STANDALONE[name], who))
elif name in live:
results.append(ModuleAudit(name, "load_bearing", f"reachable from {', '.join(ROOTS[:3])} or the CLI", who))
elif who:
results.append(ModuleAudit(name, "load_bearing", f"imported by {', '.join(who)}", who))
else:
results.append(ModuleAudit(name, "orphaned", "no importer, no entry point, no surviving requirement", who))
return tuple(results)
def to_dict(results: Iterable[ModuleAudit]) -> dict[str, object]:
rows = list(results)
counts: dict[str, int] = {}
for row in rows:
counts[row.classification] = counts.get(row.classification, 0) + 1
return {
"modules": len(rows),
"counts": dict(sorted(counts.items())),
"orphaned": [row.module for row in rows if row.classification == "orphaned"],
"standalone": {row.module: row.reason for row in rows if row.classification == "standalone_capability"},
"rows": [
{"module": row.module, "classification": row.classification, "reason": row.reason, "importers": list(row.importers)}
for row in rows
],
}
__all__ = ["ModuleAudit", "SPEC_OWNED", "STANDALONE", "audit", "import_graph", "module_name", "to_dict"]
src/openultrasast/learning/candidates.py → src/openultrasast/model/candidates.py
+45 -8
View File
@@ -15,7 +15,7 @@ the sequence of model calls, are determined by the input alone.
from __future__ import annotations
from collections.abc import Sequence
from collections.abc import Mapping, Sequence
from dataclasses import dataclass, field
from pathlib import Path
from typing import Literal
@@ -263,22 +263,20 @@ def ceiling(cases: Sequence[object], *, taxonomy: object, generator: str = "ir")
import tempfile
from ..pairs import _materialize_side, _overlay_scan, _quick_scan
from .rounds import _family_of
from .scoring import _spans
families: dict[str, dict[str, int]] = {}
slices: dict[str, dict[str, int]] = {}
gaps: list[tuple[str, str, str]] = []
sites: dict[str, list[int]] = {}
for case in cases:
family = _family_of(case, taxonomy) # type: ignore[arg-type]
family = labeled_family(case)
slice_name = str(getattr(case, "slice", "") or "")
region_path = str(getattr(case, "relpath", "") or "")
with tempfile.TemporaryDirectory(prefix="ousast-ceiling-") as scratch:
root = _materialize_side(Path(scratch) / "vuln", case, side="vuln") # type: ignore[arg-type]
if generator == "ruleset":
ranges = _overlay_scan(root).ranges
spans = _spans(case, ranges)
spans = labeled_spans(case, ranges)
found = [
finding
for finding in _quick_scan(root)
@@ -289,7 +287,7 @@ def ceiling(cases: Sequence[object], *, taxonomy: object, generator: str = "ir")
labeled = [str(getattr(row, "function", "") or "") for row in getattr(case, "expected", ()) or ()]
result = enumerate_candidates(
root,
_Region(region_path, next((name for name in labeled if name), None)),
Region(region_path, next((name for name in labeled if name), None)),
family,
taxonomy=taxonomy,
limit=10**6, # the ceiling is about coverage, never about the batch bound
@@ -331,9 +329,45 @@ def _distribution(counts: list[int]) -> dict[str, int]:
}
def labeled_family(case: object) -> str:
"""The family a pair is labeled with, or ``unknown``.
The deleted ``learning/rounds.py`` fell back to the LLM classifier here. Nothing does now: an unlabeled
pair is honestly ``unknown`` rather than guessed, which is the same discipline the evidence ladder applies
to a claim the model cannot arbitrate.
"""
for row in getattr(case, "expected", ()) or ():
family = getattr(row, "family", None)
if family:
return str(family)
return "unknown"
def labeled_spans(
case: object, ranges: Mapping[str, Sequence[tuple[str, int, int]]], *, function: str = ""
) -> tuple[tuple[str, int, int], ...]:
"""The (path, start, end) spans of the labeled functions; ``function`` overrides the label for the fixed side.
Re-homed from the deleted ``learning/scoring.py`` — the enumerator's ceiling needs it to know which region a
candidate had to fall inside, and it carries none of the K-run machinery that surrounded it.
"""
wanted = {function} if function else {str(getattr(row, "function", "") or "") for row in getattr(case, "expected", ()) or ()}
wanted.discard("")
found: list[tuple[str, int, int]] = []
for path, entries in ranges.items():
for name, start, end in entries:
if name in wanted:
found.append((path, start, end))
return tuple(found)
@dataclass(frozen=True)
class _Region:
"""The two fields `enumerate_candidates` reads, so the ceiling does not import the detector module."""
class Region:
"""The two fields ``enumerate_candidates`` reads: where to look, and which function bounds the search.
Public since the detector module that used to own this type was removed with the noise architecture — the
enumerator still needs a name for its input, and callers still need one to construct.
"""
path: str
function: str | None
@@ -347,5 +381,8 @@ __all__ = [
"CandidateKind",
"CandidateReport",
"CandidateSet",
"Region",
"enumerate_candidates",
"labeled_family",
"labeled_spans",
]
src/openultrasast/learning/endpoint.py → src/openultrasast/model/endpoint.py
+5 -5
View File
@@ -154,7 +154,7 @@ def resolve_chat_endpoint(config: ResolvedConfig, *, override: ChatClient | None
if configured or key_env:
key = os.environ.get(key_env or DEEPSEEK_KEY_ENV, "")
base_url = configured or DEEPSEEK_BASE_URL
return _deepseek_client(key, base_url, model, provider=_provider_of(base_url), thinking=config.learning.thinking)
return _deepseek_client(key, base_url, model, provider=_provider_of(base_url), thinking=config.models.thinking)
deepseek_key = os.environ.get(DEEPSEEK_KEY_ENV)
if deepseek_key:
return _deepseek_client(
@@ -162,7 +162,7 @@ def resolve_chat_endpoint(config: ResolvedConfig, *, override: ChatClient | None
os.environ.get(DEEPSEEK_BASE_ENV, DEEPSEEK_BASE_URL),
model,
provider="deepseek",
thinking=config.learning.thinking,
thinking=config.models.thinking,
)
if os.environ.get("OPENROUTER_API_KEY"):
try:
@@ -175,9 +175,9 @@ def resolve_chat_endpoint(config: ResolvedConfig, *, override: ChatClient | None
def resolve_models(config: ResolvedConfig) -> tuple[str, str]:
"""(detector model, judge model). `[learning]` wins, then `[models]`, then the shipped defaults."""
detector = config.learning.detector_model or config.models.hunter or DEFAULT_DETECTOR_MODEL
judge = config.learning.judge_model or config.models.judge or DEFAULT_JUDGE_MODEL
"""(detector model, judge model) from `[models]`, then the shipped defaults."""
detector = config.models.hunter or DEFAULT_DETECTOR_MODEL
judge = config.models.judge or DEFAULT_JUDGE_MODEL
return detector, judge
src/openultrasast/model/specs.py
+156
View File
@@ -0,0 +1,156 @@
"""The security vocabularies the model layer arbitrates with (model-grounded-detection, Req 7.2-7.4).
Two kinds of knowledge live here, and they are held differently on purpose.
The **guard vocabulary** is ported verbatim from ``semantic/variants.py``, whose optimisation loop this
feature deleted. Nothing else holds it: it is the closed set of ways a fix discharges a flow, and it tells
the model what a *fixed* twin looks like. Order matters — first match wins.
The **taint and dominance specs** are *derived* from data that survives: the source/sink/sanitizer facts in
``ruleset/semantic`` and the obligated operations and dischargers in ``ruleset/obligations``. Copying them
here would create a second place to update, so this module reads them through the retained loaders and
groups them by family via the taxonomy's CWE map.
Read-only by construction (Req 7.4, the verifier boundary): this module exposes no writer, opens no file for
writing, and holds no mutable module state. An LLM or a future optimiser has nothing here to rewrite.
"""
from __future__ import annotations
import re
from collections.abc import Mapping
from dataclasses import dataclass
from ..semantic.facts import SemanticFacts, load_facts
from ..semantic.obligations.facts import ObligationFacts, load_obligation_facts
from .taxonomy import FamilyTaxonomy, load_families
# --- The guard vocabulary (ported verbatim; nothing else holds it) ----------------------------------------
GUARD_KINDS = ("null_test", "bounds_test", "allowlist_test", "auth_check", "parameterized_call", "type_change", "none")
# Closed guard classification over the statements a fix added inside the labeled function.
# Order matters: the first matching kind wins. These classify the *fix*, they never detect.
GUARD_PATTERNS: tuple[tuple[str, re.Pattern[str]], ...] = (
("parameterized_call", re.compile(r"execute\w*\([^,]+,\s*[\(\[]|prepareStatement\(|\?\s*['\"]?\s*[,)]|\$\d\b|%s['\"]\s*,\s*[\(\[]")),
(
"auth_check",
re.compile(
r"\b(request\.user|req\.user|current_user|request\.state|session\[|is_authenticated|authoriz|require_auth"
r"|login_required|verify_token|check_permission|owner_id\s*[!=]=)"
),
),
(
"allowlist_test",
re.compile(
r"\bnot in\b|\bin\s*[\(\{\[]|\.includes\(|\.has\(|\bstartswith\(|\bstartsWith\(|\bendswith\(|\bendsWith\("
r"|re\.(?:match|fullmatch)\(|\.test\(|allow_?list|whitelist|\bALLOWED\b|\bsafe_join\(|secure_filename\("
r"|\brealpath\(|\bnormpath\(|\bresolve\("
),
),
("bounds_test", re.compile(r"\blen\(|\.length\b|\bsizeof\(|\bstrn?len\(|\bsnprintf\(|\bstrlcpy\(|\bmin\(|\bmax\(|\s(?:<|>|<=|>=)\s")),
(
"null_test",
re.compile(
r"\bis None\b|\bis not None\b|[!=]= *NULL\b|[!=]==? *null\b|\bnullptr\b|\bif\s*\(\s*!\s*\w|\bif not \w+\s*:"
r"|\bif\s+\w+\s+is\s+None|\?\?|\?\."
),
),
(
"type_change",
re.compile(r"\bint\(|\bparseInt\(|\bNumber\(|\(int\)|\(size_t\)|\(unsigned\b|\bstr\(|\.toString\(|\bstatic_cast<|\bBigInt\("),
),
)
def classify_guard_text(text: str) -> str:
"""The guard kind the given statements express, or ``none`` when no known guard is recognised."""
for kind, pattern in GUARD_PATTERNS:
if pattern.search(text):
return kind
return "none"
# --- The specs the CPG queries are parameterised by ------------------------------------------------------
@dataclass(frozen=True)
class TaintSpec:
"""What a flow family's source→sink−sanitizer query matches on, for one language."""
family: str
language: str
sources: tuple[str, ...]
sinks: tuple[str, ...]
sanitizers: tuple[str, ...]
@dataclass(frozen=True)
class DominanceSpec:
"""What an absence family's guard-dominates-operation query matches on, for one language.
``dischargers`` are the guards that discharge an obligation; the query asks whether one of them dominates
the operation on every path, which is the arbiter for the classes that never crash.
"""
family: str
language: str
operations: tuple[str, ...]
dischargers: tuple[str, ...]
def taint_specs(
*,
language: str,
facts: SemanticFacts | None = None,
taxonomy: FamilyTaxonomy | None = None,
) -> Mapping[str, TaintSpec]:
"""One ``TaintSpec`` per flow family that has a sink in this language, grouped through the taxonomy's CWE map.
Sources and sanitizers are language-wide: a family does not get its own notion of what untrusted input is,
and a sanitizer that breaks one flow breaks it whatever the sink's family.
"""
scoped = (facts if facts is not None else load_facts()).for_language(language)
families = taxonomy if taxonomy is not None else load_families()
sources = tuple(sorted({pattern for fact in scoped.sources for pattern in fact.patterns}))
sanitizers = tuple(sorted({call for fact in scoped.sanitizers for call in fact.calls}))
by_family: dict[str, set[str]] = {}
for sink in scoped.sinks:
family = families.family_of_cwe(sink.cwe)
if family is None:
continue
by_family.setdefault(family.id, set()).update(sink.calls)
return {
family_id: TaintSpec(family=family_id, language=language, sources=sources, sinks=tuple(sorted(calls)), sanitizers=sanitizers)
for family_id, calls in sorted(by_family.items())
}
def dominance_specs(*, language: str, facts: ObligationFacts | None = None) -> Mapping[str, DominanceSpec]:
"""One ``DominanceSpec`` per absence family for this language.
The obligation facts are authored for authorization, so every obligated operation routes to
``access_control``; the shape generalises when a second absence family is added to the taxonomy.
"""
scoped = (facts if facts is not None else load_obligation_facts()).for_language(language)
if not scoped.operations:
return {}
operations = tuple(sorted({call for fact in scoped.operations for call in fact.calls}))
dischargers = tuple(
sorted(
{call for fact in scoped.dischargers for call in fact.calls}
| {decorator for fact in scoped.dischargers for decorator in fact.decorators}
)
)
return {"access_control": DominanceSpec(family="access_control", language=language, operations=operations, dischargers=dischargers)}
__all__ = [
"GUARD_KINDS",
"GUARD_PATTERNS",
"DominanceSpec",
"TaintSpec",
"classify_guard_text",
"dominance_specs",
"taint_specs",
]
src/openultrasast/learning/families.py → src/openultrasast/model/taxonomy.py
View File
src/openultrasast/pairs.py
+1 -197
View File
@@ -20,7 +20,7 @@ import os
import shutil
import tempfile
import tomllib
from collections.abc import Callable, Iterable, Mapping, Sequence
from collections.abc import Iterable, Mapping, Sequence
from dataclasses import asdict, dataclass, field, replace
from functools import lru_cache
from pathlib import Path
@@ -98,7 +98,6 @@ _DEFAULT_TIER = {
_DEFAULT_PROVENANCE = {"sast": "synthetic", "local": "human", "github": "human", "vfc": "human"}
_MANIFEST_MECHANISM = "source_reaches_sink" # cheat-sheet fixtures referenced via expected_from
HunterScan = Callable[[Path], list[StaticFinding]]
class CatalogError(ValueError):
@@ -173,7 +172,6 @@ class PairOutcome:
unadjudicated_vuln: int = 0
unadjudicated_fixed: int = 0
detection_kinds: tuple[str, ...] = ()
family: str = "" # the family this pair was scored under (hunter path only)
unscorable: str | None = None # why it could not be scored; it then leaves every denominator
unresolved_labels: int = 0 # expected rows whose function names no range on the vulnerable side
review_tier: str = "advisory"
@@ -205,7 +203,6 @@ class PairEvalResult:
per_profile: dict[str, PairCorpusMetrics] = field(default_factory=dict)
per_tier: dict[str, PairCorpusMetrics] = field(default_factory=dict)
per_mechanism: dict[str, PairCorpusMetrics] = field(default_factory=dict)
per_family: dict[str, dict[str, object]] = field(default_factory=dict) # learning-harness Req 3.3: family metrics beside the rest
achievable: dict[str, PairCorpusMetrics] = field(default_factory=dict)
known_limit: tuple[str, ...] = ()
loss: dict[str, dict[str, int]] = field(default_factory=dict)
@@ -427,15 +424,11 @@ def _review_tier(item: dict[str, object], slice_name: str, name: str) -> tuple[s
def evaluate_pair(
case: PairCase,
*,
hunter: HunterScan | None = None,
ruleset: tuple[PatternRule, ...] | None = None,
k_runs: int | None = None,
) -> PairOutcome:
with tempfile.TemporaryDirectory(prefix="ousast-pair-") as scratch:
vuln_root = _materialize_side(Path(scratch) / "vuln", case, side="vuln")
fix_root = _materialize_side(Path(scratch) / "fixed", case, side="fixed")
if hunter is not None:
return _evaluate_hunter_pair(case, vuln_root, fix_root, hunter, _k_runs(k_runs))
if case.slice in OVERLAY_SLICES:
return _evaluate_overlay_pair(case, vuln_root, fix_root, ruleset)
return _evaluate_inventory_pair(case, vuln_root, fix_root, ruleset=ruleset)
@@ -444,19 +437,15 @@ def evaluate_pair(
def evaluate_catalog(
cases: Sequence[PairCase],
*,
hunter: HunterScan | None = None,
ruleset: tuple[PatternRule, ...] | None = None,
inventory_sidecar: bool = True,
pointers: bool | None = None,
k_runs: int | None = None,
) -> PairEvalResult:
"""Score every case. ``ruleset`` lets the improve loop score a candidate ledger; ``inventory_sidecar=False`` skips the second scan.
Pointer pairs (``vendored = false``) are harvested into the cache when ``pointers`` (or OPENULTRASAST_PAIRS_NETWORK=1) allows the
network, otherwise skipped with a ``pointer_pair_skipped`` degradation (Req 10.2, 10.3).
"""
if hunter is not None:
_k_runs(k_runs) # refuse an impossible K before spending a single call
degradations: list[dict[str, object]] = []
ready: list[PairCase] = []
for case in cases:
@@ -468,18 +457,11 @@ def evaluate_catalog(
cases = tuple(ready)
outcomes = tuple(evaluate_pair(case, ruleset=ruleset) for case in cases)
scorers: dict[str, dict[str, PairCorpusMetrics]] = {}
hunter_outcomes: list[PairOutcome] = [] # the family scorer runs on the hunter path only
for slice_name in sorted({case.slice for case in cases if case.slice in OVERLAY_SLICES}):
slice_cases = tuple(case for case in cases if case.slice == slice_name)
scorers[slice_name] = {"overlay": _metrics(tuple(item for item in outcomes if item.slice == slice_name))}
if inventory_sidecar:
scorers[slice_name]["inventory"] = _metrics(tuple(_inventory_only(case, ruleset) for case in slice_cases))
if hunter is not None:
hunted = tuple(evaluate_pair(case, hunter=hunter, k_runs=k_runs) for case in slice_cases)
hunter_outcomes.extend(hunted)
scorers[slice_name]["hunter"] = _metrics(hunted)
if hunter is None and scorers:
degradations.append({"stage": "pairs", "reason": "hunter_model_unavailable"})
achievable_outcomes = tuple(item for item in outcomes if item.known_limit is None)
per_mechanism: dict[str, list[PairOutcome]] = {}
for item in outcomes:
@@ -503,7 +485,6 @@ def evaluate_catalog(
for tier in sorted({item.review_tier for item in outcomes})
},
per_mechanism={mechanism: _metrics(tuple(items)) for mechanism, items in sorted(per_mechanism.items())},
per_family=_family_metrics(hunter_outcomes),
achievable={
slice_name: _metrics(tuple(item for item in achievable_outcomes if item.slice == slice_name))
for slice_name in sorted({item.slice for item in achievable_outcomes})
@@ -514,43 +495,6 @@ def evaluate_catalog(
)
def _family_metrics(outcomes: Sequence[PairOutcome]) -> dict[str, dict[str, object]]:
"""Per-family numbers for the pairs the family scorer ran, with unscorable rows out of the denominator."""
from .learning.families import load_families
from .learning.scoring import PairFamilyScore, aggregate
scored = [item for item in outcomes if item.family]
if not scored:
return {}
taxonomy = load_families()
scores = [
PairFamilyScore(
pair=item.name,
family=item.family,
runs=()
if item.unscorable
else (
(
("pair_correct" if item.pair_correct else "both_flagged")
if item.detected_vuln
else ("reversed" if not item.silent_fix else "both_silent")
),
),
outcome="unscorable"
if item.unscorable
else (
("pair_correct" if item.pair_correct else "both_flagged")
if item.detected_vuln
else ("reversed" if not item.silent_fix else "both_silent")
),
slice=item.slice,
unscorable_reason=item.unscorable,
)
for item in scored
]
return {name: metrics.to_dict() for name, metrics in aggregate(scores, taxonomy=taxonomy).items()}
def build_pair_signals(outcomes: Sequence[PairOutcome], *, split: str | None = None) -> list[dict[str, object]]:
"""Same miss/fp vocabulary as ``build_rule_signals``, tagged with the pair name, profile, and mechanisms.
@@ -737,7 +681,6 @@ def result_payload(result: PairEvalResult) -> dict[str, object]:
"per_profile": {name: asdict(metrics) for name, metrics in result.per_profile.items()},
"per_tier": {name: asdict(metrics) for name, metrics in result.per_tier.items()},
"per_mechanism": {name: asdict(metrics) for name, metrics in result.per_mechanism.items()},
"per_family": dict(result.per_family),
"achievable": {name: asdict(metrics) for name, metrics in result.achievable.items()},
"known_limit": list(result.known_limit),
"loss": {name: dict(counts) for name, counts in result.loss.items()},
@@ -964,53 +907,6 @@ def _with_functions(root: Path, findings: list[StaticFinding]) -> list[StaticFin
return out
def make_hunter_scan(client: object, model: str, *, max_steps: int | None = None, taxonomy: object | None = None) -> HunterScan:
"""Run the tool hunter over every file of an isolated pair tree (Req 1.7).
With a ``taxonomy`` the hunter is asked to name the family of every finding, from the closed list, and the answer
becomes the finding's `family:` tag. The class-aware scorer credits a detection only to a family, so a generalist
that names none can never be scored — it would report recall zero by construction rather than by measurement.
"""
from .complexity.map import Hotspot
from .tool_hunter import _SYSTEM_PROMPT, DEFAULT_MAX_STEPS, run_tool_hunter
prompt = _SYSTEM_PROMPT
if taxonomy is not None:
ids = ", ".join(family.id for family in taxonomy.families) # type: ignore[attr-defined]
prompt = (
f'{_SYSTEM_PROMPT}\n\nGive every finding a "family" field, chosen from exactly this list: {ids}. '
'Use "unknown" when none of them fits; do not invent a name.'
)
def scan(root: Path) -> list[StaticFinding]:
hotspots = [
Hotspot(
path=target.path,
function_name=None,
score=1.0,
band="high",
signals={},
rationale="pair corpus function",
test_hint=None,
inventory_finding_ids=(),
)
for target in _targets(root)
]
return _with_functions(
root,
run_tool_hunter(
root,
hotspots,
client=client, # type: ignore[arg-type]
model=model,
max_steps=max_steps or DEFAULT_MAX_STEPS,
system_prompt=prompt,
),
)
return scan
def _inventory_outcome(
case: PairCase,
vuln_findings: list[StaticFinding],
@@ -1146,98 +1042,6 @@ def _evaluate_overlay_pair(case: PairCase, vuln_root: Path, fix_root: Path, rule
)
def _k_runs(value: int | None) -> int:
"""Req 3.5: three runs per pair per side is the floor, and the default, because one run misses most changes."""
from .learning.rounds import MIN_K_RUNS
runs = MIN_K_RUNS if value is None else int(value)
if runs < MIN_K_RUNS:
raise ValueError(f"the hunter path scores at least three runs per pair; {runs} was asked for")
return runs
def _evaluate_hunter_pair(case: PairCase, vuln_root: Path, fix_root: Path, hunter: HunterScan, runs: int) -> PairOutcome:
"""Score the model-driven hunter by family and function (learning-harness Req 3.1-3.3, 3.5).
Detection is a finding of the labeled family inside the labeled function; a finding of another family
or outside it is counted and never called a leak; the words in a finding are never read. A pair that
cannot be scored is reported with its reason and the detector is never run on it.
"""
from .learning.families import load_families
from .learning.scoring import score_pair_family, unscorable_reason
taxonomy = load_families()
vuln_ranges = _overlay_scan(vuln_root).ranges
reason = unscorable_reason(case, parse_ok=bool(vuln_ranges), ranges=vuln_ranges)
family = _family_of(case, taxonomy)
if reason is not None:
return _hunter_outcome(case, family=family, unscorable=reason)
fix_ranges = _overlay_scan(fix_root).ranges
vuln_runs = [hunter(vuln_root) for _ in range(runs)]
fix_runs = [hunter(fix_root) for _ in range(runs)]
score = score_pair_family(case, family, vuln_runs, fix_runs, ranges=vuln_ranges, taxonomy=taxonomy, fixed_ranges=fix_ranges)
return _hunter_outcome(case, family=family, score=score, vuln=vuln_runs[0], fixed=fix_runs[0])
def _family_of(case: PairCase, taxonomy: object) -> str:
"""The family this pair is scored under: its label when it has one, else what the classifier says."""
from .learning.classify import classify_pair
for row in case.expected:
label = getattr(row, "family", None)
if label:
return str(label)
answer = classify_pair(case, taxonomy, client=None) # type: ignore[arg-type]
return answer.families[0] if answer.families else "unknown"
def _hunter_outcome(
case: PairCase,
*,
family: str,
score: object | None = None,
vuln: Sequence[StaticFinding] = (),
fixed: Sequence[StaticFinding] = (),
unscorable: str | None = None,
) -> PairOutcome:
outcome = getattr(score, "outcome", "unscorable")
detected = outcome in {"pair_correct", "both_flagged"}
silent = outcome in {"pair_correct", "both_silent"}
leaks = tuple(finding.finding_id for finding in fixed) if outcome in {"both_flagged", "reversed"} else ()
misses = () if detected else tuple(f"{row.rule_id or '-'}:{row.path}:{row.cwe}" for row in case.expected)
return PairOutcome(
name=case.name,
slice=case.slice,
language=case.language,
origin=case.origin,
detected_vuln=detected,
silent_fix=silent,
pair_correct=outcome == "pair_correct",
vuln_matched=1 if detected else 0,
vuln_expected=len(case.expected),
vuln_findings=len(vuln),
fix_findings=len(fixed),
fix_leaks=len(leaks),
recall=1.0 if detected else 0.0,
misses=misses,
leaks=leaks,
commit_url=case.commit_url,
cve=case.cve,
provenance=case.provenance,
review_tier=case.review_tier,
mechanisms=_mechanisms(case),
known_limit=case.known_limit,
split=case.split,
weak_labels=0,
detection_kinds=("hunter",) if detected else (),
family=family,
unscorable=unscorable or (str(getattr(score, "unscorable_reason", "") or "") or None),
)
# --- matching ---------------------------------------------------------------
def _adjudicated(records: Sequence[OverlayRecord]) -> bool:
return any(record.disposition in {"promote", "demote", "coverage"} for record in records)
src/openultrasast/reports.py
-23
View File
@@ -22,7 +22,6 @@ def write_markdown_report(
mechanisms: Mapping[str, Mapping[str, object]] | None = None,
obligations: Mapping[str, Mapping[str, object]] | None = None,
obligations_summary: Mapping[str, object] | None = None,
learning: Mapping[str, Mapping[str, object]] | None = None,
) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
verification_by_id = _verification_by_id(verifications or [])
@@ -73,8 +72,6 @@ def write_markdown_report(
_append_mechanisms(lines, cited)
if cited_obligations:
_append_obligations(lines, cited_obligations, obligations_summary)
if learning:
_append_learning(lines, learning)
if overlay:
_append_overlay(lines, overlay)
if complexity_map is not None:
@@ -158,7 +155,6 @@ def write_sarif_report(
overlay: Sequence[object] | None = None,
mechanisms: Mapping[str, Mapping[str, object]] | None = None,
obligations: Mapping[str, Mapping[str, object]] | None = None,
learning: Mapping[str, Mapping[str, object]] | None = None,
) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
verification_by_id = _verification_by_id(verifications)
@@ -183,7 +179,6 @@ def write_sarif_report(
overlay_by_id.get(finding.finding_id),
mechanisms=mechanisms,
obligation=(obligations or {}).get(finding.finding_id),
learning=(learning or {}).get(finding.finding_id),
)
for finding in findings
],
@@ -209,7 +204,6 @@ def write_manifest(
provenance: dict[str, object] | None = None,
variants: dict[str, object] | None = None,
obligations: dict[str, object] | None = None,
learning: dict[str, object] | None = None,
) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
verification_by_id = _verification_by_id(verifications)
@@ -250,8 +244,6 @@ def write_manifest(
payload["variants"] = variants
if obligations is not None:
payload["obligations"] = obligations
if learning:
payload["learning"] = learning
path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
@@ -295,7 +287,6 @@ def _sarif_result(
*,
mechanisms: Mapping[str, Mapping[str, object]] | None = None,
obligation: Mapping[str, object] | None = None,
learning: Mapping[str, object] | None = None,
) -> dict[str, object]:
properties: dict[str, object] = {
"finding_id": finding.finding_id,
@@ -327,10 +318,6 @@ def _sarif_result(
properties["obligation_evidence"] = _strings_of(obligation, "evidence")
properties["obligation_known_fix"] = obligation.get("known_fix")
properties["obligation_intent"] = obligation.get("intent")
if learning is not None:
properties["family"] = str(learning.get("family", ""))
properties["detector"] = str(learning.get("detector", ""))
properties["verifier"] = str(learning.get("verifier", ""))
return {
"ruleId": _rule_id(finding),
"level": _sarif_level(finding.severity),
@@ -486,16 +473,6 @@ def _append_obligations(lines: list[str], cited: Mapping[str, Mapping[str, objec
lines.append("")
def _append_learning(lines: list[str], cited: Mapping[str, Mapping[str, object]]) -> None:
"""Which family found a claim, which detector version produced it, and what a verifier said about it."""
lines.extend(["## Classified detectors", "", "Findings from the per-family detectors, with what confirmed them.", ""])
for finding_id, info in sorted(cited.items()):
lines.append(
f"- `{finding_id}`: family `{info.get('family')}`, detector `{info.get('detector')}`, verifier `{info.get('verifier')}`"
)
lines.append("")
def _strings_of(info: Mapping[str, object], key: str) -> list[str]:
value = info.get(key)
return [str(item) for item in value] if isinstance(value, list | tuple) else []
src/openultrasast/semantic/__init__.py
-5
View File
@@ -3,26 +3,21 @@
from .engines import joern_available, tree_sitter_available
from .extra import grammar_for, has_semantic_extra
from .facts import FactLoadError, SemanticFacts, load_facts
from .mechanisms import Mechanism, MechanismStore, append_mechanism, order_promotions
from .overlay import OverlayError, OverlayRecord, adjudicate, finding_from_coverage, write_overlay
from .prove_filter import filter_promoted_hotspots
__all__ = [
"FactLoadError",
"Mechanism",
"MechanismStore",
"OverlayError",
"OverlayRecord",
"SemanticFacts",
"adjudicate",
"append_mechanism",
"filter_promoted_hotspots",
"finding_from_coverage",
"grammar_for",
"has_semantic_extra",
"joern_available",
"load_facts",
"order_promotions",
"tree_sitter_available",
"write_overlay",
]
src/openultrasast/semantic/ir.py
+12
View File
@@ -3,6 +3,7 @@
from __future__ import annotations
import ast
import re
from dataclasses import dataclass
@@ -230,3 +231,14 @@ def _unparse(node: ast.AST) -> str:
return ast.unparse(node)
except Exception:
return ""
def trailing_name(callee: str) -> str:
"""``os.path.join`` -> ``join``; ``Runtime.getRuntime().exec`` -> ``exec``; keeps the last identifier segment.
Re-homed here from the deleted ``semantic/variants.py``: it is a pure helper over ``CallSite.name`` and has
nothing to do with the mechanism search that used to surround it.
"""
tail = callee.rsplit(".", 1)[-1].rsplit("::", 1)[-1].rsplit("->", 1)[-1]
match = re.search(r"[A-Za-z_$][\w$]*\s*$", tail)
return match.group(0).strip() if match else tail
src/openultrasast/semantic/loo.py
-304
View File
@@ -1,304 +0,0 @@
"""Leave-one-out: the pair corpus's own detection rate (corpus-seeded-mechanisms, Req 4).
For every scorable pair, a temporary mechanism store is seeded from every *other* trusted pair, both excerpts are
materialized and searched for variants, and the pair is scored with the pair rules: detected when a hit lies inside the
labeled function and names the labeled mechanism (or sink), silent when the fixed side yields no hit. Only ``seeded`` and
``reviewed`` pairs teach; ``advisory`` and ``title`` pairs are scored as held-out targets but never seed. Reported, never
gated.
Obligation-labeled rows (authorization-obligations, Req 7.4) run the obligation checker on both sides with the held-out
store's obligation shapes: detected when a finding tagged with the labeled obligation lies inside the labeled function *and*
carries a ``known_fix`` supplied by the store (the checker's own facts-only finding is not the corpus teaching), silent when
the fixed side carries no such finding; ``found_by`` names the shape record and the pairs that taught it.
"""
from __future__ import annotations
import tempfile
from collections.abc import Sequence
from dataclasses import dataclass, field
from pathlib import Path
from ..findings import StaticFinding
from ..pairs import PairCase, _materialize, _targets
from .facts import FactLoadError, SemanticFacts, load_facts
from .functions import named_function_ranges, spans_named
from .mechanisms import Mechanism, MechanismStore, append_from_pair, corpus_mechanisms
from .obligations import check_obligations, findings_to_static, load_obligation_facts, obligation_mechanisms
from .obligations.dominance import OrderDominance
from .seed import Lesson, _parse_language, pair_lessons
from .variant_search import search_tree
from .variants import Shape, trailing_name
@dataclass(frozen=True)
class LooOutcome:
pair: str
slice: str
provenance: str
mechanisms: tuple[str, ...] # labeled mechanism ids of the held-out pair
detected: bool
silent: bool
found_by: tuple[tuple[str, tuple[str, ...]], ...] # (mechanism record id, pairs that taught it) for detecting hits
hits_vuln: int
hits_fixed: int
teaches: bool # the pair contributed at least one shape when others were held out
leaked_by: tuple[str, ...] = () # mechanism record ids that hit the fixed side
degradations: tuple[dict[str, object], ...] = () # from variant search on either side
obligations: tuple[str, ...] = () # labeled obligation kinds of the held-out pair (Req 7.4)
def to_dict(self) -> dict[str, object]:
return {
"pair": self.pair,
"slice": self.slice,
"provenance": self.provenance,
"mechanisms": list(self.mechanisms),
"detected": self.detected,
"silent": self.silent,
"found_by": [{"mechanism_id": mechanism_id, "pairs": list(pairs)} for mechanism_id, pairs in self.found_by],
"hits_vuln": self.hits_vuln,
"hits_fixed": self.hits_fixed,
"teaches": self.teaches,
"leaked_by": list(self.leaked_by),
"obligations": list(self.obligations),
}
@dataclass(frozen=True)
class LooResult:
outcomes: tuple[LooOutcome, ...]
per_slice: dict[str, dict[str, float]]
per_profile: dict[str, dict[str, float]]
per_mechanism: dict[str, dict[str, float]]
skipped: tuple[tuple[str, str], ...] = ()
per_obligation_kind: dict[str, dict[str, float]] = field(default_factory=dict)
teaching_pairs: int = 0
degradations: tuple[dict[str, object], ...] = field(default=())
def to_dict(self) -> dict[str, object]:
return {
"per_slice": self.per_slice,
"per_profile": self.per_profile,
"per_mechanism": self.per_mechanism,
"per_obligation_kind": self.per_obligation_kind,
"outcomes": [outcome.to_dict() for outcome in self.outcomes],
"skipped": [{"pair": pair, "reason": reason} for pair, reason in self.skipped],
"teaching_pairs": self.teaching_pairs,
"degradations": list(self.degradations),
}
def evaluate_loo(cases: Sequence[PairCase], *, facts: SemanticFacts | None = None) -> LooResult:
loaded = facts if facts is not None else _facts()
targets: list[PairCase] = []
skipped: list[tuple[str, str]] = []
for case in cases:
if case.unscorable or case.known_limit:
# `PairCase.unscorable` verbatim when the loader computed one: one spelling per corpus fact, and it
# already carries the `known_limit:` prefix when the limit is a maintainer's own word rather than one
# of the vocabulary's. A case built by hand may carry only the declaration.
skipped.append((case.name, case.unscorable or f"known_limit:{case.known_limit}"))
elif not case.vuln_file.is_file() or not case.fixed_file.is_file():
skipped.append((case.name, "pointer_pair_not_cached" if not case.vendored else "excerpt_missing"))
else:
targets.append(case)
from ..learning.split import GATING_TIERS, holdout_names, is_teacher, refuse_if_holdout
# learning-harness Req 5.1: every pair is still scored, but only a train-split teacher may seed a store.
lessons = {case.name: pair_lessons(case, loaded) for case in targets if is_teacher(case)}
outcomes = [_hold_out(case, targets, lessons, loaded) for case in targets]
# Req 5.2: the holdout pairs kept out of the teaching set are named, not silently dropped. Non-teachers held
# back for their tier or their unscorable reason are already in `skipped` with that reason.
excluded = [case.name for case in targets if case.name not in lessons]
refusal = refuse_if_holdout(excluded, targets)
refusals = (refusal.degradation(),) if refusal is not None else ()
withheld = tuple(
(case.name, "tier" if case.review_tier not in GATING_TIERS else "unscorable")
for case in targets
if case.name in set(excluded) - set(holdout_names(targets))
)
skipped.extend((name, f"not_a_teacher:{why}") for name, why in withheld)
return LooResult(
outcomes=tuple(outcomes),
per_slice=_group(outcomes, lambda item: (item.slice,)),
per_profile=_group(outcomes, lambda item: (item.provenance,)),
per_mechanism=_group(outcomes, lambda item: item.mechanisms),
per_obligation_kind=_group(outcomes, lambda item: item.obligations),
skipped=tuple(skipped),
teaching_pairs=sum(1 for shapes in lessons.values() if shapes),
degradations=(*refusals, *_unique_degradations(outcomes)),
)
def _unique_degradations(outcomes: Sequence[LooOutcome]) -> tuple[dict[str, object], ...]:
seen: set[str] = set()
unique: list[dict[str, object]] = []
for outcome in outcomes:
for item in outcome.degradations:
key = repr(sorted(item.items()))
if key not in seen:
seen.add(key)
unique.append(item)
return tuple(unique)
def _hold_out(case: PairCase, targets: Sequence[PairCase], lessons: dict[str, list[Lesson]], facts: SemanticFacts) -> LooOutcome:
with tempfile.TemporaryDirectory(prefix="ousast-loo-") as scratch:
root = Path(scratch)
store = MechanismStore(root / "mechanisms.jsonl")
for teacher in targets:
if teacher.name == case.name:
continue
for lesson in lessons.get(teacher.name, ()):
append_from_pair(
store,
lesson.shape,
summary=lesson.summary,
cwe=lesson.cwe,
pair=teacher.name,
provenance=teacher.provenance,
tier=teacher.review_tier,
)
scored = score_pair_with_store(case, store, facts, root=root)
return LooOutcome(**{**scored.__dict__, "teaches": bool(lessons.get(case.name))})
def score_pair_with_store(case: PairCase, store: MechanismStore, facts: SemanticFacts, *, root: Path | None = None) -> LooOutcome:
"""Search both sides of one pair with a fixed store and score it with the pair rules (shared by LOO and the lever)."""
labeled_mechanisms = tuple(sorted({row.mechanism for row in case.expected if row.mechanism}))
labeled_obligations = tuple(sorted({str(row.obligation) for row in case.expected if getattr(row, "obligation", None)}))
loaded = store.load()
records = {record.id: record for record in corpus_mechanisms(loaded)}
obligation_vuln: list[StaticFinding] = []
obligation_fix: list[StaticFinding] = []
with tempfile.TemporaryDirectory(prefix="ousast-loo-pair-") as scratch:
base = root if root is not None else Path(scratch)
vuln_root = _materialize(base / "vuln", case.vuln_file, case.relpath)
fix_root = _materialize(base / "fixed", case.fixed_file, case.relpath)
vuln_search = search_tree(vuln_root, _targets(vuln_root), store, facts, max_mechanisms=len(records) or 1)
fix_search = search_tree(fix_root, _targets(fix_root), store, facts, max_mechanisms=len(records) or 1)
vuln_hits, fix_hits = vuln_search.hits, fix_search.hits
if labeled_obligations:
shapes = obligation_mechanisms(loaded)
obligation_vuln = _obligation_findings(vuln_root, shapes)
obligation_fix = _obligation_findings(fix_root, shapes)
text = case.vuln_file.read_text(errors="ignore")
ranges = named_function_ranges(case.relpath, text, _parse_language(case)) or ()
found: list[tuple[str, tuple[str, ...]]] = []
for finding in obligation_vuln:
record = _known_fix_record(finding, records)
if record is None or finding.line is None or not _inside_label(case, finding.line, ranges):
continue
if any(f"obligation:{kind}" in finding.tags for kind in labeled_obligations):
found.append((record.id, tuple(record.pairs)))
obligation_leaks = tuple(
sorted({record.id for finding in obligation_fix if (record := _known_fix_record(finding, records)) is not None})
)
for hit in vuln_hits:
record = records.get(hit.mechanism_id)
if record is None or not _inside_label(case, hit.line, ranges):
continue
if _names_label(case, record, hit.sink_name):
found.append((record.id, tuple(record.pairs)))
return LooOutcome(
pair=case.name,
slice=case.slice,
provenance=case.provenance,
mechanisms=labeled_mechanisms,
detected=bool(found),
silent=not fix_hits and not obligation_leaks,
found_by=tuple(found),
hits_vuln=len(vuln_hits) + len(obligation_vuln),
hits_fixed=len(fix_hits) + len(obligation_fix),
teaches=False,
leaked_by=tuple(sorted({hit.mechanism_id for hit in fix_hits})) + obligation_leaks,
degradations=tuple(vuln_search.degradations) + tuple(fix_search.degradations),
obligations=labeled_obligations,
)
def _obligation_findings(root: Path, shapes: Sequence[object]) -> list[StaticFinding]:
"""Function-local obligation findings for one materialized side, with the held-out store's obligation shapes as known fixes."""
from ..config import ObligationsConfig
from ..mapping import analyze_entry_points
from .ir import parse_file
targets = _targets(root)
irs: dict[str, tuple[object, str]] = {}
texts: dict[str, str] = {}
for target in targets:
try:
text = (root / target.path).read_text(errors="ignore")
except OSError:
continue
texts[target.path] = text
irs[target.path] = (parse_file(target.path, text, target.language), text)
result = check_obligations(
irs=irs, # type: ignore[arg-type]
entries=analyze_entry_points(root, targets),
facts=load_obligation_facts(),
flow_facts=_facts(),
policy=None,
paths=(),
dominance=OrderDominance(texts=texts),
store_shapes=shapes,
min_siblings=ObligationsConfig().min_siblings,
)
return findings_to_static(result)
def _known_fix_record(finding: StaticFinding, records: dict[str, Mechanism]) -> Mechanism | None:
for tag in finding.tags:
if tag.startswith("mechanism:"):
return records.get(tag.split(":", 1)[1])
return None
def _inside_label(case: PairCase, line: int, ranges: Sequence[tuple[str, int, int]]) -> bool:
functions = {row.function for row in case.expected if row.function}
if not functions:
return True
return any(start <= line <= end for function in functions for start, end in spans_named(ranges, function))
def _names_label(case: PairCase, record: Mechanism, sink_name: str) -> bool:
"""The hit names the pair's labeled mechanism, or its labeled sink when the row carries one."""
shape = Shape.from_dict(record.shape or {})
for row in case.expected:
if row.mechanism and row.mechanism == shape.mechanism:
return True
if row.sink and trailing_name(row.sink) == sink_name:
return True
return False
def _group(outcomes: Sequence[LooOutcome], keys: object) -> dict[str, dict[str, float]]:
buckets: dict[str, list[LooOutcome]] = {}
for outcome in outcomes:
for key in keys(outcome): # type: ignore[operator]
buckets.setdefault(str(key), []).append(outcome)
return {name: _metrics(items) for name, items in sorted(buckets.items())}
def _metrics(outcomes: Sequence[LooOutcome]) -> dict[str, float]:
pairs = len(outcomes)
detected = sum(1 for item in outcomes if item.detected)
silent = sum(1 for item in outcomes if item.silent)
recall = detected / pairs if pairs else 0.0
silence = silent / pairs if pairs else 0.0
return {
"pairs": pairs,
"detected": detected,
"silent": silent,
"recall": recall,
"silence": silence,
"youden": recall + silence - 1.0 if pairs else 0.0,
"teaching": sum(1 for item in outcomes if item.teaches),
}
def _facts() -> SemanticFacts:
try:
return load_facts()
except FactLoadError:
return SemanticFacts(version="", sources=(), sinks=(), sanitizers=())
src/openultrasast/semantic/mechanisms.py
-363
View File
@@ -1,363 +0,0 @@
"""Append-only mechanism memory. OpenRouter embeddings rank prove budget; never demote."""
from __future__ import annotations
import hashlib
import json
import math
import os
import uuid
from collections.abc import Sequence
from dataclasses import asdict, dataclass
from pathlib import Path
from ..complexity.map import Hotspot
from ..findings import StaticFinding
from ..index import CodeChunk, EmbeddingClient, VectorIndex, VectorRecord, write_vector_index
from .obligations.shapes import ObligationShape
from .overlay import OverlayRecord
from .variants import Shape
DEFAULT_MECHANISM_LOG = Path(".openultrasast/mechanisms.jsonl")
@dataclass(frozen=True)
class Mechanism:
id: str
summary: str
cwe: str
language: str
tags: tuple[str, ...]
keywords: tuple[str, ...]
what_made_it_exploitable: str
source_finding_id: str = ""
source_repo: str = ""
# corpus-seeded-mechanisms (additive; old rows load with these defaults)
origin: str = "sandbox" # "sandbox" (proven finding) | "corpus" (derived from a trusted pair)
review_tier: str = ""
pairs: tuple[str, ...] = ()
shape: dict[str, object] | None = None # Shape.to_dict(); None on sandbox rows (not searchable)
guard: str = "none"
retracted: bool = False # tombstone row appended by the improve lever; load() folds the record away
class MechanismStore:
def __init__(self, log_path: Path, cache_path: Path | None = None) -> None:
self.log_path = log_path
self.cache_path = cache_path if cache_path is not None else log_path.with_suffix(".index.json")
def load(self) -> tuple[Mechanism, ...]:
if not self.log_path.is_file():
return ()
by_id: dict[str, Mechanism] = {} # append-only log; the latest row for an id is its current state
for line in self.log_path.read_text().splitlines():
if not line.strip():
continue
payload = json.loads(line)
record = _mechanism_from_payload(payload)
if record.retracted:
by_id.pop(record.id, None)
else:
by_id[record.id] = record
return tuple(by_id.values())
def append(self, mechanism: Mechanism) -> None:
self.log_path.parent.mkdir(parents=True, exist_ok=True)
with self.log_path.open("a") as handle:
handle.write(json.dumps(asdict(mechanism), sort_keys=True) + "\n")
def append_mechanism(
store: MechanismStore,
*,
summary: str,
cwe: str,
language: str,
tags: Sequence[str],
what_made_it_exploitable: str,
source_finding_id: str = "",
source_repo: str = "",
keywords: Sequence[str] = (),
) -> Mechanism:
mechanism = Mechanism(
id=str(uuid.uuid4()),
summary=summary,
cwe=cwe,
language=language,
tags=tuple(tags),
keywords=tuple(keywords),
what_made_it_exploitable=what_made_it_exploitable,
source_finding_id=source_finding_id,
source_repo=source_repo,
)
store.append(mechanism)
return mechanism
_TIER_RANK = {"seeded": 1, "reviewed": 2}
def corpus_mechanism_id(shape: Shape | ObligationShape) -> str:
"""Deterministic id from the shape key, so the same shape gets the same record on every machine (lever-addressable)."""
return "corpus:" + hashlib.sha1(shape.key().encode()).hexdigest()[:16]
def append_from_pair(
store: MechanismStore,
shape: Shape | ObligationShape,
*,
summary: str,
cwe: str,
pair: str,
provenance: str,
tier: str,
) -> Mechanism:
"""Second writer (Req 1.3, 1.4): one record per shape, ``origin = "corpus"``, provenance lists every pair that taught it."""
mechanism_id = corpus_mechanism_id(shape)
existing = next((item for item in store.load() if item.id == mechanism_id), None)
pairs = tuple(existing.pairs) if existing is not None else ()
if pair not in pairs:
pairs = (*pairs, pair)
if existing is not None and _TIER_RANK.get(existing.review_tier, 0) > _TIER_RANK.get(tier, 0):
tier = existing.review_tier # a shape keeps the strongest review it ever had
inherited = tuple(tag for tag in (existing.tags if existing is not None else ()) if tag.startswith(("provenance:", "tier:")))
if isinstance(shape, Shape):
positions = ", ".join(f"arg{p} ({k})" for p, k in zip(shape.source_positions, shape.source_kinds, strict=True))
exploitable = f"{positions} reaches {shape.sink_name}/{shape.arity}; the fix added {shape.guard}"
keywords: tuple[str, ...] = (shape.sink_name, shape.guard, *shape.source_kinds)
guard = shape.guard
else: # obligation shape: the discharger the fix carries is the 'guard' the reports show
exploitable = f"{shape.operation_kind} reached without {shape.discharger_kind}; the fix binds {shape.provenance}"
keywords = (shape.operation_kind, shape.discharger_kind, shape.provenance, shape.resource_class)
guard = shape.discharger_kind
record = Mechanism(
id=mechanism_id,
summary=summary,
cwe=cwe,
language=shape.language,
tags=(
shape.mechanism,
f"mechanism:{shape.mechanism}",
f"guard:{guard}",
*sorted({*inherited, f"provenance:{provenance}", f"tier:{tier}"}),
),
keywords=keywords,
what_made_it_exploitable=exploitable,
source_finding_id="",
source_repo="",
origin="corpus",
review_tier=tier,
pairs=pairs,
shape=shape.to_dict(),
guard=guard,
)
store.append(record)
return record
def corpus_mechanisms(records: Sequence[Mechanism]) -> tuple[Mechanism, ...]:
"""Records that variant search may use: corpus origin with a shape."""
return tuple(item for item in records if item.origin == "corpus" and item.shape is not None)
def order_promotions(
hotspots: Sequence[Hotspot],
records: Sequence[OverlayRecord],
findings: Sequence[StaticFinding],
*,
store: MechanismStore | None = None,
client: EmbeddingClient | None = None,
model: str | None = None,
) -> tuple[tuple[Hotspot, ...], str | None]:
"""Heuristic order, then optional OpenRouter cosine boost. Never drops a promotion."""
heuristic = tuple(sorted(hotspots, key=lambda item: (-item.score, item.path, item.function_name or "")))
if store is None:
return heuristic, None
mechanisms = store.load()
if not mechanisms:
return heuristic, None
if client is None or not model:
return heuristic, "embeddings_unavailable"
overlay_by_id = {record.proposal_id: record for record in records}
finding_by_id = {finding.finding_id: finding for finding in findings}
try:
ranked = _rank_with_embeddings(heuristic, overlay_by_id, finding_by_id, mechanisms, store, client, model)
except Exception:
return heuristic, "embed_failed"
return ranked, None
def overlay_text(record: OverlayRecord) -> str:
parts = [record.cwe, record.reason, *record.sources, *record.sinks, *record.sanitizers]
return " ".join(part for part in parts if part)
def mechanism_text(mechanism: Mechanism) -> str:
return " ".join(
part
for part in (mechanism.summary, mechanism.what_made_it_exploitable, mechanism.cwe, *mechanism.tags, *mechanism.keywords)
if part
)
def _rank_with_embeddings(
hotspots: tuple[Hotspot, ...],
overlay_by_id: dict[str, OverlayRecord],
finding_by_id: dict[str, StaticFinding],
mechanisms: tuple[Mechanism, ...],
store: MechanismStore,
client: EmbeddingClient,
model: str,
) -> tuple[Hotspot, ...]:
vectors = _mechanism_vectors(mechanisms, store, client, model)
scored: list[tuple[float, Hotspot]] = []
for hotspot in hotspots:
record = _overlay_for_hotspot(hotspot, overlay_by_id)
finding = _finding_for_hotspot(hotspot, finding_by_id)
language = record.language if record is not None else ""
cwe = record.cwe if record is not None else ""
tags = tuple(finding.tags) if finding is not None else ()
filtered = [item for item in mechanisms if _hard_filter(item, language, cwe, tags)]
boost = 0.0
if record is not None and filtered:
query = client.embed(model=model, inputs=[overlay_text(record)])[0]
for mechanism in filtered:
vector = vectors.get(mechanism.id)
if vector is None:
continue
boost = max(boost, _cosine(query, vector))
scored.append((hotspot.score + boost, hotspot))
scored.sort(key=lambda item: (-item[0], item[1].path, item[1].function_name or ""))
return tuple(item[1] for item in scored)
def _mechanism_vectors(
mechanisms: tuple[Mechanism, ...],
store: MechanismStore,
client: EmbeddingClient,
model: str,
) -> dict[str, list[float]]:
vectors: dict[str, list[float]] = {}
missing: list[Mechanism] = []
cached = _read_cache(store.cache_path, model)
for mechanism in mechanisms:
key = f"{mechanism.id}:{model}"
if key in cached:
vectors[mechanism.id] = cached[key]
else:
missing.append(mechanism)
if missing:
embeddings = client.embed(model=model, inputs=[mechanism_text(item) for item in missing])
for mechanism, embedding in zip(missing, embeddings, strict=True):
vectors[mechanism.id] = embedding
records = [
VectorRecord(
chunk=CodeChunk(
chunk_id=f"{mechanism.id}:{model}",
namespace="mechanisms",
path=mechanism.id,
language=mechanism.language or "text",
start_line=1,
end_line=1,
text=mechanism_text(mechanism),
metadata={"mechanism_id": mechanism.id, "model": model, "cwe": mechanism.cwe, "language": mechanism.language},
),
embedding=vectors[mechanism.id],
)
for mechanism in mechanisms
if mechanism.id in vectors
]
index = VectorIndex(
store="json-local",
embedding_model=model,
repo_root=str(store.log_path.parent),
repo_commit=None,
records=records,
)
write_vector_index(index, store.cache_path)
return vectors
def _read_cache(path: Path, model: str) -> dict[str, list[float]]:
if not path.is_file():
return {}
try:
payload = json.loads(path.read_text())
except json.JSONDecodeError:
return {}
if str(payload.get("embedding_model")) != model:
return {}
cached: dict[str, list[float]] = {}
for item in payload.get("records", []):
chunk = item.get("chunk", {})
metadata = chunk.get("metadata", {})
mechanism_id = str(metadata.get("mechanism_id", ""))
if mechanism_id:
cached[f"{mechanism_id}:{model}"] = [float(value) for value in item.get("embedding", [])]
return cached
def _hard_filter(mechanism: Mechanism, language: str, cwe: str, tags: Sequence[str]) -> bool:
if mechanism.language and language and mechanism.language != language:
return False
if mechanism.cwe and cwe and mechanism.cwe != cwe:
return False
tag_ok = not mechanism.tags or not tags or bool(set(mechanism.tags) & set(tags))
return tag_ok
def _overlay_for_hotspot(hotspot: Hotspot, overlay_by_id: dict[str, OverlayRecord]) -> OverlayRecord | None:
for finding_id in hotspot.inventory_finding_ids:
record = overlay_by_id.get(finding_id)
if record is not None:
return record
return None
def _finding_for_hotspot(hotspot: Hotspot, finding_by_id: dict[str, StaticFinding]) -> StaticFinding | None:
for finding_id in hotspot.inventory_finding_ids:
finding = finding_by_id.get(finding_id)
if finding is not None:
return finding
return None
def _cosine(left: list[float], right: list[float]) -> float:
if len(left) != len(right):
return 0.0
numerator = sum(a * b for a, b in zip(left, right, strict=True))
left_norm = math.sqrt(sum(a * a for a in left))
right_norm = math.sqrt(sum(b * b for b in right))
if left_norm == 0 or right_norm == 0:
return 0.0
return numerator / (left_norm * right_norm)
def _mechanism_from_payload(payload: dict[str, object]) -> Mechanism:
tags = payload.get("tags") or []
keywords = payload.get("keywords") or []
pairs = payload.get("pairs") or []
shape = payload.get("shape")
if not isinstance(tags, list) or not isinstance(keywords, list) or not isinstance(pairs, list):
raise ValueError("mechanism tags, keywords and pairs must be lists")
return Mechanism(
id=str(payload["id"]),
summary=str(payload.get("summary", "")),
cwe=str(payload.get("cwe", "")),
language=str(payload.get("language", "")),
tags=tuple(str(item) for item in tags),
keywords=tuple(str(item) for item in keywords),
what_made_it_exploitable=str(payload.get("what_made_it_exploitable", "")),
source_finding_id=str(payload.get("source_finding_id", "")),
source_repo=str(payload.get("source_repo", "")),
origin=str(payload.get("origin", "sandbox")),
review_tier=str(payload.get("review_tier", "")),
pairs=tuple(str(item) for item in pairs),
shape=dict(shape) if isinstance(shape, dict) else None, # validated where it is used (search_tree, the lever)
guard=str(payload.get("guard", "none")),
retracted=bool(payload.get("retracted", False)),
)
def embeddings_configured(model: str | None) -> bool:
return bool(model) and bool(os.environ.get("OPENROUTER_API_KEY"))
src/openultrasast/semantic/obligations/operations.py
+1 -2
View File
@@ -14,8 +14,7 @@ from collections.abc import Sequence
from dataclasses import dataclass
from ..facts import SemanticFacts
from ..ir import FileIR, FunctionIR
from ..variants import trailing_name
from ..ir import FileIR, FunctionIR, trailing_name
from .facts import DischargerFact, ObligationFacts
from .shapes import _call_matches, _constraint_provenance, _is_literal, _operations, _permissive
src/openultrasast/semantic/obligations/shapes.py
+1 -2
View File
@@ -17,8 +17,7 @@ from dataclasses import dataclass
from ..facts import SemanticFacts
from ..functions import named_ranges_from_ir
from ..ir import CallSite, FileIR, FunctionIR
from ..variants import trailing_name
from ..ir import CallSite, FileIR, FunctionIR, trailing_name
from .facts import DISCHARGER_KINDS, OPERATION_KINDS, PROVENANCE_KINDS, DischargerFact, ObligationFacts, OperationFact
FAMILY = "obligation"
src/openultrasast/semantic/prove_budget.py
-12
View File
@@ -1,12 +0,0 @@
"""Prove-budget ranker: heuristic order plus optional mechanism recall."""
from .mechanisms import MechanismStore, append_mechanism, order_promotions
from .prove_filter import filter_promoted_hotspots, promoted_findings
__all__ = [
"MechanismStore",
"append_mechanism",
"filter_promoted_hotspots",
"order_promotions",
"promoted_findings",
]
src/openultrasast/semantic/seed.py
-231
View File
@@ -1,231 +0,0 @@
"""Export trusted pairs as mechanism records (corpus-seeded-mechanisms, Req 1).
Only ``seeded`` and ``reviewed`` pairs seed; ``advisory`` and ``title`` pairs, ``known_limit`` pairs, pointer pairs
without a cached excerpt, and excerpts that fail to parse are skipped with a reason. Runs offline over the vendored
catalog; nothing here fetches, and no code text enters the store.
"""
from __future__ import annotations
from collections.abc import Sequence
from dataclasses import dataclass, field
from pathlib import Path
from ..pairs import GATING_TIERS, PairCase
from ..preprocess import LANGUAGE_BY_EXTENSION
from .facts import FactLoadError, SemanticFacts, load_facts
from .ir import FileIR, parse_file
from .mechanisms import MechanismStore, append_from_pair
from .obligations.facts import ObligationFacts, load_obligation_facts
from .obligations.shapes import ABSENCE_MECHANISMS, ObligationShape, derive_obligation, explain_skip
from .variants import Shape, derive_shape
LANGUAGE_UNSUPPORTED = "variants_language_unsupported"
@dataclass(frozen=True)
class ExportReport:
seeded: int # pairs that produced at least one record
records: int # distinct records written (deduped by shape)
skipped: tuple[tuple[str, str], ...] = () # (pair, reason)
degradations: tuple[dict[str, object], ...] = ()
seeded_pairs: tuple[str, ...] = field(default=())
def to_dict(self) -> dict[str, object]:
return {
"seeded": self.seeded,
"records": self.records,
"skipped": [{"pair": pair, "reason": reason} for pair, reason in self.skipped],
"degradations": list(self.degradations),
"seeded_pairs": list(self.seeded_pairs),
}
def export_mechanisms(cases: Sequence[PairCase], store: MechanismStore, *, facts: SemanticFacts | None = None) -> ExportReport:
"""Derive one shape per labeled expected row of every trusted pair and append it to ``store`` (deduped by shape)."""
loaded = facts if facts is not None else _facts()
skipped: list[tuple[str, str]] = []
degradations: list[dict[str, object]] = []
seeded: list[str] = []
record_ids: set[str] = set()
unsupported_languages: set[str] = set()
for case in cases: # catalog order; the store lists pairs in the order they taught
reason = _skip_reason(case)
if reason is not None:
skipped.append((case.name, reason))
continue
vuln_text = _read(case.vuln_file)
fixed_text = _read(case.fixed_file)
language = _parse_language(case)
vuln_ir = parse_file(case.relpath, vuln_text, language)
fixed_ir = parse_file(case.relpath, fixed_text, language)
if not vuln_ir.parse_ok or not fixed_ir.parse_ok:
bad = vuln_ir if not vuln_ir.parse_ok else fixed_ir
if bad.reason == "language_unsupported":
skipped.append((case.name, LANGUAGE_UNSUPPORTED))
if language not in unsupported_languages:
unsupported_languages.add(language)
degradations.append({"stage": "mechanisms", "reason": LANGUAGE_UNSUPPORTED, "language": language})
else:
skipped.append((case.name, "parse_failed"))
continue
if not any(row.function and row.mechanism for row in case.expected):
skipped.append((case.name, "no_labeled_function"))
continue
lessons = pair_lessons(case, loaded, vuln_ir=vuln_ir, fixed_ir=fixed_ir, vuln_text=vuln_text, fixed_text=fixed_text)
for lesson in lessons:
record = append_from_pair(
store,
lesson.shape,
summary=lesson.summary,
cwe=lesson.cwe,
pair=case.name,
provenance=case.provenance,
tier=case.review_tier,
)
record_ids.add(record.id)
if lessons:
seeded.append(case.name)
else:
skipped.append((case.name, _skip_detail(case, vuln_ir, fixed_ir, vuln_text, fixed_text, loaded)))
return ExportReport(
seeded=len(seeded),
records=len(record_ids),
skipped=tuple(skipped),
degradations=tuple(degradations),
seeded_pairs=tuple(seeded),
)
@dataclass(frozen=True)
class Lesson:
"""One shape a trusted pair teaches, with the CWE and a text-free summary (a sink shape or an obligation shape)."""
shape: Shape | ObligationShape
cwe: str
summary: str
def pair_lessons(
case: PairCase,
facts: SemanticFacts,
*,
vuln_ir: FileIR | None = None,
fixed_ir: FileIR | None = None,
vuln_text: str | None = None,
fixed_text: str | None = None,
) -> list[Lesson]:
"""Shapes derived from every labeled row of a pair (shared by the exporter and leave-one-out). Empty when a side fails to parse."""
vuln_text = vuln_text if vuln_text is not None else _read(case.vuln_file)
fixed_text = fixed_text if fixed_text is not None else _read(case.fixed_file)
language = _parse_language(case)
vuln_ir = vuln_ir if vuln_ir is not None else parse_file(case.relpath, vuln_text, language)
fixed_ir = fixed_ir if fixed_ir is not None else parse_file(case.relpath, fixed_text, language)
if not vuln_ir.parse_ok or not fixed_ir.parse_ok:
return []
lessons: list[Lesson] = []
for row in case.expected:
if not row.function or not row.mechanism:
continue
obligation_kind = getattr(row, "obligation", None)
if obligation_kind or row.mechanism in ABSENCE_MECHANISMS:
obligation = derive_obligation(
vuln_ir,
fixed_ir,
function=row.function,
obligation=obligation_kind,
mechanism=row.mechanism,
facts=_obligation_facts(),
flow_facts=facts,
vuln_text=vuln_text,
fixed_text=fixed_text,
)
if obligation is not None:
summary = (
f"{obligation.mechanism}: {obligation.operation_kind} without {obligation.discharger_kind} ({row.cwe}); "
f"fix binds {obligation.provenance}"
)
lessons.append(Lesson(shape=obligation, cwe=row.cwe, summary=summary))
continue
# an absence row that teaches no obligation may still teach a sink shape (a weak literal at a sink, say)
shape = derive_shape(
vuln_ir,
fixed_ir,
function=row.function,
sink=row.sink,
line=row.line,
mechanism=row.mechanism,
facts=facts,
vuln_text=vuln_text,
fixed_text=fixed_text,
cwe=row.cwe or None,
)
if shape is not None:
summary = f"{shape.mechanism}: {'/'.join(shape.source_kinds)} into {shape.sink_name} ({row.cwe}); fix adds {shape.guard}"
lessons.append(Lesson(shape=shape, cwe=row.cwe, summary=summary))
return lessons
def _skip_detail(case: PairCase, vuln_ir: FileIR, fixed_ir: FileIR, vuln_text: str, fixed_text: str, flow_facts: SemanticFacts) -> str:
"""Why a parsing, labeled pair taught nothing: the obligation derivation's reason for absence rows, else the sink reason."""
for row in case.expected:
if row.function and row.mechanism and (getattr(row, "obligation", None) or row.mechanism in ABSENCE_MECHANISMS):
return explain_skip(
vuln_ir,
fixed_ir,
function=row.function,
vuln_text=vuln_text,
fixed_text=fixed_text,
obligation=getattr(row, "obligation", None),
facts=_obligation_facts(),
flow_facts=flow_facts,
)
return "no_labeled_sink_call_site"
_OBLIGATION_FACTS: list[ObligationFacts] = []
def _obligation_facts() -> ObligationFacts:
if not _OBLIGATION_FACTS:
_OBLIGATION_FACTS.append(load_obligation_facts())
return _OBLIGATION_FACTS[0]
def _skip_reason(case: PairCase) -> str | None:
from ..learning.split import is_teacher
if case.review_tier not in GATING_TIERS:
return f"tier:{case.review_tier}"
if case.unscorable or case.known_limit:
# `PairCase.unscorable` verbatim when the loader computed one, the same string every other path reports:
# `identical_twin` declared in the catalog and `identical_twin` computed from the bytes are one fact, and
# calling either of them `split:train` is simply untrue. A case built by hand may carry only the
# declaration (Req 5.1, 10.3).
return case.unscorable or f"known_limit:{case.known_limit}"
if not is_teacher(case) and case.vendored:
return f"split:{case.split}"
if not case.vuln_file.is_file() or not case.fixed_file.is_file():
return "pointer_pair_not_cached" if not case.vendored else "excerpt_missing"
return None
def _parse_language(case: PairCase) -> str:
by_ext = LANGUAGE_BY_EXTENSION.get(case.vuln_file.suffix.lower())
if by_ext:
return str(by_ext)
return {"c_cpp": "c"}.get(case.language, case.language)
def _read(path: Path) -> str:
try:
return path.read_text(errors="ignore")
except OSError:
return ""
def _facts() -> SemanticFacts:
try:
return load_facts()
except FactLoadError:
return SemanticFacts(version="", sources=(), sinks=(), sanitizers=())
src/openultrasast/semantic/variant_search.py
-125
View File
@@ -1,125 +0,0 @@
"""Variant search over a scanned tree and its merge with overlay flows (corpus-seeded-mechanisms, Req 3).
Depends on the shapes (``variants``), the store (``mechanisms``), the overlay records and ``StaticFinding``; the shape
module itself stays free of those imports.
"""
from __future__ import annotations
from collections.abc import Mapping, Sequence
from dataclasses import dataclass, replace
from pathlib import Path
from ..findings import StaticFinding
from ..preprocess import FileTarget
from .facts import SemanticFacts
from .ir import parse_file
from .mechanisms import MechanismStore, corpus_mechanisms
from .overlay import OverlayRecord
from .variants import Shape, VariantHit, match_shapes
LANGUAGE_UNSUPPORTED = "variants_language_unsupported"
@dataclass(frozen=True)
class SearchResult:
hits: tuple[VariantHit, ...]
mechanisms_searched: int
files_searched: int
degradations: tuple[dict[str, object], ...] = ()
def search_tree(
root: Path,
targets: Sequence[FileTarget],
store: MechanismStore,
facts: SemanticFacts,
*,
max_mechanisms: int,
) -> SearchResult:
"""Variant search over the parsed files of a tree with the store's corpus shapes (bounded by ``max_mechanisms``)."""
records = [record for record in corpus_mechanisms(store.load()) if (record.shape or {}).get("family", "sink") == "sink"]
records = records[: max(max_mechanisms, 0)]
shapes: list[Shape] = []
ids: dict[str, str] = {}
degradations: list[dict[str, object]] = []
for record in records:
assert record.shape is not None
try:
shape = Shape.from_dict(record.shape)
except (ValueError, TypeError) as exc: # one malformed operator-visible row must not abort the scan
degradations.append(
{"stage": "variants", "reason": "variants_store_row_invalid", "mechanism_id": record.id, "detail": str(exc)[:120]}
)
continue
shapes.append(shape)
ids[shape.key()] = record.id
if not shapes:
return SearchResult(hits=(), mechanisms_searched=0, files_searched=0, degradations=tuple(degradations))
hits: list[VariantHit] = []
files = 0
unsupported: set[str] = set()
for target in targets:
try:
text = (root / target.path).read_text(errors="ignore")
except OSError:
continue
ir = parse_file(target.path, text, target.language)
if not ir.parse_ok:
if ir.reason == "language_unsupported" and target.language not in unsupported:
unsupported.add(target.language)
degradations.append({"stage": "variants", "reason": "variants_language_unsupported", "language": target.language})
continue
files += 1
hits.extend(match_shapes(ir, shapes, facts, mechanism_ids=ids))
return SearchResult(hits=tuple(hits), mechanisms_searched=len(shapes), files_searched=files, degradations=tuple(degradations))
def hits_to_findings(
hits: Sequence[VariantHit],
records: Sequence[OverlayRecord],
summaries: Mapping[str, tuple[str, tuple[str, ...], str]],
) -> tuple[list[StaticFinding], list[OverlayRecord]]:
"""Merge hits into overlay records at the same call site (Req 3.3); the rest become ``suspicion`` findings (Req 3.2)."""
by_site: dict[tuple[str, int | None], list[int]] = {}
for index, record in enumerate(records):
if record.disposition in {"promote", "coverage"} and record.sources and record.sinks:
by_site.setdefault((record.path, record.line), []).append(index)
updated = list(records)
findings: list[StaticFinding] = []
emitted: set[str] = set()
for hit in hits:
indexes = by_site.get((hit.path, hit.line))
if indexes:
for index in indexes:
if updated[index].mechanism_id is None:
updated[index] = replace(updated[index], mechanism_id=hit.mechanism_id)
continue
finding_id = f"variant:{hit.mechanism_id}:{hit.path}:{hit.line}"
if finding_id in emitted:
continue
emitted.add(finding_id)
summary, pairs, cwe = summaries.get(hit.mechanism_id, (hit.mechanism_id, (), ""))
taught_by = ", ".join(pairs) if pairs else "the mechanism store"
findings.append(
StaticFinding(
finding_id=finding_id,
path=hit.path,
title=f"Variant of known mechanism: {hit.sink_name}",
severity="medium",
confidence="low",
evidence_level="suspicion",
rationale=(
f"{summary}. Structural variant ({hit.source_kind} reaches {hit.sink_name}) of a mechanism learned from "
f"{taught_by}; suspicion until the overlay or the sandbox agrees."
),
line=hit.line,
function_name=None,
reachability_status="unknown",
reachability_evidence=[],
reachability_conditions=[],
tags=["variant", f"mechanism:{hit.mechanism_id}", *([f"cwe:{cwe}"] if cwe else [])],
ranking_priority=0.0,
)
)
return findings, updated
src/openultrasast/semantic/variants.py
-347
View File
@@ -1,347 +0,0 @@
"""Structural mechanism shapes derived from labeled pairs (corpus-seeded-mechanisms, Req 2).
A shape is the closed, text-free description of how a known vulnerable call looked: the
sink's trailing callee name, its arity, which argument positions carried source-derived
values and of what kind, and the kind of guard the fix added. Shapes are derived from
``FileIR`` (never from regex over code) and compared structurally, so a variant in another
project matches even when every identifier differs. Stdlib plus ``.ir``, ``.facts`` and
``.functions`` only; never imports the overlay, the scorer, or the store.
"""
from __future__ import annotations
import re
from collections.abc import Mapping, Sequence
from dataclasses import dataclass
from .facts import SemanticFacts, SinkFact
from .functions import named_ranges_from_ir
from .ir import Bind, CallSite, FileIR, FunctionIR
GUARD_KINDS = ("null_test", "bounds_test", "allowlist_test", "auth_check", "parameterized_call", "type_change", "none")
SOURCE_KINDS = ("fact_source", "parameter", "container_read")
_MAX_BIND_DEPTH = 8
# Closed guard classification over the statements the fix added inside the labeled function.
# Order matters: the first matching kind wins. These classify the *fix*, they never detect.
_GUARD_PATTERNS: tuple[tuple[str, re.Pattern[str]], ...] = (
("parameterized_call", re.compile(r"execute\w*\([^,]+,\s*[\(\[]|prepareStatement\(|\?\s*['\"]?\s*[,)]|\$\d\b|%s['\"]\s*,\s*[\(\[]")),
(
"auth_check",
re.compile(
r"\b(request\.user|req\.user|current_user|request\.state|session\[|is_authenticated|authoriz|require_auth"
r"|login_required|verify_token|check_permission|owner_id\s*[!=]=)"
),
),
(
"allowlist_test",
re.compile(
r"\bnot in\b|\bin\s*[\(\{\[]|\.includes\(|\.has\(|\bstartswith\(|\bstartsWith\(|\bendswith\(|\bendsWith\("
r"|re\.(?:match|fullmatch)\(|\.test\(|allow_?list|whitelist|\bALLOWED\b|\bsafe_join\(|secure_filename\("
r"|\brealpath\(|\bnormpath\(|\bresolve\("
),
),
("bounds_test", re.compile(r"\blen\(|\.length\b|\bsizeof\(|\bstrn?len\(|\bsnprintf\(|\bstrlcpy\(|\bmin\(|\bmax\(|\s(?:<|>|<=|>=)\s")),
(
"null_test",
re.compile(
r"\bis None\b|\bis not None\b|[!=]= *NULL\b|[!=]==? *null\b|\bnullptr\b|\bif\s*\(\s*!\s*\w|\bif not \w+\s*:"
r"|\bif\s+\w+\s+is\s+None|\?\?|\?\."
),
),
(
"type_change",
re.compile(r"\bint\(|\bparseInt\(|\bNumber\(|\(int\)|\(size_t\)|\(unsigned\b|\bstr\(|\.toString\(|\bstatic_cast<|\bBigInt\("),
),
)
# Design §Shape: a container read is a subscript or a `.get(` read; source names themselves live in facts, never here.
_CONTAINER_READ = re.compile(r"\[[^\]]+\]|\.get\(")
@dataclass(frozen=True)
class Shape:
language: str
sink_name: str # trailing callee segment
arity: int
source_positions: tuple[int, ...]
source_kinds: tuple[str, ...]
guard: str # GUARD_KINDS
mechanism: str # closed vocabulary id from the pair label
def key(self) -> str:
"""Stable identity of the shape; carries no path, line, or literal."""
positions = ",".join(str(p) for p in self.source_positions)
kinds = ",".join(self.source_kinds)
return f"{self.language}|{self.sink_name}/{self.arity}|pos={positions}|kinds={kinds}|guard={self.guard}|{self.mechanism}"
def to_dict(self) -> dict[str, object]:
return {
"language": self.language,
"sink_name": self.sink_name,
"arity": self.arity,
"source_positions": list(self.source_positions),
"source_kinds": list(self.source_kinds),
"guard": self.guard,
"mechanism": self.mechanism,
}
@classmethod
def from_dict(cls, payload: dict[str, object]) -> Shape:
positions = payload.get("source_positions") or []
kinds = payload.get("source_kinds") or []
if not isinstance(positions, list) or not isinstance(kinds, list):
raise ValueError("shape positions and kinds must be lists")
guard = str(payload.get("guard", "none"))
if guard not in GUARD_KINDS:
raise ValueError(f"unknown guard kind {guard!r}")
return cls(
language=str(payload.get("language", "")),
sink_name=str(payload.get("sink_name", "")),
arity=int(str(payload.get("arity", 0))),
source_positions=tuple(int(p) for p in positions),
source_kinds=tuple(str(k) for k in kinds),
guard=guard,
mechanism=str(payload.get("mechanism", "other")),
)
def trailing_name(callee: str) -> str:
"""``os.path.join`` -> ``join``; ``Runtime.getRuntime().exec`` -> ``exec``; keeps the last identifier segment."""
tail = callee.rsplit(".", 1)[-1].rsplit("::", 1)[-1].rsplit("->", 1)[-1]
match = re.search(r"[A-Za-z_$][\w$]*\s*$", tail)
return match.group(0).strip() if match else tail
def derive_shape(
vuln: FileIR,
fixed: FileIR,
*,
function: str,
sink: str | None,
line: int | None,
mechanism: str,
facts: SemanticFacts,
vuln_text: str = "",
fixed_text: str = "",
cwe: str | None = None,
) -> Shape | None:
"""The shape of the labeled sink call in ``function`` on the vulnerable side, with the guard the fixed side added.
None when either side failed to parse, the function or the labeled call site is not found, or no argument of the
call carries a source (a constant-only call teaches nothing).
"""
if not vuln.parse_ok or not fixed.parse_ok:
return None
target = _function_named(vuln, function, vuln_text)
if target is None:
return None
scoped = facts.for_language(vuln.language)
call = _labeled_call(target, sink, line, scoped, cwe)
if call is None:
return None
kinds = [_source_kind_of_argument(target, call, index, scoped) for index in range(len(call.arg_texts))]
positions = tuple(index for index, kind in enumerate(kinds) if kind is not None)
if not positions:
return None
guard = classify_guard(vuln, fixed, function=function, vuln_text=vuln_text, fixed_text=fixed_text)
return Shape(
language=vuln.language,
sink_name=trailing_name(call.name),
arity=len(call.arg_texts),
source_positions=positions,
source_kinds=tuple(kind for kind in kinds if kind is not None),
guard=guard,
mechanism=mechanism,
)
def classify_guard(vuln: FileIR, fixed: FileIR, *, function: str, vuln_text: str, fixed_text: str) -> str:
"""Closed guard kind from the statements inside ``function`` that exist only on the fixed side (``none`` when unknown)."""
before = _function_lines(vuln, function, vuln_text)
after = _function_lines(fixed, function, fixed_text)
added = [line for line in after if line not in set(before)]
for kind, pattern in _GUARD_PATTERNS:
if any(pattern.search(line) for line in added):
return kind
return "none"
def source_kind_of_name(function: FunctionIR, name: str, facts: SemanticFacts, *, before_line: int, depth: int = 0) -> str | None:
"""How ``name`` came to carry attacker data inside ``function``: a fact source, a parameter, or a container read."""
if depth > _MAX_BIND_DEPTH:
return None
bind = _last_bind(function, name, before_line)
if bind is not None:
if _source_in(bind.value_text, facts):
return "fact_source"
if _CONTAINER_READ.search(bind.value_text):
return "container_read"
for inner in bind.names:
if inner == name:
continue
kind = source_kind_of_name(function, inner, facts, before_line=bind.line, depth=depth + 1)
if kind is not None:
return kind
if name in function.params:
return "parameter"
return None
def _source_kind_of_argument(function: FunctionIR, call: CallSite, index: int, facts: SemanticFacts) -> str | None:
text = call.arg_texts[index]
if _source_in(text, facts):
return "fact_source"
if call.arg_is_constant[index] and not call.arg_names[index]:
return None
if _CONTAINER_READ.search(text) and any(name in function.params for name in call.arg_names[index]):
return "container_read"
found: list[str] = []
for name in call.arg_names[index]:
kind = source_kind_of_name(function, name, facts, before_line=call.line + 1)
if kind is not None:
found.append(kind)
for preferred in SOURCE_KINDS:
if preferred in found:
return preferred
return None
def _source_in(text: str, facts: SemanticFacts) -> bool:
return any(pattern and pattern in text for source in facts.sources for pattern in source.patterns)
def _last_bind(function: FunctionIR, name: str, before_line: int) -> Bind | None:
candidates = [bind for bind in function.binds if bind.name == name and bind.line < before_line]
return max(candidates, key=lambda bind: bind.line) if candidates else None
def _labeled_call(function: FunctionIR, sink: str | None, line: int | None, facts: SemanticFacts, cwe: str | None) -> CallSite | None:
"""The call the label names: line and sink intersected when both exist, else either; else the fact sink in the function.
Labels that carry only a function and a mechanism (Real-Vuln rows) fall back to the fact-sink call inside the labeled
function, preferring the sink whose CWE matches the row and a call with a non-constant argument.
"""
wanted = trailing_name(sink) if sink else None
on_line = [call for call in function.calls if line is not None and call.line == line]
if on_line and wanted:
exact = [call for call in on_line if trailing_name(call.name) == wanted]
if exact:
return exact[0]
if on_line:
return on_line[0]
if wanted:
for call in function.calls:
if trailing_name(call.name) == wanted:
return call
return None
ranked: list[tuple[int, int, CallSite]] = []
for order, call in enumerate(function.calls):
fact = _sink_fact_for(call.name, facts)
if fact is None:
continue
cwe_match = 0 if (cwe and fact.cwe == cwe) else 1
non_constant = 0 if any(not const or names for const, names in zip(call.arg_is_constant, call.arg_names, strict=True)) else 1
ranked.append((cwe_match + non_constant, order, call))
ranked.sort(key=lambda item: (item[0], item[1]))
return ranked[0][2] if ranked else None
def _sink_fact_for(name: str, facts: SemanticFacts) -> SinkFact | None:
for fact in facts.sinks:
if any(_call_matches(name, call) for call in fact.calls):
return fact
return None
def _call_matches(name: str, fact_call: str) -> bool:
"""Same rule as the taint walker: exact, dotted suffix, or a bare name equal to the fact's last segment."""
if name == fact_call or name.endswith("." + fact_call):
return True
return "." not in name and name == fact_call.split(".")[-1]
def _function_named(ir: FileIR, function: str, text: str) -> FunctionIR | None:
for item in ir.functions:
if item.name == function:
return item
if text: # labels use declarator names; the parser may name the function differently (or <anon>)
for name, start, _end in named_ranges_from_ir(ir, text):
if name == function:
for item in ir.functions:
if item.start_line == start:
return item
return None
def _function_lines(ir: FileIR, function: str, text: str) -> list[str]:
target = _function_named(ir, function, text)
if target is None or not text:
return []
lines = text.splitlines()
body = lines[target.start_line - 1 : target.end_line]
return [line.strip() for line in body if line.strip()]
def shapes_by_sink(shapes: Sequence[Shape]) -> dict[tuple[str, str], list[Shape]]:
"""Index for matching: (language, sink_name) -> shapes."""
index: dict[tuple[str, str], list[Shape]] = {}
for shape in shapes:
index.setdefault((shape.language, shape.sink_name), []).append(shape)
return index
@dataclass(frozen=True)
class VariantHit:
path: str
line: int
mechanism_id: str
sink_name: str
source_kind: str
def match_shapes(
ir: FileIR,
shapes: Sequence[Shape],
facts: SemanticFacts,
*,
mechanism_ids: Mapping[str, str],
) -> list[VariantHit]:
"""Call sites of ``ir`` whose trailing callee name, arity and source-carrying positions match a shape (Req 3.1).
Any source kind at the shape's positions matches: a variant learned from ``request.args`` still fires when the same
sink takes a parameter. Constant-only arguments never match. ``mechanism_ids`` maps ``Shape.key()`` to the store id.
"""
if not ir.parse_ok:
return []
index = shapes_by_sink([shape for shape in shapes if shape.language == ir.language])
if not index:
return []
scoped = facts.for_language(ir.language) if hasattr(facts, "for_language") else facts
hits: list[VariantHit] = []
seen: set[tuple[int, str]] = set()
for function in ir.functions:
for call in function.calls:
candidates = index.get((ir.language, trailing_name(call.name)))
if not candidates:
continue
for shape in candidates:
if shape.arity != len(call.arg_texts) or (call.line, shape.key()) in seen:
continue
kinds = [
_source_kind_of_argument(function, call, position, scoped)
for position in shape.source_positions
if position < len(call.arg_texts)
]
if len(kinds) != len(shape.source_positions) or any(kind is None for kind in kinds):
continue
seen.add((call.line, shape.key()))
hits.append(
VariantHit(
path=ir.path,
line=call.line,
mechanism_id=mechanism_ids.get(shape.key(), shape.key()),
sink_name=shape.sink_name,
source_kind=str(kinds[0]),
)
)
return hits
src/openultrasast/vectorstore.py
-69
View File
@@ -1,69 +0,0 @@
from __future__ import annotations
from dataclasses import asdict, dataclass
from pathlib import Path
from .index import VectorIndex, index_reuse_key, query_vector_index, read_vector_index, write_vector_index
from .preprocess import RepoSnapshot
SELECTED_LOCAL_VECTOR_STORE = "json-local"
@dataclass(frozen=True)
class VectorStoreBakeoffResult:
store: str
metadata_filtering: bool
local_persistence: bool
incremental_keying: bool
export_import: bool
retrieval_quality: bool
selected_default: bool
@property
def passed(self) -> bool:
return all(
(
self.metadata_filtering,
self.local_persistence,
self.incremental_keying,
self.export_import,
self.retrieval_quality,
self.selected_default,
)
)
def bakeoff_json_local(index: VectorIndex, output_path: Path, *, query_embedding: list[float]) -> VectorStoreBakeoffResult:
write_vector_index(index, output_path)
loaded = read_vector_index(output_path)
metadata_matches = query_vector_index(loaded, query_embedding, metadata_filter={"language": "python"}, limit=1)
unfiltered_matches = query_vector_index(loaded, query_embedding, limit=1)
snapshot = RepoSnapshot(
root=loaded.repo_root,
commit=loaded.repo_commit,
file_count=len({record.chunk.path for record in loaded.records}),
languages=_language_counts(loaded),
)
result = VectorStoreBakeoffResult(
store=SELECTED_LOCAL_VECTOR_STORE,
metadata_filtering=bool(metadata_matches) and metadata_matches[0][1].metadata.get("language") == "python",
local_persistence=output_path.exists() and output_path.stat().st_size > 0,
incremental_keying=index_reuse_key(snapshot, loaded.embedding_model) == index_reuse_key(snapshot, loaded.embedding_model),
export_import=len(loaded.records) == len(index.records) and loaded.embedding_model == index.embedding_model,
retrieval_quality=bool(unfiltered_matches) and unfiltered_matches[0][0] > 0,
selected_default=SELECTED_LOCAL_VECTOR_STORE == "json-local",
)
return result
def bakeoff_payload(result: VectorStoreBakeoffResult) -> dict[str, object]:
payload = asdict(result)
payload["passed"] = result.passed
return payload
def _language_counts(index: VectorIndex) -> dict[str, int]:
counts: dict[str, int] = {}
for record in index.records:
counts[record.chunk.language] = counts.get(record.chunk.language, 0) + 1
return counts