fix(pre-push-safety-net): stop semantic planning at the shared deadline

This commit is contained in:
Marius Ciepluch
2026-09-13 15:11:42 +02:00
parent 31e5ec7430
commit 8fe3a7e51e
8 changed files with 24687 additions and 11 deletions
@@ -64,3 +64,44 @@ RED evidence: `/tmp/ousast-replay-tierzero-red.log` and `/tmp/ousast-replay-snap
- CLAIM: Explicit local replay traverses the production mapper/ranker/arbiter/comparison/report path and retains uncertainty and shared deadlines.
- EVIDENCE: Final packaged controlled security change yields new/source_connection_absent_from_comparable_base. Fixed twin emits no modeled head finding. The packaged deadline control returns incomplete in 30.785 seconds, within 30 + 2 seconds. Exact final package policy/core/facts/query digests match the workspace. Complete commands and captured raw answers live in the measurement artifact.
- GAPS: Aggregate coverage remains incomplete, real capabilities are disabled, and lab timings do not satisfy the latency gate. Source-mounted fixed and earlier packaged deadline provenance is explicitly separated in the artifact. PMPro/Node transfer is task 5.2, not inferred from a small PHP control.
## Predecessor deadline regression discovered by task 5.2
The first frozen trial completed six CLI invocations, retaining all 11 declared cases.
All three PMPro cases timed out during preparation; both snapshots were fully materialized.
Node hit the engine deadline but then took 34.6–37.4 seconds and could not publish its artifact.
Task 5.2 is blocked pending the owning 3.1/5.1 deadline integration repair and a newly frozen run.
The initial trial overlaps regression testing; it is diagnostic evidence, not the final latency sample.
### Debug report
- ROOT_CAUSE: After an engine timeout, the generic scan driver's failure census reloaded semantic facts and taxonomy for every question. The replay runner then launched an unbudgeted engine-availability subprocess before attempting base comparison. Main-thread stack samples at 30–34 seconds prove both paths.
- CATEGORY: LOGIC_ERROR
- FIX_PLAN: Route failure bookkeeping through the existing driver without loading new semantic facts after a failed build; instantiate the budget-aware Joern backend once rather than running legacy availability probes in replay. Preserve every declared question as deferred and keep ordinary backend selection unchanged outside replay.
- VERIFICATION: Two failing regressions (including 400 declared failed questions), real Node deadline replay, full suite and packaged validation before re-freezing the experiment.
- NEXT_ACTION: RETRY_TASK
- CONFIDENCE: HIGH
- NOTES: This repairs the already-approved generic deadline integration in task 3.1 and replay in 5.1. It changes no detector facts, ranking features or abstract capabilities. No workaround is hidden in the evaluation harness; task 5.2 stays blocked until this contract is repaired. Manual Kiro debug/review fallback applies after the recorded agent thread limit. Runtime trace proves a local logic error; no external dependency/version hypothesis is needed.
RED: `/tmp/ousast-node-deadline-flag-red.log`, two failures with flags OFF. Flags were
removed after GREEN; the focused scope/replay suite passed 30 tests.
The first repair exposed the same repeated fact-read path after evidence planning rather than
build failure. The absolute budget now also reaches `_collect` and `_scope_work`; planning
stops loading facts when time expires but retains the entire question census. No ranking weights,
facts or detector semantics changed. The additional regression failed before repair in
`/tmp/ousast-node-planning-red.log` and then passed.
### Predecessor repair review verdict
- VERDICT: APPROVED
- TASK: 3.1/5.1 deadline integration regression
- MECHANICAL_RESULTS: Full current suite 1,242 passed, nine skipped (including three pending experiment-harness checks), exit 0; Ruff/format/mypy/compileall and all regression gates PASS; behavioral RED VERIFIED; no new placeholders/secrets; scope is the existing driver/deadline integration.
- FINDINGS: Both measured post-deadline paths are repaired, including successful-build/evidence-planning cancellation; all declared questions remain recorded. No outstanding repair blocker.
- SUMMARY: Manual review approves the predecessor repair; task5.2 may resume with a new frozen profile.
### Predecessor repair verification
- STATUS: VERIFIED
- CLAIM_TYPE: FIX
- CLAIM: The reproduced Node cancellation path retains diagnostics and returns within 30 + 2 seconds.
- EVIDENCE: Final packaged real Node replay returned in 30.1255 seconds with one complete diagnostic artifact and incomplete coverage. Final policy/core/facts/query hashes match the workspace. See `benchmarks/measurements/2026-09-13-replay-deadline-repair.json` for initial/partial-repair traces and final raw artifact.
- GAPS: This proves the reproduced cancellation path, not representative warm performance or security coverage. Normal capabilities remain disabled.
+2 -2
View File
@@ -1,7 +1,7 @@
{
"feature_name": "pre-push-safety-net",
"created_at": "2026-09-12T15:35:00Z",
"updated_at": "2026-09-13T12:50:25.249104+00:00",
"updated_at": "2026-09-13T13:11:42.850127+00:00",
"language": "en",
"phase": "implementation",
"approvals": {
@@ -19,7 +19,7 @@
}
},
"ready_for_implementation": true,
"implementation_approval_note": "Group 5 authorized. Task5.1 verified through manual Kiro fallback review; 5.2 next. Real capabilities remain disabled.",
"implementation_approval_note": "Group5 authorized. Task5.1 and its observed deadline regression repaired and verified with manual Kiro fallback; task5.2 resumes under a newly frozen core identity.",
"depends_on": [
"contributor-scan"
],
+1
View File
@@ -149,6 +149,7 @@ Task 5.1 is also verified (15/27 total); task 5.2 and groups 6–8 remain pendin
- _Depends: 1.3, 3.1, 3.4, 4.3_
- _Requirements: 1.1, 2.1, 3.1, 3.4, 4.1, 4.2, 5.5, 7.1, 7.3_
- [ ] 5.2 Measure PMPro and frozen-core Node transfer
- _Prerequisite repaired 2026-09-13:_ 3.1/5.1 deadline regression verified in the packaged Node replay; re-freeze the experiment after this core identity change.
- Replay each prepared security/fixed/benign case, first using the declared hook deadline; record timeouts and stage costs. A separately named extended lab run may obtain witnesses but cannot satisfy the hook latency gate.
- Freeze the core after PMPro, run Node with existing facts, and classify instrument, mapping, fact, scope and domain gaps without tuning on an untouched holdout.
- Done when one joint artifact reports actual evidence and outcomes for both workloads, with unresolved cases retained and an explicit feasibility decision; no normal capability is enabled here.
File diff suppressed because it is too large Load Diff
+27 -6
View File
@@ -245,7 +245,15 @@ def _scan_repository_impl(
if cpg is not None:
owned.append(cpg)
scope, _ = _scope_work(
regions, {}, limits, unit, ranking_mode, population_complete, failure="cpg_build_failed", change_context=change_context
regions,
{},
limits,
unit,
ranking_mode,
population_complete,
failure="cpg_build_failed",
change_context=change_context,
execution_budget=execution_budget,
)
scope = replace(scope, unresolved_boundaries=tuple(dict.fromkeys((*scope.unresolved_boundaries, *getattr(cpg, "boundaries", ())))))
return ModelScanResult(
@@ -319,7 +327,7 @@ def _scan_repository_impl(
tier_started = time.monotonic()
evidence_by_pair, failed = _evidence_pass(
cpg,
_collect(evidence_candidates),
_collect(evidence_candidates, execution_budget=execution_budget),
hooks,
degradations=degradations,
context_rows=context_rows if change_context is not None else None,
@@ -369,6 +377,7 @@ def _scan_repository_impl(
population_complete,
change_context=change_context,
context_gaps=context_gaps,
execution_budget=execution_budget,
)
outcomes: dict[str, QuestionOutcome] = {}
@@ -573,6 +582,7 @@ def _scope_work(
failure: str | None = None,
change_context: ChangeContext | None = None,
context_gaps: set[QuestionIdentity] | None = None,
execution_budget: ExecutionBudget | None = None,
) -> tuple[ScopeDecision, list[tuple[str, ScanRegion, ArbiterSpec]]]:
"""Consume the ranker's final order once; the returned work IS selected scope."""
selected: list[RankedQuestion] = []
@@ -586,10 +596,15 @@ def _scope_work(
for index, region in enumerate(regions):
for family in region.families:
identity = _identity(unit, region, family)
spec = _spec_for(family, region.language)
if failure is None and not _within_deadline(execution_budget):
failure = "deadline_exhausted"
boundaries.append(failure)
spec = None if failure else _spec_for(family, region.language)
vector = evidence.get((region.path, region.function or "", family))
facts: tuple[str, ...] = ("static_rank=" + str(region.rank),)
if vector is not None:
if failure:
facts += ("analysis_not_started=" + failure,)
elif vector is not None:
facts += ("normalized_evidence=" + _rows_json(asdict(vector)), "tier=" + str(vector.tier), "score=" + str(vector.score))
elif isinstance(spec, TaintSpec):
facts += ("evidence_unknown",)
@@ -616,7 +631,9 @@ def _scope_work(
if identity in (context_gaps or set()):
facts += ("change_context_incomplete",)
reason = (
"unsupported_family"
failure
if failure
else "unsupported_family"
if spec is None
else failure
or (
@@ -676,11 +693,15 @@ def _pair_key(region: ScanRegion, spec: ArbiterSpec) -> tuple[str, str, str]:
return (region.path, region.function or "", getattr(spec, "family", ""))
def _collect(regions: Sequence[ScanRegion]) -> list[tuple[str, ScanRegion, ArbiterSpec]]:
def _collect(
regions: Sequence[ScanRegion], *, execution_budget: ExecutionBudget | None = None
) -> list[tuple[str, ScanRegion, ArbiterSpec]]:
"""Every (region, family) question with an arbiter, in the order given, with batch-local ids."""
work: list[tuple[str, ScanRegion, ArbiterSpec]] = []
for region in regions:
for family in region.families:
if not _within_deadline(execution_budget):
return work
spec = _spec_for(family, region.language)
if spec is None:
continue
+5 -3
View File
@@ -14,7 +14,7 @@ from pathlib import Path
from typing import Any, Literal, TypeVar, cast
from openultrasast.config import PushConfig
from openultrasast.cpg.backend import joern_version, resolve_cpg_backend
from openultrasast.cpg.backend import JoernBackend, joern_version
from openultrasast.mapping import analyze_entry_points
from openultrasast.model.contracts import ExecutionBudget
from openultrasast.model.regions import ScanRegion, regions_for
@@ -156,6 +156,8 @@ def replay(
started = time.monotonic()
budget = ExecutionBudget(started + config.deadline_seconds, config.cancellation_allowance_seconds)
scan_budget = ScanBudget(max_model_calls=0, max_regions=max_regions, order_by_evidence=True)
if backend is None:
backend = JoernBackend()
records: list[ScanRecord] = []
manifests: list[SnapshotManifest] = []
reasons: list[str] = []
@@ -195,7 +197,7 @@ def replay(
head_scan = scan_repository(
tip.root,
head_regions,
backend=backend or resolve_cpg_backend(),
backend=backend,
budget=scan_budget,
execution_budget=budget,
ranking_mode="evidence",
@@ -212,7 +214,7 @@ def replay(
head_regions=head_regions,
base_regions=base_regions,
context=context,
backend=backend or resolve_cpg_backend(),
backend=backend,
execution_budget=budget,
scan_budget=scan_budget,
head_semantics=provenance["semantics"],
+44
View File
@@ -271,3 +271,47 @@ def test_unbatched_failure_is_not_a_completed_empty_answer(tmp_path):
assert result.question_outcomes[0].status == "unanswered"
assert result.regions_scanned == 0
assert result.family_coverage[0].completed == 0
def test_failed_build_does_not_reload_facts_after_deadline(tmp_path, monkeypatch):
import time
import openultrasast.model.scan as driver
from openultrasast.model.contracts import ExecutionBudget
root = source(tmp_path, "api.py")
def forbidden_facts(*args, **kwargs):
raise AssertionError("failed build must not perform fresh semantic work")
monkeypatch.setattr(driver, "_spec_for", forbidden_facts)
regions = [ScanRegion("api.py", f"handler{i}", "python", ("injection", "access_control"), 1.0, "entry_point") for i in range(200)]
started = time.monotonic()
result = scan_repository(root, regions, backend=Backend(fail=True), execution_budget=ExecutionBudget(started + 0.01, 0.5))
assert time.monotonic() - started < 0.5
assert len(result.scope.deferred) == 400 and not result.scope.selected
assert all(q.reason == "cpg_build_failed" for q in result.scope.deferred)
assert not result.question_outcomes and "cpg_build_failed" in result.scope.unresolved_boundaries
def test_expired_planning_keeps_census_without_more_fact_reads(tmp_path, monkeypatch):
import time
import openultrasast.model.scan as driver
from openultrasast.model.contracts import ExecutionBudget
root = source(tmp_path, "api.py")
original = driver._spec_for
deadline = time.monotonic() + 0.05
def checked_facts(*args, **kwargs):
assert time.monotonic() < deadline, "fact load started after the deadline"
time.sleep(0.015)
return original(*args, **kwargs)
monkeypatch.setattr(driver, "_spec_for", checked_facts)
regions = [ScanRegion("api.py", f"handler{i}", "python", ("injection",), 1.0, "entry_point") for i in range(200)]
result = scan_repository(root, regions, backend=Backend(), execution_budget=ExecutionBudget(deadline, 0.5))
assert time.monotonic() < deadline + 0.5
assert len(result.scope.selected) + len(result.scope.deferred) == 200
assert "deadline_exhausted" in result.scope.unresolved_boundaries
+22
View File
@@ -196,3 +196,25 @@ def test_snapshot_boundaries_reach_engine_change_context(tmp_path):
data = json.loads(artifact.read_text())
assert result.result.coverage_status == "incomplete"
assert "snapshot:symlink_not_materialized" in data["change_context"]["unresolved_boundaries"]
def test_default_replay_does_not_launch_unbudgeted_availability_probe(tmp_path, monkeypatch):
from openultrasast.push import runner
from openultrasast.semantic import engines
root, base, head, _ = history(tmp_path)
builds = []
class Backend:
def build(self, root, **kwargs):
assert any(p.read_bytes() for p in root.rglob("*.js"))
builds.append(kwargs["execution_budget"])
return None
def forbidden_probe(*args, **kwargs):
raise AssertionError("availability probe has no shared deadline")
monkeypatch.setattr(engines, "joern_available", forbidden_probe)
monkeypatch.setattr(runner, "JoernBackend", Backend, raising=False)
result = replay(root, base=base, head=head, artifact=tmp_path / "probe.json")
assert builds and result.result.coverage_status == "incomplete"