mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
fix(pre-push-safety-net): stop semantic planning at the shared deadline
This commit is contained in:
@@ -64,3 +64,44 @@ RED evidence: `/tmp/ousast-replay-tierzero-red.log` and `/tmp/ousast-replay-snap
|
||||
- CLAIM: Explicit local replay traverses the production mapper/ranker/arbiter/comparison/report path and retains uncertainty and shared deadlines.
|
||||
- EVIDENCE: Final packaged controlled security change yields new/source_connection_absent_from_comparable_base. Fixed twin emits no modeled head finding. The packaged deadline control returns incomplete in 30.785 seconds, within 30 + 2 seconds. Exact final package policy/core/facts/query digests match the workspace. Complete commands and captured raw answers live in the measurement artifact.
|
||||
- GAPS: Aggregate coverage remains incomplete, real capabilities are disabled, and lab timings do not satisfy the latency gate. Source-mounted fixed and earlier packaged deadline provenance is explicitly separated in the artifact. PMPro/Node transfer is task 5.2, not inferred from a small PHP control.
|
||||
|
||||
## Predecessor deadline regression discovered by task 5.2
|
||||
|
||||
The first frozen trial completed six CLI invocations, retaining all 11 declared cases.
|
||||
All three PMPro cases timed out during preparation; both snapshots were fully materialized.
|
||||
Node hit the engine deadline but then took 34.6–37.4 seconds and could not publish its artifact.
|
||||
Task 5.2 is blocked pending the owning 3.1/5.1 deadline integration repair and a newly frozen run.
|
||||
The initial trial overlaps regression testing; it is diagnostic evidence, not the final latency sample.
|
||||
|
||||
### Debug report
|
||||
- ROOT_CAUSE: After an engine timeout, the generic scan driver's failure census reloaded semantic facts and taxonomy for every question. The replay runner then launched an unbudgeted engine-availability subprocess before attempting base comparison. Main-thread stack samples at 30–34 seconds prove both paths.
|
||||
- CATEGORY: LOGIC_ERROR
|
||||
- FIX_PLAN: Route failure bookkeeping through the existing driver without loading new semantic facts after a failed build; instantiate the budget-aware Joern backend once rather than running legacy availability probes in replay. Preserve every declared question as deferred and keep ordinary backend selection unchanged outside replay.
|
||||
- VERIFICATION: Two failing regressions (including 400 declared failed questions), real Node deadline replay, full suite and packaged validation before re-freezing the experiment.
|
||||
- NEXT_ACTION: RETRY_TASK
|
||||
- CONFIDENCE: HIGH
|
||||
- NOTES: This repairs the already-approved generic deadline integration in task 3.1 and replay in 5.1. It changes no detector facts, ranking features or abstract capabilities. No workaround is hidden in the evaluation harness; task 5.2 stays blocked until this contract is repaired. Manual Kiro debug/review fallback applies after the recorded agent thread limit. Runtime trace proves a local logic error; no external dependency/version hypothesis is needed.
|
||||
|
||||
RED: `/tmp/ousast-node-deadline-flag-red.log`, two failures with flags OFF. Flags were
|
||||
removed after GREEN; the focused scope/replay suite passed 30 tests.
|
||||
|
||||
|
||||
The first repair exposed the same repeated fact-read path after evidence planning rather than
|
||||
build failure. The absolute budget now also reaches `_collect` and `_scope_work`; planning
|
||||
stops loading facts when time expires but retains the entire question census. No ranking weights,
|
||||
facts or detector semantics changed. The additional regression failed before repair in
|
||||
`/tmp/ousast-node-planning-red.log` and then passed.
|
||||
|
||||
### Predecessor repair review verdict
|
||||
- VERDICT: APPROVED
|
||||
- TASK: 3.1/5.1 deadline integration regression
|
||||
- MECHANICAL_RESULTS: Full current suite 1,242 passed, nine skipped (including three pending experiment-harness checks), exit 0; Ruff/format/mypy/compileall and all regression gates PASS; behavioral RED VERIFIED; no new placeholders/secrets; scope is the existing driver/deadline integration.
|
||||
- FINDINGS: Both measured post-deadline paths are repaired, including successful-build/evidence-planning cancellation; all declared questions remain recorded. No outstanding repair blocker.
|
||||
- SUMMARY: Manual review approves the predecessor repair; task5.2 may resume with a new frozen profile.
|
||||
|
||||
### Predecessor repair verification
|
||||
- STATUS: VERIFIED
|
||||
- CLAIM_TYPE: FIX
|
||||
- CLAIM: The reproduced Node cancellation path retains diagnostics and returns within 30 + 2 seconds.
|
||||
- EVIDENCE: Final packaged real Node replay returned in 30.1255 seconds with one complete diagnostic artifact and incomplete coverage. Final policy/core/facts/query hashes match the workspace. See `benchmarks/measurements/2026-09-13-replay-deadline-repair.json` for initial/partial-repair traces and final raw artifact.
|
||||
- GAPS: This proves the reproduced cancellation path, not representative warm performance or security coverage. Normal capabilities remain disabled.
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"feature_name": "pre-push-safety-net",
|
||||
"created_at": "2026-09-12T15:35:00Z",
|
||||
"updated_at": "2026-09-13T12:50:25.249104+00:00",
|
||||
"updated_at": "2026-09-13T13:11:42.850127+00:00",
|
||||
"language": "en",
|
||||
"phase": "implementation",
|
||||
"approvals": {
|
||||
@@ -19,7 +19,7 @@
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": true,
|
||||
"implementation_approval_note": "Group 5 authorized. Task5.1 verified through manual Kiro fallback review; 5.2 next. Real capabilities remain disabled.",
|
||||
"implementation_approval_note": "Group5 authorized. Task5.1 and its observed deadline regression repaired and verified with manual Kiro fallback; task5.2 resumes under a newly frozen core identity.",
|
||||
"depends_on": [
|
||||
"contributor-scan"
|
||||
],
|
||||
|
||||
@@ -149,6 +149,7 @@ Task 5.1 is also verified (15/27 total); task 5.2 and groups 6–8 remain pendin
|
||||
- _Depends: 1.3, 3.1, 3.4, 4.3_
|
||||
- _Requirements: 1.1, 2.1, 3.1, 3.4, 4.1, 4.2, 5.5, 7.1, 7.3_
|
||||
- [ ] 5.2 Measure PMPro and frozen-core Node transfer
|
||||
- _Prerequisite repaired 2026-09-13:_ 3.1/5.1 deadline regression verified in the packaged Node replay; re-freeze the experiment after this core identity change.
|
||||
- Replay each prepared security/fixed/benign case, first using the declared hook deadline; record timeouts and stage costs. A separately named extended lab run may obtain witnesses but cannot satisfy the hook latency gate.
|
||||
- Freeze the core after PMPro, run Node with existing facts, and classify instrument, mapping, fact, scope and domain gaps without tuning on an untouched holdout.
|
||||
- Done when one joint artifact reports actual evidence and outcomes for both workloads, with unresolved cases retained and an explicit feasibility decision; no normal capability is enabled here.
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -245,7 +245,15 @@ def _scan_repository_impl(
|
||||
if cpg is not None:
|
||||
owned.append(cpg)
|
||||
scope, _ = _scope_work(
|
||||
regions, {}, limits, unit, ranking_mode, population_complete, failure="cpg_build_failed", change_context=change_context
|
||||
regions,
|
||||
{},
|
||||
limits,
|
||||
unit,
|
||||
ranking_mode,
|
||||
population_complete,
|
||||
failure="cpg_build_failed",
|
||||
change_context=change_context,
|
||||
execution_budget=execution_budget,
|
||||
)
|
||||
scope = replace(scope, unresolved_boundaries=tuple(dict.fromkeys((*scope.unresolved_boundaries, *getattr(cpg, "boundaries", ())))))
|
||||
return ModelScanResult(
|
||||
@@ -319,7 +327,7 @@ def _scan_repository_impl(
|
||||
tier_started = time.monotonic()
|
||||
evidence_by_pair, failed = _evidence_pass(
|
||||
cpg,
|
||||
_collect(evidence_candidates),
|
||||
_collect(evidence_candidates, execution_budget=execution_budget),
|
||||
hooks,
|
||||
degradations=degradations,
|
||||
context_rows=context_rows if change_context is not None else None,
|
||||
@@ -369,6 +377,7 @@ def _scan_repository_impl(
|
||||
population_complete,
|
||||
change_context=change_context,
|
||||
context_gaps=context_gaps,
|
||||
execution_budget=execution_budget,
|
||||
)
|
||||
outcomes: dict[str, QuestionOutcome] = {}
|
||||
|
||||
@@ -573,6 +582,7 @@ def _scope_work(
|
||||
failure: str | None = None,
|
||||
change_context: ChangeContext | None = None,
|
||||
context_gaps: set[QuestionIdentity] | None = None,
|
||||
execution_budget: ExecutionBudget | None = None,
|
||||
) -> tuple[ScopeDecision, list[tuple[str, ScanRegion, ArbiterSpec]]]:
|
||||
"""Consume the ranker's final order once; the returned work IS selected scope."""
|
||||
selected: list[RankedQuestion] = []
|
||||
@@ -586,10 +596,15 @@ def _scope_work(
|
||||
for index, region in enumerate(regions):
|
||||
for family in region.families:
|
||||
identity = _identity(unit, region, family)
|
||||
spec = _spec_for(family, region.language)
|
||||
if failure is None and not _within_deadline(execution_budget):
|
||||
failure = "deadline_exhausted"
|
||||
boundaries.append(failure)
|
||||
spec = None if failure else _spec_for(family, region.language)
|
||||
vector = evidence.get((region.path, region.function or "", family))
|
||||
facts: tuple[str, ...] = ("static_rank=" + str(region.rank),)
|
||||
if vector is not None:
|
||||
if failure:
|
||||
facts += ("analysis_not_started=" + failure,)
|
||||
elif vector is not None:
|
||||
facts += ("normalized_evidence=" + _rows_json(asdict(vector)), "tier=" + str(vector.tier), "score=" + str(vector.score))
|
||||
elif isinstance(spec, TaintSpec):
|
||||
facts += ("evidence_unknown",)
|
||||
@@ -616,7 +631,9 @@ def _scope_work(
|
||||
if identity in (context_gaps or set()):
|
||||
facts += ("change_context_incomplete",)
|
||||
reason = (
|
||||
"unsupported_family"
|
||||
failure
|
||||
if failure
|
||||
else "unsupported_family"
|
||||
if spec is None
|
||||
else failure
|
||||
or (
|
||||
@@ -676,11 +693,15 @@ def _pair_key(region: ScanRegion, spec: ArbiterSpec) -> tuple[str, str, str]:
|
||||
return (region.path, region.function or "", getattr(spec, "family", ""))
|
||||
|
||||
|
||||
def _collect(regions: Sequence[ScanRegion]) -> list[tuple[str, ScanRegion, ArbiterSpec]]:
|
||||
def _collect(
|
||||
regions: Sequence[ScanRegion], *, execution_budget: ExecutionBudget | None = None
|
||||
) -> list[tuple[str, ScanRegion, ArbiterSpec]]:
|
||||
"""Every (region, family) question with an arbiter, in the order given, with batch-local ids."""
|
||||
work: list[tuple[str, ScanRegion, ArbiterSpec]] = []
|
||||
for region in regions:
|
||||
for family in region.families:
|
||||
if not _within_deadline(execution_budget):
|
||||
return work
|
||||
spec = _spec_for(family, region.language)
|
||||
if spec is None:
|
||||
continue
|
||||
|
||||
@@ -14,7 +14,7 @@ from pathlib import Path
|
||||
from typing import Any, Literal, TypeVar, cast
|
||||
|
||||
from openultrasast.config import PushConfig
|
||||
from openultrasast.cpg.backend import joern_version, resolve_cpg_backend
|
||||
from openultrasast.cpg.backend import JoernBackend, joern_version
|
||||
from openultrasast.mapping import analyze_entry_points
|
||||
from openultrasast.model.contracts import ExecutionBudget
|
||||
from openultrasast.model.regions import ScanRegion, regions_for
|
||||
@@ -156,6 +156,8 @@ def replay(
|
||||
started = time.monotonic()
|
||||
budget = ExecutionBudget(started + config.deadline_seconds, config.cancellation_allowance_seconds)
|
||||
scan_budget = ScanBudget(max_model_calls=0, max_regions=max_regions, order_by_evidence=True)
|
||||
if backend is None:
|
||||
backend = JoernBackend()
|
||||
records: list[ScanRecord] = []
|
||||
manifests: list[SnapshotManifest] = []
|
||||
reasons: list[str] = []
|
||||
@@ -195,7 +197,7 @@ def replay(
|
||||
head_scan = scan_repository(
|
||||
tip.root,
|
||||
head_regions,
|
||||
backend=backend or resolve_cpg_backend(),
|
||||
backend=backend,
|
||||
budget=scan_budget,
|
||||
execution_budget=budget,
|
||||
ranking_mode="evidence",
|
||||
@@ -212,7 +214,7 @@ def replay(
|
||||
head_regions=head_regions,
|
||||
base_regions=base_regions,
|
||||
context=context,
|
||||
backend=backend or resolve_cpg_backend(),
|
||||
backend=backend,
|
||||
execution_budget=budget,
|
||||
scan_budget=scan_budget,
|
||||
head_semantics=provenance["semantics"],
|
||||
|
||||
@@ -271,3 +271,47 @@ def test_unbatched_failure_is_not_a_completed_empty_answer(tmp_path):
|
||||
assert result.question_outcomes[0].status == "unanswered"
|
||||
assert result.regions_scanned == 0
|
||||
assert result.family_coverage[0].completed == 0
|
||||
|
||||
|
||||
def test_failed_build_does_not_reload_facts_after_deadline(tmp_path, monkeypatch):
|
||||
import time
|
||||
|
||||
import openultrasast.model.scan as driver
|
||||
from openultrasast.model.contracts import ExecutionBudget
|
||||
|
||||
root = source(tmp_path, "api.py")
|
||||
|
||||
def forbidden_facts(*args, **kwargs):
|
||||
raise AssertionError("failed build must not perform fresh semantic work")
|
||||
|
||||
monkeypatch.setattr(driver, "_spec_for", forbidden_facts)
|
||||
regions = [ScanRegion("api.py", f"handler{i}", "python", ("injection", "access_control"), 1.0, "entry_point") for i in range(200)]
|
||||
started = time.monotonic()
|
||||
result = scan_repository(root, regions, backend=Backend(fail=True), execution_budget=ExecutionBudget(started + 0.01, 0.5))
|
||||
assert time.monotonic() - started < 0.5
|
||||
assert len(result.scope.deferred) == 400 and not result.scope.selected
|
||||
assert all(q.reason == "cpg_build_failed" for q in result.scope.deferred)
|
||||
assert not result.question_outcomes and "cpg_build_failed" in result.scope.unresolved_boundaries
|
||||
|
||||
|
||||
def test_expired_planning_keeps_census_without_more_fact_reads(tmp_path, monkeypatch):
|
||||
import time
|
||||
|
||||
import openultrasast.model.scan as driver
|
||||
from openultrasast.model.contracts import ExecutionBudget
|
||||
|
||||
root = source(tmp_path, "api.py")
|
||||
original = driver._spec_for
|
||||
deadline = time.monotonic() + 0.05
|
||||
|
||||
def checked_facts(*args, **kwargs):
|
||||
assert time.monotonic() < deadline, "fact load started after the deadline"
|
||||
time.sleep(0.015)
|
||||
return original(*args, **kwargs)
|
||||
|
||||
monkeypatch.setattr(driver, "_spec_for", checked_facts)
|
||||
regions = [ScanRegion("api.py", f"handler{i}", "python", ("injection",), 1.0, "entry_point") for i in range(200)]
|
||||
result = scan_repository(root, regions, backend=Backend(), execution_budget=ExecutionBudget(deadline, 0.5))
|
||||
assert time.monotonic() < deadline + 0.5
|
||||
assert len(result.scope.selected) + len(result.scope.deferred) == 200
|
||||
assert "deadline_exhausted" in result.scope.unresolved_boundaries
|
||||
|
||||
@@ -196,3 +196,25 @@ def test_snapshot_boundaries_reach_engine_change_context(tmp_path):
|
||||
data = json.loads(artifact.read_text())
|
||||
assert result.result.coverage_status == "incomplete"
|
||||
assert "snapshot:symlink_not_materialized" in data["change_context"]["unresolved_boundaries"]
|
||||
|
||||
|
||||
def test_default_replay_does_not_launch_unbudgeted_availability_probe(tmp_path, monkeypatch):
|
||||
from openultrasast.push import runner
|
||||
from openultrasast.semantic import engines
|
||||
|
||||
root, base, head, _ = history(tmp_path)
|
||||
builds = []
|
||||
|
||||
class Backend:
|
||||
def build(self, root, **kwargs):
|
||||
assert any(p.read_bytes() for p in root.rglob("*.js"))
|
||||
builds.append(kwargs["execution_budget"])
|
||||
return None
|
||||
|
||||
def forbidden_probe(*args, **kwargs):
|
||||
raise AssertionError("availability probe has no shared deadline")
|
||||
|
||||
monkeypatch.setattr(engines, "joern_available", forbidden_probe)
|
||||
monkeypatch.setattr(runner, "JoernBackend", Backend, raising=False)
|
||||
result = replay(root, base=base, head=head, artifact=tmp_path / "probe.json")
|
||||
assert builds and result.result.coverage_status == "incomplete"
|
||||
|
||||
Reference in New Issue
Block a user