From 3259d0469f08ec19f2d4077e0b615390a8c56f09 Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Sun, 6 Sep 2026 08:07:44 +0200 Subject: [PATCH] docs(learning-harness): brief for a classified, self-improving detector set; fold closed-loop-integrity Measured 2026-09-06 on the vibe-py holdout: the existing tool hunter, with no learning, scores 12/14 pair-correct (Youden +0.857, no leaks) once detection and silence are keyed on the labeled class and function and the byte-identical twins are excluded; the overlay scores 5/14 on the same rows. The published hunter number (0/17, Youden -0.588) was a scoring artifact: the scorer wanted the literal CWE id in the finding text and the prompt never asked for it. The brief proposes an auto-classifier as the routing key, one evolved hunter configuration per class, a class-aware scorer with unscorable denominators, the split enforced by construction, and the HarnessX evolve loop with per-class attribution as the round primitive. Raw measurements are committed under reports/measurements. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG --- .kiro/specs/closed-loop-integrity/brief.md | 73 - .kiro/specs/learning-harness/brief.md | 90 ++ ...09-06-vibe-py-holdout-hunter-findings.json | 841 ++++++++++++ ...-09-06-vibe-py-holdout-hunter-sonnet5.json | 1220 +++++++++++++++++ 4 files changed, 2151 insertions(+), 73 deletions(-) delete mode 100644 .kiro/specs/closed-loop-integrity/brief.md create mode 100644 .kiro/specs/learning-harness/brief.md create mode 100644 benchmarks/measurements/2026-09-06-vibe-py-holdout-hunter-findings.json create mode 100644 benchmarks/measurements/2026-09-06-vibe-py-holdout-hunter-sonnet5.json diff --git a/.kiro/specs/closed-loop-integrity/brief.md b/.kiro/specs/closed-loop-integrity/brief.md deleted file mode 100644 index 3e1ee37..0000000 --- a/.kiro/specs/closed-loop-integrity/brief.md +++ /dev/null @@ -1,73 +0,0 @@ -# Brief: closed-loop-integrity - -## Problem - -The closed loop cannot tell a better detector from a corpus artifact. Measured 2026-09-05 while wiring the obligation checker to the real pair corpus: - -- **The lever trains on the test set.** `ousast mechanisms export` seeds candidate shapes from every vendored pair of a slice, and `improve.propose_mechanism_edits` admits a candidate because it recovers a missed holdout pair. A shape taught by a holdout pair can recover itself. On vibe-py (35 vendored, 18 train, 17 holdout) 5 of 11 candidates were taught by a holdout pair; holdout Youden is +0.059 when every pair teaches and **−0.059 when only the train split teaches**. The reported +0.353 holdout gain from `corpus-seeded-mechanisms` is inflated by an unknown amount. `evaluate_loo` is honest; only the lever leaks. -- **Five vibe-py twins are byte-identical** once the provenance header is stripped (`dsvw-dsvw-do-get`, `python-insecure-app-main-try-hack-me`, `pythonssti-main-read-root`, `vampi-books-get-by-title`, `vulnpy-deserialization-do-pickle-load`). No detector can be loud on one side and silent on the other; the slice ceiling is 30/35 and every published vibe-py number carries those five as permanent losses. -- **TypeScript has no grammar.** The semantic extra declares python, javascript, c, cpp and java. Every `.ts`/`.tsx` row parses as `language_unsupported`: 18 agent-vfc rows plus most of the absence rows. The row count is scored as failure, not as unscorable. -- **Excerpts carry no registration context.** Vendored absence excerpts are the handler function alone, so no entry point is named and anything gated on reachability reports nothing. The `handler_context` harvest mode exists (2026-09-05) but no row has been re-harvested with it. -- **Unit fixtures are easier than the corpus.** Fixtures bind identity in one hop; the real vampi fix binds it in two. Green suites therefore do not evidence corpus behaviour, and nothing in CI compares a fixture's difficulty against the rows it stands in for. - -The consequence is that three specs (`corpus-seeded-mechanisms`, `authorization-obligations`, and the unwritten `guard-dominance-regime`) each added a detector and a measurement, and none of the measurements is trustworthy. Detection logic is ~16% of the source; the other 84% is harness that currently cannot say whether the 16% works. - -## Desired Outcome - -One number per slice that a maintainer can believe, before any detector is tuned against it. Concretely: the improve lever never sees a holdout excerpt while it learns; unscorable rows are excluded from the denominator and listed by reason; every language the corpus carries either parses or is counted as unsupported rather than as a miss; every absence excerpt carries the context its label needs; and a fixture-difficulty check fails CI when a unit fixture is strictly easier than the corpus rows its test claims to cover. - -Then, and only then, the evolutionary loop (HarnessX `MetaAgent.evolve` or the existing bounded levers) is given more search freedom, because a fitness function that rewards memorizing the holdout will be optimized for exactly that. - -Principle for the roadmap: **the ruler is repaired before the thing it measures is tuned; no published number without its unscorable denominator.** - -## Approach - -1. **Split enforcement.** The exporter takes a split and defaults to `train`; the lever refuses a candidate whose `record.pairs` intersect the holdout set and records the refusal. Re-measure `corpus-seeded-mechanisms` and rewrite its roadmap row with the honest number. -2. **Unscorable rows.** The pair loader flags `identical_twin` (excerpt bodies equal after header strip) and `language_unsupported` rows; scorer and LOO report them under a `unscorable` counter with the reason and drop them from recall, silence and Youden denominators. Nothing is deleted; the catalog row gains a `known_limit` reason instead, which the existing achievable-metric machinery already honors. -3. **TypeScript grammar.** `tree-sitter-typescript` joins the semantic extra; the walker maps `typescript`/`tsx` to it. Existing grammars unchanged. -4. **Context re-harvest.** The absence rows of vibe-py and agent-vfc are re-harvested in `handler_context` mode (maintainer, network, license already on the recipe); pointer rows through the cache. Task 4.3 of `authorization-obligations` moves here. -5. **Fixture-difficulty gate.** A test-only check derives a small difficulty vector from a fixture (bind depth to the identity source, registration present, cross-file registration) and compares it against the corpus rows named in the test's docstring; strictly easier fails. -6. **Evolution under the repaired ruler.** Only after 1–5: expose the HarnessX evolve loop over the existing levers (rules, policy, mechanisms) with the per-profile holdout clause as the fitness function and the split enforced by construction. This step gets its own requirements once 1–5 are measured. - -## Scope - -- **In**: exporter split filter and lever refusal; unscorable classification in the loader, scorer and LOO; TypeScript grammar in the semantic extra; re-harvest of absence rows; fixture-difficulty check; re-measurement and roadmap rewrite; a closure decision on `authorization-obligations` tasks 4.3, 5 and 6 (they measure against this repaired ruler or not at all). -- **Out**: new detectors; new shape families; changing dispositions or evidence rungs; any merge gate on non-local slices; vendoring new pairs beyond re-harvesting existing recipes. - -## Boundary Candidates - -- Corpus honesty (loader, scorer, LOO, catalogs) vs detectors (unchanged) -- Lever admission rule (split-aware) vs lever validator and store schema (unchanged) -- Semantic extra dependency set (one grammar added) vs walker semantics (unchanged) - -## Out of Boundary - -- Taint semantics, facts, dominance (`guard-dominance-regime`) -- Call graph, path records (`reachability-flow-model`) -- Obligation checker internals (`authorization-obligations`), except that its corpus tasks are re-homed here - -## Upstream / Downstream - -- **Upstream**: `pair-corpus-honesty` (labels, tiers, splits, `known_limit`), `corpus-seeded-mechanisms` (exporter, lever, LOO), `tree-sitter-overlay-extra` (grammar set), `authorization-obligations` (`handler_context`, obligation label). -- **Downstream**: every roadmap number; `harnessx-self-improving-rulesets` (evolve loop gets a trustworthy fitness); `authorization-obligations` 4.3/5/6. - -## Existing Spec Touchpoints - -- **Extends**: `pair-corpus-honesty` Req 9/10 (one more row classification), `corpus-seeded-mechanisms` Req 4/5 (split-aware seeding and admission), `tree-sitter-overlay-extra` (grammar list). -- **Re-homes**: `authorization-obligations` task 4.3 and the corpus half of 6.1. -- **Blocks**: `harnessx-self-improving-rulesets` evolve step until 1–5 are measured. - -## Constraints - -Core `dependencies = []`; the grammar lands in the optional extra only. No row is deleted; unscorable rows stay in the catalog with a reason. No LLM-authored labels, shapes or policy. Gates (`gate`, `map_gate`, `pair_gate`) stay byte-identical: the local slice has no identical twins, no TypeScript and no lever. Every re-harvested excerpt is re-redacted and keeps its license line. All numbers in the roadmap are rewritten in one commit with the measurement command that produced them. - -## Viability of `authorization-obligations` under this brief - -The detector is sound where it can see: with a registration line present it fires on the real vampi vulnerable excerpt and stays silent on a constrained twin. It cannot be evaluated today because its inputs (named entry points, registration context, TypeScript parse) are exactly the defects above. Recommendation: keep tasks 1–3 and 4.1/4.2 (implemented, reviewed, pushed), re-home 4.3 and 6.1's corpus half here, and hold 5 (the lever) until the split is enforced, because a lever admitted under the leaking rule would be admitted for the wrong reason. - -## Open Questions for the maintainer - -1. Treat the five identical twins as `known_limit` (excluded from achievable, kept for silence) or drop them from the vibe-py catalog? -2. Is `tree-sitter-typescript` acceptable in the semantic extra, or does TypeScript wait for a broader overlay-ir-completeness spec? -3. Should the fixture-difficulty gate be CI-failing from day one or reported first? -4. Does the HarnessX evolve step (approach 6) get its own spec, or stays as the final task group here once 1–5 are measured? diff --git a/.kiro/specs/learning-harness/brief.md b/.kiro/specs/learning-harness/brief.md new file mode 100644 index 0000000..a39a229 --- /dev/null +++ b/.kiro/specs/learning-harness/brief.md @@ -0,0 +1,90 @@ +# Brief: learning-harness + +## Problem + +The tool is a hand-authored static analyzer with a learning loop bolted on the side, and the loop cannot learn. Three deliberate choices cap it: closed vocabularies (sinks, sources, obligation kinds) that grow only by writing a new spec; a corpus that is the only teacher and a shape that is the only lesson; and a ban on retaining anything the model discovers. Results have ceilings by construction, the loop trains on its own test set, and there is no failure analysis feeding the next round. Detection logic is ~16% of the source and the other 84% cannot say whether the 16% works. + +Measured 2026-09-06 on the vibe-py holdout (17 vendored pairs), the existing tool hunter (Sonnet 5 through OpenRouter, 4 steps, tools `read_file`, `grep_repo`, `find_refs`, prompted from file-level hotspots, no learning of any kind) against the overlay: + +| scorer | published number | class-aware, identical twins excluded (n = 14) | +|---|---|---| +| tool hunter | 0/17 correct, Youden −0.588 | **12/14 correct, Youden +0.857, 0 leaks** | +| overlay (facts + shapes) | 5/17 correct, Youden +0.294 | 5/14 correct, Youden +0.357 | + +The published hunter number is a scoring artifact: the pair scorer counts a hunter finding only when the finding text contains the literal CWE id and the prompt never asks for one, so 24 of 25 vulnerable-side findings that sit inside the labeled function and name the labeled bug (SQL injection in `resolve_pastes`, BOLA in `get_by_title`, SSTI in `read_root`, missing authorization on the profile lookup) were thrown away. The 24 "leaks" split into 3 findings on byte-identical twins (unscorable by construction) and 21 findings outside the labeled function on the trap twin, most of them real defects in a deliberately vulnerable repository. The two remaining misses are `before_request` (`loads(b64decode(cookie))` with no import in the excerpt) and `bad_mvc.do_GET` (labeled CWE-79, the hunter reported the SQL concatenation on the same lines; the label may be wrong). + +Three consequences follow. + +1. **The model with tools is already the best detector we own, by a wide margin, without any learning.** The roadmap thesis ("recall from the model with tools plus proof; precision and silence from static checks") is confirmed, and the harness has been optimizing the wrong component. +2. **Detection, silence and leak are meaningless without a class.** The same finding is a detection in one class, a leak in another, and noise in a third. Every scorer, lever and gate in the tool currently uses text tokens (CWE strings, sink names) as a proxy for class, and the corpus's own class labels are 32 free-text strings across 7 slices, with 130 of 176 vfc rows labeled `other`. +3. **Improvements without a class dimension oscillate.** A change that helps SQL injection and hurts access control reads as "no change" on an aggregate Youden. The loop then reverts it, or keeps it, for the wrong reason, and the system degrades in ways nobody predicted. This is the maintainer's observation and it is exactly what the corpus-seeded lever does today. + +Repairs already measured and required regardless (from the folded `closed-loop-integrity` brief): the mechanism lever seeds candidates from holdout pairs (holdout Youden +0.059 with every pair teaching, −0.059 with the train split only); five vibe-py twins are byte-identical; TypeScript has no grammar; absence excerpts carry no registration context; unit fixtures are easier than the corpus rows they stand in for. + +## Desired Outcome + +A harness that improves a **set of classified detectors** round by round, where every change is attributed to the class it was meant to move, every unpredicted regression in another class counts against it, and nothing is published without its unscorable denominator. + +- **Auto-classifier.** Every unit of work (a labeled pair, a finding, a scan target region) is assigned to a class from a closed, versioned taxonomy by a classifier that is itself measured: agreement with maintainer labels on the reviewed pairs, confusion matrix per round. The classifier is the routing key for everything downstream: which detector runs, which skills load, which holdout the change is validated on, which journal row records it. +- **Per-class detectors.** One hunter configuration per class: system prompt, skills, tool set, step budget, evidence checklist, known counterexamples. A round evolves one class's configuration and is validated on that class's holdout plus a regression sweep over every other class. The static machinery (facts, overlay, obligations, dominance, sandbox) becomes tools the detector calls and verifiers that check its claims; it stops being the detector. +- **Class-aware scorer.** Detection = a finding in the labeled function whose class matches the label. Silence = no finding of the labeled class in the labeled function on the fixed side. Unscorable rows (identical twin, unsupported language, unresolved label) are listed by reason and removed from the denominator, never deleted. The text-token proxy goes away. +- **Learning unit = skill, not regex.** A round's output is a natural-language skill, a tool recipe, a checklist or a counterexample, drafted from a failure analysis, tested on the class's train split, validated on its holdout, journaled with a hypothesis, the predicted affected class, and the measured attribution. HarnessX's `MetaAgent.evolve` with its journal (`hypothesis_id`, `levers`, `predicted_affected`, precision discounted by `regressed_unpredicted`) is the loop primitive; the tool supplies tasks, trajectories, the evaluator and the class routing. +- **Honest fitness.** Split enforced by construction (a candidate touching a holdout pair is refused, not scored); per-class Youden on scorable rows; sandbox proof as the tiebreaker where a class has recipes; the roadmap rewritten from one command with the unscorable denominator next to every number. + +Principle for the roadmap: **classify first, then detect, then improve one class at a time under a fitness that punishes what it did not predict.** + +## Approach + +1. **Class-aware scorer and unscorable rows** (repairs the ruler; smallest change, biggest number). Detection and silence keyed on class and labeled function; identical-twin and unsupported-language rows flagged with a reason and excluded from denominators; the hunter's findings carry a class field the scorer reads. Re-publish every slice number from one command. +2. **Taxonomy and auto-classifier.** A closed, versioned class taxonomy (seeded from the 32 free-text labels, the mechanism vocabulary and the obligation kinds; CWE families as the spine). A classifier with two tiers: a deterministic tier from static signals (rule id, sink family, obligation kind, mechanism) and a model tier for what static signals cannot place, each answer carrying which tier decided. Measured against maintainer labels on the reviewed pairs; disagreements are the review queue. The vfc slice's 130 `other` rows are its first job. +3. **Per-class hunter configurations.** Split the single hunter into one configuration per class: prompt, skills directory, tools, budget, evidence checklist. Start by cloning the current hunter into every class so round zero is a controlled baseline, then let rounds diverge them. The classifier routes a target region to its detector; a region can route to several. +4. **Split enforcement and lever refusal.** The exporter and every future lever take the train split by construction and refuse candidates that touch a holdout pair. Re-measure `corpus-seeded-mechanisms` and rewrite its roadmap row. +5. **Evolve loop on HarnessX.** Tasks = classified pairs (`RLTask.task_type` = class); trajectories = the hunter's journaled runs; evaluator = the class-aware scorer; `MetaAgent.evolve` proposes one configuration change per round for one class with `predicted_affected` set to that class; the round is accepted only if the class's holdout improves and no other class regresses beyond tolerance; the journal's attribution table is the round report. Budget-capped per round. +6. **Corpus repairs that the loop needs.** TypeScript grammar in the semantic extra; absence rows re-harvested with `handler_context`; identical twins marked `known_limit` with reason `identical_twin`; a fixture-difficulty check that fails when a unit fixture is strictly easier than the corpus rows its test claims to cover. +7. **Static machinery as tools and verifiers.** Facts, overlay dispositions, obligations, entry points and the sandbox are exposed to the hunter as tools (`entry_points`, `flows`, `obligations`, `prove`) and to the scorer as verifiers that can raise a hunter suspicion to `static_corroboration` or a proof rung. No detector logic is deleted; it changes role. + +## Scope + +- **In**: class-aware scorer; taxonomy and classifier with its own measurement; per-class hunter configurations; split enforcement; HarnessX evolve loop wiring with tasks, trajectories, evaluator and journal; the corpus repairs listed in 6; static machinery exposed as hunter tools; roadmap rewrite from one command. +- **Out**: new static shape families; new hand-written facts beyond what a round proposes and validates; changing the evidence ladder or dispositions; any merge gate on non-local slices; vendoring new pairs beyond re-harvesting existing recipes; fine-tuning model weights (skills and configurations only). + +## Boundary Candidates + +- Taxonomy and classifier (new, owns class assignment) vs pair labels (unchanged, the reference the classifier is measured against) +- Per-class detector configurations (new, evolved) vs static analyzers (unchanged internally, exposed as tools) +- Evolve loop (new, HarnessX) vs existing bounded levers (rules, policy, mechanisms; kept, subordinated to the split and the class attribution) +- Class-aware scorer (replaces the text-token rule) vs catalog schema (one additive `unscorable` reason) + +## Out of Boundary + +- Taint semantics, walker, dominance internals (`guard-dominance-regime`, `overlay-ir-completeness`) +- Call graph and path records (`reachability-flow-model`), except as a future hunter tool +- Obligation checker internals (`authorization-obligations`); its corpus tasks 4.3 and 6.1 move here, its lever task 5 waits for the split + +## Upstream / Downstream + +- **Upstream**: `pair-corpus-honesty` (labels, tiers, splits, `known_limit`), `corpus-seeded-mechanisms` (exporter, LOO, lever), `tool_hunter` and `provider/openrouter`, `harness_ext` seam, HarnessX 0.1.0 (`meta_harness`, `rl`, `tracing.journal`), `tree-sitter-overlay-extra`. +- **Downstream**: every roadmap number; `harnessx-self-improving-rulesets` (its evolve step is this spec); `authorization-obligations` 4.3/5/6; scan reports (findings carry a class and the detector that produced them). + +## Existing Spec Touchpoints + +- **Extends**: `pair-corpus-honesty` Req 1 (scorer rule) and Req 9/10 (row classification); `corpus-seeded-mechanisms` Req 4/5 (split by construction); `three-stage-scan` MAP (per-class hunters as proposers); `harnessx-self-improving-rulesets` (the evolve loop lands here). +- **Folds in**: the `closed-loop-integrity` brief (its five repairs are approach items 1, 4 and 6). +- **Re-homes**: `authorization-obligations` 4.3 and the corpus half of 6.1; holds 5. + +## Constraints + +Core `dependencies = []`; HarnessX and grammars stay optional extras; every loop step degrades to a recorded reason without them. No row deleted; unscorable rows keep a reason. Labels stay maintainer-owned: the classifier proposes, disagreements queue for review, and a class change on a reviewed pair is a reviewed edit. Model-authored skills are retained only after held-out validation and are journaled with their attribution; nothing a round proposes reaches a scan without passing its class holdout and the cross-class regression sweep. Split enforced by construction, never by convention. Gates (`gate`, `map_gate`, `pair_gate`) stay byte-identical: the local slice is not touched by any of this. Redaction applies to every prompt and every journaled trajectory. Per-round cost cap; a round that exceeds it is recorded and reverted. + +## Open Questions for the maintainer + +1. Taxonomy spine: CWE families, the mechanism vocabulary, or a new closed list with both as attributes? +2. Round zero: clone the current hunter into every class (controlled baseline) or start each class from a hand-written seed prompt? +3. Cross-class regression tolerance: zero, or a per-class floor proportional to class size? +4. Does `authorization-obligations` task 5 (the lever) stay a spec task or become one class's evolve round? +5. Model for the detectors and for the meta-agent: same model, or a cheaper detector with a stronger meta-agent? + +## Evidence + +- `benchmarks/measurements/2026-09-06-vibe-py-holdout-hunter-sonnet5.json`: `ousast pairs --slice vibe-py --split holdout --hunter --hunter-model anthropic/claude-sonnet-5 --json` (2026-09-06). +- `benchmarks/measurements/2026-09-06-vibe-py-holdout-hunter-findings.json`: every hunter finding per side for the 17 holdout pairs with `in_labeled_function` and the scorer's verdict; the class-aware re-score uses a keyword classifier written for this measurement only and is the reason approach item 2 exists. diff --git a/benchmarks/measurements/2026-09-06-vibe-py-holdout-hunter-findings.json b/benchmarks/measurements/2026-09-06-vibe-py-holdout-hunter-findings.json new file mode 100644 index 0000000..059d211 --- /dev/null +++ b/benchmarks/measurements/2026-09-06-vibe-py-holdout-hunter-findings.json @@ -0,0 +1,841 @@ +[ + { + "pair": "damn-vulnerable-graphql-application-views-resolve-pastes", + "expected": [ + { + "cwe": "CWE-89", + "vulnerability_class": "sql_injection", + "path": "core/views.py", + "evidence": "SQL injection via unsanitized 'filter' GraphQL argument directly interpolated into raw SQL using Python string formatting: result.filter(text(\"title = '%s' or content = '%s'\" % (filter, filter))). An attacker can inject arbitrary SQL via th", + "rule_id": null, + "line": null, + "function": "resolve_pastes", + "sink": null, + "mechanism": "source_reaches_sink", + "obligation": null + } + ], + "sides": { + "vuln": [ + { + "path": "core/views.py", + "line": 20, + "title": "SQL Injection via unsanitized 'filter' parameter in resolve_pastes", + "rationale": "The GraphQL resolver builds a raw SQL WHERE clause using Python string interpolation with the user-supplied 'filter' argument: `text(\"title = '%s' or content = '%s'\" % (filter, filter))`. Since 'filter' is passed directly from the GraphQL query without escaping or parameterization, an attacker can i", + "in_labeled_function": true, + "matches": false + } + ], + "fixed": [ + { + "path": "core/views.py", + "line": 14, + "title": "Plaintext password comparison enables credential stuffing/timing attacks", + "rationale": "User.query.filter_by(username=username, password=password) compares raw password directly against a stored value with no hashing (e.g., bcrypt) and no constant-time comparison, suggesting passwords are stored in plaintext in the database. This exposes credentials to trivial theft on any DB read acce", + "in_labeled_function": false, + "matches": false + }, + { + "path": "core/views.py", + "line": 14, + "title": "No account lockout / rate limiting on login mutation", + "rationale": "The mutate() resolver performs an unrestricted DB lookup with username+password on every call, with no rate limiting or lockout logic visible, making brute-force credential guessing against this GraphQL mutation straightforward given GraphQL's higher-density request surface.", + "in_labeled_function": false, + "matches": false + }, + { + "path": "core/views.py", + "line": 16, + "title": "Generic exception message but no differentiation prevents user enumeration only weakly", + "rationale": "While raising 'Authentication Failure' avoids leaking whether the username or password was wrong, the surrounding logic still allows unlimited attempts, and any variance in Audit.create_audit_entry timing/behavior before the check could still leak information about valid usernames.", + "in_labeled_function": false, + "matches": false + }, + { + "path": "core/views.py", + "line": 19, + "title": "Tokens minted from username only, without verifying additional session/secret binding", + "rationale": "create_access_token(username) and create_refresh_token(username) are generated using just the username string; if the underlying JWT implementation lacks proper signing/expiry configuration (not shown here), tokens could be forged or reused improperly, and there's no per-session revocation reference", + "in_labeled_function": false, + "matches": false + } + ] + } + }, + { + "pair": "djangoat-analytics-hits-by-ip", + "expected": [ + { + "cwe": "CWE-89", + "vulnerability_class": "sql_injection", + "path": "app/models/Analytics/analytics.py", + "evidence": "SQL injection via string formatting in raw query. Both ip and col parameters are interpolated directly into SQL: 'SELECT %s FROM %s WHERE ip_address=\\'%s\\' ORDER BY id DESC' % (col, table_name, ip). The ip parameter comes from request.GET['", + "rule_id": null, + "line": null, + "function": "hits_by_ip", + "sink": null, + "mechanism": "source_reaches_sink", + "obligation": null + } + ], + "sides": { + "vuln": [ + { + "path": "app/models/Analytics/analytics.py", + "line": 13, + "title": "SQL Injection via unparameterized string interpolation in hits_by_ip", + "rationale": "The 'ip' and 'col' parameters are directly interpolated into a raw SQL string using Python %-formatting instead of using parameterized queries (e.g., cursor.execute(cmd, [ip])). If 'ip' or 'col' originates from user-controllable input (such as an HTTP request parameter), an attacker could inject arb", + "in_labeled_function": true, + "matches": false + } + ], + "fixed": [ + { + "path": "app/models/Analytics/analytics.py", + "line": 13, + "title": "Orphaned/incomplete analytics function with no visible callers or surrounding query logic", + "rationale": "The file only contains a nested parse_field() helper that whitelists field names ('ip_address','referrer','user_agent') defaulting to '1' for anything else, evidently used to sanitize a column name for dynamic SQL (as suggested by the header comment referencing hits_by_ip and 'source_reaches_sink').", + "in_labeled_function": false, + "matches": false + } + ] + } + }, + { + "pair": "djangoat-views-user-view", + "expected": [ + { + "cwe": "CWE-89", + "vulnerability_class": "sql_injection", + "path": "app/views/users/views.py", + "evidence": "SQL injection via string formatting in raw query. user_id_form from POST data is interpolated directly into SQL: User.objects.raw('SELECT * FROM %s WHERE user_id=\\'%s\\' ORDER BY is_admin=\\'0\\'' % (table_name, user_id_form)). An attacker can", + "rule_id": null, + "line": null, + "function": "user_view", + "sink": null, + "mechanism": "source_reaches_sink", + "obligation": null + } + ], + "sides": { + "vuln": [ + { + "path": "app/views/users/views.py", + "line": 18, + "title": "Broad exception swallowing hides injected/malformed query errors", + "rationale": "The `try/except` around `users[0]` catches all exceptions silently and just returns a generic 'NOT FOUND' message. This makes it harder to detect exploitation attempts (e.g., malformed SQL injection payloads causing query errors) and could mask underlying security-relevant failures, aiding attackers", + "in_labeled_function": true, + "matches": false + }, + { + "path": "app/views/users/views.py", + "line": 21, + "title": "SQL Injection via raw() query using unsanitized user_id from POST data", + "rationale": "The function takes `user_id` directly from `request.POST['user_id']` (form['user_id']) and interpolates it unsanitized into a raw SQL string via Python string formatting (`% (table_name, user_id_form)`), then executes it with `User.objects.raw(...)`. An attacker controlling the POST body can inject ", + "in_labeled_function": true, + "matches": false + }, + { + "path": "app/views/users/views.py", + "line": 36, + "title": "Mass-assignment / unchecked mutable update dict onto arbitrary model instance", + "rationale": "The `update` dict is built from `User.validate_update_form(form, user, update)` and then applied directly via `User.objects.filter(pk=user.pk).update(**update)` or `user.__dict__.update(update)` + `user.save()`. If the validation function does not strictly whitelist fields (e.g., it could allow `is_", + "in_labeled_function": true, + "matches": false + } + ], + "fixed": [ + { + "path": "app/views/users/views.py", + "line": 13, + "title": "Whitelist-based field validator with no visible call site or sink usage", + "rationale": "The file only contains a nested `parse_field` helper that whitelists `ip_address`, `referrer`, `user_agent` and falls back to the literal `'1'` for anything else. This pattern is typical of sanitizing a column/field name before interpolating it into a raw SQL query (e.g., ORDER BY or WHERE clause bu", + "in_labeled_function": false, + "matches": false + } + ] + } + }, + { + "pair": "dsvpwa-dsvpwa-main", + "expected": [ + { + "cwe": "CWE-326", + "vulnerability_class": "security_misconfiguration", + "path": "dsvpwa.py", + "evidence": "Insecure SSL/TLS configuration in main(). Three dangerous SSL options are explicitly enabled: (1) SSLv3 re-enabled via ctx.options &= ~ssl.OP_NO_SSLv3 (vulnerable to POODLE), (2) TLS compression re-enabled via &= ~ssl.OP_NO_COMPRESSION (vul", + "rule_id": null, + "line": null, + "function": "main", + "sink": null, + "mechanism": "permissive_default", + "obligation": null + } + ], + "sides": { + "vuln": [ + { + "path": "dsvpwa.py", + "line": 22, + "title": "SSL enablement defaults from an environment variable with weak type handling", + "rationale": "The --ssl flag defaults to os.getenv('DSVPWA_SSL', 0) but is defined as store_true with a non-boolean default; combined with the insecure SSL context modifications above, any deployment relying on environment-based configuration is likely to end up with a downgraded/misconfigured TLS setup by defaul", + "in_labeled_function": true, + "matches": false + }, + { + "path": "dsvpwa.py", + "line": 33, + "title": "SSL context re-enables SSLv3 support", + "rationale": "The code explicitly clears ssl.OP_NO_SSLv3 from ctx.options, re-enabling the obsolete and insecure SSLv3 protocol which is vulnerable to POODLE and other downgrade attacks. This is a permissive default that weakens transport security for any client that can negotiate SSLv3.", + "in_labeled_function": true, + "matches": false + }, + { + "path": "dsvpwa.py", + "line": 34, + "title": "SSL context disables compression protection", + "rationale": "Clearing ssl.OP_NO_COMPRESSION re-enables TLS compression, which exposes the connection to CRIME-style compression side-channel attacks that can leak sensitive session data (e.g., cookies).", + "in_labeled_function": true, + "matches": false + }, + { + "path": "dsvpwa.py", + "line": 35, + "title": "Server cipher preference explicitly disabled", + "rationale": "Clearing ssl.OP_CIPHER_SERVER_PREFERENCE allows the client to dictate the negotiated cipher suite instead of the server, letting a malicious or misconfigured client force use of a weak cipher, undermining the server's ability to enforce strong cryptography.", + "in_labeled_function": true, + "matches": false + } + ], + "fixed": [ + { + "path": "dsvpwa.py", + "line": 17, + "title": "Unvalidated/unsanitized comment input accepted without length or content checks", + "rationale": "The 'comment' parameter is taken directly from request params with only a default empty-string fallback and no validation, sanitization, or length limiting before being persisted to the comments table, allowing arbitrary and potentially malicious content (e.g., script payloads, oversized data) to be", + "in_labeled_function": false, + "matches": false + }, + { + "path": "dsvpwa.py", + "line": 20, + "title": "Stored XSS via unescaped comment output in guestbook", + "rationale": "User-supplied 'comment' parameter is inserted into the database via a parameterized query (safe from SQLi) but later rendered directly into HTML via string formatting ('{}'.format(column)) without any HTML encoding. Any comment containing HTML/JS (e.g.,