docs(learning-harness): amend the taxonomy name and carry the flat-taxonomy caveat

The maintainer settled both items the group 1 review escalated.

The family that covers server-side request forgery is named untrusted
destination and also covers open redirect, because both are untrusted input
choosing where a request goes and the corpus carries 34 open-redirect rows that
would otherwise be unclassifiable. The requirement, the brief and the design
example now say so, and the approval note records the amendment.

The shipped taxonomy is flat, so no answer can earn hierarchical partial credit.
Rather than leave that as prose, two new acceptance criteria require every
agreement figure and every family figure to say it, and both report types now
carry the flag, computed from the taxonomy rather than asserted. A taxonomy that
later declares a parent flips it without any other change.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG

.kiro/specs/learning-harness/brief.md

@@ -96,7 +96,7 @@ The target is to catch the taxonomy families below **reliably** on real-world ap
 
 ## Decisions (maintainer, 2026-09-06)
 
-1. **Taxonomy spine.** Ten families keyed to danger and to a verifier we can run, with CWE and the mechanism vocabulary as attributes, never the routing key: injection into a query or command (SQL, command, code, template); file path from user input; deserialization of untrusted data; access control (IDOR, missing guard, identity from request body; the obligations checker is its verifier); output encoding (XSS, template to HTML); server-side request forgery; prototype pollution and unsafe object merge; permissive configuration and secrets (CORS, debug, cookie flags, weak crypto, hardcoded credentials); memory safety as one coarse family for the vfc slice only; unknown. Multi-label routing.
+1. **Taxonomy spine.** Ten families keyed to danger and to a verifier we can run, with CWE and the mechanism vocabulary as attributes, never the routing key: injection into a query or command (SQL, command, code, template); file path from user input; deserialization of untrusted data; access control (IDOR, missing guard, identity from request body; the obligations checker is its verifier); output encoding (XSS, template to HTML); untrusted destination (server-side request forgery and open redirect, which share the shape of untrusted input choosing where a request goes and account for 34 corpus rows); prototype pollution and unsafe object merge; permissive configuration and secrets (CORS, debug, cookie flags, weak crypto, hardcoded credentials); memory safety as one coarse family for the vfc slice only; unknown. Multi-label routing.
 2. **Regression budget.** Measured, not chosen: round zero runs the cloned hunter K = 5 times per family at temperature 0 over the vendored corpus; the per-family negative-flip rate between seeds is the budget. A round that exceeds it in any family is rejected regardless of its target gain.
 3. **Models.** Detectors and meta-agent on `deepseek-v4-flash` through DeepSeek's own OpenAI-compatible endpoint (`https://api.deepseek.com`); `deepseek-v4-pro` only as the second independent judge above suspicion and for classifier disagreements queued for review; embeddings stay on OpenRouter. The chat endpoint and the embedding endpoint are configured separately (today one base-URL variable serves both). The 1M context lets the detector read whole files instead of hotspot excerpts.
 4. **Round-zero verifiers.** Snippet templates with an oracle per family, on the existing per-language sandbox recipes: canary row in an in-memory database for injection, canary file for command execution and traversal, canary object for deserialization. Access control gets the obligations checker as static corroboration. Output encoding, SSRF, prototype pollution and configuration report at `suspicion` only until they have an oracle. A family without a verifier never reports above suspicion in a scan but is still scored class-aware on the corpus: the corpus is the verifier for learning, the sandbox is the verifier for reporting.

.kiro/specs/learning-harness/design.md

@@ -263,7 +263,7 @@ Decisions: the proposer never sees holdout inputs, outputs or scores; cost is me
 ```python
 @dataclass(frozen=True)
 class Family:
-    id: str                      # e.g. "injection", "path", "deserialization", "access_control", "output_encoding", "ssrf", "prototype", "config_secrets", "memory", "unknown"
+    id: str                      # e.g. "injection", "path", "deserialization", "access_control", "output_encoding", "untrusted_destination", "prototype", "config_secrets", "memory", "unknown"
     cwes: frozenset[str]
     mechanisms: frozenset[str]
     verifier: Literal["canary", "static", "none"]

.kiro/specs/learning-harness/requirements.md

@@ -18,7 +18,7 @@ OpenUltraSAST's detection numbers cannot be trusted and cannot be improved. The
 
 #### Acceptance Criteria
 
-1. The Learning Harness shall define a closed, versioned taxonomy of exactly these families: injection into a query or command; file path from user input; deserialization of untrusted data; access control; output encoding; server-side request forgery; prototype pollution and unsafe object merge; permissive configuration and secrets; memory safety; unknown.
+1. The Learning Harness shall define a closed, versioned taxonomy of exactly these families: injection into a query or command; file path from user input; deserialization of untrusted data; access control; output encoding; untrusted destination (server-side request forgery and open redirect); prototype pollution and unsafe object merge; permissive configuration and secrets; memory safety; unknown.
 2. The Learning Harness shall record for every family the CWE identifiers it covers, the mechanism ids it covers, and the verifier that can confirm a claim in that family or the statement that none exists.
 3. When the taxonomy changes, the Learning Harness shall bump its version and every published number shall name the taxonomy version it was scored under.
 4. The Learning Harness shall treat CWE identifiers and mechanism ids as attributes of a family and shall never use them as the key that routes work or scores a pair.
@@ -34,6 +34,7 @@ OpenUltraSAST's detection numbers cannot be trusted and cannot be improved. The
 3. When the classifier is evaluated, the Learning Harness shall report agreement with maintainer labels on reviewed pairs with partial credit for a parent–child relation and no credit for a lateral or fabricated family, a confusion matrix per family, and its accuracy against both a random router and an oracle router.
 4. When the classifier disagrees with a maintainer label, the Learning Harness shall queue the disagreement for review and shall not change the label.
 5. The Learning Harness shall classify every labeled row in the corpus, including rows whose current class is free text or `other`, and shall report the count classified per family and the count left `unknown`.
+6. While no family in the taxonomy declares a parent, the Learning Harness shall state beside every agreement figure that hierarchical partial credit could not apply.
 
 ### Requirement 3: A class-aware, pair-wise scorer
 
@@ -47,6 +48,7 @@ OpenUltraSAST's detection numbers cannot be trusted and cannot be improved. The
 4. The Learning Harness shall report a fixed-false-positive-rate view per family beside Youden.
 5. When a detector is evaluated, the Learning Harness shall run it at least three times per pair and shall report a change against the last blessed round only when a reliable-change test passes, together with the negative-flip count per family.
 6. The Learning Harness shall never count a text token, a CWE string or a sink name in a finding's text as evidence of detection.
+7. While no family in the taxonomy declares a parent, the Learning Harness shall state beside every published family figure that hierarchical partial credit could not apply.
 
 ### Requirement 4: Unscorable rows and corpus hygiene
 

.kiro/specs/learning-harness/spec.json

@@ -1,7 +1,7 @@
 {
   "feature_name": "learning-harness",
   "created_at": "2026-09-06T06:32:41Z",
-  "updated_at": "2026-09-06T07:00:52Z",
+  "updated_at": "2026-09-06T08:20:56Z",
   "language": "en",
   "phase": "implementation",
   "approvals": {
@@ -26,5 +26,5 @@
     "tree-sitter-overlay-extra",
     "harnessx-self-improving-rulesets"
   ],
-  "approval_note": "2026-09-06 requirements approved by the maintainer in session ('approve the requirements and run the design phase'); 2026-09-06 design approved by the maintainer in session ('approve the design and generate the tasks'); 2026-09-06 tasks approved by the maintainer in session ('approve the tasks and go on with the implementation')"
+  "approval_note": "2026-09-06 requirements approved by the maintainer in session ('approve the requirements and run the design phase'); 2026-09-06 design approved by the maintainer in session ('approve the design and generate the tasks'); 2026-09-06 tasks approved by the maintainer in session ('approve the tasks and go on with the implementation'); 2026-09-06 requirements amended by the maintainer in session ('amend the requirements to untrusted_destination and note the flat taxonomy'): Req 1.1 names untrusted destination instead of server-side request forgery, and Req 2.7 and 3.7 require every agreement and family figure to state that hierarchical partial credit could not apply while the taxonomy is flat"
 }

src/openultrasast/learning/classify.py

@@ -262,6 +262,7 @@ class ClassifierReport:
     per_family: dict[str, int] = field(default_factory=dict)
     unknown: int = 0
     queue: tuple[QueueEntry, ...] = ()
+    hierarchical_credit: bool = False  # False: the taxonomy is flat, so no answer could earn partial credit (Req 2.6)
 
     def to_dict(self) -> dict[str, object]:
         return {
@@ -275,6 +276,7 @@ class ClassifierReport:
             "per_tier": dict(sorted(self.per_tier.items())),
             "per_family": dict(sorted(self.per_family.items())),
             "unknown": self.unknown,
+            "hierarchical_credit": self.hierarchical_credit,
             "queue": [entry.to_dict() for entry in self.queue],
         }
 
@@ -334,6 +336,7 @@ def measure_classifier(
         per_family=per_family,
         unknown=unknown,
         queue=tuple(queue),
+        hierarchical_credit=taxonomy.hierarchical,
     )
 
 

src/openultrasast/learning/families.py

@@ -68,6 +68,11 @@ class FamilyTaxonomy:
     def family_of_mechanism(self, mechanism: str) -> Family | None:
         return next((family for family in self.families if mechanism in family.mechanisms), None)
 
+    @property
+    def hierarchical(self) -> bool:
+        """True when some family declares a parent. While this is False, partial credit can never be earned."""
+        return any(family.parent for family in self.families)
+
     def related(self, left: str, right: str) -> Relation:
         """How two family answers relate. Abstention is not partial credit: ``unknown`` against a real
         family is ``lateral``, never ``parent_child``."""

src/openultrasast/learning/scoring.py

@@ -181,6 +181,7 @@ class FamilyMetrics:
     positive_flips: int = 0
     p_value: float | None = None  # two-sided sign test over the discordant pairs
     reliable_change: bool = False
+    hierarchical_credit: bool = False  # False: the taxonomy is flat, so no answer could earn partial credit (Req 3.7)
 
     def to_dict(self) -> dict[str, object]:
         return {
@@ -199,6 +200,7 @@ class FamilyMetrics:
             "positive_flips": self.positive_flips,
             "p_value": self.p_value,
             "reliable_change": self.reliable_change,
+            "hierarchical_credit": self.hierarchical_credit,
         }
 
 
@@ -258,6 +260,7 @@ def aggregate(
             positive_flips=positive,
             p_value=p_value,
             reliable_change=bool(p_value is not None and p_value < alpha and positive != negative),
+            hierarchical_credit=taxonomy.hierarchical,
         )
     return out
 

tests/test_learning_aggregation.py

@@ -96,3 +96,21 @@ def test_the_sign_test_is_two_sided_and_symmetric() -> None:
     assert sign_test(0, 0) == 1.0
     assert sign_test(6, 0) == sign_test(0, 6) < 0.05
     assert sign_test(1, 0) > 0.05 and sign_test(3, 3) == 1.0
+
+
+def test_every_family_figure_says_when_hierarchical_credit_could_not_apply() -> None:
+    """Req 3.7: with a flat taxonomy no answer can earn partial credit, and the number must carry that."""
+    from openultrasast.learning.families import Family, FamilyTaxonomy
+    from openultrasast.learning.scoring import aggregate
+
+    flat = load_families()
+    metrics = aggregate([_score("a", "pair_correct")], taxonomy=flat)["access_control"]
+    assert metrics.hierarchical_credit is False and metrics.to_dict()["hierarchical_credit"] is False
+    nested = FamilyTaxonomy(
+        version="test",
+        families=(
+            Family(id="injection", description="d", cwes=frozenset(), mechanisms=frozenset(), verifier="none"),
+            Family(id="access_control", description="d", cwes=frozenset(), mechanisms=frozenset(), verifier="none", parent="injection"),
+        ),
+    )
+    assert aggregate([_score("a", "pair_correct")], taxonomy=nested)["access_control"].hierarchical_credit is True

tests/test_learning_classifier_report.py

@@ -99,3 +99,13 @@ def test_only_reviewed_and_seeded_rows_count_towards_agreement(tmp_path: Path) -
     report = measure_classifier(cases, taxonomy, client=None)
     assert report.labeled == 1 and report.classified == 2  # an unreviewed label is not a reference answer
     assert [entry.pair for entry in report.queue] == ["trusted"]
+
+
+def test_the_report_says_when_hierarchical_credit_could_not_apply(tmp_path: Path) -> None:
+    """Req 2.6: the shipped taxonomy is flat, so partial credit is inert and every figure must say so."""
+    from openultrasast.learning.classify import measure_classifier
+
+    taxonomy = load_families()
+    report = measure_classifier([_case(tmp_path, "a", cwe="CWE-89", family="injection")], taxonomy, client=None)
+    assert report.hierarchical_credit is False
+    assert report.to_dict()["hierarchical_credit"] is False