fix(learning-harness): a preprocessor directive is code, not a provenance header

Review of task 1.3 found the twin detector stripping every leading comment
line, which in C and C++ swallows include and define directives, and in any
language swallows an ordinary leading comment. Two files whose only difference
was a buffer size on a define hashed equal and were marked unscorable, which
removed a real pair from every denominator: the exact dishonesty this spec
exists to end, running backwards. Measured before the fix: twelve file sides
across six vendored pairs had directives inside the stripped prefix, including
the Juliet stack-overflow pair whose whole point is the buffer size.

The strip is now anchored to the grammar the harvester writes: a leading
comment line counts as header only when it carries one of the provenance keys
followed by a colon, and the scan stops at the first line that is code. Block
and hash comment forms both work. Three regression tests pin the boundary in
both directions, and a fourth asserts no vendored excerpt loses a directive.
The five identical twins, the ten duplicate groups and the nineteen straddling
repositories are unchanged, so the committed hygiene artifact still holds.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG

src/openultrasast/pairs.py

@@ -502,6 +502,40 @@ class SplitReport:
     undated: int = 0  # assigned rows with no fix date, ordered by name instead
 
 
+# The keys the harvester writes into an excerpt's provenance header (benchmarks/pairs/harvest.py).
+# Only a leading comment line carrying one of these keys is metadata; anything else is code. A C
+# preprocessor directive and an ordinary leading comment are code, and stripping them would let two
+# different files hash equal and vanish from every denominator.
+_PROVENANCE_KEYS = frozenset(
+    {"provenance", "repo", "commit", "parent", "commit_url", "cve", "license", "function", "relpath", "mechanism", "upstream_start"}
+)
+_COMMENT_MARKERS = ("#", "//", "*/", "/*", "*")
+
+
+def _is_header_line(line: str) -> bool:
+    body = line.strip()
+    if not body:
+        return True
+    for marker in _COMMENT_MARKERS:
+        if body.startswith(marker):
+            body = body[len(marker) :].strip()
+            break
+    else:
+        return False
+    if not body:
+        return True  # the framing line of a block comment
+    key, separator, _rest = body.partition(":")
+    return bool(separator) and key.strip().lower() in _PROVENANCE_KEYS
+
+
+def _header_lines(lines: Sequence[str]) -> int:
+    """How many leading lines are the provenance header. Stops at the first line that is code."""
+    index = 0
+    while index < len(lines) and _is_header_line(lines[index]):
+        index += 1
+    return index
+
+
 @lru_cache(maxsize=4096)
 def _body_digest(path: Path) -> str | None:
     """Hash of an excerpt with its provenance header and trailing whitespace removed; None when unreadable or empty."""
@@ -510,10 +544,7 @@ def _body_digest(path: Path) -> str | None:
     except OSError:
         return None
     lines = text.splitlines()
-    index = 0
-    while index < len(lines) and (not lines[index].strip() or lines[index].lstrip().startswith(("#", "//", "/*", "*"))):
-        index += 1
-    body = "\n".join(line.rstrip() for line in lines[index:]).strip()
+    body = "\n".join(line.rstrip() for line in lines[_header_lines(lines) :]).strip()
     return hashlib.sha256(body.encode()).hexdigest() if body else None