Review of task 1.3 found the twin detector stripping every leading comment
line, which in C and C++ swallows include and define directives, and in any
language swallows an ordinary leading comment. Two files whose only difference
was a buffer size on a define hashed equal and were marked unscorable, which
removed a real pair from every denominator: the exact dishonesty this spec
exists to end, running backwards. Measured before the fix: twelve file sides
across six vendored pairs had directives inside the stripped prefix, including
the Juliet stack-overflow pair whose whole point is the buffer size.
The strip is now anchored to the grammar the harvester writes: a leading
comment line counts as header only when it carries one of the provenance keys
followed by a colon, and the scan stops at the first line that is code. Block
and hash comment forms both work. Three regression tests pin the boundary in
both directions, and a fourth asserts no vendored excerpt loses a directive.
The five identical twins, the ten duplicate groups and the nineteen straddling
repositories are unchanged, so the committed hygiene artifact still holds.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 1.3. An expected row may carry a family label, stored opaquely because the
classifier owns the taxonomy and the corpus module never imports the learning
package. Every pair carries a computed reason when it cannot be scored: a
declared known limit wins, otherwise identical twin when the two excerpts are
the same code once the provenance header is stripped. No row is ever dropped.
Two corpus defects the new accounting surfaces, committed as an artifact beside
the numbers they cap. Nineteen repositories sit on both sides of the split, so a
learning path could be taught by one file of a repository and evaluated on
another. Ten groups of pairs share a vulnerable excerpt byte for byte, mostly
the same fix harvested twice at different granularity.
The split helper keeps every declared split, lets rows that named none inherit
their repository's side, and otherwise assigns a whole repository at once,
oldest first, so nothing straddles by our doing. Pair signals can now be
restricted to one split, which is what lets a learning path teach from train
rows only.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
