Skip to content

A space-padded comparison table is code the classifier still calls prose #35

Description

@tamnd

One entry left in tier 1 after the code rules landed. tutorial/datastructures.po:

(1, 2, 3)              < (1, 2, 4)
[1, 2, 3]              < [1, 2, 4]
'ABC' < 'C' < 'Pascal' < 'Python'
(1, 2, 3, 4)           < (1, 2, 4)
(1, 2)                 < (1, 2, -1)
(1, 2, 3)             == (1.0, 2.0, 3.0)
(1, 2, ('aa', 'ab'))   < (1, 2, ('abc', 'a'), 4)

It is a table of tuple comparisons, laid out with runs of spaces so the operators line up. Nothing in it is translatable and there is nothing a model can usefully do with it.

Every rule in classify.py misses it, and each for a reason worth keeping:

  • No line is indented, so the block rule does not see it.
  • The lines are not calls or assignments or imports, so no code shape matches.
  • The tight rule requires that every space follows a comma, which is what stops it swallowing popen() (in module os). Here the alignment padding is exactly what breaks that.
  • It has letters in it, ABC and Pascal and Python, so it is not a no-op either.

The shape that would catch it is a line made of bracketed literals joined by comparison operators, or more generally an entry whose lines are all bare expressions with no natural-language words in them. Both need measuring against the whole corpus before either goes in, the same way the code rules were, because this is one entry and a rule that gets it wrong costs paragraphs.

It has cost four model calls over two runs. Low priority next to that, and worth doing when the next tier turns up more of the same shape rather than on the strength of one entry.

Metadata

Metadata

Assignees

No one assigned

    Labels

    segmentationMarkup protection, classification, batching

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions