Fix comment marker swallowed by preceding operator run

[mayuriphad]

Bug

Fixes #722.

sqlparse's operator regex ([+/@#%^&|^-]+ in sqlparse/keywords.py) is
greedy and includes - in its character class. When a -- (or # )
single-line comment marker immediately follows an operator with no
intervening whitespace, e.g. ||--comment, the operator regex consumes the
whole ||-- run as a single Operator token before the comment regex ever
gets a chance to match. The comment is then invisible to the tokenizer.

Repro:

import sqlparse
sql = "myval := oneval || otherval ||--;\ncontinuedvals;\n"
print(sqlparse.format(sql, strip_comments=True))
# strip_comments leaves the "--;" comment in place instead of stripping it

This can also make sqlparse.split() misjudge statement boundaries: a ;
that lives inside an un-recognized "comment" gets treated as a real
statement terminator only in other contexts, but more directly, comment
content that should have been dropped is not.

Fix

Add a negative lookahead to the operator regex so it stops consuming
characters right before a -- or # comment marker, letting the earlier
comment regex handle it as intended.

Test plan

Added regression tests:

• tests/test_format.py::TestFormat::test_strip_comments_single_no_space_before_operator
• tests/test_split.py::test_split_comment_directly_after_operator

Both fail on master and pass with the fix. Ran the full test suite
locally:

python -m pytest tests/
508 passed, 2 xfailed, 1 xpassed

Also ran ruff check sqlparse/keywords.py — clean (pre-existing lint
findings elsewhere in the repo are unrelated to this change).

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.