Skip to content

Fix search query syntax broken on the way to Solr - #526

Merged
mitchelljkotler merged 14 commits into
masterfrom
fix-solr-query-serialization
Nov 9, 2026
Merged

mitchelljkotler merged 14 commits into
masterfrom
fix-solr-query-serialization

Conversation

@mitchelljkotler

@mitchelljkotler mitchelljkotler commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

Fixes several places where a user's search query, or the note subqueries built from it, reached Solr as syntax it misreads or rejects, or crashed while being parsed.

Fixes #418
Fixes #515

Behaviour changes

  • Boosts and fuzzy degrees ending in zero are sent as typed: "taser"^50 stays ^50 instead of becoming ^5E+1 (a boost of 5 plus a stray search term E+1, which affected both results and highlighting). Same for foo~10. (Broken tokenization of boost #418)
  • A boolean query made only of search fields (e.g. user:100014 AND +data__tag:"piano") no longer 503s for pro users with highlighting on. The note-highlight query used to become title: description:; it is now skipped because there is no text to highlight. (Broken boolean operators on notes search #515)
  • A boosted search field (user:1^2 AND tag:a) no longer leaves the literal term None in the note query.
  • Malformed boost or fuzzy numbers in q (foo^1.2.3, foo^.) are escaped like any other unparseable query instead of returning a 500.
  • A fractional phrase slop is truncated the way Solr itself does ("test pdf"~1.5 → ~1, ~. → no slop), instead of returning a 500.
  • Highlighting uses only the text of the query, not its filter values: foo AND (user:… OR tag:email) no longer highlights "email" in page text. A separate highlight query with filter fields removed (text fields such as title: are kept) is now sent as hl.q for every user, not just pro users. A query with nothing but filters has highlighting turned off.
  • Unparseable numeric or date filter params (page_count=1^, id=1^1.2.3) are dropped as invalid instead of returning a 500.
  • Queries containing an apostrophe (don't) now find notes for pro users. Before, the apostrophe ended the inline note subquery early, and Solr silently matched no notes. The note subqueries are now sent as separate request params (note_q_public, note_q_organization), referenced with v=$param, so no escaping is needed. These two params now appear in the staff/debug output.
  • Note highlights work for multi-term queries. The highlight query is now title:(...) description:(...), where before only the first term was limited to the note fields.
  • The note query is built once, from q, before title=/description=/source=/text=/page_no_N= params are appended. Those params are always field searches, so they never searched notes, but the old code re-parsed the combined query, and values like title=bar), title=foo\ or page_no_1= gave pro users a 500. Those params still go to Solr exactly as before.
  • For pro users, the text query is sent to Solr as its own param (text_q, referenced with {!edismax v=$text_q}) instead of being inlined ahead of the note subqueries. An unbalanced paren in a text param (title=bar)) used to close the group early. Solr then reparsed the whole query leniently and made the organization-note subquery's clauses required for every result, so the search returned nothing. text_q now appears in the staff/debug output, and text_query there shows the wrapper.
  • Pro users with no projects they can edit now match organization-access notes. Their organization-note subquery contained projects_edit_access:(), which is invalid, and Solr silently dropped the whole subquery. This bug is also on master.

Migrations
None

What to check

  • "death" AND "taser"^50 with hl=true: debug text_query shows ^50, and no E+1 term appears in results or highlights
  • user:<id> AND +data__tag:"<tag>" with hl=true, as a pro user: returns results, no 503
  • A normal text query with notes (hl=true, pro user) still returns note highlights
  • Mixed query such as foo AND (user:<id> OR tag:x): notes are still searched and highlighted for foo
  • As a pro user, search for a word in one of your notes plus a word with an apostrophe (<word> OR don't): the document is found through the note
  • Organization-access notes are still only searchable by people who can edit the document (the subquery is now passed differently)
  • Two-word query matching a note (<word1> <word2>, hl=true): note_highlights covers both words
  • q=foo^1.2.3 and ?page_count=1^: no 500, for anonymous and pro users
  • "test pdf"~1.5 with hl=true: behaves like "test pdf"~1, and "1.5" is not highlighted
  • foo AND (user:<you> OR tag:<tag>) with hl=true: "foo" is highlighted in page text, the tag value is not (check as anonymous, free and pro users)
  • A fielded text search such as foo AND title:bar with hl=true still highlights both words
  • As a pro user (logged in, not an anonymous API request), ?q=<word in a note>&title=bar): the document is found through the note, and documents matching the word in their own text are found too. No 500
  • As a pro user who has no projects with edit access, search for a word in an organization-access note written by someone else in your organization: the document is found

🤖 Generated with Claude Code

mitchelljkotler and others added 7 commits October 30, 2026 15:01
luqum normalizes boost and fuzzy values with Decimal.normalize(), so
"taser"^50 was re-serialized as "taser"^5E+1, which Solr reads as a
boost of 5 plus a stray term E+1 in both the query and the highlighter.
Format these values in plain notation after every parse whose tree is
turned back into a query string.

Fixes #418

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
NoteExtractor only dropped emptied operations and groups below the root,
so a boolean query made only of search fields (user:1 AND +tag:"x")
became an empty AndOperation rather than None. The highlight query then
became "title: description:", which Solr rejects, failing the whole
search with a 503 for pro users with highlighting on.

Fixes #515

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A boosted search field such as user:1^2 left Boost(None) behind once
NoteExtractor removed the field, so the note query searched for the
literal term "None" (e.g. "foo OR None^2"), adding documents whose
notes contain "none" and breaking note highlights.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
luqum builds boosts and fuzzy degrees with Decimal(), which raises
InvalidOperation for input such as foo^1.2.3 or foo^. rather than a
ParseError, so the search returned a 500 for any user. Treat it like
any other unparseable query and escape it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The note query was inlined in single-quoted local params (v='...') with
only double quotes escaped, so any apostrophe ended the value early.
Solr's lenient parser swallowed the error and the note joins matched
nothing, silently dropping note results for queries like "don't".
Escaping the apostrophe is not enough, as the value passes through two
layers of unescaping and a user's own backslashes are still mangled, so
send each subquery as its own request param and reference it with
v=$param instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The highlight query was built as "title:<q> description:<q>", so for a
multi-term query only the first term was searched in title or
description; the rest went to the default field, and notes matched by
the main search came back without highlights.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
title=, description=, source=, text= and page_no_N= values were
appended to the query unchecked. Solr tolerates something like
title:(bar)), but for pro users the note query re-parses the whole
query with luqum, which raised an uncaught ParseError and returned a
500. Escape any value luqum cannot parse before appending it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mitchelljkotler mitchelljkotler changed the title Fix query syntax broken on the way to Solr (#418, #515) Fix search query syntax broken on the way to Solr Oct 30, 2026
mitchelljkotler and others added 3 commits October 30, 2026 16:41
luqum reads a proximity slop with int(), so "a b"~1.5 or "a b"~. raised
ValueError rather than a ParseError and the search returned a 500 for
any user. Treat it like the other unparseable queries and escape it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_validate_range let luqum errors escape, so a filter param such as
page_count=1^, id=1^1.2.3 or created_at="1 2"~1.5 returned a 500 for
any user. Fail validation instead, which drops the value like any
other invalid filter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Text params are always field searches, which the note extractor drops,
so they never contribute to the note query. They only broke it because
_add_note_query and _highlight_notes re-parsed the combined query, and
escaping each param missed cases such as title=foo\ and page_no_1=.
Extract the note query once in _parse from the parsed q, before params
are appended, and pass it to both, which also avoids parsing it twice.
Text params go to Solr unescaped again, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@duckduckgrayduck

Copy link
Copy Markdown
Contributor

We probably need to strip the fielded from hl.q. This is an example query with unexpected behavior due to the lack of this:

image It correctly highlights foo in the notes, but also highlights email, which is a tag on the document and what we are looking to filter on, not highlight on.

@duckduckgrayduck

duckduckgrayduck commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

As a pro user, ?q=&title=bar): the document is found through the note, no 500

I don't know if this query is achievable in the front-end? In the API it is this (with the paren):
https://api.staging.documentcloud.org/api/documents/search/?q=test&title=bar%29&version=2.0
and although it doesn't 500, it doesn't return the doc with a note with the word "test" in it. So that testing step fails

image image

@duckduckgrayduck

Copy link
Copy Markdown
Contributor

Similar for the highlighting, it seems to allow for ~1.5 to escape like so:
image

@duckduckgrayduck duckduckgrayduck left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some edge cases we'll want to fix before merging to production

mitchelljkotler and others added 4 commits November 5, 2026 09:55
luqum cannot parse a fractional proximity slop, so "test pdf"~1.5 fell
back to the escaped query and searched for and highlighted "1.5" as a
term. Solr itself truncates the slop ("~1.5" is "~1", "~." is none), so
do the same before parsing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
In a boolean query the filter fields stay in q, and both q and the pro
user hl.q were used for highlighting, so foo AND (user:1 OR tag:email)
also highlighted "email" in page text. Build a separate highlight query
with the filter fields removed (text fields such as title: are kept),
use it as hl.q for every user, and turn highlighting off when only
filters are left.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
For pro users the text query was inlined in parens ahead of the note
join subqueries, so an unbalanced paren from a text param such as
title=bar) closed the group early. Solr then reparsed the query
leniently and flattened the required clauses of the organization note
subquery into the top level, returning no results at all. Pass the
text query as text_q and reference it with {!edismax v=$text_q}, as
the note queries already are, so nothing a user types can change the
structure around it. The test now also runs as a user with an edit
project, where the organization note subquery is valid.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A pro user with no projects they can edit got projects_edit_access:()
in the organization note subquery. That is invalid syntax, and Solr's
lenient parsing silently dropped the whole subquery, so those users
never matched organization access notes on documents they can edit.
Only add the projects clause when there are projects. A new seeded
organization note, written by another member on a document user 2 can
edit, tests the case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@duckduckgrayduck

duckduckgrayduck commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Document is organization, note is organization. Both parties are professional users:

  • A non-org viewer can’t see the org note.
  • A non-org editor can see it.
  • An org member who is not a viewer/editor/admin of a shared project the document is in can see the org level note.
  • A professional user who is otherwise able to view the document(public), but is not in the org nor is added as a project collaborator does not see the note in results

Document is public, note is organization:

  • A non-org viewer can’t see the org note.
  • A non-org editor CAN'T see it. They also can't edit the title, so I think this is correct behavior to prevent defacing of public documents?
  • An org member who is not a viewer/editor/admin of the project the document is in can see the org level note
  • A professional user who is otherwise able to view the document(public), but is not in the org nor is added as a project collaborator does not see the note in results

@duckduckgrayduck
duckduckgrayduck self-requested a review November 6, 2026 21:32
@mitchelljkotler
mitchelljkotler merged commit e16dd36 into master Nov 9, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Broken boolean operators on notes search Broken tokenization of boost

2 participants