Repository navigation
Fix search query syntax broken on the way to Solr - #526
Merged
Merged
Conversation
luqum normalizes boost and fuzzy values with Decimal.normalize(), so "taser"^50 was re-serialized as "taser"^5E+1, which Solr reads as a boost of 5 plus a stray term E+1 in both the query and the highlighter. Format these values in plain notation after every parse whose tree is turned back into a query string. Fixes #418 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
NoteExtractor only dropped emptied operations and groups below the root, so a boolean query made only of search fields (user:1 AND +tag:"x") became an empty AndOperation rather than None. The highlight query then became "title: description:", which Solr rejects, failing the whole search with a 503 for pro users with highlighting on. Fixes #515 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A boosted search field such as user:1^2 left Boost(None) behind once NoteExtractor removed the field, so the note query searched for the literal term "None" (e.g. "foo OR None^2"), adding documents whose notes contain "none" and breaking note highlights. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
luqum builds boosts and fuzzy degrees with Decimal(), which raises InvalidOperation for input such as foo^1.2.3 or foo^. rather than a ParseError, so the search returned a 500 for any user. Treat it like any other unparseable query and escape it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The note query was inlined in single-quoted local params (v='...') with only double quotes escaped, so any apostrophe ended the value early. Solr's lenient parser swallowed the error and the note joins matched nothing, silently dropping note results for queries like "don't". Escaping the apostrophe is not enough, as the value passes through two layers of unescaping and a user's own backslashes are still mangled, so send each subquery as its own request param and reference it with v=$param instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The highlight query was built as "title:<q> description:<q>", so for a multi-term query only the first term was searched in title or description; the rest went to the default field, and notes matched by the main search came back without highlights. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
title=, description=, source=, text= and page_no_N= values were appended to the query unchecked. Solr tolerates something like title:(bar)), but for pro users the note query re-parses the whole query with luqum, which raised an uncaught ParseError and returned a 500. Escape any value luqum cannot parse before appending it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
luqum reads a proximity slop with int(), so "a b"~1.5 or "a b"~. raised ValueError rather than a ParseError and the search returned a 500 for any user. Treat it like the other unparseable queries and escape it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_validate_range let luqum errors escape, so a filter param such as page_count=1^, id=1^1.2.3 or created_at="1 2"~1.5 returned a 500 for any user. Fail validation instead, which drops the value like any other invalid filter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Text params are always field searches, which the note extractor drops, so they never contribute to the note query. They only broke it because _add_note_query and _highlight_notes re-parsed the combined query, and escaping each param missed cases such as title=foo\ and page_no_1=. Extract the note query once in _parse from the parsed q, before params are appended, and pass it to both, which also avoids parsing it twice. Text params go to Solr unescaped again, as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Contributor
I don't know if this query is achievable in the front-end? In the API it is this (with the paren):
|
Contributor
duckduckgrayduck
requested changes
Nov 4, 2026
duckduckgrayduck
left a comment
Contributor
There was a problem hiding this comment.
Left some edge cases we'll want to fix before merging to production
luqum cannot parse a fractional proximity slop, so "test pdf"~1.5 fell
back to the escaped query and searched for and highlighted "1.5" as a
term. Solr itself truncates the slop ("~1.5" is "~1", "~." is none), so
do the same before parsing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
In a boolean query the filter fields stay in q, and both q and the pro user hl.q were used for highlighting, so foo AND (user:1 OR tag:email) also highlighted "email" in page text. Build a separate highlight query with the filter fields removed (text fields such as title: are kept), use it as hl.q for every user, and turn highlighting off when only filters are left. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
For pro users the text query was inlined in parens ahead of the note
join subqueries, so an unbalanced paren from a text param such as
title=bar) closed the group early. Solr then reparsed the query
leniently and flattened the required clauses of the organization note
subquery into the top level, returning no results at all. Pass the
text query as text_q and reference it with {!edismax v=$text_q}, as
the note queries already are, so nothing a user types can change the
structure around it. The test now also runs as a user with an edit
project, where the organization note subquery is valid.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A pro user with no projects they can edit got projects_edit_access:() in the organization note subquery. That is invalid syntax, and Solr's lenient parsing silently dropped the whole subquery, so those users never matched organization access notes on documents they can edit. Only add the projects clause when there are projects. A new seeded organization note, written by another member on a document user 2 can edit, tests the case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
|
Document is organization, note is organization. Both parties are professional users:
Document is public, note is organization:
|
duckduckgrayduck
self-requested a review
November 6, 2026 21:32
duckduckgrayduck
approved these changes
Nov 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.




Fixes several places where a user's search query, or the note subqueries built from it, reached Solr as syntax it misreads or rejects, or crashed while being parsed.
Fixes #418
Fixes #515
Behaviour changes
"taser"^50stays^50instead of becoming^5E+1(a boost of 5 plus a stray search termE+1, which affected both results and highlighting). Same forfoo~10. (Broken tokenization of boost #418)user:100014 AND +data__tag:"piano") no longer 503s for pro users with highlighting on. The note-highlight query used to becometitle: description:; it is now skipped because there is no text to highlight. (Broken boolean operators on notes search #515)user:1^2 AND tag:a) no longer leaves the literal termNonein the note query.q(foo^1.2.3,foo^.) are escaped like any other unparseable query instead of returning a 500."test pdf"~1.5→~1,~.→ no slop), instead of returning a 500.foo AND (user:… OR tag:email)no longer highlights "email" in page text. A separate highlight query with filter fields removed (text fields such astitle:are kept) is now sent ashl.qfor every user, not just pro users. A query with nothing but filters has highlighting turned off.page_count=1^,id=1^1.2.3) are dropped as invalid instead of returning a 500.don't) now find notes for pro users. Before, the apostrophe ended the inline note subquery early, and Solr silently matched no notes. The note subqueries are now sent as separate request params (note_q_public,note_q_organization), referenced withv=$param, so no escaping is needed. These two params now appear in the staff/debug output.title:(...) description:(...), where before only the first term was limited to the note fields.q, beforetitle=/description=/source=/text=/page_no_N=params are appended. Those params are always field searches, so they never searched notes, but the old code re-parsed the combined query, and values liketitle=bar),title=foo\orpage_no_1=gave pro users a 500. Those params still go to Solr exactly as before.text_q, referenced with{!edismax v=$text_q}) instead of being inlined ahead of the note subqueries. An unbalanced paren in a text param (title=bar)) used to close the group early. Solr then reparsed the whole query leniently and made the organization-note subquery's clauses required for every result, so the search returned nothing.text_qnow appears in the staff/debug output, andtext_querythere shows the wrapper.projects_edit_access:(), which is invalid, and Solr silently dropped the whole subquery. This bug is also on master.Migrations
None
What to check
"death" AND "taser"^50withhl=true: debugtext_queryshows^50, and noE+1term appears in results or highlightsuser:<id> AND +data__tag:"<tag>"withhl=true, as a pro user: returns results, no 503hl=true, pro user) still returns note highlightsfoo AND (user:<id> OR tag:x): notes are still searched and highlighted forfoo<word> OR don't): the document is found through the note<word1> <word2>,hl=true):note_highlightscovers both wordsq=foo^1.2.3and?page_count=1^: no 500, for anonymous and pro users"test pdf"~1.5withhl=true: behaves like"test pdf"~1, and "1.5" is not highlightedfoo AND (user:<you> OR tag:<tag>)withhl=true: "foo" is highlighted in page text, the tag value is not (check as anonymous, free and pro users)foo AND title:barwithhl=truestill highlights both words?q=<word in a note>&title=bar): the document is found through the note, and documents matching the word in their own text are found too. No 500🤖 Generated with Claude Code