OPENNLP-1877: Static embedding engine for modern distilled embedding tables (opennlp-embeddings) - #1152
Draft
krickert wants to merge 162 commits into
Draft
OPENNLP-1877: Static embedding engine for modern distilled embedding tables (opennlp-embeddings)#1152krickert wants to merge 162 commits into
krickert wants to merge 162 commits into
Conversation
mawiesne
marked this pull request as draft
July 8, 2026 04:32
krickert
force-pushed
the
static-embeddings
branch
2 times, most recently
from
July 8, 2026 14:44
2085efd to
7f6f67f
Compare
krickert
marked this pull request as ready for review
July 8, 2026 15:00
krickert
force-pushed
the
static-embeddings
branch
2 times, most recently
from
July 8, 2026 19:05
e60bdba to
7f6f67f
Compare
Contributor
|
This probably requires an update to the docs? Probably doesn't need to be extensive - just a description of when to use it and how. |
jzonthemtn
reviewed
Jul 9, 2026
Contributor
Author
I'll do that now |
krickert
force-pushed
the
static-embeddings
branch
from
July 9, 2026 20:02
7f6f67f to
e60bdba
Compare
Contributor
Author
|
Done in 76d58c5: a Static Embeddings chapter in the Dev Manual (when to use a static table over a contextual model, both load calls, thread safety, the no-bundled-model license note) plus a module README with the same ground covered for people landing in the source tree. |
Contributor
|
Hi @krickert - as mentioned on Slack I currently dont have the time for a manual review but I just let Fable do a comprehensive review on this PR. Here is the result: Blocking / should be addressed before merge:
Minor:
Human review will follow. |
Contributor
Author
|
Thanks. Addressed in 9dd6ff7, one pushback on the marker question: Blocking:
Minor:
|
krickert
marked this pull request as draft
July 11, 2026 13:22
Contributor
Author
|
Moved this to draft - interfacing to match Onnx impl. Onnx's impl is better for accuracy, this one is for speed. Update incoming. |
krickert
force-pushed
the
static-embeddings
branch
from
July 13, 2026 00:51
8230a2c to
6c216b7
Compare
krickert
force-pushed
the
static-embeddings
branch
from
July 13, 2026 08:18
4f7a84e to
6184021
Compare
Contributor
Author
|
Before this leaves draft: re-verify the two embed() throughput rows in the README performance table against the original JMH runs. The surrounding prose was reworked, but the numbers were deliberately left untouched pending that check. |
…rivate readFloat32(String) and metadata() have no main-source callers; only the module's own tests use them. Shrink the experimental public surface.
…e manual Add a Command Line Tools section to the embeddings chapter covering the bin/embeddings launcher and both tools: purpose, invocation shape, and the load-based verification each run ends with, in the style of the manual's other tool sections.
The distiller writes the pooling field but the loader never read it, so a third-party model declaring another pooling silently mean-pooled. Only mean pooling is implemented, so the loader now rejects any other declared value with an InvalidFormatException naming it; a declared mean still loads.
mostSimilar and analogy sized the candidate arrays by the raw topK before any clamping, so mostSimilar(word, Integer.MAX_VALUE) failed with an OutOfMemoryError. The scan can never yield more than one neighbor per row, so the capacity is now the smaller of topK and the row count.
The distiller path zeroes non-finite teacher values, but the loader accepted any bytes, and a single NaN row defeats the zero-norm guard and the TopK comparison, silently corrupting similarity rankings. Loading a matrix that holds a NaN or infinity now throws InvalidFormatException naming the row.
do_lower_case=false was never tested; the new case asserts that a cased model matches lower-case vocabulary entries as-is and folds upper-case text to the skipped unknown token instead of lower-casing it first.
The chapter's second programlisting (the explicit load and loadSentencePiece overloads) had no test pinning it. testExplicitOverloads loads both fixture layouts exactly as the listing shows, backed by a new minimal SentencePiece directory fixture, and the chapter now cites the test the way it already cites the directory-load listing.
…test The headline use case, ranking documents against a query by cosine similarity, existed only in the module README. The chapter gains a Semantic Search section whose listing scores each document with the public similarity method and sorts by descending score, and the new StaticEmbeddingSearchExampleTest asserts the exact ranking on the analogy fixture's known geometry.
The hub cache compiled two Patterns for the org/model@revision reference and the hex shape of commit shas and digests. Hand scans over the same ASCII grammars replace them; the module's parsing is now regex-free throughout.
…d pieces A distillation can now take a term list, a learned corpus vocabulary of whole words and multi-word phrases. Each term is segmented by the teacher's own tokenizer, encoded through the teacher as one sequence, and appended to the table after the subword rows, recorded as terms.txt in the model directory. The same PCA and Zipf pipeline spans all rows, and a term equal to a surviving vocabulary token is dropped as a duplicate row. At embed time the model matches text against its terms greedily longest-first over case-folded word runs (StringUtil.toLowerCase, one code point to one code point) and pools a matched term's single row in place of its words' subword pieces; text between matches tokenizes as before, and a model without a terms file embeds exactly as it did. Terms are neighbor candidates in mostSimilar like any token. The DistillModel tool gains -terms, documented in the manual. TeacherTokenizer's two Patterns (the unused-token filter and the template splitter) are replaced with cursor scans along the way.
Red evidence: Model2VecUnigramTokenizerTest failed because missing model.vocab leaked a NullPointerException, and StaticEmbeddingModelSentencePieceTest failed because an incomplete legacy tokenizer did not explain the supported alternatives. The loader now reconstructs the published Unigram tokenizer in memory from tokenizer.json, including its precompiled normalizer and supported Model2Vec post-normalization steps. The pinned potion-multilingual-128M artifact loads without a borrowed teacher model and matches the reference Python vector.
* OPENNLP-1897: Add a term vector layer rolling tokens up to (term, frequency, offsets) A new opennlp.tools.termvector package aggregates the document token layer into a document-scoped layer of TermVector records for index consumers, without touching the opennlp.tools.document container. TermVector carries the term string, the occurrence count, and the occurrence spans in original text coordinates. It comes in two shapes: full (one span per occurrence) and scoring-only (counts only, no offset storage). TermVectorAnnotator implements DocumentAnnotator: it requires Layers.TOKENS and provides its own opennlp:term-vectors key. Term identity is delegated, never analyzed: without a normalizer the token's covered text groups as-is; with an OffsetAwareNormalizer the document text is normalized once and each token span is mapped through the alignment, so tokens differing only by a normalization fold (case, eszett expansion, collapsed whitespace) group together while every emitted occurrence span still points into the original text.
Add a general subword tokenizer contract with original-text UTF-16 offsets, plus a dependency-free WordPiece implementation and BERT compatibility layer. Document the API and cover vocabulary validation, reference sequences, Unicode, and offset behavior. Red evidence: - A supplementary-plane word at 100 code points was rejected because UTF-16 code units were counted. - Negative piece ids and empty vocabulary pieces were accepted.
krickert
added a commit
to ai-pipestream/opennlp
that referenced
this pull request
Sep 4, 2026
Red evidence: after merging the current apache#1152 tip, opennlp-embeddings failed compilation because StaticEmbeddingModel still called the removed raw-array constructor and rowNorms helper.
…e#1190) * OPENNLP-1893: Hunspell-format affix engine over user-supplied dictionaries A clean-room reader for .dic and .aff files with PFX/SFX rules, strip strings, character-class conditions matched by a single scan, cross products, and char, long, and num flag modes. No dictionary data is bundled: users point at their own files, so dictionary licenses never attach to the jar. Unsupported affix features fail closed, missing analyses rather than inventing them. (cherry picked from commit 0ecc39c) * OPENNLP-1893: Twofold suffix analysis through Hunspell continuation classes Suffix rules now carry the continuation flags declared on their affix text, and analysis undoes a stacked pair when the inner rule's classes allow the outer one, so derived-then-inflected forms reduce to their dictionary word. (cherry picked from commit b543dea) * OPENNLP-1893: Usage, threading, and malformed-input tests for the Hunspell engine, precise javadoc * OPENNLP-1893: Document Hunspell dictionary acquisition with a license-preserving download helper * OPENNLP-1893: Cut morphology like hunspell does, accept UTF-8 flags and strip-only rules, and read the parser through the whitespace seam * OPENNLP-1893: Read flags as code points, tolerate trailing morphology, and stem nothing from nothing Loading the Spanish dictionary of the LibreOffice collection, the same collection this module's README recommends, exposed three gaps against real data. Flags under FLAG UTF-8 are now one code point each instead of one UTF-16 unit, since that dictionary names prefix rules with supplementary characters that would otherwise split into two flags and abort the load; a variation selector after a flag character selects presentation, not identity, and is dropped, which the same file also relies on. A numeric or long flag run ends at the first space or tabulator, the separators the word-list format defines, so trailing morphological text without a tag no longer aborts the load; the morphology cut itself now splits on exactly those two separators, the set the reference implementation's hashmgr.cxx uses, which the javadoc previously claimed while scanning wider whitespace. Stemming the empty word answers the empty word instead of letting a strip-only rule conjure a stem from nothing. All four downloaded dictionaries of the collection, English, Spanish, Hungarian, and German, now load and stem; new tests pin the escaped slash, the multi-word entry with trailing tags, and each corrected behavior. * OPENNLP-1893: Resolve numeric dictionary flags through the AF alias table The published Hungarian dictionary flags all of its entries as numeric references into an AF alias table, so without alias support every entry loaded flagless and stemming answered the surface form unchanged. The affix parser now reads the AF table, the first line as the declared count and every further line as one flag run with trailing comments discarded, and a purely numeric flag field in the word list resolves as a 1-based reference into it, failing loud with the line and table size when the reference is out of range. Without an AF table numeric fields keep their FLAG num meaning. The Hungarian dictionary of the LibreOffice collection now stems inflected forms; remaining gaps there are compound territory, which is tracked separately. * OPENNLP-1893: Walk only the affix rules that can apply, bucketed by their boundary character Undoing a suffix requires the word to end with the rule's affix material, so only rules whose material ends in the word's last character can ever apply, and likewise for prefixes and the first character. The dictionary now buckets its rules by that boundary character at load, and every scan in the stem path, including the twofold and cross-product inner scans, walks the one bucket plus the strip-only rules instead of the whole inventory. Measured on the LibreOffice dictionaries at 4,000 words each: English 553k to 1,024k words per second, Spanish 9.6k to 28.9k, German 132k to 287k. * OPENNLP-1893: Decompose unanalyzed words into two flagged compound parts When the affix analysis finds nothing and the affix file declares compounding, a word now splits into two listed parts that the COMPOUNDFLAG or the positional COMPOUNDBEGIN and COMPOUNDEND flags allow in their positions, honoring COMPOUNDMIN, with the parts reported left to right. Affix analyses keep precedence, listed words never decompose, and unflagged parts block a split. Against the published Hungarian dictionary the unlisted kutyahaz decomposes into its two nouns while listed compounds and inflected forms keep their regular analyses. Longer chains, syllable rules, and the compound-only flags stay unimplemented and simply leave such words unanalyzed. * OPENNLP-1893: Honor the blocking flags, circumfixes, and compound positioning A NEEDAFFIX (or PSEUDOROOT) entry is a virtual stem that exists only to be affixed, an ONLYINCOMPOUND entry appears only inside compounds, and a FORBIDDENWORD entry is listed to be blocked; none of them is a standalone analysis anymore, per homonym flag set, and an affix carrying NEEDAFFIX among its continuation classes yields no single-removal analysis while its twofold and cross-product removals stand, the other affix being exactly the further one required. A cross-product now also requires both removed affixes' flags in the same homonym's flag set, and CIRCUMFIX binds marked prefix and suffix halves to one another, so neither half analyzes alone and a marked half never combines with an unmarked affix. Decomposition grows from two verbatim parts to the compound machinery the published German dictionary actually uses: any number of parts under the positional COMPOUNDBEGIN/COMPOUNDMIDDLE/COMPOUNDEND flags and COMPOUNDWORDMAX, parts standing on an entry plus one affix with COMPOUNDPERMITFLAG required at internal boundaries and COMPOUNDFORBIDFLAG barring marked forms, zero and dash linking suffixes included, an uppercased retry for capitalized entries spelled lowercase inside a compound, and the CHECKCOMPOUNDDUP, CHECKCOMPOUNDCASE, and CHECKCOMPOUNDTRIPLE junction guards, case judged against the original surface. A listed forbidden word never decomposes, and a fixed part-licensing budget keeps adversarial input bounded, missing analyses rather than stalling. Abbildungsverzeichnis, Haustuer, and Kinderzimmer now decompose against de_DE_frami at 137k words/s single-threaded. An opt-in test class checks everyday morphology against downloaded dictionaries under -Dopennlp.hunspell.dict.dir; nothing is bundled. * OPENNLP-1893: Add hunspell manual coverage with mirror-tested examples Extend docbkx/stemmer.xml with the hunspell affix-stemmer section, wire the chapter into the manual, and add StemmerFactoryUsageExampleTest and HunspellManualExampleTest asserting the load-and-stem values the chapter prints. Point the dictionary README at the new manual example. * OPENNLP-1893: Apply the review-convention pass: factual license prose, split null contracts, thread-safety annotations * OPENNLP-1893: Add {@inheritdoc} to the stemmer overrides and trim empty javadoc lines * OPENNLP-1893: Address review: fold the affix twins, extract tags, complete javadoc - Fold the two per-kind bucketing loops in the HunspellDictionary constructor into a single bucketByBoundary helper that takes the rule list, the kind, and the sink for the rules with empty affix material. - Fold collectSuffixedPartStem and collectPrefixedPartStem, which differed only in the boundary they face, into one collectAffixedPartStem with a suffix marker and an atEdge marker; document what atEdge means at each end. - Extract a parseValue helper for the single-integer directives so COMPOUNDMIN and COMPOUNDWORDMAX no longer share one case body that re-tests which directive it is. - Extract PREFIX_TAG, SUFFIX_TAG, and NO_MATERIAL constants and use them at the affix block header, the rule lines, and the strip and affix material checks. - Give FORBIDDENWORD its own case in the flag directive switch instead of letting the catch-all default assign it, and make that default throw for a directive listed on the outer switch but not handled on the inner one. - Add the missing javadoc on the AffixCondition and HunspellDictionary constructors, the Affix record components, and the splitLines, splitOn, and split helpers. - Convert the single-line accessor javadoc on the compounding and affix bucket getters to the {@return ...} form, and replace the hand-written prose on HunspellStemmerFactory.newStemmer with {@inheritdoc} plus the instancing note. - Trim commentary that restates the code: the bucketing rationale duplicated in HunspellStemmer, the LibreOffice Spanish anecdote on the code point flag reader, and the sentence left dangling in testGermanCompoundsDecompose. - Drop the defensive null and directory guards from the test helpers writeAndLoadFixture and load, which no caller can trip, and document what the real dictionary tests assert. - Fold the repeated ByteArrayInputStream plumbing in HunspellStemmerTest into two load overloads, one UTF-8 and one taking the charset the SET declaration test needs. - Turn the four table-style stemming tests into parameterized tests over their word and expected stem pairs, so a failing row names itself. - Add testNullArgumentsAreRejected, pinning the exact IllegalArgumentException message of every public entry point including the argument names the stream loader reports. - Correct the stemmer manual: name the example files after the fixture the test loads rather than en_US, and state that the printed stems are the fixture's, since which stem a published dictionary yields is that dictionary's decision. * OPENNLP-1893: Fail loud on result-altering unsupported affix directives Reject ICONV, OCONV, and COMPLEXPREFIXES at load time; keep skipping cosmetic tables such as REP. Copy lookup results defensively and document the compound search budget on HunspellStemmer. * OPENNLP-1893: Bound stream size and match affix conditions by code point Reject affix and dictionary streams above MAX_STREAM_BYTES. Affix conditions and boundary bucketing use Unicode code points so supplementary characters agree with FLAG UTF-8. Document the ceiling in the stemmer chapter and pin both behaviors in tests. * OPENNLP-1893: Verify dictionary downloads by SHA-512 and add an opt-in URL catalog * OPENNLP-1893: Sync shared DownloadUtil with the startup-overridable download ceiling The 512 MiB download ceiling becomes a default that opennlp.download.max.bytes can raise at JVM startup; absent or invalid values fall back. Keeps the file identical to the copy in the MeCab PR. * OPENNLP-1893: Trigger CI for the DownloadUtil sync commit * OPENNLP-1893: Address review: unbox the boundary lookups and publish the stream ceiling The affix boundary buckets move from a boxed Integer map to a sorted int index answered by binary search, closing the per-call boxing note from the review; the strip-only rule lists are frozen at load so no internal mutable list is handed out. MAX_STREAM_BYTES becomes public, matching its citation in the load javadoc, the manual chapter, and the dev README. A parameterized test pins trimming of word-list lines edged by the no-break space and the ideographic space on both edges, and the download and catalog test classes get the javadoc coverage the hunspell tests already carry. * OPENNLP-1893: Use a numeric character reference for the no-break space in the manual DocBook XML defines no nbsp entity, so the PDF build rejects it. * OPENNLP-1893: Reconcile the shared download test files with the sibling PR Both PRs carry byte-identical copies; this folds the sibling's parameterized invalid-limit test into this side's documented fixtures so the copies match again. * OPENNLP-1893: Expose silent COMPOUNDRULE, IGNORE, KEEPCASE and ungated full-strip rules with failing tests COMPOUNDRULE, IGNORE, and KEEPCASE alter analyses when ignored, so the fail-closed loader policy requires them to fail at load time like ICONV, OCONV, and COMPLEXPREFIXES; the loader currently accepts them silently. A suffix rule whose strip string is the whole stem is applied without the FULLSTRIP declaration hunspell requires for it, inventing a stem for a surface form the dictionary does not license. * OPENNLP-1893: Fail loud on COMPOUNDRULE, IGNORE, KEEPCASE and gate full-strip rules behind FULLSTRIP COMPOUNDRULE licenses pattern compounds, IGNORE drops characters before matching, and KEEPCASE forbids the capitalized variants this stemmer analyzes through lowercasing, so ignoring any of them would change stems with no signal; they now join ICONV, OCONV, and COMPLEXPREFIXES in the load-time rejection. An affix rule whose strip string consumes the whole stem is now undone only when the affix file declares FULLSTRIP; without the declaration the rule is skipped at match time, which is what hunspell itself does rather than rejecting the file. The manual and the dictionary README follow the loader. * OPENNLP-1893: Address Hunspell review feedback Add named format constants, lower the download default to 64 MiB, and require applications to supply URL catalogs. Red: the focused test compile failed for missing suffix constants and the caller-supplied catalog overload before the implementation was added.
… and Chinese (apache#1191) * OPENNLP-1894: Lattice segmentation over user-supplied mecab-format dictionaries A Viterbi decoder over word and connection costs segments languages written without spaces; the same engine serves Japanese and Korean because the language lives entirely in the dictionary. Unknown text is handled through the dictionary's character categories, and every span stays in original text coordinates. An installer fetches and unpacks a user-chosen dictionary archive at install time: nothing is bundled, no location is built in, and entry names are flattened so no archive path escapes the target directory. (cherry picked from commit a699c8a) * OPENNLP-1894: Character trie for lattice prefix search Common-prefix lookup walks a trie built at load time instead of probing substrings per length, terminating on the first missing prefix and allocating nothing per position. (cherry picked from commit e10ce4b) * OPENNLP-1894: Frequency-driven segmentation over user-supplied lexicons A Viterbi search maximizing summed word log-probabilities segments Chinese and similar scripts from a plain word-count lexicon, with unlisted characters falling back to single-character words. The user supplies the lexicon and thereby accepts its license; nothing is bundled. (cherry picked from commit bff3f23) * OPENNLP-1894: Usage example and edge-case tests for the lattice and unigram segmenters, corrected javadoc * OPENNLP-1894: Keep lattice test sources ASCII-only via Unicode escapes, add an EUC-JP loading example * OPENNLP-1894: Document dictionary acquisition with a checksum-verifying download helper * OPENNLP-1894: Categorize by code point, keep unknown candidates inside their category run, and validate context ids at load * OPENNLP-1894: Precompute category runs, unbox the trie, and reject inexpressible dictionary values The lattice tokenizer rescanned the same-category run from every position, so a run of L characters cost on the order of L squared category lookups; a 16,000-character katakana run measured around half a second. One right-to-left pass per stretch now fixes every position's category and run end, and the same 16,000-character run tokenizes in about half a millisecond at 31 million characters per second. The character table holds Category instances instead of names, so the per-character path compares by identity with no name-map lookup, and a char.def mapping to an undefined category now fails at load naming the code point. The lexicon trie's children are sorted character arrays found by binary search, so a descent no longer boxes a Character per step. matrix.def loading rejects connection costs outside the 16-bit range instead of silently truncating them, and dimension products beyond the addressable array size fail at the header. The unigram segmenter's unknown-character fallback advances one code point, never one code unit, so an unknown supplementary character is stepped over whole and no span can split its surrogate halves. * OPENNLP-1894: Hold the lexicon in a double-array trie with frequency-recoded labels The lexicon trie's per-node child lookup, a binary search over the node's fan-out, paid about a dozen comparisons at the root of a real dictionary; the classic base/check double-array makes every transition one array read and one comparison. Characters are recoded into dense labels ordered by descending frequency before the array is built, so the array stays compact although CJK surfaces draw on tens of thousands of distinct characters, and a character the lexicon never uses misses in the recode table before the array is consulted. On the IPADIC harness the prefix walk now matches the fastest previous implementation at 5.6M chars/s with strictly constant-time transitions, and building the array adds about a quarter second to the 392k-entry load. * OPENNLP-1894: Chain lattice nodes intrusively instead of allocating per-position lists The Viterbi lattice held one ArrayList per text position plus one fresh candidate list per position, pure allocation churn on long stretches. Nodes ending at a position now chain through their own link field behind a single head reference per position, and candidate gathering fills one scratch list reused across positions, so building the lattice allocates nothing besides the nodes themselves. IPADIC throughput on the 400k-character harness rises from 5.6M to 6.5M characters per second with identical output. * OPENNLP-1894: Document lattice CJK tokenization with a mirror-tested example Add a lattice tokenizer section to the manual citing LatticeUsageExampleTest. * OPENNLP-1894: Apply the review-convention pass and drop unreferenced lexicon accessors * OPENNLP-1894: Trim parsed lines as Unicode whitespace and document the tokenizer overrides The frequency lexicon was trimmed with String.trim(), which strips only ASCII control characters and the space. A line starting with an ideographic space (U+3000), ordinary in hand-edited CJK text files, therefore kept that space as part of the word and pushed the count field one token to the right, so the load failed as a malformed count. The lexicon reader now trims with StringUtil.trimUnicodeWhitespace, matching the White_Space convention the rest of the tokenizer already scans by, and a test pins the leading U+3000 case. The mecab reader's line and numeric-field trims move to the same call so one class does not mix two whitespace judgments; those fields are ASCII in valid dictionaries, so the behavior there is unchanged. Both tokenizer views also gain {@inheritdoc} and their null contract, and the unknown-candidate helper drops a static modifier it did not need. * OPENNLP-1894: Address review: complete javadoc, hoist constants, and fold fixture duplication - Document the private lattice helpers decode, relax, and candidates, and the installer's boundedStream, with the parameter, return, and exception contracts the review expects every method to carry. - Document the WordEntry and Category record components and the double-array builder's findBase and ensureCapacity helpers. - Record on analyze, tokenize, and tokenizePos that a unk.def without a DEFAULT template leaves the lattice disconnected and makes them throw IllegalStateException. - State on readLines that it never returns an empty list, which is what lets the matrix.def header be read before the emptiness check. - Rename the Tokenizer override parameter from s to text in LatticeTokenizer and UnigramSegmenter, so the javadoc names a parameter that exists. - Hoist the matrix.def, char.def, and unk.def file names, the DEFAULT category name, the 0x code point prefix, the .. range separator, and the flag value into named constants in MecabDictionary, and let LatticeTokenizer reach the DEFAULT name through MecabDictionary instead of repeating the literal. - Name the tar block size, header field offsets, and field lengths in the TarGzArchives test helper instead of writing 512, 124, and 148 inline. - Replace the boolean[1] capture in candidates with a check that the candidate list is still empty, which is the same signal without the array. - Track the best boundary total in decode instead of recomputing the incumbent's connection cost on every comparison. - Drop the categories map field from MecabDictionary, which nothing read once the constructor resolved the DEFAULT category out of it. - Match the char.def code point prefix once, case insensitively, rather than testing 0x and 0X separately, and cut the range at the separator's own length. - Trim the matrix.def header before parsing it and report an empty first line as an empty matrix.def, since readLines never yields the empty list the previous check was looking for. - Split the omnibus malformed-dictionary test into named cases that pin the messages for a missing definition file, a char.def without DEFAULT, a lexicon with no entries, and an empty matrix.def. - Parameterize the malformed char.def cases and the malformed unigram lexicon cases, which were repeated assertThrows calls over one fixture shape. - Add a Morpheme test pinning the null and empty argument rejections and the defensive copy of the feature list. - Extend the invalid-argument tests to the entry points that were uncovered: MecabDictionary.load with a null directory or charset, the installer's null target, UnigramSegmenter's path and stream overloads, and both tokenize methods of each tokenizer. - Fold the repeated Files.write fixture calls into one write helper and hoist the shared lexicon, matrix, char.def, and unk.def fixture text into constants. - Correct dev/README-mecab-dictionaries.md to say that dicrc is the configuration file the distributions ship alongside the csv and def files a MecabDictionary reads, rather than implying the dictionary reads dicrc itself. * OPENNLP-1894: Move lattice tokenizer types into opennlp-api Place LatticeTokenizer, UnigramSegmenter, MecabDictionary, and Morpheme with the other resource-driven tokenizers. Keep MecabDictionaryInstaller in runtime. Lift MAX_ENTRIES into ResourceLimits so api loaders can share the bound, and reject incomplete or oversized matrix.def payloads. * OPENNLP-1894: Cap archive extract budgets and harden MeCab load edges Bound per-entry size, total bytes, entry count, and gzip expansion during install. Fail loud on undefined unk.def categories; accept quoted CSV fields. Pin with installer and load tests; note the limits in the manual. * OPENNLP-1894: Verify dictionary downloads by SHA-512 and add an opt-in URL catalog * OPENNLP-1894: Make download and extraction byte budgets overridable at startup Keep the 512 MiB download and tar-entry ceilings and the 2 GiB total extraction ceiling as defaults, but read them from the system properties opennlp.download.max.bytes, opennlp.install.max.entry.bytes, and opennlp.install.max.total.bytes at class load, so dictionaries larger than the defaults, such as UniDic, install without a code change. Absent or invalid values fall back to the defaults. Tests pin the parsing and the defaults; the README and manual document the overrides. * OPENNLP-1894: Trigger CI for the budget-override commit * OPENNLP-1894: Address review: cite named formats and align test conventions Link MeCab, IPADIC, UniDic, mecab-ko-dic, and the POSIX ustar format where the javadoc names them, and write MeCab in its own casing across the prose. Document the entryCount parameter and its limit rejection on MecabDictionary.readLexicon. Replace three literal kanji in LatticeTokenizerTest with Unicode escapes, matching the file's ASCII-only convention. Convert DownloadUtilFileTest's invalid-limit loop to a parameterized test so a failing value is identifiable. * OPENNLP-1894: Reconcile the shared download test files with the sibling PR Both PRs carry byte-identical copies; this folds the sibling's test javadoc into this side's parameterized fixtures so the copies match again. * OPENNLP-1894: Add a remote-gated end-to-end test over the catalog dictionaries Downloads the two pinned distributions with digest verification, installs them, loads them, and checks segmentation. Opt-in via -Dopennlp.download.remote=true like the catalog itself; CI never touches the network. At the previous tip the mecab-ko-dic case failed twice: the matrix cell bound rejected its genuine 3822 x 2693 matrix, and the flattened user-dic templates broke the lexicon parse. * OPENNLP-1894: Bound matrix cells separately so real distributions load The cell-count check reused ResourceLimits.MAX_ENTRIES, sized for record-shaped entries, and rejected mecab-ko-dic 2.1.1, whose genuine matrix declares 3822 x 2693 = 10,292,646 cells of two bytes each. A new ResourceLimits.MAX_MATRIX_CELLS bounds two-dimensional cost tables at 2^27 cells (256 MiB of shorts), overridable at startup, and still refuses the roughly 4 GiB allocation a crafted 46340 46340 header would force. Unit tests pin the new bound on both sides with the ko-dic dimensions as the accepted case. * OPENNLP-1894: Extract dictionary files from the archive root only The installer flattened every csv and def file in the archive into the target directory, so mecab-ko-dic's nested user-dic templates, whose numeric fields are empty because they are mecab-dict-index input, landed beside the real lexicon and failed the load. On a case-insensitive file system a template could even overwrite a real lexicon file of the same base name. Entries deeper than one leading directory are now skipped. * OPENNLP-1894: Address lattice tokenizer review feedback
Add a general subword tokenizer contract with original-text UTF-16 offsets, plus a dependency-free WordPiece implementation and BERT compatibility layer. Document the API and cover vocabulary validation, reference sequences, Unicode, and offset behavior. Red evidence: - A supplementary-plane word at 100 code points was rejected because UTF-16 code units were counted. - Negative piece ids and empty vocabulary pieces were accepted.
Red evidence on the prior implementation: Greek final sigma and Unicode control categories did not match the BERT reference sequence. The sibling tokenizers also disagreed on the 100-code-point limit and model callers rebuilt non-contiguous ids.
Remove the pre-release BERT wrapper, preserve source offsets through reference-compatible normalization, and pass explicit vocabulary ids through the ONNX model callers. Align both tokenizers on Unicode code-point limits and document the public subword contract.
* OPENNLP-1894: Cover dictionary validation and deep tries Add tests for zero-length categories, multiple character categories, duplicate definitions, invalid word costs, malformed input encoding, and long lexicon entries. Red evidence on current main: eight assertions failed and a 20,000-character surface raised StackOverflowError. * OPENNLP-1894: Harden CJK dictionary loading Honor all categories on char.def mappings, use the primary category for unknown-word settings, reject duplicate definitions and invalid word costs, and build lexicon tries without recursive calls. * OPENNLP-1894: Cover maintainer review cases Red evidence: LatticeTokenizerTest reported three failures for the MeCab category limit and mapping declaration diagnostics. * OPENNLP-1894: Apply CJK dictionary review Red evidence: testCategoryOverlapCanConnectRun returned 2 tokens instead of 1, and testRejectsCharacterCategoryLengthAboveMecabLimit accepted LENGTH 16.
…y resources (apache#1211) * OPENNLP-1909: Add verified third-party resource installer Add bounded downloads, SHA-512 verification, staged publication, tar and zip extraction, catalog entries, and adapters for the Hunspell and MeCab features merged through apache#1190 and apache#1191. Red evidence recorded while developing and reviewing this branch covered unsafe redirects and schemes, archive traversal and expansion limits, incomplete tar data, checksum failures, replacement of existing files, malformed catalog values, and validation order. * OPENNLP-1909: Cover installer validation edges Red evidence on the rebased branch: ZIP extraction on a non-default file system threw UnsupportedOperationException, catalog filenames accepted empty and path-like values, tar metadata bypassed maxEntries, and null timeout messages did not match the public validation contract. * OPENNLP-1909: Harden installer validation Validate catalog filenames, support ZIP archives on non-default file systems, and count tar extension headers against the entry limit. Align timeout errors and update the installer documentation. * OPENNLP-1909: Fold the duplicated path walk and tar fixture helpers ResourceInstaller walked the directories below the target twice with the same link and file checks, once to prove a destination vacant and once to create the path for the move; one helper now does both. The lattice tests kept a second copy of the gzip tar builders that the installer tests had, so TarArchives owns them and both packages call it. The MeCab README names the catalog example as the test resource it is. * OPENNLP-1909: Return DownloadUtil to its model-only surface The per-feature download path that apache#1190 and apache#1191 added to DownloadUtil (the download(URI, Path, sha512) method, the opennlp.download.remote gate, the byte ceiling and its property, and configuredLimit) merged with those PRs and is deleted here, with its test. DictionaryCatalog owns the gate and installs through ResourceInstaller, and the byte ceilings live in ResourceInstaller.Limits, so nothing in the tree called the old path. Model downloads and the cached-model checksum verification are untouched. * OPENNLP-1909: Pin the review findings on staging, expansion, and zip listings (failing tests) Red evidence on the rebased branch: an atomic move replaced an existing destination on POSIX, so the vacancy check was advisory; a 4 MiB gzip of zeros expanded in full because no ratio bounded it; *.BIN and *.GZ names did not match the lower-case rules; a download file and a staging directory left by a killed installation stayed in the target forever; a failed first installation left the target directory it created; and a zip whose local headers and central directory listed different files installed the local-header content. The MeCab installer also left a stale scratch directory in the target. The move is extracted into a helper so the promotion guarantee can be tested on its own. * OPENNLP-1909: Stage on the target filesystem, bound gzip expansion, and enforce vacancy at the move The MeCab installer unpacks into a hidden scratch directory beneath the target instead of the system temporary directory, so the download, the unpacked tree, and the installed files share one filesystem; stale scratch directories from a killed run are removed first. Gzip content is bounded to 100 times its compressed size, with a 1 MiB floor for small sources, before the absolute expansion limit applies. The promotion move is a plain move, since an atomic move renames over an existing file on POSIX and made the vacancy check advisory. Zip file names read from the local headers must match the central directory's file entries, on the default filesystem through ZipFile and elsewhere through the zip filesystem. The *.bin and *.gz name rules match in any letter case. Work files left in the target by a killed installation are removed at the next installation into it, and a failed installation removes the empty target directory it created; the class documents that concurrent installations into one target are not supported. The two ResourceLimits archive constants and their properties, which nothing read after this change, are deleted so the expansion property has one meaning. The tokenizer manual names both accepted digests, and the model loading chapter states the ratio rule. * OPENNLP-1909: Bound zip expansion by the same ratio as gzip The expansion-ratio guard applied only to gzip content, so a zip archive was bounded by the absolute expansion limit alone. Deflate reaches roughly 1000 to 1, so a few megabytes of crafted zip expanded to the full 4 GiB ceiling in the caller's target directory, and install(URI, Path) accepts a file: archive with no checksum at all. The ratio now applies to every compressed source: gzip and zip alike expand to at most 100 times the compressed size, with the 1 MiB floor for small sources. * OPENNLP-1909: Make the expansion ratio a configurable limit The ratio was a private constant, so 100 to 1 was a hard ceiling no caller could lift. Raising opennlp.install.max.total.bytes does not help, because the ratio and the absolute limit are enforced independently and the tighter one applies, so a resource that legitimately compresses better than 100 to 1 could not be installed at all. maxExpansionRatio joins the other six values on Limits, with the builder method and the opennlp.install.max.expansion.ratio property the three byte and entry ceilings already have, and must be positive like them. The manual states that the ratio and the expansion limit are separate, since raising the byte limit alone reads like the fix and is not. --------- Co-authored-by: Richard Zowalla <rzo1@apache.org>
Red tests covered deterministic neighbors, invalid model content, cache provenance, and tokenizer validation. Green affected reactor: 3,227 tests with one expected skip; package, documentation, Javadoc, Checkstyle, forbidden API, XML, and policy checks passed.
Preserve the reviewed subword API and WordPiece changes while integrating merged CJK and ResourceInstaller updates. The API, runtime, deep-learning, and documentation reactor package passes 3,209 tests with no failures or errors; four optional dictionary cases are skipped.
All published non-merge patches are present locally by patch ID. Retain later local validation, tokenizer alignment and review fixes.
Use the current WordPiece API for single and batched DL encoding while retaining TextEmbedder, dimensions and SentencePiece support. API, runtime, DL, subword and embeddings tests pass. Removed a duplicate tokenizer manual section exposed by the package build; the repaired manual package passes.
A relative path with a '..' segment is ambiguous as a teacher reference: Windows collapses '..' lexically without checking that the segment before it exists, so a misspelled hub id such as BAAI/.. silently names the working directory there while POSIX reports it as nonexistent. This is the windows-latest CI failure in HuggingFaceModelCacheTest testMalformedTeacherReferenceIsRejectedBeforeAnyRequest[14]. Red evidence on Linux: the new test fails against the current code with 'Expected java.lang.IllegalArgumentException to be thrown, but nothing was thrown', because resolve(src/..) returns the module directory.
Windows collapses '..' lexically without checking that the segment before it exists, so resolve(BAAI/..) treated the working directory as the teacher directory there while POSIX rejects the reference; this was the windows-latest CI failure in HuggingFaceModelCacheTest testMalformedTeacherReferenceIsRejectedBeforeAnyRequest[14]. The local directory shortcut now refuses relative paths containing a '..' segment, so the rejection is identical on every platform. Absolute paths and clean relative paths are unaffected, and a local directory still wins over the hub. Also rework the teacher-name escaping test, which created a directory named teacher"quoted: the quote is illegal in Windows file names, so the test could not even start there. The name-to-config plumbing is now covered with a plain directory name, and the JSON escaper behind tokenizer_name is exercised directly with quotes, backslashes, and control characters.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a new
opennlp-extensions/opennlp-embeddingsmodule: a pure-JVM engine for modern static embedding tables (Model2Vec-family distillations, the 2024/2025 successors to word2vec/GloVe: same flat per-token table shape, sentence-transformer semantics, inference is pure lookup).Positioning: this is the speed tier of text embedding, not a replacement for anything. The ONNX
SentenceVectorsDLin opennlp-dl runs the actual transformer and produces contextual vectors, so this isn't made to be a replacement as that's the OOTB accuracy tier.This module trades that context for a static table: no native runtime, no GPU, and throughput in the hundreds of thousands of texts per second per core. Different pipeline stages want different points on that curve. Matches the philosophy of this style of embedding.
What's in it:
TextEmbedder(new interface inopennlp-api, packageopennlp.tools.embeddings): the text-level embedding contract.embed(CharSequence),embedAll(List)(default implementation loops; runtimes that batch efficiently should override), anddimension(). It is the text-level counterpart of the existing word-levelWordVectorTable, and the javadoc states that layer difference explicitly.StaticEmbeddingModelis the first implementation.SentenceVectorsDLis the natural second one: adopting the interface is purely additive (its existing constructors andgetVectorsare untouched), and its ONNX runtime is exactly what the overridableembedAllbatch method exists for. That adoption is a separate discussion, not part of this PR.SafetensorsFile: reads the safetensors format with a purpose-built cursor parser for the JSON header (no third-party JSON dependency; decodes F32, F16, and BF16, widening the 16-bit types to float). safetensors carries no executable content, unlike pickle-based checkpoints, so loading is safe by construction. The embedding matrix is auto-detected as the single 2-D float tensor, failing loud and listing candidates on ambiguity rather than guessing a key-name convention.WordpieceVocabulary: BERT-stylevocab.txt, line number = embedding row id. (Named to match the existingWordpieceTokenizercasing.)StaticEmbeddingModel: embeds through the existingBertTokenizer/WordpieceTokenizer, reused unchanged. The pooling formula is verified against the Moust reference implementations:[CLS]/[SEP]never pooled, unknown tokens dropped from sum and denominator, optional per-rowweightstensor, token-count denominator, epsilon-floored normalization.similarity,mostSimilar(bounded top-K over precomputed row norms),analogy(inputterms excluded by folding through the mode
Posture: code only, bring your own tab fetched at build or run time, no new dependencies.
Thread safety: immutable,
@ThreadSafe, with an 8-thread concurrency test comparing every result against thesingle-threaded reference.
Measured (JMH, opt-in
jmhprofile matching opennlp-runtime's pattern; fixture at real published-table scale, 29,528 x 256):embed~766k short sentences/s on one core (1.04M ops/s of 5 sentences at 32 threads); full-vocabulary top-10 scan 649/s per core, ~9.2k/s at 32 threads.Verification: opennlp-embeddings 43/0 plus the new interface contract test,
mvn verifygreen including checkstyle and forbiddenapis.Follow-ups (deliberately out of this PR):
TextEmbedderwith a real batchedembedAll(pending discussion with its author), the gRPC backend in opennlp-sandbox, a concurrent-load comparison against a Python baseline, an ANN index formostSimilar, and bundled-default-model license diligence.https://issues.apache.org/jira/browse/OPEN-1877