OPENNLP-1887: Lexical expansion over the knowledge base seam: synonyms, hypernyms, optional hyponyms - #1167
Draft
krickert wants to merge 55 commits into
Draft
OPENNLP-1887: Lexical expansion over the knowledge base seam: synonyms, hypernyms, optional hyponyms#1167krickert wants to merge 55 commits into
krickert wants to merge 55 commits into
Conversation
krickert
changed the title
krickert
changed the title
krickert
force-pushed
the
wordnet-expansion
branch
6 times, most recently
from
July 17, 2026 20:45
5db6a8a to
917e4be
Compare
krickert
force-pushed
the
wordnet-expansion
branch
2 times, most recently
from
July 19, 2026 11:41
1b90059 to
f4cc90c
Compare
krickert
added a commit
to ai-pipestream/opennlp
that referenced
this pull request
Jul 24, 2026
…ENNLP-1895 recorded Restate the map against apache main a864230, cut as 3.0.0-M5 on 2026-07-24. apache#1177 (OPENNLP-1870) merged upstream and moves into the merged box, apache#1190 and apache#1191 are marked ready for review, and OPENNLP-1895 (quantized embedding tables) joins the diagram in its own colour: filed in JIRA with the pull request deliberately held until apache#1165 and apache#1152 move. Statuses now carry the measured GitHub draft flag and how far each head sits behind main, which surfaces three things the old text did not: apache#1182 is a draft again, apache#1167 is based on main rather than on apache#1155 and carries the seam and isBlank commits as copies, and apache#1152 reports conflicts only because its apache-hosted sentencepiece base has diverged from the refreshed head.
krickert
force-pushed
the
wordnet-expansion
branch
from
July 24, 2026 19:27
ff066b0 to
b1aef66
Compare
krickert
added a commit
to ai-pipestream/opennlp
that referenced
this pull request
Jul 24, 2026
…est head All nine open heads now sit directly on a864230 and report mergeable. Two were reporting conflicts and both cleared: apache#1167 through a plain rebase, and apache#1152 by pointing its apache-hosted sentencepiece base branch at the refreshed head it had drifted away from, which shrinks its diff back to the 30 commits it owns. apache#1166 shed the 13 OPENNLP-1883 commits it carried, since apache#1163 is upstream as a single squash, and is 3 commits now. Also correct what the draft flag on apache#1182 means: the branch is review-ready and waits on the upstream queue, not on unfinished work.
krickert
added a commit
to ai-pipestream/opennlp
that referenced
this pull request
Jul 24, 2026
The clean-room CFSA2/FSA5 readers and the PoliMorf lemmatizer had no node because until today they had no branch: they were written on the OPENNLP-1887 line under formats: and lemmatizer: titles and were only visible as a sentence in that pull request's notes. They now sit on their own branch off current main and get a node and a row like every other staged feature. The note on apache#1167 says what is actually true: the commits were copied out, and dropping them from that branch is a separate rebase that has not happened.
krickert
force-pushed
the
wordnet-expansion
branch
2 times, most recently
from
July 31, 2026 10:38
89a5a27 to
840d4fd
Compare
krickert
force-pushed
the
wordnet-expansion
branch
2 times, most recently
from
August 9, 2026 13:09
af19962 to
be24524
Compare
krickert
added a commit
to ai-pipestream/opennlp
that referenced
this pull request
Aug 9, 2026
Adds the symbol-joiner and dehyphenation rows, marks the apache#1167 restack and the morfologik reconciliation, and moves the state line to fc9824a.
krickert
force-pushed
the
wordnet-expansion
branch
from
August 10, 2026 10:24
be24524 to
3dddfed
Compare
krickert
added a commit
that referenced
this pull request
Aug 15, 2026
# Conflicts: # opennlp-api/src/main/java/opennlp/tools/util/StringUtil.java # opennlp-tools/src/test/java/opennlp/tools/util/StringUtilTest.java
krickert
added a commit
that referenced
this pull request
Aug 16, 2026
krickert
added a commit
that referenced
this pull request
Aug 30, 2026
krickert
added a commit
to ai-pipestream/opennlp
that referenced
this pull request
Sep 4, 2026
Adds the symbol-joiner and dehyphenation rows, marks the apache#1167 restack and the morfologik reconciliation, and moves the state line to fc9824a.
…s and a Morphy lemmatizer Adds the LexicalKnowledgeBase contract in opennlp.tools.wordnet and the opennlp-wordnet module implementing it twice: WnLmfReader for WN-LMF XML and WndbReader for the legacy WNDB database files, with reader-equivalence coverage over miniature fixtures of both formats. The WN-LMF reader skips DOCTYPE declarations unresolved with DTD support off, so Open English WordNet releases parse unmodified while entity expansion stays closed; the WNDB fixtures are pinned to LF so their embedded byte offsets survive Windows checkout. The Morphy lemmatizer resolves inflected forms through suffix rules and the format's exception lists. Null arguments fail loudly with IllegalArgumentException, malformed data raises the checked InvalidFormatException, and the public seam carries no brand name: WordNet stays in the names of the classes that actually read WordNet formats.
Add docbkx/wordnet.xml, wire it into the manual, and add WordNetUsageExampleTest asserting the lookups the chapter prints.
…ment with the review conventions
Reword the expander, similarity, and typer Javadoc and the manual chapter in plainer terms, state the Leacock-Chodorow example in the manual with a matching assertion, and name the tests after the behavior they pin.
krickert
force-pushed
the
wordnet-expansion
branch
from
September 4, 2026 21:58
a68762f to
1d13232
Compare
…s and a Morphy lemmatizer Adds the LexicalKnowledgeBase contract in opennlp.tools.wordnet and the opennlp-wordnet module implementing it twice: WnLmfReader for WN-LMF XML and WndbReader for the legacy WNDB database files, with reader-equivalence coverage over miniature fixtures of both formats. The WN-LMF reader skips DOCTYPE declarations unresolved with DTD support off, so Open English WordNet releases parse unmodified while entity expansion stays closed; the WNDB fixtures are pinned to LF so their embedded byte offsets survive Windows checkout. The Morphy lemmatizer resolves inflected forms through suffix rules and the format's exception lists. Null arguments fail loudly with IllegalArgumentException, malformed data raises the checked InvalidFormatException, and the public seam carries no brand name: WordNet stays in the names of the classes that actually read WordNet formats.
Add docbkx/wordnet.xml, wire it into the manual, and add WordNetUsageExampleTest asserting the lookups the chapter prints.
…ment with the review conventions
…undary, pin WNDB rejections - Narrow the LexicalKnowledgeBase javadoc so the interface stops prescribing what only an implementation can promise: lemma matching semantics and thread safety are now stated as implementation specific and documented where they hold, on InMemoryWordNetLexicon, which already carries @threadsafe and describes the folding it applies. - Reword the contains() javadoc to say plainly that the default implementation delegates to lookup(), instead of speculating about cheaper overrides. - Move the null-element checks in MorphyLemmatizer up to the public lemmatize() overloads, both the array form and the list form, so validation happens once at the boundary the caller sees; the private lemmasOf() no longer repeats them and now documents that its arguments are validated by the caller. - Reject a null argument in LemmaFolding.splitOnSpaces() rather than letting it fail later as a NullPointerException, and capitalize the fold() message so it matches the wording the other validators use. - Document the throws clauses that the explicit validation adds, on LemmaKey.of() and on splitOnSpaces(). - Extract the repeated WN-LMF attribute names into ID_ATTRIBUTE, PART_OF_SPEECH_ATTRIBUTE, REL_TYPE_ATTRIBUTE, and TARGET_ATTRIBUTE, and the shared error opening into MALFORMED_PREFIX, so the element handlers stop repeating string literals. - Extract the WNDB offset failure detail into MALFORMED_OFFSET, shared by the length check and the digit check. - Fold the duplicated WNDB message construction into malformedMessage(), so the tokenizer builds the text directly instead of constructing an InvalidFormatException only to read getMessage() back off it. - Drop the redundant fileName parameter from WndbReader.readAll(), which already names the full path it failed to open. - Reduce the visibility of the Parser helpers in WnLmfReader: malformed() and line() are now private instance methods like every other helper in that class. - Make MorphyLemmatizer.rulesFor() an instance method for the same reason, so the lemmatizer's private helpers are consistent. - Add the missing javadoc on the RELATION_NAMES and POINTER_SYMBOLS lookup tables and on both RawSynset holders, the last undocumented members in the readers. - Correct two stale comments: the build() comment now points at memberLemmas(), where the synset and member part-of-speech agreement is really checked, and the mutate() comment in the tests states the actual constraint, that an edit which changes a line's length is only safe when the reader is expected to fail on that line before it reads the ones after it. - Add a parameterized WNDB test pinning eight field-level rejections that had no coverage: the offset length and digit checks, the synset and index part of speech mismatches, the base-16 word count field, the minimum word count, the pointer pos, the gloss separator, and the syntactic marker. - Add pinning tests for the newly explicit validation: splitOnSpaces() on null, and the list lemmatize() overload with a null token and with a null tag. - Share the fixtures instead of duplicating them: WndbReaderTest now exposes DOG_ID, CANID_ID, and its fixtureDirectory(), WnLmfReaderTest exposes fixture(), and LexiconConcurrencyTest and WordNetUsageExampleTest use those instead of their own loader copies and hardcoded ids. - Document the package-private test fixture helpers and switch WordNetUsageExampleTest to static assertion imports, matching the other tests in the module.
…reader-test gaps Add a Navigating relations section to the WordNet chapter and mirror it, pin the chapter's Path-based loading listing with a usage-test method, give the exception fixture a two-base-form entry (axes to axis, ax) so the list-form lemmatizer path is exercised with multiple candidates, and fold the copy-paste WN-LMF rejection tests into one parameterized ladder with identical documents and message pins.
…-lexicon resources Extends WordNetRelation with every typed synset and sense relation in the WN-LMF 1.4 DTD and rejects relations placed on the wrong element kind. Adds WnLmfReader.readResource with WnLmfResource, WnLmfLexicon, and WnLmfDependency so multi-lexicon documents keep per-language lookup indexes and Requires metadata; the single-lexicon read methods reject that shape instead of merging, and LexiconExtension is rejected clearly. Unlexicalized synsets used by real OMW releases are retained as relation targets with empty lemma lists. Includes relation-conformance and resource tests, an optional pinned OMW 2.0 integration check behind dev/test-omw-wordnets.sh, and manual coverage with mirror tests.
The canonical lemma fold used String.toLowerCase(Locale.ROOT), which applies the SpecialCasing one-to-many mappings. Fold with StringUtil.toLowerCase instead so the sense-index keys, the Morphy exception keys, and the query path all share the locale-independent one-to-one mapping.
…plainly Repeated exception message literals become constants, and the seam, reader, and lemmatizer Javadoc and the manual chapter say what the types do in plainer terms.
Import the upstream CI concurrency change. Production code, tests, and manual content are unchanged. The affected reactor compiles successfully.
Retain expansion, typing and similarity while adopting reviewed shared API and reader updates. WordNet and documentation clean package build passes.
krickert
added a commit
to ai-pipestream/opennlp
that referenced
this pull request
Sep 5, 2026
# Conflicts: # opennlp-api/src/main/java/opennlp/tools/wordnet/LexicalKnowledgeBase.java # opennlp-api/src/main/java/opennlp/tools/wordnet/Synset.java # opennlp-api/src/main/java/opennlp/tools/wordnet/WordNetPOS.java # opennlp-api/src/main/java/opennlp/tools/wordnet/WordNetRelation.java # opennlp-api/src/test/java/opennlp/tools/wordnet/LexicalKnowledgeBaseTest.java # opennlp-distr/pom.xml # opennlp-distr/src/main/assembly/bin.xml # opennlp-docs/src/docbkx/wordnet.xml # opennlp-extensions/opennlp-wordnet/pom.xml # opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/HypernymTyper.java # opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/InMemoryWordNetLexicon.java # opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/LemmaFolding.java # opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/LexicalExpander.java # opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/MorphyExceptions.java # opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/MorphyLemmatizer.java # opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/SynsetSimilarity.java # opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/WnLmfDependency.java # opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/WnLmfLexicon.java # opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/WnLmfReader.java # opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/WnLmfResource.java # opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/WndbReader.java # opennlp-extensions/opennlp-wordnet/src/test/java/opennlp/wordnet/HypernymTyperTest.java # opennlp-extensions/opennlp-wordnet/src/test/java/opennlp/wordnet/InMemoryWordNetLexiconTest.java # opennlp-extensions/opennlp-wordnet/src/test/java/opennlp/wordnet/LemmaFoldingTest.java # opennlp-extensions/opennlp-wordnet/src/test/java/opennlp/wordnet/LexicalExpanderLexiconTest.java # opennlp-extensions/opennlp-wordnet/src/test/java/opennlp/wordnet/LexicalExpanderTest.java # opennlp-extensions/opennlp-wordnet/src/test/java/opennlp/wordnet/LexicalExpansionUsageExampleTest.java # opennlp-extensions/opennlp-wordnet/src/test/java/opennlp/wordnet/LexiconConcurrencyTest.java # opennlp-extensions/opennlp-wordnet/src/test/java/opennlp/wordnet/MorphyLemmatizerTest.java # opennlp-extensions/opennlp-wordnet/src/test/java/opennlp/wordnet/ReaderEquivalenceTest.java # opennlp-extensions/opennlp-wordnet/src/test/java/opennlp/wordnet/SynsetSimilarityTest.java # opennlp-extensions/opennlp-wordnet/src/test/java/opennlp/wordnet/WnLmfReaderTest.java # opennlp-extensions/opennlp-wordnet/src/test/java/opennlp/wordnet/WndbReaderTest.java # opennlp-extensions/opennlp-wordnet/src/test/java/opennlp/wordnet/WordNetUsageExampleTest.java # opennlp-extensions/opennlp-wordnet/src/test/resources/opennlp/wordnet/mini-wndb/.gitattributes # pom.xml # rat-excludes
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebased on main. This branch stacks on #1155 (the
LexicalKnowledgeBaseinterface and readers) and includes those 18 commits until that PR merges; the expansion work is the commits after them. #1182 (StringUtil.isBlank) is merged.Adds
LexicalExpanderto theopennlp-wordnetmodule: given a term, it produces weighted related terms from aLexicalKnowledgeBase, the synonyms sharing its synsets, the lemmas of hypernym ancestors up to a configured depth (following both the direct and the instance relation, visited-checked so cyclic data terminates), and optionally direct hyponyms. Each expansion has a deterministic heuristic weight (sense-rank decay times depth decay, both configurable), results are deduplicated case-insensitively keeping the highest weight, the input term is excluded, and ordering is stable. An optionalLemmatizerfallback expands inflected input; the Morphy lemmatizer of this module plugs in directly, sodogsexpands viadogandmiceviamouse.SynsetSimilarityscores synset pairs on the hypernym graph with path, Wu-Palmer (depths counted in nodes, so a shared ancestor scores above zero), and Leacock-Chodorow measures, andHypernymTypertypes a noun by the nearest caller-registered anchor in its hypernym graph.Tests cover the behavior over a hand-built graph with a controlled shape (sense ranking, depth decay, dedupe, cycle termination, hyponym opt-in, configuration validation, taxonomy depth, missing parents) plus integration over the miniature WN-LMF and WNDB fixtures asserting both readers expand identically. The manual chapter's examples are mirrored by
LexicalExpansionUsageExampleTest.Review round of 2026-09-04
Javadoc on every method and constructor with
@throwson the validating paths, validation at the public boundary, the Wu-Palmer depth fix with a failing test ahead of it, and the expander, similarity, and typer Javadoc reworded in plainer terms.