OPENNLP-1880: Add a Lexical knowledge base seam with WN-LMF and WNDB readers and a Morphy lemmatizer

[krickert]

Rebased on main; the diff is this change only (53 files, all new except one line in the extensions pom).

Adds the format-agnostic lexical knowledge base interface to opennlp-api (opennlp.tools.wordnet: LexicalKnowledgeBase, Synset, WordNetPOS, WordNetRelation), and a new opennlp-extensions/opennlp-wordnet module with two clean-room readers behind it:

• WnLmfReader: a StAX reader for WN-LMF XML, Open English WordNet's interchange format. XXE-hardened per the OWASP posture for DOCTYPE-bearing formats: the DOCTYPE line real OEWN releases ship is tokenized and skipped while external entities and the external DTD subset stay fully disabled, so an unmodified download parses directly; a dedicated test proves a DOCTYPE-declared internal-subset entity payload fails loud rather than expanding.
• WndbReader: a reader for legacy Princeton WNDB directories, validating the format's documented offset contract.

Both produce equivalent immutable, thread-safe lexicon views, pinned by a structural equivalence test over matching miniature fixtures. On top of the interface, MorphyLemmatizer implements the existing Lemmatizer interface with the documented Morphy algorithm: exception-list lookup first, then per-POS detachment rules with each candidate validated against the lexicon.

Naming: the interface is LexicalKnowledgeBase rather than a WordNet-branded name; the contract is generic and only the format-specific readers use the WordNet name.

The module ships code only; users point it at a WordNet database they downloaded. Verified against Princeton 3.0 (117,659 synsets) and OEWN 2024 (120,630 synsets). No third-party WordNet library is used or referenced.

Review rounds

The 2026-07-10 findings are in (see the reply below them). The 2026-09-04 conventions from the sibling PRs are in as well: Javadoc on every method and constructor with @throws on the validating paths, validation at the public boundary, repeated message literals as constants, StringUtil.toLowerCase for lemma folding, bounded WNDB index counts and markers, relation and index integrity checks, caller-owned streams preserved, and failing-test commits ahead of each fix.

Verification

opennlp-api and opennlp-wordnet build with checkstyle and forbidden APIs; the module's 347 tests pass (4 skipped without a downloaded wordnet) plus the 18 interface tests in api. The manual chapter's examples are mirrored by WordNetUsageExampleTest.

Small upstream note found along the way: DictionaryLemmatizer javadoc says unknown = "0" but the code returns the letter "O"; Morphy matches the code. Filed separately.

https://issues.apache.org/jira/browse/OPENNLP-1880

[krickert]

There's a minor windows-specific build issue due to line endings and Windoze. Will flip back to review once I find my windows machine and test. Code is the same though.

[rzo1]

Hi @krickert - as mentioned on Slack I currently dont have the time for a manual review but I just let Fable do a comprehensive review on this PR. Here is the result:

• opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/WnLmfReader.java: SynsetRelation elements with relType="other" are rejected with "Unknown relation type other", although the class javadoc documents that "other" relations are skipped. The skip filter is only applied to SenseRelations in build(); SynsetRelations go through parseRelation unconditionally, so a WN-LMF document that uses relType="other" on a SynsetRelation (permitted by the DTD, emitted by several OMW-family wordnets) fails to load. Tests only cover the SenseRelation case.
• opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/MorphyLemmatizer.java: posFromTag's first-character fallback is broader than its javadoc claims. The single-letter WordNet codes "a" and "s" are matched as prefixes, so multi-letter tags like UD AUX, ADP, SCONJ or Penn SYM are misclassified as ADJECTIVE instead of yielding the documented unknown-word result (e.g. AUX "was" is looked up as an adjective and returns "O"). The "a"/"s" cases should require tag.length() == 1, and testPosFromTagMapping should exercise such tags.
• pom.xml: The new opennlp-wordnet artifact is not registered in the root pom's dependencyManagement and not added as a dependency of opennlp-distr, unlike the other extension modules (opennlp-morfologik, opennlp-spellcheck, opennlp-uima). As-is it will not ship in the binary distribution or aggregated apidocs, and reactor modules cannot reference it without an explicit version.
• opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/WndbReader.java: The new public loading entry points (WndbReader.read, WnLmfReader.read, MorphyExceptions.load) signal I/O failures as UncheckedIOException and malformed data as IllegalArgumentException, diverging from the established convention of checked IOException / InvalidFormatException used by comparable loaders (DictionaryLemmatizer, model loaders). Once released this contract cannot be tightened without breaking source compatibility, so it should be decided deliberately before the first release.
• opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/WnLmfReader.java: The "similar"-on-verb-synset to VERB_GROUP mapping has zero test coverage in either reader; the WN-LMF fixture only carries "similar" on adjective synsets and the WNDB fixture has no "$" pointer, so a regression in the only POS-dependent relType mapping would pass the whole suite.
• opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/WnLmfReader.java: Duplicate Sense ids and duplicate LexicalEntry ids are silently overwritten (plain map.put) while duplicate Synset ids fail loud, so a malformed document can have sense relations lifted onto the wrong synset without any error.
• opennlp-api/src/main/java/opennlp/tools/wordnet/WordNetPos.java: The name uses mixed-case "Pos" while the rest of the codebase uppercases the abbreviation (POSTagger, POSModel, POSSample); this public type cannot be renamed after release, so consider WordNetPOS or a spelled-out name.
• opennlp-api/src/main/java/opennlp/tools/wordnet/Synset.java: Synset is a public record whose canonical five-component constructor becomes frozen API; since the seam explicitly anticipates additive extension, adding a component later requires permanently keeping an overload of the old constructor. Worth confirming the v1 component set (per-sense data, lexicographer file) before release.
• opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/WndbReader.java: Relation-target synset id strings are minted fresh per pointer and never canonicalized against the synset table (~660K pointers in Princeton 3.0), retaining tens of MB of duplicate short Strings in a lexicon designed to be held in memory long-term. Interning against the canonical RawSynset.id during resolve() would collapse them; the same applies to WnLmfReader.resolveRelations.
• opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/InMemoryWordNetLexicon.java: The constructor's referential-integrity validation (key/id mismatch, dangling relation target, sense-order entry referencing unknown synset) has no direct test; a future third reader would rely on these checks and a regression would go unnoticed.
• opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/WndbReader.java: The RawPointer record's lineNumber component is dead code; it is populated at every parse but never read since resolve() reports errors with the owning synset's location. Either drop it or use it for pointer-precise error messages.
• opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/MorphyExceptions.java: The lemma-folding logic (replace('_',' ').toLowerCase(Locale.ROOT)) is duplicated verbatim in InMemoryWordNetLexicon.LemmaKey.of, and splitOnSpaces is duplicated in WnLmfReader.Parser. The fold agreement is load-bearing for MorphyLemmatizer, but nothing (shared helper or cross-referencing test) prevents the copies from drifting apart.
• opennlp-extensions/opennlp-wordnet/src/test/java/opennlp/wordnet/WnLmfReaderTest.java: Several documented fail-loud paths are untested: duplicate synset id, synset/member POS mismatch, SenseRelation targeting an undeclared sense, and the structural-misplacement errors (Lemma outside LexicalEntry, Sense before Lemma, SenseRelation outside Sense, SynsetRelation outside Synset).

Human review will follow.

[krickert]

Thanks, every finding is addressed in 69ead68. Point by point:

• SynsetRelation relType="other": now skipped exactly like the SenseRelation case (documented skip, not rejection), applied at parse time. A test covers the SynsetRelation case alongside the existing SenseRelation one.
• posFromTag over-matching: the WordNet letter codes a and s match only when tag.length() == 1, so AUX, ADP, SCONJ, and SYM map to no part of speech and yield the unknown marker. The tag-mapping test pins all four, lemmatizer-level cases pin that AUX "was" is unknown and AUX "taller" no longer detaches to "tall", and the javadoc matches the implementation.
• Root pom and distribution wiring: opennlp-wordnet is in the root dependencyManagement, in opennlp-distr's dependencies, and in the bin assembly's apidocs fileSets, mirroring opennlp-spellcheck.
• Exception contract: all three loading entry points (WndbReader.read, WnLmfReader.read, MorphyExceptions.load) declare IOException; malformed content raises InvalidFormatException, matching DictionaryLemmatizer and the model loaders, while null and argument validation stays IllegalArgumentException. A failing-stream test pins that an underlying I/O error surfaces as IOException rather than a malformed-document report.
• similar-on-verb to VERB_GROUP: covered in both readers. WN-LMF parses an inline document with similar between two verb synsets and asserts VERB_GROUP with SIMILAR_TO absent; WNDB builds a minimal database in a temp directory with programmatically computed byte offsets so the $ pointer resolves, asserting VERB_GROUP in both directions.
• Duplicate Sense and LexicalEntry ids: both fail loud with InvalidFormatException naming the id, symmetric with the duplicate-synset check; tests for both.
• WordNetPos naming: renamed to WordNetPOS across the API and the module, consistent with POSTagger and POSModel.
• Synset record components: kept as the deliberate v1 contract. Per-sense data and lexicographer files are reader-level concerns the seam intentionally omits; a later addition arrives as a new canonical constructor with the old one kept as a documented overload, the standard record-evolution path.
• Relation-target id duplication: relation resolution in both readers stores the synset table's canonical id instance per target, collapsing a full lexicon's pointer strings to the table's instances; tests assert instance identity between a related() target and the target synset's id() in both readers.
• InMemoryWordNetLexicon validation: direct constructor tests for key/id mismatch, dangling relation target, sense-order entry referencing an unknown synset, null maps, and the accepting case.
• RawPointer.lineNumber: now used; dangling-pointer errors report the pointer's own line, tested.
• Fold and split duplication: extracted into a package-private LemmaFolding helper used by MorphyExceptions, the lexicon's lemma key, and the WN-LMF members parsing, with a test pinning the fold and the key/query agreement.
• Untested WN-LMF fail-loud paths: tests for duplicate synset id, synset/member POS mismatch, SenseRelation to an undeclared sense, Lemma outside LexicalEntry, Sense before Lemma, SenseRelation outside Sense, and SynsetRelation outside Synset.

Module tests go from 86 to 112; reactor verify is clean.