Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
97 commits
Select commit Hold shift + click to select a range
49b9612
OPENNLP-1888: Document annotation container: typed offset-anchored la…
krickert Jul 15, 2026
3e382b3
OPENNLP-1888: Lemma and stem layer adapters
krickert Jul 15, 2026
e8a2ca4
OPENNLP-1888: Pipeline example and contract tests for the document co…
krickert Jul 16, 2026
e9b53a7
OPENNLP-1888: Parse per sentence in the adapters, distinguish empty f…
krickert Jul 17, 2026
6b384b6
OPENNLP-1888: Lemmatize per sentence, validate adapter outputs, clear…
krickert Jul 17, 2026
99e0b83
OPENNLP-1888: Add StringUtil.isBlank following the toolkit whitespace…
krickert Jul 17, 2026
eb277fb
OPENNLP-1888: Manual chapter, review-convention pass, and key placeme…
krickert Jul 17, 2026
1ba2fcd
OPENNLP-1888: Namespace the standard layer key ids
krickert Jul 17, 2026
f8b2ee9
OPENNLP-1888: Declare per-key positional or document scope
krickert Jul 17, 2026
b39dd25
OPENNLP-1888: State the index-reference invariants in the specificati…
krickert Jul 17, 2026
e4d6ed7
OPENNLP-1888: Document the gold-layer convention
krickert Jul 17, 2026
266e780
OPENNLP-1888: Create toolkit layer keys through namespace-applying fa…
krickert Jul 17, 2026
d43c1ec
OPENNLP-1888: Javadoc container overrides and cite the pipeline examp…
krickert Jul 20, 2026
0453217
OPENNLP-1888: Anchor the Document thread-safety note to immutability
krickert Jul 20, 2026
15102d7
OPENNLP-1888: Address review: shared annotator helpers, hardened cont…
krickert Jul 28, 2026
e906225
OPENNLP-1888: Address review: fold duplicate test fixtures and pin nu…
krickert Jul 28, 2026
6896361
OPENNLP-1888: Name the null argument, avoid per-call key set wrapping…
krickert Aug 8, 2026
6084309
OPENNLP-1888: Rework the document chapter after docs review
krickert Aug 17, 2026
9442b91
OPENNLP-1888: Pin the Document.merge contract
krickert Aug 17, 2026
919cf59
OPENNLP-1888: Implement Document.merge as a default method
krickert Aug 17, 2026
aa5aeba
OPENNLP-1888: Document the parallel fan-out join in the manual
krickert Aug 17, 2026
adb15b4
OPENNLP-1888: Pin the duplicate-layer policy contract for merge
krickert Aug 17, 2026
1297f72
OPENNLP-1888: Add a duplicate-layer policy to Document.merge
krickert Aug 17, 2026
3245eff
OPENNLP-1888: Render the layer figure monospaced and tighten the prose
krickert Aug 17, 2026
4f5c0ca
OPENNLP-1888: Pin a differing-contents message for KEEP_EQUAL rejection
krickert Aug 17, 2026
844128f
OPENNLP-1888: Name the differing contents when KEEP_EQUAL rejects a l…
krickert Aug 17, 2026
75fcac0
OPENNLP-1888: Pin what layer equality compares for KEEP_EQUAL
krickert Aug 17, 2026
562af87
OPENNLP-1888: Pin merge semantics across Document implementations
krickert Aug 17, 2026
09cd47d
OPENNLP-1888: Merge with a single map copy in ImmutableDocument
krickert Aug 17, 2026
0705c5d
OPENNLP-1888: State span offsets in UTF-16 units in the manual
krickert Aug 18, 2026
550bc44
OPENNLP-1888: Pin that adapters name themselves by class name
krickert Aug 18, 2026
f4dd343
OPENNLP-1888: Name adapters by their simple class name
krickert Aug 18, 2026
ffb99d0
depparse: Arc-standard dependency parser skeleton with CoNLL-U reader…
krickert Jul 15, 2026
c828b86
depparse: DependencyModel persistence and a gated Universal Dependenc…
krickert Jul 15, 2026
f7182f6
depparse: Read CoNLL-U raw so multiword-token sentences keep their de…
krickert Jul 15, 2026
14a226a
depparse: Valency and child features for the transition classifier
krickert Jul 15, 2026
cf9080b
depparse: Pure-Java feedforward neural tier, training and inference i…
krickert Jul 15, 2026
944cff4
depparse: Pretrained word embedding seeding for the feedforward trainer
krickert Jul 15, 2026
ecfdc9e
depparse: Beam search decoding for the feedforward parser
krickert Jul 15, 2026
63cd289
depparse: Global refinement with beam search and early update
krickert Jul 15, 2026
2734990
depparse: State, transition, usage, and edge-case tests; expanded tra…
krickert Jul 16, 2026
752c05c
depparse: Document treebank acquisition for the gated evaluation with…
krickert Jul 16, 2026
d1072b7
depparse: Refine into a new model, fail loud on unknown transitions, …
krickert Jul 17, 2026
09e82da
depparse: Restore the final-sigma rule in vocabulary normalization an…
krickert Jul 17, 2026
03a005d
depparse: Cache hidden-layer contributions per feature pair on frozen…
krickert Jul 17, 2026
299b16f
depparse: Validate relation labels through StringUtil.isBlank
krickert Jul 17, 2026
0e13326
dependency: Add a dependency parser manual chapter with a mirror-test…
krickert Jul 20, 2026
ef853a5
depparse: Address review: decode outcomes once, fold duplication, nam…
krickert Jul 28, 2026
89825ab
depparse: Address review: cite specs and document private helpers
krickert Aug 8, 2026
e69d298
OPENNLP-547: Parse CoNLL-U fields without regex
krickert Aug 22, 2026
0e610d7
depparse: DependencyAnnotator, the container's first graph-shaped layer
krickert Jul 15, 2026
c174f7b
depparse: Pipeline and contract tests for the dependency annotator, d…
krickert Jul 16, 2026
edaa98f
depparse: Parse each sentence separately in the dependency annotator
krickert Jul 16, 2026
4f264f9
depparse: Validate the parsed graph size and pin the annotator's walk…
krickert Jul 17, 2026
fcf1571
depparse: Mint the dependencies layer key in the toolkit namespace
krickert Jul 17, 2026
f7a1d9e
dependency: Document the dependency annotator with a mirror-tested ex…
krickert Jul 20, 2026
8aac58b
dependency: Align annotator programlisting CDATA with sibling listings
krickert Jul 20, 2026
3fa4412
depparse: Address review: absent-versus-empty layers, javadoc and tes…
krickert Jul 28, 2026
d17b74a
dependency: Cite the annotator pipeline test in the manual section
krickert Aug 8, 2026
e380352
OPENNLP-1888: Move concrete annotators to runtime
krickert Aug 26, 2026
22780da
OPENNLP-1888: Place annotators with their components
krickert Aug 26, 2026
5115aaa
depparse: Arc-standard dependency parser skeleton with CoNLL-U reader…
krickert Jul 15, 2026
f5c11dc
depparse: DependencyModel persistence and a gated Universal Dependenc…
krickert Jul 15, 2026
1d8c366
depparse: Read CoNLL-U raw so multiword-token sentences keep their de…
krickert Jul 15, 2026
dc070db
depparse: Valency and child features for the transition classifier
krickert Jul 15, 2026
282e886
depparse: Pure-Java feedforward neural tier, training and inference i…
krickert Jul 15, 2026
70987c1
depparse: Pretrained word embedding seeding for the feedforward trainer
krickert Jul 15, 2026
b0a3ed6
depparse: Beam search decoding for the feedforward parser
krickert Jul 15, 2026
2be2809
depparse: Global refinement with beam search and early update
krickert Jul 15, 2026
d1334ad
depparse: State, transition, usage, and edge-case tests; expanded tra…
krickert Jul 16, 2026
a70ae38
depparse: Document treebank acquisition for the gated evaluation with…
krickert Jul 16, 2026
e48a5c9
depparse: Refine into a new model, fail loud on unknown transitions, …
krickert Jul 17, 2026
e83f03c
depparse: Restore the final-sigma rule in vocabulary normalization an…
krickert Jul 17, 2026
c8584a2
depparse: Cache hidden-layer contributions per feature pair on frozen…
krickert Jul 17, 2026
9eb6c1b
depparse: Validate relation labels through StringUtil.isBlank
krickert Jul 17, 2026
d894cfc
dependency: Add a dependency parser manual chapter with a mirror-test…
krickert Jul 20, 2026
e15e35e
depparse: Address review: decode outcomes once, fold duplication, nam…
krickert Jul 28, 2026
62a412d
depparse: Address review: cite specs and document private helpers
krickert Aug 8, 2026
84f059e
OPENNLP-547: Parse CoNLL-U fields without regex
krickert Aug 22, 2026
651e9ca
depparse: DependencyAnnotator, the container's first graph-shaped layer
krickert Jul 15, 2026
716e7f3
depparse: Pipeline and contract tests for the dependency annotator, d…
krickert Jul 16, 2026
96fb606
depparse: Parse each sentence separately in the dependency annotator
krickert Jul 16, 2026
9bb5ff3
depparse: Validate the parsed graph size and pin the annotator's walk…
krickert Jul 17, 2026
6464aaa
depparse: Mint the dependencies layer key in the toolkit namespace
krickert Jul 17, 2026
4b05301
dependency: Document the dependency annotator with a mirror-tested ex…
krickert Jul 20, 2026
367038f
dependency: Align annotator programlisting CDATA with sibling listings
krickert Jul 20, 2026
567cf8f
depparse: Address review: absent-versus-empty layers, javadoc and tes…
krickert Jul 28, 2026
48137b9
dependency: Cite the annotator pipeline test in the manual section
krickert Aug 8, 2026
2013861
OPENNLP-1919: Update annotator imports
krickert Aug 26, 2026
4d8ec02
OPENNLP-547: Reconcile branch after document API cascade
krickert Aug 30, 2026
63fee71
OPENNLP-1919: Cascade updated dependency parser branch
krickert Aug 30, 2026
3b438c3
OPENNLP-1919: Reconcile branch after dependency stack cascade
krickert Aug 30, 2026
65c215b
OPENNLP-1888: Add a ChunkerAnnotator providing a chunks layer
krickert Aug 30, 2026
1b24dad
OPENNLP-1888: Add a ParserAnnotator providing a phrases layer
krickert Aug 30, 2026
e37f9c1
OPENNLP-1888: List the chunker and parser adapters in the manual
krickert Aug 30, 2026
71da201
OPENNLP-547: Merge updated document container branch
krickert Aug 30, 2026
75bf330
OPENNLP-1919: Cascade updated dependency parser branch
krickert Aug 30, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package opennlp.tools.depparse;

import opennlp.tools.util.StringUtil;

/**
* One labeled edge of a {@link DependencyGraph}: the token at {@code dependent} is governed
* by the token at {@code head} under the given {@code relation}.
*
* <p>Indices are zero-based positions in the token array the graph was built over. A
* {@code head} of {@link #ROOT_HEAD} marks the dependent as the sentence root, which is
* attached to the artificial root node rather than to another token.</p>
*
* @param head The zero-based index of the governing token, or {@link #ROOT_HEAD} when the
* dependent is the sentence root.
* @param dependent The zero-based index of the governed token.
* @param relation The dependency relation label, for example {@code nsubj}.
*
* @since 3.0.0
*/
public record DependencyArc(int head, int dependent, String relation) {

/**
* The {@code head} value marking an arc from the artificial root node.
*/
public static final int ROOT_HEAD = -1;

/**
* Validates the arc invariants.
*
* @throws IllegalArgumentException Thrown if {@code dependent} is negative, {@code head}
* is less than {@link #ROOT_HEAD}, the arc is a self-loop, or {@code relation}
* is {@code null} or blank.
*/
public DependencyArc {
if (dependent < 0) {
throw new IllegalArgumentException("dependent must not be negative: " + dependent);
}
if (head < ROOT_HEAD) {
throw new IllegalArgumentException("head must be a token index or ROOT_HEAD: " + head);
}
if (head == dependent) {
throw new IllegalArgumentException("arc must not be a self-loop: " + head);
}
if (relation == null || StringUtil.isBlank(relation)) {
throw new IllegalArgumentException("relation must not be null or blank");
}
}
}
190 changes: 190 additions & 0 deletions opennlp-api/src/main/java/opennlp/tools/depparse/DependencyGraph.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,190 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package opennlp.tools.depparse;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;

import opennlp.tools.util.StringUtil;

/**
* An immutable dependency tree over one sentence: for every token, the index of its head
* and the label of the relation to that head.
*
* <p>Token indices are zero-based positions in the sentence the graph was built for.
* Exactly one token carries the head value {@link DependencyArc#ROOT_HEAD}, marking it as
* the sentence root. Instances are immutable and safe to share between threads.</p>
*
* @since 3.0.0
*/
public final class DependencyGraph {

private final int[] heads;
private final String[] relations;

/**
* Wraps already validated arrays; instances are created through {@link #of}.
*/
private DependencyGraph(int[] heads, String[] relations) {
this.heads = heads;
this.relations = relations;
}

/**
* Creates a {@link DependencyGraph} from parallel head and relation arrays.
*
* @param heads For each token, the zero-based index of its head token, or
* {@link DependencyArc#ROOT_HEAD} for the sentence root. Must not be
* {@code null} or empty, every value must be a valid token index or
* {@link DependencyArc#ROOT_HEAD}, no token may head itself, and exactly
* one token must be the root.
* @param relations For each token, the label of the relation to its head. Must not be
* {@code null}, must have the same length as {@code heads}, and no
* entry may be {@code null} or blank.
* @return A validated {@link DependencyGraph}. Never {@code null}.
* @throws IllegalArgumentException Thrown if any of the above constraints is violated.
*/
public static DependencyGraph of(int[] heads, String[] relations) {
if (heads == null || relations == null) {
throw new IllegalArgumentException("heads and relations must not be null");
}
if (heads.length == 0) {
throw new IllegalArgumentException("a dependency graph needs at least one token");
}
if (heads.length != relations.length) {
throw new IllegalArgumentException("heads and relations must have the same length: "
+ heads.length + " != " + relations.length);
}
int roots = 0;
for (int i = 0; i < heads.length; i++) {
if (heads[i] == DependencyArc.ROOT_HEAD) {
roots++;
} else if (heads[i] < 0 || heads[i] >= heads.length) {
throw new IllegalArgumentException("head of token " + i
+ " is out of range: " + heads[i]);
} else if (heads[i] == i) {
throw new IllegalArgumentException("token " + i + " must not head itself");
}
if (relations[i] == null || StringUtil.isBlank(relations[i])) {
throw new IllegalArgumentException("relation of token " + i + " must not be blank");
}
}
if (roots != 1) {
throw new IllegalArgumentException("expected exactly one root, found " + roots);
}
return new DependencyGraph(heads.clone(), relations.clone());
}

/**
* @return The number of tokens the graph spans.
*/
public int size() {
return heads.length;
}

/**
* Retrieves the head of a token.
*
* @param index The zero-based token index. Must be within {@code [0, size())}.
* @return The zero-based index of the head token, or {@link DependencyArc#ROOT_HEAD}
* when the token is the sentence root.
* @throws IllegalArgumentException Thrown if {@code index} is out of range.
*/
public int headOf(int index) {
checkIndex(index);
return heads[index];
}

/**
* Retrieves the relation label of a token.
*
* @param index The zero-based token index. Must be within {@code [0, size())}.
* @return The label of the relation between the token and its head. Never {@code null}.
* @throws IllegalArgumentException Thrown if {@code index} is out of range.
*/
public String relationOf(int index) {
checkIndex(index);
return relations[index];
}

/**
* @return The zero-based index of the sentence root token.
*/
public int root() {
for (int i = 0; i < heads.length; i++) {
if (heads[i] == DependencyArc.ROOT_HEAD) {
return i;
}
}
throw new IllegalStateException("graph invariant violated: no root present");
}

/**
* @return All arcs of the graph in token order, one per token. Never {@code null}.
*/
public List<DependencyArc> arcs() {
final List<DependencyArc> arcs = new ArrayList<>(heads.length);
for (int i = 0; i < heads.length; i++) {
arcs.add(new DependencyArc(heads[i], i, relations[i]));
}
return Collections.unmodifiableList(arcs);
}

/**
* Rejects a token index outside {@code [0, size())}.
*
* @param index The zero-based token index to check.
* @throws IllegalArgumentException Thrown if {@code index} is out of range.
*/
private void checkIndex(int index) {
if (index < 0 || index >= heads.length) {
throw new IllegalArgumentException("token index out of range: " + index
+ ", size: " + heads.length);
}
}

@Override
public boolean equals(Object obj) {
if (this == obj) {
return true;
}
if (!(obj instanceof DependencyGraph other)) {
return false;
}
return Arrays.equals(heads, other.heads) && Arrays.equals(relations, other.relations);
}

@Override
public int hashCode() {
return 31 * Arrays.hashCode(heads) + Arrays.hashCode(relations);
}

@Override
public String toString() {
final StringBuilder sb = new StringBuilder();
for (int i = 0; i < heads.length; i++) {
if (i > 0) {
sb.append(' ');
}
sb.append(i).append("<-").append(heads[i]).append(':').append(relations[i]);
}
return sb.toString();
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package opennlp.tools.depparse;

/**
* The interface for dependency parsers, which assign every token of a sentence a syntactic
* head and a relation label, forming a single-rooted tree over the sentence.
*
* <p>Dependency parsing complements the constituency {@link opennlp.tools.parser.Parser}:
* where a constituency parse groups tokens into nested phrases, a dependency parse links
* each token directly to the token it modifies. The result is a {@link DependencyGraph}
* whose indices refer back to the input token array, so spans computed for those tokens
* remain valid for the parse.</p>
*
* <p>Thread safety is implementation specific.</p>
*
* @see DependencyGraph
* @since 3.0.0
*/
public interface DependencyParser {

/**
* Parses a sentence into a {@link DependencyGraph}.
*
* @param tokens The tokens of one sentence. Must not be {@code null} and must contain
* at least one token.
* @param tags The part-of-speech tags aligned with {@code tokens}. Must not be
* {@code null} and must have the same length as {@code tokens}.
* @return A {@link DependencyGraph} over the given tokens. Never {@code null}.
* @throws IllegalArgumentException Thrown if {@code tokens} or {@code tags} is
* {@code null}, empty, or of mismatched length.
*/
DependencyGraph parse(String[] tokens, String[] tags);
}
Loading