feat: implement MIME type whitelist filtering in ParserBolt for Tika-…

[Riddhish1]

Fix parser.mimetype.whitelist evaluating against the HTTP response header instead of the detected content type

The problem

ParserBolt.execute() checks parser.mimetype.whitelist before parsing. When parse.Content-Type
is present in the metadata (written by JSoupParserBolt when detect.mimetype is true), it is used
for the whitelist check and also drives Tika's AutoDetectParser — the two are in agreement.

When parse.Content-Type is absent — because detect.mimetype is false, or because the topology
feeds ParserBolt directly without JSoupParserBolt upstream — the code fell back to the
Content-Type response header supplied by the fetched server:

// otherwise rely on what could have been obtained from HTTP
if (mimeType == null) {
    mimeType = metadata.getFirstValue(HttpHeaders.CONTENT_TYPE, this.protocolMDprefix);
}

The whitelist was then evaluated against this server-declared value, while Tika's
AutoDetectParser dispatched on the raw content bytes. A server can claim any MIME type in its
response header, so the two sources can disagree. In the worst case a server reports a whitelisted
type (e.g. application/vnd.openxmlformats-officedocument.wordprocessingml.document) while
serving an entirely different payload (e.g. HTML). The whitelist gate opened, and Tika parsed
whatever the bytes actually were.

The practical impact is limited in the common archetype setup because JSoupParserBolt runs ahead
of the Tika bolt with detection enabled, so parse.Content-Type is almost always present.
The gap opens in two real scenarios:

• detect.mimetype: false in the crawler configuration.
• A custom topology that wires FetcherBolt directly to ParserBolt without a JSoup stage.

In both cases the whitelist was not doing the job its name and the archetype documentation imply:
controlling which document types this bolt parses.

What this PR changes

ParserBolt.execute() — detect from bytes when parse.Content-Type is absent

When the metadata key parse.Content-Type is missing, the bolt now calls tika.detect() on the
content bytes before evaluating the whitelist, rather than trusting the server header:

if (mimeType == null) {
    String httpCTHint =
            metadata.getFirstValue(HttpHeaders.CONTENT_TYPE, this.protocolMDprefix);
    org.apache.tika.metadata.Metadata detectionMd = new org.apache.tika.metadata.Metadata();
    if (StringUtils.isNotBlank(httpCTHint)) {
        // pass the header as a hint only — detect() weighs it but bytes take precedence
        detectionMd.set(org.apache.tika.metadata.Metadata.CONTENT_TYPE, httpCTHint);
    }
    // pass the filename so detection matches what the parser dispatches on
    URL _url = URLUtil.toURL(url);
    detectionMd.set(TikaCoreProperties.RESOURCE_NAME_KEY, _url.getFile());
    mimeType = tika.detect(new ByteArrayInputStream(content), detectionMd);
    if (mimeType != null) {
        metadata.setValue("parse.Content-Type", mimeType);
    }
}

The HTTP response header is still passed to Tika as a hint. Content bytes take precedence when
the two disagree. The filename extracted from the URL is also passed as a hint — without it,
ambiguous bytes (e.g. plain text at a .html URL) can resolve differently in the whitelist check
vs. at parse dispatch time. The result is written back into parse.Content-Type so both the
whitelist gate and Tika's AutoDetectParser are bound to the same type.

Behaviour in the common case is unchanged. When parse.Content-Type is already present (the
normal path with JSoupParserBolt upstream), the new block is not entered.

Behaviour changes worth noting in release notes

Type mismatch rejection. Documents whose server-declared Content-Type matched the whitelist
but whose bytes are detected as a different type will now be rejected where they were
previously parsed. This is the correct outcome — the whitelist was not enforcing what operators
expected — but operators who relied on the previous behaviour should be aware.

Undetectable content (application/octet-stream) rejection. Previously, truly undetectable
binary content (no magic bytes Tika can match) would pass the whitelist if the server header
claimed a whitelisted type, because the whitelist checked that header. Now the whitelist checks
the byte-detected type, which for undetectable content is application/octet-stream. Unless the
whitelist explicitly includes application/octet-stream, such documents will be rejected. This is
the stricter and more correct behaviour, but operators should be aware the rejection boundary has
changed.

Restricting the parser set in tika-config.xml is a complementary defence in depth: it bounds
which parsers can be selected at all, regardless of what the whitelist or detection step resolves.

Tests

ParserBoltWhitelistDetectionTest (new, external/tika)

whitelistAppliesToTheDetectedType — reproduces the original bug:

• Whitelist: application/.+word.* (the pattern shipped by the archetypes).
• Server Content-Type header: application/vnd.openxmlformats-officedocument.wordprocessingml.document.
• Body bytes: plain HTML — <html><body><p>not a word document</p></body></html>.
• No parse.Content-Type in metadata (simulates a topology without JSoupParserBolt upstream).

Before this fix the bolt parsed the HTML and emitted a document. After this fix the bolt detects
text/html from the bytes, the whitelist does not match, and the tuple is emitted on the status
stream with Status.ERROR.

whitelistUsesPreexistingParsedContentType — sanity check for the unchanged common path:

• parse.Content-Type is set to text/html; charset=UTF-8 (as JSoupParserBolt would write it).
• Whitelist: text/html.*.
• Asserts that the document is accepted and no ERROR status is emitted.

filenameHintInfluencesDetection — pins the RESOURCE_NAME_KEY fix:

• URL: https://example.org/page.html (.html extension).
• Body: plain-text bytes with no HTML markup — ambiguous without the filename hint.
• No parse.Content-Type, no Content-Type header.
• Whitelist: text/html.*.

Without the filename hint the bytes alone resolve to text/plain and the whitelist rejects the
document. With the hint, Tika resolves text/html, the whitelist accepts it, and the detection
matches what the parser would dispatch on.

Verification

# install core (editorconfig check is a LF/CRLF issue on Windows; bypass validate phase)
mvn -pl core compiler:compile compiler:testCompile jar:jar install:install -DskipTests

# run the new tests
mvn -pl external/tika test -Dtest=ParserBoltWhitelistDetectionTest

[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS

Detected types printed during the test run:

detected type: text/html; charset=ISO-8859-1   ← test 1: HTML bytes correctly identified, Word header ignored
detected type: text/html                        ← test 3: plain bytes + .html filename → text/html
emitted documents: 0                            ← test 1: mismatched document rejected

The bolt rejects the mismatched document and the filename hint correctly resolves ambiguous content.

[dpol1]

You missed this: https://github.com/apache/stormcrawler/blame/e1d54a6572438a5518d772cb88ac51e8dada57fb/README.md#L43 - Format the code pls

[Riddhish1]

Fixed the formatting. Thanks for catching that!

You missed this: https://github.com/apache/stormcrawler/blame/e1d54a6572438a5518d772cb88ac51e8dada57fb/README.md#L43 - Format the code pls

[dpol1]

This closes the scenario from #2104 — server claims a whitelisted type, bytes say otherwise, doc rejected — and the new test pins exactly that. Rejection path and ack behaviour unchanged, and detection only runs when a whitelist is configured, so no cost for everyone else.

Two questions inline. One observation: undetectable content (application/octet-stream) is now rejected when a whitelist is set, where a whitelisted header used to let it through — stricter is right for this fix, but maybe worth a line in the PR description so the behaviour change is on record.

[dpol1]

The parse-time detection further down also gets the filename via RESOURCE_NAME_KEY, this one doesn't — so the two can still disagree (I could reproduce it with plain-text bytes and a .html URL: text/plain here, text/html at dispatch). Passing the same filename hint into detectionMd would make the check genuinely match what the parser dispatches on.

[Riddhish1]

updated detectionMd to pass TikaCoreProperties.RESOURCE_NAME_KEY from the URL as well, and added a test case verifying that ambiguous content (like plain text with a .html URL) resolves consistently

[dpol1]

Genuine question: is a pre-existing parse.Content-Type trusted by design? Coming from JSoupParserBolt's own byte detection that seems fine, but any other upstream writing a server-influenced value there skips the new check entirely. If it's intentional, worth a comment saying so.

[Riddhish1]

JSoupParserBolt.guessMimeType() does byte detection too (header is just a hint) so basically the value is trustworthy in the standard topology, a custom upstream writing a server copied value there would bypass it worth a comment will add it