Skip to content

source_filters.md

Latest commit

 

History

History
872 lines (666 loc) · 32.7 KB

source_filters.md

File metadata and controls

872 lines (666 loc) · 32.7 KB
Raw

Source Filters Implementation Design

Overview

This document describes the design for supporting Perl source filters in PerlOnJava, specifically enabling modules like Filter::Simple and Filter::Util::Call to work when filters are installed via use Module qw(:tag) statements.

Problem Statement

Current Limitation

PerlOnJava tokenizes the entire source file upfront before any code executes. This creates a fundamental incompatibility with Perl source filters:

use Log::Log4perl qw(:resurrect);  # Installs filter in import()

###l4p DEBUG "hidden logging";  # Should be transformed to: DEBUG "hidden logging";
print "hello\n";

What happens in PerlOnJava:

  1. Lexer tokenizes all source (including ###l4p comments - unchanged)
  2. Parser builds AST
  3. use Log::Log4perl executes, calls import(':resurrect')
  4. import() installs the filter via Filter::Util::Call::filter_add()
  5. Problem: Source is already tokenized - filter has nothing to filter!

What happens in standard Perl:

  1. Perl reads source incrementally (line by line or block by block)
  2. use Log::Log4perl is parsed and executed immediately
  3. import() installs the filter
  4. Perl continues reading source through the filter
  5. The filter transforms ###l4p DEBUG "first"; → DEBUG "first";
  6. Transformed source is then tokenized and compiled

Real-World Impact

Modules that rely on source filters:

  • Log::Log4perl (:resurrect tag) - Test: t/049Unhide.t
  • Filter::Simple - Simplified filter interface
  • Switch (deprecated but common)
  • Lingua::Romana::Perligata - Perl in Latin
  • Acme::* modules

Current Implementation

FilterUtilCall.java

PerlOnJava has a partial implementation in FilterUtilCall.java:

public class FilterUtilCall extends PerlModuleBase {
    // Thread-local filter stack
    private static final ThreadLocal<FilterContext> filterContext;
    
    // XS function implementations
    public static RuntimeList real_import(...)  // Called by filter_add()
    public static RuntimeList filter_read(...)  // Read next chunk
    public static RuntimeList filter_del(...)   // Remove filter
    
    // Apply installed filters to source
    public static String applyFilters(String sourceCode) {...}
    
    // Workaround: preprocess BEGIN blocks containing filter_add
    public static String preprocessWithBeginFilters(String sourceCode) {...}
}

preprocessWithBeginFilters() Workaround

The current workaround handles explicit BEGIN { filter_add(...) } blocks:

  1. Scans source for BEGIN { ... filter ... }
  2. Extracts and executes the BEGIN block
  3. Applies any installed filters to remaining source
  4. Returns filtered source for parsing

Limitation: This only works for explicit BEGIN blocks, NOT for filters installed via use Module qw(:tag) because:

  • The use statement is not recognized as a filter installer
  • The import() method that installs the filter is called after tokenization

Perl Source Filter Semantics

Filter::Util::Call API

# Install a filter (usually in import())
filter_add($coderef_or_object);

# Inside the filter: read next chunk into $_
$status = filter_read();      # Line mode
$status = filter_read($size); # Block mode

# Remove current filter
filter_del();

# Return values:
# > 0 : More data available
# = 0 : EOF
# < 0 : Error

Filter Types

  1. Closure Filter: Anonymous sub passed to filter_add()

    filter_add(sub {
    my $status = filter_read();
    s/old/new/g if $status > 0;
    return $status;
});

  2. Method Filter: Blessed object with filter() method

    filter_add(bless {}, $class);
# Calls $obj->filter() repeatedly

Filter::Simple API

Higher-level interface that collects all source then transforms:

use Filter::Simple;

FILTER { s/BANG/die/g };  # Transform all source

FILTER_ONLY
    code => sub { ... },      # Transform code only
    string => sub { ... };    # Transform strings only

Proposed Solution: Token Rejoin and Re-tokenize

Key Insight

The Lexer produces an array of LexerToken objects, each with a text field containing the original characters. These tokens can be rejoined back into source text by concatenating their text values, then re-tokenized after filtering.

This is much simpler than incremental parsing because:

  1. The parser already has access to parser.tokens (the token array)
  2. Tokens can be rejoined from any position: tokens[i].text + tokens[i+1].text + ...
  3. After filtering, we just re-tokenize and replace the remaining tokens

Architecture Overview

┌─────────────────────────────────────────────────────────────────┐
│                 Parser processes use statement                   │
│                 (tokenIndex points to position after use)        │
└─────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│                 import() installs a filter                       │
│                 FilterState.markFilterInstalled()                │
└─────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│              Check: FilterState.wasFilterInstalled()?            │
└─────────────────────────────────────────────────────────────────┘
                                │ Yes
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│  1. Rejoin remaining tokens: tokens[i..end].map(t => t.text)    │
│  2. Apply filters: FilterUtilCall.applyFilters(rejoined)        │
│  3. Re-tokenize: Lexer.tokenize(filtered)                       │
│  4. Replace: parser.tokens = tokens[0..i-1] + newTokens         │
└─────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│                 Continue parsing with new tokens                 │
└─────────────────────────────────────────────────────────────────┘

Key Components

1. FilterState Tracking

// In FilterUtilCall.java or new FilterState.java
public class FilterState {
    // Track if filters were installed during current use statement
    private static final ThreadLocal<Boolean> filterInstalledDuringUse = 
        ThreadLocal.withInitial(() -> false);
    
    public static void markFilterInstalled() {
        filterInstalledDuringUse.set(true);
    }
    
    public static boolean wasFilterInstalled() {
        boolean result = filterInstalledDuringUse.get();
        filterInstalledDuringUse.set(false);  // Reset
        return result;
    }
}

2. Modified filter_add()

In FilterUtilCall.real_import():

public static RuntimeList real_import(RuntimeArray args, int ctx) {
    // ... existing code to add filter to stack ...
    
    // Mark that a filter was installed
    FilterState.markFilterInstalled();
    
    return scalarTrue.getList();
}

3. Token Rejoin Utility

// In Parser.java or new utility class
public static String rejoinTokens(List<LexerToken> tokens, int fromIndex) {
    StringBuilder sb = new StringBuilder();
    for (int i = fromIndex; i < tokens.size(); i++) {
        sb.append(tokens.get(i).text);
    }
    return sb.toString();
}

4. Post-import Filter Check

In StatementParser.parseUseDeclaration(), after calling import():

// After import() completes (around line 708)
if (FilterState.wasFilterInstalled()) {
    // A filter was installed - rejoin remaining tokens, filter, and re-tokenize
    int currentPos = parser.tokenIndex;
    
    // Step 1: Rejoin remaining tokens back to source text
    String remainingSource = rejoinTokens(parser.tokens, currentPos);
    
    // Step 2: Apply the installed filters
    String filteredSource = FilterUtilCall.applyFilters(remainingSource);
    
    // Step 3: Re-tokenize the filtered source
    Lexer lexer = new Lexer(filteredSource);
    List<LexerToken> newTokens = lexer.tokenize();
    
    // Step 4: Replace remaining tokens with filtered tokens
    // Keep tokens[0..currentPos-1], append newTokens
    List<LexerToken> updatedTokens = new ArrayList<>(
        parser.tokens.subList(0, currentPos));
    updatedTokens.addAll(newTokens);
    parser.tokens = updatedTokens;
    
    // Clear the filter after applying (it's been consumed)
    FilterUtilCall.clearFilters();
}

Algorithm

function parseUseDeclaration(parser):
    1. Parse and execute `use Module qw(args)`
       - This calls require() then import()
       - import() may call filter_add()
    
    2. After import() returns, check FilterState.wasFilterInstalled()
    
    3. If a filter was installed:
       a. currentPos = parser.tokenIndex  // Position after use statement
       b. remainingSource = rejoinTokens(parser.tokens, currentPos)
       c. filteredSource = FilterUtilCall.applyFilters(remainingSource)
       d. newTokens = Lexer.tokenize(filteredSource)
       e. parser.tokens = parser.tokens[0..currentPos-1] + newTokens
       f. FilterUtilCall.clearFilters()  // Filter consumed
    
    4. Continue parsing (with potentially filtered tokens)

Example Walkthrough

Given this source:

use Log::Log4perl qw(:resurrect);
###l4p DEBUG "hello";
print "world\n";

Before filtering:

tokens = [
  "use", " ", "Log::Log4perl", " ", "qw(:resurrect)", ";", "\n",
  "###l4p DEBUG \"hello\";", "\n",
  "print", " ", "\"world\\n\"", ";", "\n"
]

After use statement executes:

  1. import(':resurrect') installs a filter that removes ###l4p prefixes
  2. FilterState.wasFilterInstalled() returns true
  3. parser.tokenIndex is at position 7 (after the semicolon and newline)

Rejoin remaining tokens:

remainingSource = "###l4p DEBUG \"hello\";\nprint \"world\\n\";\n"

Apply filter:

filteredSource = "DEBUG \"hello\";\nprint \"world\\n\";\n"

Re-tokenize and replace:

newTokens = ["DEBUG", " ", "\"hello\"", ";", "\n", "print", " ", "\"world\\n\"", ";", "\n"]
parser.tokens = tokens[0..6] + newTokens

Continue parsing with the filtered tokens.

Implementation Phases

Phase 1: Filter State Tracking (Simple)

Add state tracking to know when a filter was installed:

  1. Add filterInstalledDuringUse flag to FilterUtilCall.java
  2. Set flag in real_import() when filter is added
  3. Add wasFilterInstalled() / clearFilterInstalledFlag() methods

Files to modify:

  • FilterUtilCall.java - Add state tracking

Estimated effort: ~30 minutes

Phase 2: Token Rejoin and Re-tokenize (Core Implementation)

Implement the token rejoin, filter, and re-tokenize logic:

  1. Add rejoinTokens(List<LexerToken> tokens, int fromIndex) utility
  2. After import() in parseUseDeclaration(), check if filter was installed
  3. If yes: rejoin → filter → re-tokenize → replace tokens
  4. Clear filter after applying

Files to modify:

  • StatementParser.java - Add post-import filter check (~20 lines)
  • FilterUtilCall.java - May need to expose applyFilters() better

Estimated effort: ~2-3 hours

Phase 3: Method Filters Support

Currently applyFilters() only supports closure filters. Add method filter support:

if (!isCodeRef.getBoolean()) {
    // Method filter: call $obj->filter()
    RuntimeScalar filterMethod = Universal.can(filterObj, "filter");
    // Call repeatedly until returns <= 0
}

Files to modify:

  • FilterUtilCall.java - Add method filter support in applyFilters()

Estimated effort: ~1-2 hours

Phase 4: Error Handling and Edge Cases

Handle edge cases:

  1. Multiple use statements that install filters
  2. Nested module loading where inner module installs filter
  3. no Module removing filters
  4. Error reporting with correct line numbers after re-tokenization

Files to modify:

  • Various - depends on edge cases discovered

Estimated effort: ~2-4 hours

Alternative Approaches (For Reference)

Approach A: Pre-scan for Filter-Installing Modules

Before parsing, scan source for known filter-installing modules:

  1. Build a list of modules known to install filters (Log::Log4perl with :resurrect, etc.)
  2. If source contains use KnownFilterModule, use two-pass compilation
  3. First pass: just execute use statements
  4. Second pass: parse filtered source

Pros: No runtime overhead for code without filters Cons: Requires maintaining a list of known filter modules

Approach B: Streaming Lexer (Most Correct, Most Complex)

Modify Lexer to read source incrementally through filter chain:

  1. Lexer calls filter_read() for each chunk
  2. Filters transform source during reading
  3. Exactly matches Perl's behavior

Pros: Correct semantics, handles all edge cases Cons: Major Lexer rewrite, affects all code paths

Risks and Considerations

Performance

  • Filters add overhead to compilation
  • Re-tokenization for filtered source adds latency
  • Consider caching filtered source for repeated use

Edge Cases

  1. Nested use statements: Module A uses Module B which installs a filter
  2. BEGIN blocks in filtered code: Must be executed correctly
  3. Filter affecting use statement itself: Pathological but possible
  4. Multiple filters: Must apply in correct order (LIFO stack)

Compatibility

  • Must maintain backward compatibility for code without filters
  • Filter installation via @INC hooks (less common)
  • eval and do with filters

Testing Strategy

Unit Tests

# Test filter installation via use
use FilterModule qw(:filter_tag);
# ... code that should be transformed ...

# Test Filter::Simple
use SimpleFilter;
FILTER { s/X/Y/g };
print "X";  # Should print Y

Integration Tests

  • Log::Log4perl :resurrect tag (t/049Unhide.t)
  • Custom filter modules

Related Documentation

Progress Tracking

Current Status: Implementation Complete (Closure Filters)

Completed

  • Research Perl source filter semantics
  • Analyze current PerlOnJava implementation
  • Identify root cause of t/049Unhide.t failure
  • Design token rejoin/re-tokenize solution
  • Phase 1: Add filter state tracking to FilterUtilCall.java (2026-03-27)
    • Added filterInstalledDuringUse ThreadLocal flag
    • Added markFilterInstalled(), wasFilterInstalled(), hasActiveFilters() methods
    • Modified real_import() to mark when filter is installed
  • Phase 2: Implement token rejoin and re-tokenize in StatementParser.java (2026-03-27)
    • Added applySourceFilterToRemainingTokens() method
    • Integrated with parseUseDeclaration() to check for filters after import()
    • Added updateTokens() method to ErrorMessageUtil
    • Fixed EOF token handling (skip EOF tokens when rejoining to avoid garbage characters)
  • Phase 3: Test with Log::Log4perl :resurrect tag - PASSED

Remaining Work

  • Phase 4: Add method filter support (currently returns original source for method filters)
  • Add debug environment variable documentation (JPERL_FILTER_DEBUG=1)
  • Phase 5: Fix FILTER_ONLY @transforms issue in Java instead of patching Filter::Simple (see below)

Known Issues

FILTER_ONLY @transforms Scope Issue (2026-03-27)

Problem: When multiple filter modules using FILTER_ONLY are loaded in sequence, the second filter's $multitransform closure incorrectly includes transforms from the first module.

Root Cause: In Filter::Simple, @transforms is a package variable. In native Perl, this works because filters process source incrementally - each filter completes before the next filter module is loaded. In PerlOnJava, we tokenize upfront then apply filters, so multiple filter modules may be loaded before any filter runs, causing @transforms to accumulate transforms from different modules.

Current Fix: Patched Filter::Simple.pm to make @transforms lexical in FILTER_ONLY:

sub FILTER_ONLY {
    my $caller = caller;
    my @transforms;  # Made lexical instead of package-scoped
    ...
}

TODO - Proper Java-side Fix: The ideal solution would be to fix this in PerlOnJava's module loading code:

  1. Before loading a module that may use FILTER_ONLY, save @Filter::Simple::transforms
  2. Clear @Filter::Simple::transforms
  3. After module loading completes, restore the saved value

This would allow using unmodified upstream Filter::Simple. The challenge is detecting which modules will use FILTER_ONLY before loading them. Possible approaches:

  • Clear @Filter::Simple::transforms before every require (may have side effects)
  • Track filter module loading depth and isolate transforms per level
  • Hook into Filter::Simple's FILTER_ONLY to auto-reset before each call

Files affected by current fix:

  • src/main/perl/lib/Filter/Simple.pm (marked as protected: true in config.yaml)

Files Modified

  • src/main/java/org/perlonjava/runtime/perlmodule/FilterUtilCall.java
  • src/main/java/org/perlonjava/frontend/parser/StatementParser.java
  • src/main/java/org/perlonjava/runtime/runtimetypes/ErrorMessageUtil.java

Open Questions (Resolved)

  • Line number tracking after re-tokenization: We update ErrorMessageUtil with new tokens
  • How to handle EOF tokens: Skip them when rejoining (they contain invalid characters)

Phase 6 — Per-Compilation-Unit Scoping (2026-04-28)

The leak

FilterUtilCall keeps two pieces of state in ThreadLocals:

field purpose
filterContext.filterStack stack of currently-installed source filters
filterInstalledDuringUse one-shot flag set by real_import(), consumed by wasFilterInstalled() so the parser knows to re-tokenize after a use

Both were process-global per thread — not scoped to the file currently being compiled. A filter installed by an outer use Foo (whose import() runs while the parent file is still being parsed) leaked into whatever module Foo::import() happened to require next.

The most visible victim was Spiffy (and therefore everything that builds on Spiffy: Test::Base, the bulk of the YAML test suite, Switch, Filter::Simple users, …):

package Test::Base;
use Spiffy -Base;                         # installs filter via filter_add
field _filters => [qw(norm trim)];        # ← Spiffy's filter is what makes
                                          #   `field _filters => [...]` parse

What happened:

  1. Spiffy::import called Filter::Util::Call::filter_add → real_import() pushed the filter onto the stack and set filterInstalledDuringUse = true.
  2. Spiffy::import then called Exporter::export(...) which required Exporter::Heavy.pm.
  3. The nested compilation of Exporter::Heavy.pm encountered its own use statements; each one ran wasFilterInstalled() which returned true (Spiffy's flag, set just earlier) and triggered applySourceFilterToRemainingTokens() against Exporter::Heavy.pm's source.
  4. Spiffy's filter — which injects my $self = shift; after every sub …{ — rewrote Exporter::Heavy.pm (visible as the warning "my" variable $self masks earlier declaration … at Exporter/Heavy.pm line 237). clearFilters() then emptied the stack.
  5. Control returned to parsing Test::Base.pm at the next token after use Spiffy -Base;. The flag was now false, the stack was empty, and field _filters => [qw(norm trim)] was parsed without Spiffy's filter applied → syntax error … near "=> [qw" at Test/Base.pm line 53.

Real Perl semantics

Source filters are scoped per compilation unit (PL_compiling / PL_rsfp_filters): each require, do FILE, or string-eval starts with its own initially-empty filter chain, and the outer chain is restored when the nested compilation finishes. Spiffy itself relies on this — line 82 of Spiffy.pm reads:

spiffy_filter()
  if ($args->{-selfless} or $args->{-Base}) and
     not $filtered_files->{(caller($stack_frame))[1]}++;

i.e. "have I already filtered this caller's file?". The filter is intended to be scoped to that file.

Fix

Snapshot/reset/restore the filter state at the ModuleOperators.do_file boundary — i.e. exactly when a require or do FILE switches to compiling a different source file.

// FilterUtilCall.java
public static class FilterStateSnapshot {
    final RuntimeList filterStack;
    final boolean installedDuringUse;
    ...
}

public static FilterStateSnapshot saveAndResetFilterState() {
    FilterContext context = filterContext.get();
    FilterStateSnapshot snapshot =
            new FilterStateSnapshot(context.filterStack,
                                    filterInstalledDuringUse.get());
    context.filterStack = new RuntimeList();
    filterInstalledDuringUse.set(false);
    ...
    return snapshot;
}

public static void restoreFilterState(FilterStateSnapshot snapshot) {
    if (snapshot == null) return;
    FilterContext context = filterContext.get();
    context.filterStack = snapshot.filterStack;
    filterInstalledDuringUse.set(snapshot.installedDuringUse);
    ...
}

// ModuleOperators.do_file
FilterUtilCall.FilterStateSnapshot filterSnapshot =
        FilterUtilCall.saveAndResetFilterState();
try {
    // existing require/do compilation body ...
} finally {
    FilterUtilCall.restoreFilterState(filterSnapshot);
}

This single change unblocked 27 of 35 previously-blocked tests in the bundled YAML-1.31 distribution, plus everything else that uses Spiffy / Test::Base / Filter::Simple underneath.

Why do_file and not executePerlCode?

executePerlCode is the broader funnel — it covers require/do and string-eval and the synthetic compile inside preprocessWithBeginFilters. Wiring there would seem more "thorough", but it has a subtle problem:

preprocessWithBeginFilters deliberately runs a BEGIN { filter_add(...) } prefix through executePerlCode so that the filter installed inside the BEGIN survives back to the caller and can be applied to the parent file's remaining source. A save/reset/restore wrapper around executePerlCode would undo that install before applyFilters() could use it — the recursive test perl5_t/t/op/incfilter.t then regresses from 143/153 to 14/153 (file-handle / coderef source filters from @INC break).

Working around that with a one-shot "skip save/restore" flag threaded through preprocessWithBeginFilters works, but it's ad-hoc.

do_file's placement is cleaner: it sits outside executePerlCode, so preprocessWithBeginFilters' recursive executePerlCode call (which doesn't go through do_file) is naturally unaffected. Per-compilation-unit scoping for the require/do path is exactly what we need to fix the Spiffy bug, and nothing more.

eval STRING is not wrapped — but the filter chain installed by an outer use Foo is applied to the parent file's remaining source tokens before any eval STRING runs at runtime, so an unprotected eval cannot leak into or out of an enclosing parse in any way that causes the Spiffy class of bug.

Regression test

src/test/resources/unit/source_filter_scope.t reproduces the exact bug pattern (without depending on any external CPAN module):

  • defines an inline InlineFilter package whose import() calls filter_add and then requires Cwd (mimicking what Spiffy::import does — filter_add followed by Exporter::export -> require Exporter::Heavy),
  • asserts that the filter is correctly applied to the parent file's remaining tokens (test 2),
  • asserts that Cwd itself was unaffected by the filter (test 3 — Cwd::cwd() works),
  • asserts the filter doesn't leak past the eval STRING (test 4).

Confirmed catches the bug: with saveAndResetFilterState / restoreFilterState neutralised to no-ops, test 2 fails with got: 'REPLACEME', expected: 'ok_marker' (the filter was consumed by Cwd's parsing instead of reaching the parent's source).

Phase 7 — do CODEREF and __FILE__ for filehandle/coderef do (2026-04-28)

Two related issues surfaced while bringing perl5_t/t/op/incfilter.t past the Spiffy regression. Both lived in ModuleOperators.do_file's source-generator paths.

Bug A — STORE called on user's tied $_

The do CODEREF generator loop did:

GlobalVariable.getGlobalVariable("main::_").set("");   // clear $_
RuntimeBase result = codeRef.apply(stateArgs, ...);    // call generator
String chunk = GlobalVariable.getGlobalVariable("main::_").toString();

When the user's generator tied $_ to an object with only TIESCALAR and FETCH (no STORE) — exactly the pattern in incfilter.t lines 261-268 — the next iteration's .set("") invoked the missing STORE and died with Can't locate object method "STORE" via package "main".

Real Perl handles this with local $_ in pp_require: each iteration gets its own fresh, untied $_; the caller's tied $_ is restored at end without ever being written.

Fix

// Each iteration: install a fresh untied scalar.
GlobalVariable.aliasGlobalVariable("main::_", new RuntimeScalar(""));
...
// At end (in finally): restore the caller's slot WITHOUT calling .set()
GlobalVariable.aliasGlobalVariable("main::_", savedDefaultVar);

aliasGlobalVariable swaps the slot's RuntimeScalar reference, matching local $_'s semantics exactly. No STORE is ever invoked on the user's scalar.

Bug B — __FILE__ NPE inside do FILEHANDLE / do CODEREF

actualFileName was only set in the do \$scalarref branch. The filehandle and code-ref branches left parsedArgs.fileName = null, so __FILE__ produced a StringNode with null value and crashed downstream with Cannot invoke "String.length()" because "node.value" is null.

Fix

Set actualFileName = fileName (the stringified GLOB(0x…) / CODE(0x…)) in both branches. Matches the regex assertion in incfilter.t:

like(__FILE__, qr/(?:GLOB|CODE)\(0x[0-9a-f]+\)/, "__FILE__ is valid");

Effect on op/incfilter.t

state result
master (before any of this work) 143/153
with Phase 6 fix only (Spiffy unblocked) 143/153 (regressed to 14/153, recovered with skipSaveRestore carve-out)
with Phase 6 + Phase 7 fixes 148/153

Known Residual: op/incfilter.t 148/153

Five cmp_ok calls expected by the script's hard-coded plan(tests => 153) never fire on PerlOnJava. All 148 that do fire pass; there are zero not ok lines.

Why the count varies

Most of the test count comes from cmp_ok calls inside two filter generators that run once per byte read:

# from prepend_block_counting_filter (lines 148-165)
while (--$count) {
    $_ = '';
    my $status = filter_read($amount);                # read 1 byte
    cmp_ok (length $_, '<=', $amount, "block mode works?");   # ← per byte
    $output .= $_;
    if ($status <= 0 or /\n/s) { ...; return $status; }
}

So the total ok count = (bytes through filter 1) + (bytes through filter 2) + (line count in filter 3) + fixed assertions. Real Perl's byte stream through these filters is 5 bytes longer than ours → 5 fewer cmp_ok invocations.

Where the bytes go missing

Counting label-by-label:

Label PerlOnJava Real Perl 5.42
block mode works? (43 + 51) 94 ~99
1 line at most? 8 8
You should see this line thrice 3 3
Upstream didn't alter existing data 4 4
Fixed pass / is / like / etc. 39 39
Total 148 153

Two structural mismatches account for the missing 5 bytes in the second prepend_block_counting_filter invocation (the s/s/ss/g; s/([\nS])/$1$1$1/g; return; array-form filter chain):

  1. Where preprocessWithBeginFilters splits the source. PerlOnJava cuts at the closing } of BEGIN { … } via a literal brace-match. Real Perl's tokenizer position when the BEGIN runs is just past the ; terminating the BEGIN statement — Perl's filter sees those few extra characters that PerlOnJava had already consumed before reaching the filter machinery.

  2. EOF read on the trailing newline. PerlOnJava's block-mode filter_read(1) returns the trailing newline and then immediately 0 on the next call; real Perl produces one more 0-length read at end-of-source before returning EOF. That's a +1 cmp_ok per filter invocation × 2 invocations = +2.

The 1 from #1 plus 2 from #2 plus minor \r\n vs \n framing in the second invocation accounts for the missing 5.

Why this isn't worth chasing

  • Zero not ok: every cmp_ok that ran passed. Nothing is incorrect — only the count of bytes reported differs.

  • The plan number 153 is a hard-coded count tied to one Perl implementation's filter byte-stream framing. Anything that intercepts filter_read differently (PerlOnJava, miniperl, alternative implementations) will produce a different count.

  • Aligning to exactly 153 requires either

    • reworking preprocessWithBeginFilters to find the tokenizer position the way Perl's lexer does instead of brace-matching (invasive — would re-tokenize the BEGIN prefix to know where the ; is), or
    • emitting a synthetic 0-byte filter_read cycle at EOF so the user filter can run one final cmp_ok before status=0 (changes applyFilters semantics for every filter).

    Both are big changes for cosmetic test-count parity. Current state — 148/153, 0 failures — is the right place to leave it.

What would actually improve correctness

If we ever invest in this area further, the meaningful work is:

  1. Make filter_read truly streaming, not a "split on \n then replay" emulation. Current implementation (FilterUtilCall.filter_read) splits the upstream source on (?<=\n) ahead of time and replays line-by-line; in block mode it concatenates lines until the requested byte count is hit. Filters that depend on partial-line state observe slightly different framing than real Perl.
  2. Drive the filter from the lexer, one chunk at a time, instead of applyFilters(entire-remaining-source) followed by re-tokenize. This matches Perl's PL_rsfp_filters model and removes the "rejoin tokens, filter, re-tokenize" round-trip.

Neither is needed for any currently-failing real-world module — included here as a roadmap, not a TODO.

Files

  • src/main/java/org/perlonjava/runtime/perlmodule/FilterUtilCall.java (Phase 6 FilterStateSnapshot + saveAndResetFilterState / restoreFilterState)
  • src/main/java/org/perlonjava/runtime/operators/ModuleOperators.java (Phase 6 try/finally wrapper around the require/do compile + Phase 7 local $_ semantics + __FILE__ for do FH / do CODE)
  • src/test/resources/unit/source_filter_scope.t (Phase 6 regression test)
  • src/test/resources/module/YAML/t/ (34 upstream YAML-1.31 tests unblocked by Phase 6)