tp_run() fails at large particle counts: R character strings are limited to 2^31-1 bytes

[maribraga]

Environment

• treepplr 0.14.0
• tpplc 0.4, method = "is"

Bug

At high particle counts, tp_run() fails outright:

out <- tp_run(sampler, data_path, particles = 3000000L)
#> Error in readLines(output_path) :
#>   R character strings are limited to 2^31-1 bytes

(A run at 1,000,000 particles on the same model/data succeeded and returned normally, so this is specifically a scaling limit, not a one-off.)

Root cause

tp_run <- function (compiled_model, data, dir = NULL, out_file_name = "out", ...) {
    ...
    system(command)
    json_out <- lapply(readLines(output_path), jsonlite::fromJSON, simplifyVector = FALSE)
    return(json_out)
}

tpplc writes one JSON line per sweep to output_path; readLines() reads the entire line into a single R character string before jsonlite::fromJSON parses it. When a run has enough particles (each carrying its own full sample + weight), that single line can exceed R's per-string limit (2^31-1 bytes, ~2GB), and readLines() fails hard before any parsing happens.

Impact

This caps the usable particle count for method = "is"/SMC methods well below what might otherwise be computationally tractable, for models whose per-particle output is nontrivial (e.g. returning a full sampled history/tree per particle rather than just a few scalars) — the line-size limit is hit long before compute time becomes the binding constraint.

Suggested fix

Read/parse the output incrementally rather than materializing the whole line as one R string — e.g. stream-parse with jsonlite::stream_in(), or have tpplc itself emit one JSON object per particle (newline-delimited JSON) rather than one giant array per sweep, so readLines() + per-line parsing stays within R's string-size limits regardless of particle count.