Should errors in test files cause test execution to halt?

[kevinushey]

To reproduce:

R --vanilla -s -e 'x <- 42; package.skeleton("use.tinytest")'
cd use.tinytest
R --vanilla -s -e 'tinytest::setup_tinytest(getwd())'
echo "stop()" > inst/tinytest/test_aaa.R
R --vanilla -s -e 'tinytest::run_test_dir()'

I see:

$ R --vanilla -s -e 'tinytest::run_test_dir()'
Error in eval(expr, envir = e) : 
Calls: <Anonymous> -> lapply -> FUN -> eval -> eval
Execution halted

Could tinytest instead capture errors seen when executing test files, and report those and allow other tests to continue executing? (Or at least, report what file the error was caught in, or other diagnostic information?)

[markvanderloo]

It is probably doable. I do have my doubts if you would want that. Once an error is thrown, the code after that usually doesn't run, so you'd get a whole list of errors. Don't we want to be warned ASAP?

Thinking out loud: it would require some rewiring because tinytest only captures the output of test_ statements, all other code is executed but otherwise ignored.

[kevinushey]

For what it's worth, I agree the current test should halt execution -- I was just hoping that other test files could continue execution in that scenario as well. (under the assumption that each test should be an independent execution unit)

If not, then I think a message indicating which test the error occurred in would be useful.