Server crashes after not responding during pregen events.

[johncsuti]

Expected behavior

We might need to scale down pre-generation as it seems to have caused a server to lag for so long that paper killed itself.

Paste bin of the error.
https://pastebin.com/XGr7qBaU

Observed/Actual behavior

Pregeneration wouldn't lag the server to the point of server killing the process.

Steps/models to reproduce

Start a Paper.1.21.4 jar instance with 5-7Gb of ram and allow it to pre-generate the world.

BentoBox version

3.3.0 Server is not able to respond to command. Downloaded off bentobox website today.

Plugin list

Bentobox recommended build

Poseidon
Challenges
Level
Warps
InvSwitcher

Auction House 1.0.8 (Custom)
EssentialsX
Vault

Other

No response

[tastybento]

Hmm, sorry about that. Could you share the full log? Zip files can be dropped into these boxes.

I have got a new snapshot that adds settings for the pre-gen to the top of the config.yml file and defaults to having a second between each event. Could you try it and see if it helps?

https://ci.codemc.io/job/BentoBoxWorld/job/Poseidon/20/

[johncsuti]

Testing currently will update after it has been running for 30 minutes.

[johncsuti]

Here is a full paste bin of the entire server log. It looks like the timeout for server lag is 60 sec. before it shuts itself down. Pregen also wont run again after another boot up. Maybe we should set a flag somewhere if pregen completed successfully or not set it and have pregen start again on non gened chunks?

https://pastebin.com/37RLawkY

[tastybento]

Thanks for the log. Pregen only operates if the chunks have not been generated. So if you don't see anything in the log, then it's done. It also tells you that it's done. If you like, reduce the number of pre-gen realms in config.yml to 1 and see if you get any issues and then try and increase them.

I've been running the code for weeks on various servers and I haven't seen this so I'm thinking either some set up issue or server code bug. It's definitely something around world saving (see below). Attached is a log from yesterday from my test server - no issues and it's not a big server at all. You are running on Paper build 177, which was release 2/21 so it's over a month old. Can you try the latest build?

2025-03-24-7.log.gz

Details

It's blocked on a call to CompletableFuture.join(), which is a synchronous wait for an asynchronous task to complete.

This join() call originates from:

net.minecraft.world.level.storage.DimensionDataStorage.saveAndJoin(DimensionDataStorage.java:192)

So it's saving world data, but that saveAndJoin() method is calling .join() on a CompletableFuture that hasn’t completed, causing the thread to just sit and wait. There's no trace back to BentoBox in the log. It's all server code. Although of course, chunks are being made. Anyway, CompletableFuture.join() will block the thread until the async task finishes. If that future never completes (or is taking too long due to I/O, deadlock, or overload), then the server thread freezes, and the Paper Watchdog logs this error after the timeout.

Possible causes could be:

1. Deadlock or long I/O in the saveAndJoin() method.
2. A plugin or mod might be hooking into world save events and doing something blocking.
3. A resource bottleneck (like high disk I/O or CPU usage) might cause the future to stall.
4. A bug in Paper or a plugin where the CompletableFuture is never completed.

[johncsuti]

Found the issue to be two fold. 1st issue is deadlocking I/O causing writing to fail. 2nd issue was how crafty controller handles instances and keeping them inside a container. When giving the server multiple cores without a container it runs fine. As far as i can tell crafty limits the number of CPU cores to 1 per container without a way of increasing it. Calling tasks asynchronously causes them to run on the same core as the rest of the server.

[tastybento]

Interesting. I'm not a container expert, you using Docker? The container would need to have at least 2 CPUs to avoid the deadlock. Did you find a solution?

[johncsuti]

It does use docker containers that are non-root similar to that of kubernetes. I am unsure of it it allocates more then 1 CPU core. However when looking at htop on linux it shows them running under the user crafty as a full java process. I wonder if its jailed instead of a full container. From my testing it appears to not use any type of container but it could be a non execution environment. I have it generating again with the fallowing. Paper 1.21.4-221, Ram 10Gb, along with 24 cores of a Intel Xeon CPU E5-2667 v2's I have 2 physical processors inside the machine.