PPC64: correct number conversions for the 64-bit ABI

[neomantra]

In working this OpenResty issue #1152, the root cause was related to broken PPC64 number-to-integer conversions that cause errors in FFI operations and table lengths. Non-P64 code is unchanged.

I worked with Claude and Codex to come up with this, but I will be honest, that even in trying to understand it, I don't fully. I even have this nice Claude Artifact that walks through it, but I'm not sure it is accurate.

That said, QEMU tests passed over 1.5 million conversion checks and test-suite failures decreased from 39 to 2 existing failures. I only ran it in emulation, as I don't have access to anything native.

I will leave Docker-based instructions for those interested in testing on the OpenResty issue.

The rest is dense LLM stuff, which includes a description of why lj_tab_len needed to change. I am marking this as draft and for public information, or experts to handle better.

---

Commit f80b349 ("Unify Lua number to FFI integer conversions") moved double-to-integer conversions into four assembly functions in vm_ppc.dasc: lj_vm_num2int_check, lj_vm_num2i64, lj_vm_num2u64, and lj_vm_tobit. These functions use incorrect word order and integer return conventions for ppc64le. They read the low word of a double as the high word. Thus, they read an incorrect exponent and return incorrect results.

Since v2.1-20260114, ffi.cast("int64_t", 123456789) returns 0LL on ppc64le. The error also affects int32, uint64, and intptr casts, ffi.new(), 64-bit arithmetic with Lua numbers, and integer string formatting. Incorrect integer-key checks prevent correct use of table array parts. This causes errors in table lengths and table operations.

Add P64 implementations of the four functions. Use fctidz and fcfid for conversions. Return results in r3 as the 64-bit ABI requires. These leaf functions use the red zone below sp. They keep sp and the stack back chain unchanged for asynchronous stack walkers. Do not use fctiduz, to keep compatibility with older big-endian PPC64 CPUs. The non-P64 code paths do not change.

Remove the lj_tab_len workaround from PR #274 (1b3e6c0). The incorrect integer check kept keys with integer values in the hash part. The workaround always used the slow path, which found those keys. Thus, the workaround appeared to correct the problem. The slow path assumes that the last array slot is not nil. With correct conversions, keys go into the array part. This assumption then causes incorrect lengths. For example, #{1,2,3} returns 4.

Previous tests used QEMU user emulation on ppc64le with Debian bookworm and GCC 12. The results were as follows:

• The corrected test harness passed all 125 assertions for 35 inputs. These assertions included the signed -2^63 boundary.
• All 1,530,534 extended conversion checks passed.
• During signal handling, 393 samples from the four functions showed no stack-pointer changes or invalid stack back chains.
• Failures in luajit2-test-suite decreased from 39 to 2 of 172 tests. The two failures did not change: ffi_lib_z.lua requires the missing libz library, and debug_gc.lua has an error that occurred before this change.
• The DynASM output was identical in all bytes for four non-P64 configurations.

Tests on native POWER hardware are still necessary.

Fixes openresty/openresty#1152.

[neomantra]

I worked with other LLMs on it, but I admit I still do not fully understand it. But won't be making further changes without any feedback, so undrafting this. Thanks!