dlopen offload

[sgasho]

dlopen offload to dist it in dist-x86_64-linux without enabling llvm.offload=true in CI (#159064 (comment))

I tested that this works by following steps at https://rustc-dev-guide.rust-lang.org/offload/usage.html for both LLVM22 and 23(by switch base branch to #158734 temporarily), though It might be better to land this after LLVM 23.

r? @ZuseZ4

[rustbot]

This PR changes how LLVM is built. Consider updating src/bootstrap/download-ci-llvm-stamp.

[sgasho]

I got this error while testing on LLVM23

Related to this I guess.
llvm/llvm-project#199483

View changes since the review

[ZuseZ4]

cc @kevinsala
@Sa4dUs I vaguely remember you ran into this when trying LLVM 23 patches, right? Was this the right solution?

[ZuseZ4]

The LLVM 23 update unfortunately takes a little longer than expected, but glad to hear you got it to work with both 22 and 23. In that case I'd just merge it once it's ready and not wait, since it seems unlikely that we'll break the LLVM 23 update PR with this work here. Worst case, std::offload is broken on main for a couple of days if we missed something, but that shouldn't affect too many users yet.

It's a bit unfortunate that we need a new cmake file for the offload wrapper. Hopefully I'll find time to upstream our wrappers into LLVM, so we won't need that in the future. But it would probably be part of llvm 24, so we shouldn't wait for that.

[rust-log-analyzer]

The job x86_64-gnu-tools failed! Check out the build log: (web) (plain enhanced) (plain)

Click to see the possible cause of the failure (guessed by this bot)

    LLVMConfig.cmake
    llvm-config.cmake

  Add the installation prefix of "LLVM" to CMAKE_PREFIX_PATH or set
  "LLVM_DIR" to a directory containing one of the above files.  If "LLVM"
  provides a separate development package or SDK, be sure it has been
  installed.


-- Configuring incomplete, errors occurred!
See also "/checkout/obj/build/x86_64-unknown-linux-gnu/rust-offload/build/CMakeFiles/CMakeOutput.log".
---
   0: __rustc::rust_begin_unwind
             at /rustc/08d5b675a9b2abdca5e2fe4eabe0e07bbda15d49/library/std/src/panicking.rs:679:5
   1: core::panicking::panic_fmt
             at /rustc/08d5b675a9b2abdca5e2fe4eabe0e07bbda15d49/library/core/src/panicking.rs:80:14
   2: cmake::fail
   3: cmake::run
   4: <cmake::Config>::build
   5: <bootstrap::core::build_steps::llvm::RustOffload as bootstrap::core::builder::CommandLineStep>::run
             at /checkout/src/bootstrap/src/core/build_steps/llvm.rs:1019:13
   6: <bootstrap::core::build_steps::llvm::RustOffload as bootstrap::core::builder::Step>::run
             at /checkout/src/bootstrap/src/core/builder/mod.rs:131:9
   7: <bootstrap::core::builder::Builder>::ensure::<bootstrap::core::build_steps::llvm::RustOffload>
             at /checkout/src/bootstrap/src/core/builder/mod.rs:1673:36
   8: <bootstrap::core::build_steps::llvm::RustOffload as bootstrap::core::builder::CommandLineStep>::make_run
             at /checkout/src/bootstrap/src/core/build_steps/llvm.rs:973:21
   9: <bootstrap::core::builder::CommandLineStepDescription>::maybe_run
             at /checkout/src/bootstrap/src/core/builder/mod.rs:495:13
  10: bootstrap::core::builder::cli_paths::match_paths_to_steps_and_run
             at /checkout/src/bootstrap/src/core/builder/cli_paths.rs:233:18
  11: <bootstrap::core::builder::Builder>::run_step_descriptions
             at /checkout/src/bootstrap/src/core/builder/mod.rs:1199:9
  12: <bootstrap::core::builder::Builder>::execute_cli
             at /checkout/src/bootstrap/src/core/builder/mod.rs:1157:14
  13: <bootstrap::Build>::build
             at /checkout/src/bootstrap/src/lib.rs:802:25
  14: bootstrap::main
             at /checkout/src/bootstrap/src/bin/main.rs:157:11
  15: <fn() as core::ops::function::FnOnce<()>>::call_once
             at /rustc/08d5b675a9b2abdca5e2fe4eabe0e07bbda15d49/library/core/src/ops/function.rs:250:5