chore(madmail): port to v2

[j4n]

This adds support for the v2 Rust-Version and adapt tests; opportunistically download binaries if available to skip the build process:

• Pin MadmailDriver.DEFAULT_REF to v2.20.0
• Fetch tags so RELAY_REF and --source can checkout release tags
• Run madmail E2E tests from the deployed checkout, not the cached clone
• Raise lxc-test job timeout to 60 minutes to fit potential madmail source build

Support for @latest split into separate PR #46.

[hpk42]

please

1. drop the @latest dynamic git resolving logic (does network requests during CLI parsing and is orthogonal issue to madmail v2), just default to @main or latest release tag for now.

2. Keep the go->rust madmail upgrade, and also support for downloading prebuilt binaries, the imap-test fix to work against madmail-v2 is fine as well.

3. Prepare a draft themadorg/madmail PR that uses this branch here. If i am not mistaken you can pick up this PR branch here with something like

jobs:
  lxc-test:
    name: cmlxc deploy and run tests
    uses: chatmail/cmlxc/.github/workflows/lxc-test.yml@main
    with:
      cmlxc_version: j4n/madmail-main
      cmlxc_commands: |
        cmlxc init
        cmlxc deploy-madmail --source ./ mad0
        cmlxc deploy-cmdeploy --source @main relay0
        cmlxc test-mini mad0 relay0
        cmlxc test-mini relay0 mad0
        cmlxc test-madmail mad0

[j4n]

Thank you. I have addressed your feedback, and updated PR body text accordingly; madmail PR is themadorg/madmail#144.

[hpk42]

thanks for addressing most of the feedback!

However, the v0.47.1 pin was intentional, not an oversight, so can not serve as a reason for adding @latest support. What we want i think is:

• chatmail/relay tests its PR/main against a fixed madmail version. MadmailDriver.DEFAULT_REF can stay a fixed tag, just bump it to the current latest madmail v2 one. Relay-tests should automatically use it then, and they will use the already-compiled Rust binary (madmail has tags running ahead of releases, so downloads might more easily fail with @latest but not sure)

• themadorg/madmail tests its PR/main against chatmail/relay @main (relays are often deployed from main anyway, so madmail should work against such relays, but pinning is also fine)

• cmlxc's should test against latest pinned relay/madmail or latest main ones or both (but doesn't today, a separate issue)

Despite you fixing the CLI/network issue i still prefer @latest support to come in a separate PR (unstacked! :).

[j4n]

However, the v0.47.1 pin was intentional, not an oversight, so can not serve as a reason for adding @latest support.
* chatmail/relay tests its PR/main against a fixed madmail version. MadmailDriver.DEFAULT_REF can stay a fixed tag, just bump it to the current latest madmail v2 one. Relay-tests should automatically use it then, and they will use the already-compiled Rust binary (madmail has tags running ahead of releases, so downloads might more easily fail with @latest but not sure)

Brought back, pinned to 2.20.0, indeed 2.23.0 is latest tag; thats why I did the HTTP-query; but maybe we can just cache the build somehow at a later point.

* themadorg/madmail tests its PR/main against chatmail/relay @main (relays are often deployed from main anyway, so madmail should work against such relays, but pinning is also fine)
* cmlxc's should test against latest pinned relay/madmail or latest main ones or both (but doesn't today, a separate issue)

Leaving for subsequent edits.

Despite you fixing the CLI/network issue i still prefer @latest support to come in a separate PR (unstacked! :).

I split it out now, coming up.
Before after for this round: git range-diff origin/main..0f158e5 origin/main..HEAD

[hpk42]

thanks and sorry for taking a while.

1. PR failed because i had existing (madmail Go) containers.
   I think it's better to try to detect and remove the Go-version early on. There aren't too many users of cmlxc but still some, and it will give a weird traceback. I think there should be Driver.cached_checkout_is_stale classmethod that checks REQUIRED_SOURCE_PATH. This can then be used from get_git_main_path to discard the old checkout.

2. PR description says "Fetch tags so RELAY_REF and --source can checkout release tags" but i think you can still just do "git fetch origin" and it autofollows the tags (madmail tags from main)

It' helpful for review if all lines in a PR are neccessary.

[j4n]

Thanks for re-reviewing.

I applied your suggestions, they were correct.

1. PR failed because i had existing (madmail Go) containers.
   I think it's better to try to detect and remove the Go-version early on. There aren't too many users of cmlxc but still some, and it will give a weird traceback. I think there should be Driver.cached_checkout_is_stale classmethod that checks REQUIRED_SOURCE_PATH. This can then be used from get_git_main_path to discard the old checkout.

Ah, good point. I added a commit that addresses this, separatly, for easier review, with a simple test to check it but I haven't tested with stale go containers yet.

2. PR description says "Fetch tags so RELAY_REF and --source can checkout release tags" but i think you can still just do "git fetch origin" and it autofollows the tags (madmail tags from main)
   It' helpful for review if all lines in a PR are neccessary.

Indeed, apologies.

[hpk42]

i think the timeout 30 -> 60 was really about allowing CI to finish, and you removed it in force-push. please also squash everything to a single commit, and make sure CI passes. It's getting ready for merge.

[j4n]

i think the timeout 30 -> 60 was really about allowing CI to finish, and you removed it in force-push. please also squash everything to a single commit, and make sure CI passes. It's getting ready for merge.

Yes, as I wrote in the reply to your question, I checked the times and decided that 10 minutes headroom with the current 30min limit for runs with source-build are enough, so I removed it. Pushed the squashed single commit now.