Repository navigation
Conversation
Move CONFIG_GVE=m from the amd64 defconfig to commonconfig so the Google virtual NIC driver is also built for arm64. All GCE arm64 machine types (T2A, C4A, ...) only offer gVNIC, so without the driver there is no network in the initrd and Ignition cannot fetch its config. Signed-off-by: bra-fsn <nagy.attila@gmail.com>
Bump coreos-modules and coreos-kernel so the kernel config change is not masked by already built binary packages, as only the config files changed and not the package version. Install app-admin/google-guest-configs for all architectures instead of amd64 only. It provides the dracut module with the GCE disk naming udev rules, which are needed on arm64 as well, since GCE arm64 machine types expose their disks over NVMe. Signed-off-by: bra-fsn <nagy.attila@gmail.com>
Add arm64 to KEYWORDS of oem-gce and its dependencies google-guest-configs and google-compute-engine, so that the OEM sysext is built for arm64 and the gce VM image format works there. All of them are scripts, udev rules and Python, with no architecture specific code. Use the PL011 serial console (ttyAMA0) in the GCE grub.cfg on arm64, where the ttyS0 console and GRUB serial settings of amd64 do not exist. Signed-off-by: bra-fsn <nagy.attila@gmail.com>
The GCE OEM enables OS Login on first boot by linking nsswitch.conf and other files to the ones shipped by sys-auth/google-oslogin, which coreos-base/coreos only installed on amd64. On arm64 these links would dangle, so no user lookup works and dbus and sshd fail to start. Add arm64 to KEYWORDS and install the package on every architecture. Revbump coreos-base/coreos for the dependency change. Signed-off-by: bra-fsn <nagy.attila@gmail.com>
Signed-off-by: bra-fsn <nagy.attila@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add arm64 support for GCE
All Google Cloud arm64 machine types (T2A, C4A/Axion) only offer gVNIC, and the arm64 kernel has no
CONFIG_GVE, so an arm64 image has no network in the initrd and Ignition cannot run. T2A/C4A also expose disks over NVMe, so the GCE disk naming udev rules fromgoogle-guest-configsare needed in the arm64 initrd too. The GCE OEM packages were not keyworded for arm64, so nooem-gcesysext was built for it.The commits: move
CONFIG_GVE=mtocommonconfig-6.18(and revbumpcoreos-modules/coreos-kernel, installinggoogle-guest-configson all arches); keywordoem-gce,google-guest-configsandgoogle-compute-enginefor arm64 and use the PL011 console (ttyAMA0) in the GCEgrub.cfg; keywordgoogle-osloginfor arm64 and install it on all arches, because the OEM'senable-osloginunit linksnsswitch.confto it and without it dbus and sshd fail on first boot; changelog.Part of flatcar/Flatcar#2000: this makes
image_to_vm.sh --format=gceproduce a working arm64 image. Publishing needs mantle changes (see TODO). Note thatgoogle-osloginis now installed in every arm64 image, not only the GCE one, which is relevant for the image diff.Validate what we did: https://github.com/SpareCores/flatcar-images
gce-arm64at 8f3e8ed (onstable-4757.2.3)..github/workflows/build-gce-arm64.yml, documentation:docs/gce-arm64.md.stable-4757.2.3with the image artifact (with a Sigstore bundle), the manifest (records the exact fork commit built) and the checksums.How to use
Build
arm64-usras usual and convert with./image_to_vm.sh --board=arm64-usr --format=gce. Import the resulting tarball withgcloud compute images create --architecture=ARM64 --guest-os-features=UEFI_COMPATIBLE,GVNIC ....To try the same change without building, a public community image of 4757.2.3 with this change backported is available (a community build made outside Flatcar's release infrastructure, so nodes must keep update-engine disabled: an official update would replace it with an image lacking gVNIC on arm64; OS Login SSH key login is affected by flatcar/Flatcar#2496 exactly like amd64):
(image
flatcar-stable-4757-2-3-arm64in projectflatcar-images).Testing done
build-gce-arm64.yml, run linked above) and boot-tested on GCE T2A and C4A: boot to login (~14-18 s), Ignition (platform gcp, user-data from metadata), gVNIC networking (gve driver, MTU 1460, DHCP, DNS), metadata server, root filesystem grow, NVMe disk names via/dev/disk/by-id/google-<name>, time sync from the metadata server, reboot and stop/start persistence, OS Login NSS links and user lookup (an OS Login user checked on T2A). Additionally, from a different GCP project, with a Google account outside the publishing organization, an instance was created with--image-family flatcar-stable-arm64 --image-project flatcar-imageson t2a-standard-1 and booted to the login prompt with the gve driver loaded and eth0 configured.TODO / open questions:
Mantle:
ore gcloud create-imagemust setArchitecture: ARM64, andplume releasecurrently skips GCE for arm64; both are needed to publish.Follow-up PR in this repo once mantle supports it: enable
ci-automation/vendor-testing/gce.shfor arm64 (it rejects arm64 now) with an arm64 machine type such ast2a-standard-2inci-automation/ci-config.envand--gce-gvnic.OS Login SSH login ordering (GCE: OS Login SSH fails since 4230.2.0, systemd's 20-systemd-userdb.conf shadows the OS Login AuthorizedKeysCommand Flatcar#2496) is not addressed here.
Changelog entries added in the
changelog/changes/directoryInspected CI output for image differences:
/bootand/usrsize, packages, list files for any missing binaries, kernel modules, config files, kernel modules, etc.AI use: an AI assistant was used for the investigation and to draft the patch; I reviewed the change and the test results.