state: remount and unmount filesystems after the final SIGKILL

[graysky2]

sync() does not leave a journalling filesystem clean; only a remount read-only or an unmount commits the superblock. The shutdown scripts try both, but they run while every service is still alive, so any open file makes them fail with EBUSY. On ext4 every boot then replays the journal and e2fsck reports stale free block and inode counts.

After the last SIGKILL nothing holds a file any more. Wait for the processes to be reaped, then remount every real filesystem read-only and unmount what can be unmounted, deepest mount first, repeating until a pass makes no progress. That covers the root, block-mounted data partitions, and anything stacked on them.

Network filesystems are only detached lazily, since the network is down. In a container the mounts belong to the host and are left alone.

[geldot]

(Context: original review thread is openwrt PR #24053)

@graysky2 I think this is improving with each iteration, but I'm still not comfortable with the hard-coded filesystems and teardown logic, or indeed whether the actual logic belongs directly in the init system.

The good part is that we've established that STATE_HALT is the correct place to initiate fs teardown.

But the question about whether it would be better to use / call out to existing fstools mount state management to handle this remains open.

Can you explain what barriers you hit with the other approaches?

Currently this patch is jumping across openwrt projects, so I think it's worth taking a moment to briefly document what approaches you tried, what worked, what didn't.

[graysky2]

Agreed that STATE_HALT is the right place, glad we've settled that part.

On reusing fstools: I tried it first. The previous iteration gave fstab.init a STOP=89 so that "block umount" would actually run at shutdown. Reading block.c killed that idea. block umount walks every probed block device rather than the mount table, takes only the first mountpoint it finds for a device, and has no remount-read-only operation at all. What it does is umount2(MNT_DETACH) followed by rmdir() on the mountpoint, which is device removal semantics. On a squashfs+overlay root that detaches /rom and /overlay and then rmdirs them in the live overlay, leaving whiteouts in the upper dir. The next boot fails pivot_root and falls back to a tmpfs overlay. That's why I pulled it.

busybox "umount -a -r", which is what the K90 umount script already runs is problematic: it unmounts first and only falls back to remount-ro when the unmount fails with EBUSY. After the final SIGKILL nothing is busy, so the unmount succeeds, but a superblock that is pinned outside the mount table never gets remounted. Two real cases: the overlay upper, which overlayfs holds through its own private clone of the mount, and an ext4 data partition holding a loop backed image, which the loop device pins through its backing file. Both unmount cleanly from the namespace and stay dirty. The order has to be remount-ro first, then unmount, repeated until a pass changes nothing. Nothing that exists does that, which is why the loop ended up in the patch.

On the filesystem list: I looked for a way to avoid it. AFAIK, the only classification the kernel offers is the nodev flag in /proc/filesystems, and that puts ubifs, overlay, fuse and every network filesystem in the same bucket as proc and sysfs, so it would skip what we need. Every init that does this carries a list; the one in the patch is the same set as systemd's fstype_is_api_vfs(). It can be cut down if you'd rather have fewer names in procd. The only entries that actually matter are proc, because the loop reads it, and the network types, because they hang once the network is gone. Remounting or unmounting sysfs, tmpfs or cgroup2 at that point is harmless, just noisy on the console. Say which you prefer and I'll send it that way.

On whether it belongs in init at all: procd could carry only a hook that runs something after the last SIGKILL, with the policy in base-files or fstools. That works mechanically, procd can still fork, exec and wait there. But it moves the same loop and the same list into another repo and adds a runtime dependency on the one path that must not fail. systemd-shutdown and OpenRC both do this inside init for the same reason: it's the only moment nothing else is alive.