containerd 2.2's mount manager panics on a one-mount mkfs chain

containerd 2.2 shipped a mount manager: a service that can format a file as ext4 or xfs, attach it as a loopback device, and hand the result to a runtime, all from a single Activate call instead of the usual truncate, mkfs, losetup, mount sequence. I wanted to know whether that call is actually faster than doing it by hand, so I wrote a small Go program against the manager's package directly and ran both paths three times each on the same machine.

The manual sequence came out at 23.5 to 25.9 milliseconds. The mount manager's Activate came out at 29.6 to 47.2 milliseconds. It was not faster. Along the way it also panicked once, leaked a raw BoltDB error once, and left a loop device attached with nothing able to find it again.

What the mount manager actually is

There's no ctr subcommand for any of this. The manager lives at github.com/containerd/containerd/v2/core/mount/manager and is meant to be embedded by a snapshotter or a runtime shim, not driven from a terminal. Its job is to let a mount type be built out of steps: a "transformer" can create a file, format it, and format directories, and a "handler" can attach it as a loopback device, before the result gets handed off as an ordinary system mount.

The container running this test was Docker Engine 29.3.1, whose bundled containerd reports itself as v2.2.2. I confirmed the mount manager package is at that exact version by pinning it in go.mod and building against it, not by trusting the daemon's version string.

A working activation for a 200MiB ext4 image looks like this, once I had the templating right:

mounts := []mount.Mount{
    {
        Type:   "mkfs/loop",
        Source: imgPath,
        Options: []string{
            "X-containerd.mkfs.size=200MiB",
            "X-containerd.mkfs.fs=ext4",
        },
    },
    {
        Type:   "format/ext4",
        Source: "{{ mount 0 }}",
    },
}
info, err := mgr.Activate(ctx, "demo1", mounts)

Activate returns two sets of mounts: info.Active, the ones it handled itself (the loopback attach), and info.System, the ones it expects the caller to mount with the ordinary mount(2) syscall (the ext4 filesystem on that loop device). The manager does not mount your rootfs for you. You still call Mount() on whatever comes back in info.System.

The speed comparison

Both paths format a 200MiB ext4 image, attach it as a loopback device, and mount it. I timed the manager's Activate call and, separately, the same steps run as four shell commands: truncate -s 200M, mkfs.ext4 -q -F, losetup -f --show, mount.

The manual path was faster on every run. That is not a criticism of the manager's design so much as an observation about what it's for: I checked the source of the mkfs transformer (core/mount/manager/mkfs.go) and it shells out to the real mkfs.ext4 and mkfs.xfs binaries with exec.CommandContext, exactly what the manual path calls directly. There's no custom fast-path formatter underneath. The manager's job is composability across snapshotters, not raw speed, and on this measurement it costs a small constant overhead (BoltDB writes, symlink creation, bookkeeping) rather than removing any.

Cleanup showed the same pattern: Deactivate plus umount took 32 to 45ms; umount plus losetup -d by hand took 11 to 13ms.

Disk usage was identical either way. du --apparent-size on both images read 200M; actual usage was 17M on both, since neither mkfs.ext4 nor truncate writes real data to most of the file.

What I got wrong on the way

My first version of the manual comparison used dd if=/dev/zero of=disk.img bs=1M count=200 to create the image before formatting it, because that's the version of this recipe I'd actually run in the past. It came out at 358ms to 1.6 seconds; against that, the mount manager looked ten to fifty times faster, which would have been the headline of this post.

dd was writing 200MB of real zero bytes to disk before mkfs.ext4 ever ran. truncate -s 200M creates the same size file as a sparse hole in under a millisecond, and mkfs.ext4 doesn't need the data pre-zeroed, it only writes its own metadata. The mount manager's mkfs transformer already does the equivalent of truncate, via os.OpenFile plus f.Truncate(size), then calls the real mkfs.ext4 binary on the result. Once I made the manual comparison do the same thing, the ten-times "win" disappeared and mildly reversed. The lesson: when a new API bundles three steps into one call, benchmark it against the shortest correct version of those three steps, not the version you happen to type from muscle memory.

Under concurrency

Ten goroutines calling Activate in parallel against the same manager instance, each formatting its own 50MiB image, completed in 70.1ms of wall time, with individual calls ranging from 30.5 to 69.8ms. All ten succeeded. If activations were serialized behind a lock, ten of them would have taken close to 300-400ms; they didn't, so the manager's internal locking (RLock during normal activation, held exclusively only during garbage collection) does allow real concurrent formatting.

What it refuses

An unsupported filesystem type is rejected before any file gets created:

unsupported filesystem "btrfs": invalid argument

A missing size option is also rejected cleanly:

mkfs requires mkfs.size option: invalid argument

A path outside the manager's configured root is rejected too, but with a misleading error class:

no root "/tmp/not-the-root/disk.img" configured for mkfs: not implemented

That comes back as errdefs.ErrNotImplemented, the same error category containerd uses for "this operation genuinely doesn't exist here." Code that checks errdefs.IsNotImplemented() to decide whether to fall back to a different mount path would treat "you forgot to allow this directory" the same as "this feature isn't built." I read the source (core/mount/manager/mkfs.go) to confirm this isn't a formatting quirk on my end; the transformer returns exactly that wrapped error whenever the source path doesn't match any configured root.

Activating a second time under the same name, without deactivating the first, doesn't get a clean "already exists" either:

bucket already exists

That's a raw bbolt error surfacing straight from the metadata store, with no containerd-level wrapping. It's accurate, but it tells you about the manager's storage engine rather than about your mistake.

The panic

The documentation's own examples always chain at least two mounts: something that produces a loopback device, and something that mounts a filesystem on it. I tried activating a single mkfs/loop mount on its own, with nothing consuming its output, expecting either a successful format-only activation or a clean validation error.

panic: runtime error: index out of range [1] with length 1

goroutine 1 [running]:
github.com/containerd/containerd/v2/core/mount/manager.(*mountManager).Activate(...)
    .../core/mount/manager/manager.go:421 +0x2030

I read manager.go to find out why. The function tracks firstSystemMount, the index of the first mount it expects the caller to handle. When a mount is only ever a transform target (my single mkfs/loop mount, with no second mount to hand the loop device to), that index gets set to i+1, which in a one-mount list equals len(mounts). A later loop indexes into mountConv[firstSystemMount] to apply any pending format templating, and mountConv was allocated with len(mounts) elements. Index 1 into a slice of length 1 panics.

This matters beyond the crash itself. The loopback device gets attached to the backing file before the panic point, and because Activate never returns successfully, nothing gets written to the manager's BoltDB. There is no record of this activation to recover.

Crash recovery, and where it doesn't reach

The manager persists activation state in BoltDB specifically so a restarted process can find and clean up mounts from before a crash. I tested that separately from the panic: one process called Activate and then exited hard with os.Exit(0), skipping Deactivate entirely, to simulate a daemon that died mid-operation.

$ ./mmdemo -mode=crash-activate
activated in 30.7ms, err=<nil>

$ ./mmdemo -mode=crash-recover
List() after simulated crash: 1 activations, err=<nil>
  recovered activation: crashy active=[{loop ... /tmp/mm-crash/targets/1/1}]
cleanup via Deactivate: <nil>

A second process, pointed at the same database and target directory, listed the orphaned activation and deactivated it cleanly, and the loop device it had been holding was released. That documented claim held up.

But it only works because the first Activate call returned successfully and got committed. Compare that against the one-mount panic above: I left that test running separately, and the loop device it opened is still attached to a deleted backing file with no BoltDB record anywhere pointing at it. losetup -a still shows it. No amount of restarting a mount manager pointed at any database will find it, because it was never written down. The crash-recovery mechanism protects against a daemon dying after an activation completes. It has no way to protect against the daemon crashing during one.

xfs, and a size limit that isn't the manager's

I ran the same mkfs/loop chain with X-containerd.mkfs.fs=xfs at 200MiB and got a full mkfs.xfs usage message back:

mkfs.xfs failed: Filesystem must be larger than 300MB.

That's mkfs.xfs itself refusing, not the manager. At 400MiB, formatting succeeded in 48 to 72ms. Mounting the result failed in this environment specifically:

mount source: ".../targets/1/1", target: ".../rootfs", fstype: xfs, flags: 0, data: "", err: no such device

/proc/filesystems on this container's kernel has no xfs entry and there's no modprobe to load one. That's a property of the sandbox this test ran in, not of containerd, and I'm noting it rather than counting it as a finding against the feature.

Run it yourself

This needs Go 1.24+, root (for loopback devices and mounts), and mkfs.ext4.

mkdir mounttest && cd mounttest
go mod init mounttest
go get github.com/containerd/containerd/v2@v2.2.2

Save as main.go:

package main

import (
    "context"
    "fmt"
    "os"

    "github.com/containerd/containerd/v2/core/mount"
    "github.com/containerd/containerd/v2/core/mount/manager"
    "github.com/containerd/containerd/v2/pkg/namespaces"
    bolt "go.etcd.io/bbolt"
)

func main() {
    base := "/tmp/mm-demo"
    os.RemoveAll(base)
    os.MkdirAll(base+"/targets", 0755)

    db, _ := bolt.Open(base+"/meta.db", 0644, nil)
    defer db.Close()

    mgr, err := manager.NewManager(db, base+"/targets",
        manager.WithMountHandler("loop", mount.LoopbackHandler()),
        manager.WithAllowedRoot(base),
    )
    if err != nil {
        panic(err)
    }

    ctx := namespaces.WithNamespace(context.Background(), "default")
    mounts := []mount.Mount{
        {Type: "mkfs/loop", Source: base + "/disk.img", Options: []string{
            "X-containerd.mkfs.size=200MiB",
            "X-containerd.mkfs.fs=ext4",
        }},
        {Type: "format/ext4", Source: "{{ mount 0 }}"},
    }

    info, err := mgr.Activate(ctx, "demo1", mounts)
    fmt.Printf("info=%+v err=%v\n", info, err)

    os.MkdirAll(base+"/rootfs", 0755)
    for _, sm := range info.System {
        if err := sm.Mount(base + "/rootfs"); err != nil {
            fmt.Println("mount failed:", err)
        }
    }

    // activating the same name again without deactivating first:
    _, err = mgr.Activate(ctx, "demo1", mounts)
    fmt.Println("second activate, same name:", err)
}

go build -o mmdemo .
sudo ./mmdemo
mount | grep mm-demo
sudo umount /tmp/mm-demo/rootfs

I verified every command in this section on a fresh checkout before publishing.

What to do with this

If you're building on the mount manager today, treat it as a composability primitive, not a performance one; it won't beat a shell one-liner on latency. Never construct a mount list that ends in a transform-only type like mkfs/loop without a mount that actually consumes its output, until this specific crash is fixed upstream. Don't branch on errdefs.IsNotImplemented() from this package without also checking the error text, because it currently covers both "unsupported" and "not configured." And if you're relying on its crash recovery for anything in production, test it against a process that dies mid-Activate, not just one that dies after — those are different guarantees, and only one of them is covered right now.