feature: add ublk-based block device frontend (overlaybd-ublk) - #429
Open
haolianglh wants to merge 6 commits into
Open
feature: add ublk-based block device frontend (overlaybd-ublk)#429haolianglh wants to merge 6 commits into
haolianglh wants to merge 6 commits into
Conversation
The existing TCMU frontend requires a single resident daemon for all devices on a host: netlink only notifies the registered handler process, so one crash takes down every overlaybd disk, and the code carries several netlink workarounds as the price of that architecture. Introduce overlaybd-ublk, an alternative frontend based on the kernel's io_uring-backed ublk driver, with no SCSI overhead and no single point of failure: one process serves exactly one device, so device create/delete collapses into process start/stop and failures are isolated to a single disk. * add/del/list CLI following ublk ecosystem conventions; add returns only after the device is usable and prints the device path * per-queue event loop driven by photon: ring fd polled via epoll, CQEs reaped with the non-blocking libublksrv API, IO served by coroutines calling ImageFile directly * IO mapping mirrors the TCMU frontend (READ/WRITE/FLUSH/DISCARD); writable devices honestly advertise a volatile cache, read-only devices do not * works on kernels with partial io_uring feature sets: queue setup retries without IORING_SETUP_COOP_TASKRUN, device deletion prefers DEL_DEV_ASYNC and falls back to a bounded sync del * libublksrv v1.7 and liburing 2.8 are fetched at configure time and statically linked (dual MIT/LGPL, used under MIT); building needs autotools, opt out with -DBUILD_UBLK_FRONTEND=off * verified end-to-end on ublk-capable kernels: startup sequence, idle wakeup, full-queue fio pressure with crc32c verify, FLUSH path, add/del lifecycle, and read-only semantics; 13 unit tests cover the IO mapping and CLI layers Signed-off-by: haolianglh <haolianglh@sina.com>
With one process per device, daemons sharing one cache directory can race: the file cache's eviction (truncate+unlink) and refill locking is in-process only, so concurrent daemons risk accounting drift, cache thrashing, and a small window of serving stale zeroes when eviction truncates a file another daemon is reading. Introduce `add --cache-dir <dir>`, giving each device its own registry and gzip cache directories. The override is applied by pointing create_image_service at a patched temporary copy of the service config, leaving the shared image_service code untouched. Document that multiple devices must use separate cache directories. Signed-off-by: haolianglh <haolianglh@sina.com>
The file cache's eviction and refill locking is in-process only, so
two daemons sharing a cache directory can corrupt space accounting,
thrash each other's evictions, and in a small window serve stale
zeroes. With one process per device, the unsafe configuration must
not be reachable by default.
Give every device its own cache directory, always:
<base>/<image-key>/<instance>/{registry_cache,gzip_cache}
* <base> defaults to /opt/overlaybd/ublk_cache; --cache-dir overrides
the base only
* <image-key> is a hash of the image config's realpath, so remounting
an image reuses its warm cache and distinct images never collide;
a "source" file records the origin path
* <instance> is claimed via flock (inst0, inst1, ...): mounting the
same read-only image repeatedly is a legitimate pattern and gets
automatically numbered instances, or a caller-pinned name via
--instance-id
* writable images (a config with an effective upper) are restricted
to a single exclusive mount: concurrent RW mounts would corrupt the
upper layer itself, not just the cache
Also fold the six file-scope globals of ublk_device.cpp into a
UblkDevice class (libublksrv callbacks reach it through the ctrl-dev
priv-data slot), removing the last single-device assumption from the
code in preparation for a possible multi-device daemon mode.
Unit tests cover the config patching, image key derivation, writable
detection and the CLI additions (19 cases); runtime verified on a
ublk-capable kernel: automatic slots, writable-mount rejection,
--instance-id, warm directory reuse, and the full add/IO/del
lifecycle.
Signed-off-by: haolianglh <haolianglh@sina.com>
Add a daemon that hosts all ublk devices in one process and shares a single ImageService, complementing the one-process-per-device CLI. The control plane is HTTP over a root-only unix socket (docker.sock style, curl-debuggable): POST /v1/add and /v1/del, GET /v1/list and /v1/ping, POST /v1/shutdown. add replies only when the device is usable; del replies after the device is fully torn down; SIGTERM or shutdown stops all devices in parallel (STOP first, then join) so total teardown time stays close to the slowest single device. To host several devices per process, split UblkDevice::run() into start()/wait()/stop()/teardown() with a second construction path that injects a shared ImageService, and derive the ImageService registration tag from pid plus a sequence number (a bare pid tag collides on the second add). Also fix a latent fd-0 bug the daemon exposed: on kernels without IORING_SETUP_COOP_TASKRUN, the old init-then-retry flow made every queue creation fail once, and libublksrv's failure path closes fd 0 (ublksrv_queue_deinit runs on a zero-initialized queue whose epollfd/efd are 0 behind >= 0 guards). With one process per device the victim was stdin and the retried ring landed on fd 0, working by luck; in the daemon, fd 0 was the previous device's queue ring, so every add silently wedged the device added before it. Probe the flag once at startup with a private throwaway ring instead, so the failing path is never reached; the daemon additionally parks fd 0 on /dev/null as a guard. Protocol parsing/encoding is a standalone photon-free module with unit tests (23 cases in total). Verified on a ublk-capable kernel: mixed RO/RW devices, cross-device regression probes, concurrent dual-device fio with data verification, del, and parallel shutdown. Signed-off-by: haolianglh <haolianglh@sina.com>
Land the safety items of the daemon milestone M2 (ADR-0006):
* The daemon now patches the service config to a private cache tree
<base>/daemon/{registry_cache,gzip_cache} guarded by an exclusive
flock (--cache-dir overrides the base; the service config's cacheDir
is deliberately ignored so the default never collides with tcmu's
cache directory). A second daemon on the same base is refused.
* Writable images are excluded across processes: the daemon takes the
per-image inst0 lock file (lock-only) before opening the upper, so
the daemon and single-device CLI processes cannot double-mount one
RW image; the lock is released when the device is deleted.
* The CLI `del -n N` now refuses devices owned by overlaybd-ublkd and
prints the equivalent curl command: the pidfile of such a device
points at the daemon, so the old SIGTERM semantics tore down ALL
daemon devices at once (found the hard way). `list` tags these
devices with [ublkd-managed].
* Concurrent control requests are made safe: a per-image in-flight
guard rejects duplicate adds, and a stopping state machine rejects
concurrent deletes of one device, which previously raced on the
device-table iterator across coroutine yields.
* Failure reasons of a daemonized CLI add now travel to the console
over the ready pipe (the child has no stderr and the log is not yet
redirected during early setup), and the patched service config copy
stays alive until teardown -- deleting it early silently dropped the
global default download section from image configs.
Verified on a ublk-capable kernel: dual-daemon refusal, bidirectional
RW mount exclusion, RO cross-mounting, concurrent add/del races, del
rejection with a live daemon, and full teardown via SIGTERM.
Signed-off-by: haolianglh <haolianglh@sina.com>
Add POST /v1/resize for growing a writable image online: the image layer goes through ImageFile::resize() in-process (optionally growing the ext4 inside), then the kernel is told the new capacity with UBLK_U_CMD_UPDATE_SIZE. libublksrv v1.7 has no wrapper for that command and its generic sender is private, so the uring_cmd is sent on a private throwaway SQE128 ring against /dev/ublk-control, assembled the same way as the library does. Support is negotiated: UBLK_F_UPDATE_SIZE is probed once via GET_FEATURES and requested at ADD_DEV, since the kernel only accepts UPDATE_SIZE on devices created with the flag. Only growing is allowed and only for writable images; kernels without the feature get a clear 501. The error paths (feature missing, read-only device, unknown device, bad parameters) are runtime verified; the success path needs a mainline >= 6.11 kernel and is covered by the pending mainline regression. Also ship a systemd unit template for the daemon (generous TimeoutStopSec: deletions may take seconds per device on kernels without DEL_DEV_ASYNC, and killing a flushing daemon risks a writeback deadlock) and document the daemon in the README. Signed-off-by: haolianglh <haolianglh@sina.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
This PR adds a new block device frontend
overlaybd-ublk, which exposes an overlaybd image (config.v1.json) as/dev/ublkbNvia the kernel's ublk driver (io_uring based), as an alternative to the existing TCMU/SCSI path. The TCMU frontend is not touched by this PR.Motivation
TCMU forces a single resident daemon (netlink only notifies the registered handler process): one crash takes down every overlaybd disk on the host, and the codebase carries several netlink workarounds (SO_RCVBUFFORCE, NETLINK_NO_ENOBUFS, block_netlink restart protection) as the price of that architecture. ublk has no such constraint and no SCSI protocol overhead, and is the de-facto successor to TCMU for userspace block devices.
Design
One process per device: process lifetime == device lifetime. Device create/delete collapses into process start/stop, and a failure is isolated to a single disk. CLI follows the ublk ecosystem conventions:
overlaybd-ublk add --config /path/config.v1.json # prints /dev/ublkbN when ready
overlaybd-ublk list
overlaybd-ublk del -n 0
add daemonizes and only returns success after the image is open and the block device is usable; the device path on stdout is the sync-ready contract for future snapshotter integration. Each device can write to its own log file via add --log-path ... (recommended when running multiple devices; daemons otherwise share the global log file and are tagged ublk- for disambiguation).
Single-thread hybrid event loop: the queue thread runs a photon environment and owns the loop -- the io_uring ring fd is polled via photon epoll, CQEs are reaped with the non-blocking public API ublksrv_queue_reap_events(), and IO runs in photon coroutines calling ImageFile directly (ublksrv_complete_io() is always followed by an explicit io_uring_submit(), since commit SQEs are queued but not submitted by libublksrv). On teardown the loop drains through ublksrv_process_io() until -ENODEV, matching the official daemons' exit semantics, so no io command is ever left inflight in the kernel. The IO dispatch layer is decoupled from the event loop and unit-testable in isolation.
IO mapping mirrors the TCMU frontend: READ -> preadv (retry-forever), WRITE -> pwritev (EROFS special case), FLUSH -> fdatasync, DISCARD/WRITE_ZEROES -> fallocate(PUNCH_HOLE|KEEP_SIZE). Writable devices honestly advertise UBLK_ATTR_VOLATILE_CACHE so the kernel issues FLUSH as needed; read-only devices advertise no cache (nothing volatile to flush).
Kernel compatibility: the frontend runs on any kernel exposing /dev/ublk-control, including ones where ublk was backported with a partial io_uring feature set:
queue setup retries without IORING_SETUP_COOP_TASKRUN (a pure optimization, mainline 5.19+) when the kernel rejects it; device deletion prefers UBLK_CMD_DEL_DEV_ASYNC (kernel 6.5+) and falls back to a sync del bounded at 2s; if the kernel still holds device references at that point the daemon exits and process teardown lets the kernel complete the deletion, so del never hangs and dev ids are never leaked.
Testing
Full build passes with the option on and off; the off state produces no new targets and introduces no new build requirements. 13 unit tests (ublk_dispatch_test) cover the op-mapping layer (read/write roundtrip, EIO/EROFS/short-read paths, flush, discard punch hole, unknown op) and CLI parsing -- no kernel or photon dependency. Verified end-to-end on kernels with the ublk driver available: device startup sequence, first IO after long idle (no lost wakeups), full-queue fio pressure (4k randread iodepth 128 x 4 jobs; randwrite + crc32c verify with zero errors; 256k sequential mixed), FLUSH path via fsync, full add/del lifecycle (bounded del, clean process exit, dev id reuse), and read-only semantics (getro=1, raw writes rejected, committed layer mounts ro and reads back written data).
Known limitations / follow-ups
v1 scope: single queue, no zero-copy, no UBLK_F_USER_RECOVERY (process death == device removal, by design), no snapshotter integration. End-to-end DISCARD depends on the kernel: some kernels with ublk backported apply the discard queue limits but lack the pre-5.19 QUEUE_FLAG_DISCARD glue, so fstrim/blkdiscard report "operation not supported" there (graceful degradation; the userspace path is covered by unit tests).
Hardening backlog: orphan-device cleanup in del/list (direct /dev/ublk-control DEL without pidfile).
Unmount filesystems on /dev/ublkbN before del: the daemon serves the device's IO, so tearing it down with a mounted filesystem aborts in-flight journal writes (documented in README).