QDMA Shell

A high-performance FPGA shell built on the Xilinx QDMA subsystem for Alveo boards. It provides PCIe 4.0 host DMA, 100 GbE CMAC networking, and a Dynamic Function eXchange (DFX) reconfigurable partition (RP) for user logic, so the shell stays static while the user module is reloaded at run time.

This document covers the parts that are common to every board. Board-specific details (floorplan, clocking, timing-closure notes, build status) live in each board's own README — see Supported boards.

Architecture

• Top level (src/shell_top.v) — integrates PCIe, Ethernet, and memory.
• QDMA subsystem (src/qdma_wrapper.v) — PCIe 4.0 x8 + host↔FPGA DMA.
• Ethernet subsystem (src/cmac_subsystem.v) — wraps the 100 GbE CMAC(s).
• User block (src/user_block.sv) — the user-logic interface, split into:
  • Static part (bd_user) — Vivado block design with the NIC datapath (packet routers, switches, FIFOs) and infrastructure.
  • Reconfigurable partition (rp_blk) — the DFX partition holding user logic; ships with a default loopback module.
• Host interfaces — AXI-Lite (control/config), AXI-MM (memory-mapped access), and AXI-Stream over QDMA (host packet DMA) and CMAC (network).

Repository structure

Path	Contents
boards/	Per-board params (params.tcl), constraints (base.xdc, rp.xdc), block-design scripts (bd_user.tcl, rp_user.tcl), and a board README.
src/	Verilog / SystemVerilog RTL.
ips/	Custom IP repository. The active packet router is pktrte_dual (AxisPacketRouterDual); pktrte_mono is the superseded single-output variant.
script/	Shared TCL for the build flow and IP configuration (e.g. conf_ip.tcl, abs_flow.tcl) and the check_axi.py linter.
build/	Generated Vivado projects (one subdir per PROJ).
Makefile, build.tcl, build_module.tcl	Build entry points.

Building

The flow is driven by a Makefile plus Vivado TCL scripts.

Common parameters

Variable	Default	Meaning
BOARD	au50	Target board (au50, au280, au55c).
PROJ	<BOARD>_shell	Vivado project name (build dir = build/<PROJ>).
PRFLOW	0	1 enables the Partial Reconfiguration (DFX) flow.
RUN	0	1 runs synth + impl; 0 only creates the project.
NTHRD	16	Parallel Vivado jobs (forced to 0 when RUN=0).

Source the matching Vivado before building — see the board table for the required version.

Build without PR flow

make shell BOARD=<board> PROJ=<name> PRFLOW=0   # PRFLOW=0 is the default

Generates the Vivado project and, with RUN=1, builds the full monolithic shell.

Monolithic UUID sentinel. In a non-PR build the low 32-bit word of the build-info UUID (word 3 — the last 8 hex characters of the canonical xxxxxxxx-xxxx-xxxx-xxxx-xxxxFFFFFFFF string) is forced to 0xFFFFFFFF. This marks the loaded image as a monolithic application that does not support partial reconfiguration. A PR-flow (PRFLOW=1) shell keeps a fully random low word, so a host reading the infopage can distinguish a monolithic image from a reconfigurable shell by the UUID alone.

Build with PR flow

1. Shell:

   make shell BOARD=<board> PROJ=<name> PRFLOW=1

2. Reconfigurable module (rp_blk):

   make module BOARD=<board> PROJ=<module_name>

   Produces a synthesis checkpoint (.dcp) for the RP.
3. Link (manual): use the helpers in script/abs_flow.tcl to link the RP .dcp into the PR-enabled shell and emit the final bitstreams. This step is not yet wired into the Makefile.

Default register map

The QDMA exposes one AXI-Lite master (BAR2). It fans out through the top-level axil_xbar (configured in script/conf_ip.tcl) to six master ports. This map is shared by every board.

Top-level AXI-Lite xbar

M-port	Range	Size	Destination
M00	0x000000–0x1FFFFF	2 MB	RP partition (rp_user_inst) — rpen-gated
M01	0x200000–0x2FFFFF	1 MB	Static shell (bd_user_inst.s_axil_cfg)
M02	0x310000–0x31FFFF	64 KB	CMAC0 (QSFP0)
M03	0x320000–0x32FFFF	64 KB	CMAC1 (QSFP1)
M04	0x308000–0x30FFFF	32 KB	System Management (sys_mgmt_wiz)
M05	0x300000–0x307FFF	32 KB	QDMA CSR

Static-shell config window (M01)

Within the 1 MB window at base 0x200000, the BD SmartConnect axil_sc_i decodes to:

Absolute	Sub-offset	Block (BD path)
0x208000–0x208FFF	0x8000	pkt_route_0/s_axil/reg0 (AxisPacketRouterDual, CMAC0)
0x209000–0x209FFF	0x9000	pkt_route_1/s_axil/reg0 (AxisPacketRouterDual, CMAC1) — dual-CMAC boards only
0x20C000–0x20CFFF	0xC000	h2c_dst_trans_i/s_axil/reg0 (AxisDestTrans)
0x20E000–0x20EFFF	0xE000	pr_subsys/axi_gpio_prctrl/S_AXI (PR-control GPIO)
0x20F000–0x20FFFF	0xF000	pr_subsys/axi_hwicap/S_AXI_LITE (ICAP)

Single-CMAC boards (au50) have one router and omit 0x209000 (axil_sc_i NUM_MI=4); dual-CMAC boards (au280, au55c) include it (NUM_MI=5). The PR-control GPIO drives rp_reset / rp_detach.

RP user window (M00)

The RP partition's s_axil is reached directly through xbar M00 and is gated by rpen (so a detached/blank RP cannot stall the bus):

Absolute	Sub-offset	Size	Block
0x000000–0x07FFFF	0x00000	512 KB	axi_gpio_0/S_AXI/Reg
0x080000–0x1FFFFF	—	1.5 MB	unused (RP-side reserve)

The RP also exposes AXI-MM spaces (s_axi_dma, s_axi_pcie) to axi_bram_ctrl and m_axibr; those are not reached through the top axil_xbar and are listed in each board's rp_user.tcl for reference only.

src/axil_xbar_cdc.v masks each CDC input address to its window width so downstream blocks see 0-based offsets; software still uses the absolute addresses above.

Packet dataflow (NIC)

The static block design implements the NIC datapath between the CMAC(s), the host (QDMA), and the RP. Each CMAC has its own dual-output packet router (pkt_route_N); a host-side splitter (h2c_sw) and an egress switch (nicsw) steer traffic by AXI-Stream tdest. The diagram shows a dual-CMAC board:

 eth_rx_0 ─► pkt_route_0   m_axis_0 ─► nicsw.S03      m_axis_1 ─► ethrx0 ─► RP
 eth_rx_1 ─► pkt_route_1   m_axis_0 ─► nicsw.S04      m_axis_1 ─► ethrx1 ─► RP

 QDMA H2C ─► fifo ─► h2c_dst_trans_i ─► h2c_sw
                            ├─ tdest 0xFFF0–0xFFFF ─► nicsw.S00
                            └─ tdest 0x0000–0xFFEF ─► rph2c ─► RP

 RP ─► rpout0 ─► nicsw.S01      RP ─► rpout1 ─► nicsw.S02

 nicsw routes by tdest:
   0x0000–0xFFEF ─► QDMA C2H (to host)
   0xFFF0        ─► eth_tx_0 (CMAC0 wire)
   0xFFF1        ─► eth_tx_1 (CMAC1 wire)

The RP boundary is 3 streams in (rph2c, ethrx0, ethrx1) and 2 streams out (rpout0, rpout1). On single-CMAC boards (au50) there is no pkt_route_1, ethrx1, or eth_tx_1, and 0xFFF1 is unused.

tdest routing convention

tdest value	Destination
0x0000–0xFFEF	Host (QDMA C2H)
0xFFF0	CMAC0 TX (eth_tx_0)
0xFFF1	CMAC1 TX (eth_tx_1)

Host packet access (default config)

• Receive (wire → host): each pkt_route_N (programmed at 0x208000/0x209000) by default forwards received frames out m_axis_0 with a host-range tdest, so nicsw delivers them to QDMA C2H; the host reads them off its C2H queues. Frames steered out m_axis_1 go into the RP (ethrxN) for user-logic processing instead.
• Transmit (host → wire): the host writes a packet through a QDMA H2C queue; h2c_dst_trans_i (0x20C000) stamps the AXIS tdest from the descriptor. tdest 0xFFF0 reaches CMAC0, 0xFFF1 reaches CMAC1; ≤0xFFEF is delivered into the RP via rph2c.

RX-to-host and TX-from-host do not require an RP to be loaded — the rpen gating keeps a detached/blank RP from stalling these paths. The default loopback RP merges its three inputs into one FIFO and splits back to rpout0/rpout1 by tdest bit 15.

Configuring the stream IPs (register models)

The tdest-steering blocks are custom BSV IPs configured over AXI-Lite. Their register models are summarised here so software can drive them directly. All registers are 32-bit, little-endian, word-aligned; offsets are relative to each block's window (see the address maps above). BSV source lives in the mds_ethpl repo (src/AxisPacketRouter.bsv, src/AxisDestTrans.bsv + .md).

AxisPacketRouterDual — pkt_route_N and the RP's AxisPacketRouterDual_0

A header match-action router: it matches the first beat of each packet against a table, then for the winning entry replaces the outgoing tdest and selects which downstream master (m_axis_0 / m_axis_1) the packet leaves on. The packed variant (pktrte_dual, mkAxisPacketRouterCmac27) is 512-bit data, 2 downstream ports, 8 entries (entry 0 + 7 match entries).

Offset	Access	Layout
0x000	R / W	R: IDENT [31:8]=0xC32956, [7:4]=#downstreams, [3:0]=#entries. W any non-zero = soft reset
0x004	R / W	force ctrl: [4]=force_drop (drop all), [0]=force_match enable, [11:8]=forced entry id
0x008 / 0x00C	R	upstream packet count, low / high 32 bits
0x010 + 0x10·e	R / W	entry e action word0: [3:0]=downstream port, [31:16]=TID reload
0x014 + 0x10·e	R / W	entry e action word1: [15:0]=egress tdest (overwrites the beat), [31:16]=metadata
0x018 / 0x01C + 0x10·e	R	entry e matched-packet count, low / high
0x100 + 0x80·(e-1)	R / W	entry e (≥1) 512-bit header match value, [5:2]=32-bit word index (0–15)
0x140 + 0x80·(e-1)	R / W	entry e (≥1) 512-bit mask, [5:2]=word index

Match semantics: a beat matches entry e when (data & mask) == (header & mask) over the kept bytes. Entry 0 is the miss/default action (no header) used only when nothing else matches; entries 1…N-1 are header entries, lowest index wins.

Reset quirk: entries reset to {port 0, tdest 0, mask 0, header 0}, and a zero mask matches every packet. So at reset entry 1 (lowest header entry) catches all traffic and forwards it to downstream port 0 with tdest 0. To change the default behaviour you must reprogram entry 1, not entry 0 — writing only entry 0 has no effect while a maskless entry 1 still matches everything.

AxisDestTrans — h2c_dst_trans_i

An inline tdest translator (ternary CAM). It passes every beat through unchanged except tdest, which is rewritten from the first beat's value and applied to the whole packet. The packed variant (mkAxisDestTrans15) has 15 entries.

Offset	Access	Layout
0x000	R	IDENT [31:8]=id prefix, [7:0]=entry count
0x004	R / W	DEFAULT_TRANS (miss action): [31:16]=mask, [15:0]=value
0x008 + 8·i	R / W	ENT_MATCH[i]: [31:16]=match_mask, [15:0]=match_value
0x00C + 8·i	R / W	ENT_TRANS[i]: [31:16]=trans_mask, [15:0]=trans_value

An entry is enabled only when both its match_mask and trans_mask are non-zero; lowest index wins; on a miss DEFAULT_TRANS applies. The result is tdest_out = (tdest_in & ~trans_mask) | (trans_value & trans_mask). Reset is all-zero → identity passthrough (transparent wire).

The RP's internal AxisPacketRouterDual_0 is not on the AXI-Lite xbar — it lives in the QDMA AXI-MM space at 0x08200000, reached through the PCIe-bridge BAR (s_axi_pcie) or an MM DMA (s_axi_dma), per each board's rp_user.tcl. au280's stock RP has no such router: its egress is a fixed rp_out_sw axis_switch that splits by tdest bit 15 and cannot rewrite tdest, so the stock au280 RP can only return host-range traffic to C2H — it cannot turn an RP-range packet into a 0xFFF0/0xFFF1 wire-egress one. au50/au55c can, because their RP egress is the programmable AxisPacketRouterDual_0.

Supported boards

Board	Part	CMAC / QSFP	Memory	Vivado	README
au50	xcu50-fsvh2104-2-e	1 × 100 GbE	HBM	2024.2	boards/au50/README.md
au280	xcu280-fsvh2892-2L-e	2 × 100 GbE	DDR4	2023.2	boards/au280/README.md
au55c	xcu55c-fsvh2892-2L-e	2 × 100 GbE	HBM	2024.2	boards/au55c/README.md

au280's part is absent from Vivado 2024.2 — build it with 2023.2.

AXI interface conventions

All AXI interfaces follow standard AXI4 / AXI4-Lite / AXI-Stream directional conventions. script/check_axi.py enforces this across the primary RTL files (m_ = master, s_ = slave) and checks cross-module consistency between user_block.sv and rp_blk.v:

python3 script/check_axi.py        # PASS/FAIL only
python3 script/check_axi.py -v     # include intentional differences

Known-intentional interface mismatches are documented in AGENTS.md and must not be "fixed".

Releases

A release ships pre-built bitstreams as zipped packages, produced by the packagers in script/:

Package	Built by	Contents
Shell package (<name>_support_<uuid8>.zip)	mk_support_pkg.py	Full shell bitstream (shell_top.bit/.ltx) plus the pieces needed to build an app against it (Vivado part, rp_blk.v, board_config.vh, abstract-shell .dcp).
App package (<name>_app_<uuid8>.zip)	mk_app_pkg.py	The RP partial bitstream(s) and a manifest. --rp-only packs the partial bitstream only (no full shell bitstream) — the artifact to load onto an already-running shell.

Both packages are tagged with the shell's 8-character UUID (<uuid8>). The host reads the running shell's UUID from the info page (see the monolithic-UUID note under Building) and an app package's manifest records the shell UUID it was built against, so a partial bitstream can be matched to its shell in the field and a mismatched app is refused.

To build these from a finished Vivado project, see the packaged-shell app flow in AGENTS.md.

License

Licensed under the Apache License, Version 2.0 — see LICENSE.

This shell borrows and derives from the Xilinx OpenNIC Shell, which is also Apache-2.0; those files under src/ keep their original Xilinx headers. src/crc32.v is third-party (OutputLogic.com) under its own permissive terms. Borrowed sources remain under their original licenses; see NOTICE for details.