VCK5000 ES1 (CP-AFU): timing-clean build hangs — CP CMD_LAUNCH never retires (cp_busy=0), follow-up to #333

[hwirys]

Summary

Follow-up to #333 (closed as superseded). That report targeted the pre-rewrite AFU control FSM; the close note there asked to "rebuild on the current CP-based AFU with timing met (WNS ≥ 0); reopen with a fresh trace if a timing-clean build still hangs." I rebuilt on the current Command-Processor AFU. Timing is fully met (WNS +0.233 ns) and the kernel launch still hangs — and it is not in the Vortex cores: the CP's CMD_LAUNCH never retires while the CP reports itself idle. Filing as a new issue with the fresh trace (I'm read-only on the repo, so I can't reopen #333 myself).

Build (current CP-AFU master + #363)

Item	Value
Platform	xilinx_vck5000-es1_gen3x16_2_202020_1 (Versal ES1)
Vivado/Vitis	2020.2
Config	XLEN=32, NUM_CORES=2, single-bank, no-L2, KERNEL_FREQ=200
Final routed timing	WNS +0.233 ns, WHS +0.014 ns — "All user specified timing constraints are met."

So the timing-closure hypothesis from #333 is ruled out for this bitstream.

Two VCK5000-specific runtime issues resolved first (disclosed for reproducibility)

The stock CP-AFU runtime path does not work as-is on the VCK5000 base shell; two adaptations were needed before the kernel path could be exercised at all:

1. No host-memory bridge. The driver (sw/runtime/xrt/vortex.cpp) allocates its CP-visible region (command ring + DMA staging) as xrt::bo::flags::host_only, reached by the CP's m_axi_host master over the PCIe slave bridge. The gen3x16_2 base shell has no host-mem bridge (/sys/.../host_mem_size = 0, CmaTotal: 0 kB), so host_only allocation fails with Operation not permitted. Vitis auto-maps both m_axi_host and m_axi_mem_0 to on-card DDR (MC_NOC0, 8 GB @ 0xC000000000). I adapted the driver to allocate the CP region as a normal device BO on that bank and keep it coherent with explicit bo.sync() — TO_DEVICE at the Q_TAIL doorbell, FROM_DEVICE on Q_SEQNUM advance.

2. s_axi_ctrl register window = 4 KB in metadata. The packaged xclbin declares <port name="s_axi_ctrl" … range="0x1000">, so xrt::ip rejects CP-regfile accesses at host offset 0x1000+ ("Cannot read or write outside ip register space"). The RTL is correct (C_S_AXI_CTRL_ADDR_WIDTH = 16, decodes 64 KB) and package_kernel.tcl sets range 65536, but the linked EMBEDDED_METADATA came out 0x1000 under Vitis 2020.2. Patched in place with xclbinutil (range → 0x10000); no re-synthesis required.

(Also: the kernel is ap_ctrl_none, so the driver's AP_RESET write is a no-op and XRT skips reprogramming on a same-UUID load — the CP's seqnum persists across runs and must be cleared with xbutil reset before each run.)

The hang — CP LAUNCH never retires; the CP is idle, not stalled

With the above in place, the full command stream executes and then deadlocks on exactly one command. Per-command CP trace (opcode + completion-counter target):

cmd 1–6    op=0x01  MEM_WRITE  (binary + data)   -> complete, seqnum 1..6
cmd 7–26   op=0x04  DCR_WR     (kernel config)   -> complete, seqnum 7..26
cmd 27     op=0x06  LAUNCH                        -> never completes, seqnum frozen at 26

The host then spins Q_SEQNUM forever (100M+ polls). The decisive read taken during the hang:

CP_STATUS (regfile 0x004) = 0x00000000   ->   cp_busy = 0,  cp_error = 0

cp_busy = 0 is the key observation. VX_cp_launch (IDLE → PULSE_START → WAIT_BUSY → WAIT_DRAIN) holds the KMU bid — and thus keeps cp_busy = 1 — for as long as it waits on gpu_busy to rise (kernel never started) or fall (kernel running/hung). Seeing cp_busy = 0 means the CP is idle: not stalled in the launch FSM, no in-flight kernel, no error. So this is not a core-compute hang and not a timing-latched-data wedge.

Yet the completion counter never reaches 27. So CMD_LAUNCH (op 0x06) appears to either not be consumed from the ring, or to complete in hardware without advancing Q_SEQNUM. All 26 preceding MEM_WRITE/DCR_WR commands retire normally, so the ring / doorbell / seqnum mechanics themselves work.

Questions

• On the current CP-AFU, what advances Q_SEQNUM for a CMD_LAUNCH, and can a launch complete with cp_busy already deasserted? Is there a path where VX_cp_launch returns to IDLE without the engine counting the launch as retired — e.g. a missed gpu_busy rising edge (the FSM samples gpu_busy the cycle after start)?
• Is CMD_LAUNCH expected to work when the CP's m_axi_host is mapped to device DDR (no host-mem bridge)? Could the launch path rely on host-coherent ring/staging that explicit bo.sync brackets don't cover at the granularity the CP needs?

Hardware is available for further instrumentation (ChipScope/ILA on gpu_busy, ring head/tail, and the completion counter). Happy to share the full traces, the device-mem driver diff, and the build artifacts.

---

Cross-reference: #333 (VCK5000 ES1 kernel hang; closed as superseded by the CP-AFU rewrite).