Add video encoding support with SYCL RGB→NV12 kernel and CPU fallback by eromomon · Pull Request #103 · intel/torchlib-xpu

eromomon · 2026-07-27T22:58:01Z

Adds a hardware video encoding path to the torchcodec-xpu plugin (supersedes #58)

What changes:

XpuDeviceInterface.{h,cpp} — implements three encoder overrides:

- get_encoding_pixel_format (pinned to NV12).
- setup_hardware_frame_context_for_encoding (VAAPI hw_frames_ctx, BT.709 + JPEG-range defaults, B-frames disabled, low_power=0, quality=1).
- convert_tensor_to_av_frame_for_encoding dispatching to:
- - -SYCL fast path — exports the VA surface as DMA-BUF, imports via Level Zero USM, runs convertRGBToNV12 directly on the surface (handles linear + Intel Tile-Y, and both NV12 export layouts on iHD/i915).
- - -CPU fallback — GBRP→NV12 via libswscale, then av_hwframe_transfer_data onto the VA surface. Used when SYCL is disabled or FP64 is unavailable (DG2 / ATS-M).
- find_codec extended to resolve VAAPI encoders, with H.264 / HEVC / AV1 fallback when the requested codec id has no VAAPI encoder.

ColorConversionKernel.{h,cpp} — adds convertRGBToNV12 SYCL kernel (BT.601 / BT.709, limited + full range); get_tile_offset reused for encode.
patches/0002-Add-XPU-support-to-video-encoder-tests.patch — parametrizes upstream torchcodec encoder tests over CUDA + XPU, and primes the encoder with one frame in test_write_frames_different_devices_errors (VAAPI segfaults on empty flush; NVENC doesn't). Depends on 0001-Add-XPU-support-to-tests.patch.

Compatibility
No change to decode behavior, plugin loading, or public API.

Testing
Encoder test suite (patches 0001 + 0002) passes on Intel Arc / Battlemage.

Signed-off-by: Edgar Romo Montiel <edgar.romo.montiel@intel.com>

Copilot

Pull request overview

Adds an Intel XPU (VAAPI) hardware video encoding path to the torchcodec-xpu plugin, including an on-device SYCL RGB→NV12 conversion kernel with a CPU/libswscale fallback, and updates upstream TorchCodec encoder tests (via patch) to run on XPU as well as CUDA.

Changes:

Implement XPU encoding overrides in XpuDeviceInterface (pixel format selection, VAAPI hw_frames_ctx setup, tensor→AVFrame upload path with SYCL fast-path + CPU fallback).
Add SYCL convertRGBToNV12 kernel supporting BT.601/BT.709 and full/limited range, including tiled (Intel Tile-Y) surface handling.
Extend/parameterize upstream encoder tests (patch) to exercise XPU and avoid VAAPI flush-on-empty segfault by priming with a frame.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
packages/torchcodec-xpu/src/torchcodec_xpu/XpuDeviceInterface.h	Declares XPU encoding overrides and helper conversion entry points.
packages/torchcodec-xpu/src/torchcodec_xpu/XpuDeviceInterface.cpp	Implements VAAPI frames context setup, VAAPI encoder discovery logic, and SYCL/CPU tensor→NV12 surface conversion.
packages/torchcodec-xpu/src/torchcodec_xpu/ColorConversionKernel.h	Exposes the new SYCL RGB→NV12 conversion API.
packages/torchcodec-xpu/src/torchcodec_xpu/ColorConversionKernel.cpp	Implements RGB→NV12 SYCL kernel and supporting YUV conversion matrices.
packages/torchcodec-xpu/patches/0002-Add-XPU-support-to-video-encoder-tests.patch	Updates upstream TorchCodec encoder tests to include XPU device coverage and VAAPI-specific priming behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

eromomon · 2026-07-29T00:38:50Z

+  // Encoder-only fallback: if no VAAPI encoder exists for codecId
+  // substitute a HW-capable alternative 
+  if (!is_decoder) {
+    static constexpr AVCodecID kHwEncoderFallbacks[] = {
+        AV_CODEC_ID_H264,
+        AV_CODEC_ID_HEVC,
+        AV_CODEC_ID_AV1,
+    };
+    for (AVCodecID fb : kHwEncoderFallbacks) {
+      if (fb == codec_id) {
+        continue;
+      }
+      if (const AVCodec* c = findVaapiForId(fb)) {
+        VLOG(1) << "No VAAPI encoder for codec id " << codec_id
+                << ", substituting " << c->name;
+        return c;
      }
    }
  }


see comment

+  hwFramesCtx->width     = codec_context->width;
+  hwFramesCtx->height    = codec_context->height;
+
+  // XPU quality is matterially better when using BT.709 color space and full range


eromomon · 2026-07-29T00:38:03Z

+import torchcodec
+import torchcodec_xpu
+


Patch updated

dvrogozh · 2026-07-27T23:29:02Z

+  // Disable B-frames (mirrors CUDA/NVENC delay=0) to avoid reorder issues
+  // with fragmented containers; callers can restore via extra_options={"bf":"2"}.


Reordering is not an issue - that's how media codecs behave. I don't understand what you mean by "fragmented containers", so drop it.

Suggested change

// Disable B-frames (mirrors CUDA/NVENC delay=0) to avoid reorder issues

// with fragmented containers; callers can restore via extra_options={"bf":"2"}.

// Disable B-frames (mirrors CUDA/NVENC delay=0) to avoid frames reordering.

// Callers can restore via extra_options={"bf": "2"}.

Question: extra_options is a TorchCodec option?

Yes, extra_options is a TorchCodec public API parameter on _video_encoder.py

dvrogozh · 2026-07-27T23:38:43Z

+    return c;
+  }
+
+  // Encoder-only fallback: if no VAAPI encoder exists for codecId


Remind me, please, why do we need this? What will start to fail if we will drop this?

Removing the fallback here regresses XPU encoding on any FFmpeg build that lacks h264_vaapi. The block is not a CPU fallback: it substitutes a different VAAPI encoder from the same family, so frames continue to be encoded on the Intel Media Engine — only the bitstream codec ID changes, and Encoder.cpp propagates that through codec_context->codec_id = av_codec->id, so the MP4 muxer writes the correct sample entries.

The reason this cannot simply be dropped today is that torchcodec's initialize_video_stream unconditionally invokes our setup_hardware_frame_context_for_encoding, which forces pix_fmt = AV_PIX_FMT_VAAPI and attaches an AVHWFramesContext. When find_codec returns nullopt, torchcodec falls back to avcodec_find_encoder() — a software encoder (e.g. libx264) — and avcodec_open2 then rejects the VAAPI pixel format with AVERROR(EINVAL) ("Invalid argument"), which is exactly the reproduced failure.

RuntimeError: initialize_video_stream, /__w/torchcodec/torchcodec/meta-pytorch/torchcodec/src/torchcodec/_core/Encoder.cpp:500, avcodec_open2 failed: Invalid argument

When find_codec returns nullopt

This is in the assumption that ffmpeg was built without h264_vaapi, right? find_codec should not return nullopt if ffmpeg was built with h264_vaapi, correct?

…e clarity. Update patch file for test encoding to align with autoloader changes. Signed-off-by: Edgar Romo Montiel <edgar.romo.montiel@intel.com> Co-authored-by: Dmitry Rogozhkin <dmitry.v.rogozhkin@intel.com>

eromomon added 3 commits July 27, 2026 08:32

Add XPU encoding support to Encoder

f0871cc

Signed-off-by: Edgar Romo Montiel <edgar.romo.montiel@intel.com>

Update encoder side implementation for XPU

112c56f

Signed-off-by: Edgar Romo Montiel <edgar.romo.montiel@intel.com>

Remove shutdown entry point from XPU device interface.

7b9bb37

Signed-off-by: Edgar Romo Montiel <edgar.romo.montiel@intel.com>

eromomon requested review from dvrogozh and luis-real as code owners July 27, 2026 22:58

eromomon requested review from Copilot and removed request for dvrogozh and luis-real July 27, 2026 22:58

Copilot started reviewing on behalf of eromomon July 27, 2026 22:58 View session

Copilot AI reviewed Jul 27, 2026

View reviewed changes

eromomon requested a review from dvrogozh July 27, 2026 23:29

dvrogozh requested changes Jul 27, 2026

View reviewed changes

Update TORCH_CHECK messages to align with coding standards and improv…

459723a

…e clarity. Update patch file for test encoding to align with autoloader changes. Signed-off-by: Edgar Romo Montiel <edgar.romo.montiel@intel.com> Co-authored-by: Dmitry Rogozhkin <dmitry.v.rogozhkin@intel.com>

eromomon requested a review from dvrogozh July 29, 2026 00:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add video encoding support with SYCL RGB→NV12 kernel and CPU fallback - #103

Add video encoding support with SYCL RGB→NV12 kernel and CPU fallback#103
eromomon wants to merge 4 commits into
intel:mainfrom
eromomon:eromomon/encoding_no_patch

eromomon commented Jul 27, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

eromomon Jul 29, 2026

Uh oh!

Uh oh!

Uh oh!

eromomon Jul 29, 2026

Uh oh!

Uh oh!

dvrogozh Jul 27, 2026

Uh oh!

eromomon Jul 29, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dvrogozh Jul 27, 2026

Uh oh!

eromomon Jul 29, 2026

Uh oh!

dvrogozh Jul 29, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		// Disable B-frames (mirrors CUDA/NVENC delay=0) to avoid reorder issues
		// with fragmented containers; callers can restore via extra_options={"bf":"2"}.

Uh oh!

Conversation

eromomon commented Jul 27, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

eromomon Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

eromomon Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dvrogozh Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

eromomon Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dvrogozh Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

eromomon Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

dvrogozh Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dvrogozh Jul 29, 2026 •

edited

Loading