Skip to content

Use int8 on pre-Volta GPUs instead of float16 (fixes CUDA on Pascal)#48

Open
Suijiku wants to merge 1 commit into
ObscureAintSecure:masterfrom
Suijiku:cuda-compute-type-fallback
Open

Use int8 on pre-Volta GPUs instead of float16 (fixes CUDA on Pascal)#48
Suijiku wants to merge 1 commit into
ObscureAintSecure:masterfrom
Suijiku:cuda-compute-type-fallback

Conversation

@Suijiku

@Suijiku Suijiku commented Jul 9, 2026

Copy link
Copy Markdown

Closes #45.

Summary

transcriber.py hardcodes float16 whenever the compute device is cuda:

compute_type = "float16" if device == "cuda" else "int8"

CTranslate2 (faster-whisper's backend) rejects float16 on pre-Volta NVIDIA GPUs
(Pascal / GTX 10xx and older), failing transcription with:

target device does not support efficient float16 computation

This makes the CUDA option unusable on those cards even though int8 works fine on them
(Pascal supports DP4A int8). This PR adds a compute-capability check: pre-Volta GPUs
(compute capability major < 7) fall back to int8 and surface a one-line progress note;
Volta and newer keep float16. If the capability probe fails for any reason we keep the
existing float16 behavior so the current error still surfaces rather than being masked.

The model cache key already includes compute_type, so no cache changes are needed.

Changes

  • app/transcription/transcriber.py — replace the one-line ternary with a capability check.

Testing

Reproduced on a GTX 1080 (Pascal, compute capability 6.1), Windows 11, CUDA 12.6
PyTorch, faster-whisper 1.2.1 / CTranslate2 4.8.1.

CTranslate2's supported CUDA compute types on this device — note float16 is absent:

ct2 supported compute types (cuda): {'int8_float32', 'int8', 'float32'}

TranscriptionWorker(device="cuda") before vs after, same machine:

# master (old code, hardcoded float16):
ERROR: Transcription failed: Requested float16 compute type, but the target device
       or backend do not support efficient float16 computation.
# -> transcription fails, no result

# this PR (cuda-compute-type-fallback):
Loading transcription model...
GPU is pre-Volta — using int8 compute (float16 unsupported).
Transcribing audio...
Transcription complete.
# -> succeeds on int8
  • Unit tests: python -m pytest tests/ -v green, including 4 new tests covering the
    CPU / Volta+ / pre-Volta / probe-failure branches (torch mocked).
  • The pre-fix failure is a hard ValueError from CTranslate2 (not a silent fallback), so
    the old CUDA path is genuinely unusable on pre-Volta cards; this PR makes it work.

CTranslate2 rejects float16 on pre-Volta NVIDIA GPUs (Pascal/GTX 10xx and
older) with 'target device does not support efficient float16 computation',
making the CUDA path unusable there even though int8 works (DP4A). Probe the
compute capability and fall back to int8 (major < 7), surfacing a progress
note; Volta+ keeps float16. On any probe failure we keep float16 so the
existing error still surfaces. The model cache key already includes
compute_type, so no cache change is needed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CUDA path hardcodes float16, breaking pre-Volta (Pascal) GPUs

1 participant