Your meeting ends at 5:00 PM. By 5:02, you want a full transcript, edited and searchable, without sending a single audio byte to Google or OpenAI. That’s the promise of a self-hosted Whisper Large-v3 server — and in 2026 it’s genuinely achievable on consumer hardware you already own.

Hardware requirements, backend comparison, step-by-step server setup, and the honest assessment of where the model still falls short — all below.

What Whisper Large-v3 Actually Is

OpenAI released Whisper Large-v3 on November 6, 2023. The architecture is identical to Large-v2 with two changes: the input uses 128 Mel frequency bins instead of 80, and a Cantonese language token was added. Those changes, combined with a training dataset of 1 million hours of weakly labeled audio plus 4 million hours of pseudo-labeled audio, produced a 10–20% reduction in word error rate across the supported language set compared to v2.

The model has 1.55 billion parameters and supports 99 languages (100 if you count Cantonese separately). On LibriSpeech test-clean — a clean audiobook benchmark — it hits 2.01% WER. On messier real-world audio (earnings calls, podcasts, meetings), expect 8–15% WER. Still the best open-source option by a wide margin.

The model weights on disk are ~3.0 GB. At inference time, GPU VRAM usage depends heavily on which backend you choose.

Hardware Requirements

VRAM: Lower Than You Think

The stock OpenAI Whisper package in float16 requires around 10 GB of VRAM for the Large-v3 model — and float32 doubles that. Run it naive on a 12 GB RTX 3060 and you’ll be tight.

The faster-whisper backend changes this completely:

Precision	VRAM (faster-whisper)	Notes
float16 (FP16)	~3.1 GB	Base load; add ~20% for inference overhead
int8_float16	~2.9 GB	Best accuracy-per-VRAM ratio
int8	~2.9 GB	CPU or GPU; minimal accuracy loss
Batched (batch_size=8, INT8)	~4.5 GB	Throughput mode for bulk files

Practical floor: any GPU with 6 GB VRAM runs Large-v3 comfortably via faster-whisper. A 4 GB card (GTX 1650, RX 570) is borderline — use int8 precision and keep batch size at 1.

For GPU selection context, see our GPU buying guide and the best local AI models by VRAM tier breakdown.

GPU Tier Benchmarks

The table below measures seconds of processing time per minute of audio (lower is better) using Whisper Large-v3 and the original OpenAI Whisper backend. Data sourced from 1 QuBit’s cross-GPU benchmark (measured on long-form audio files):

GPU	VRAM	Avg sec/min audio	Implied RTF	Real-time?
RTX 4090	24 GB	~7 sec	~0.12	Yes, 8× faster
RTX 3090	24 GB	~12–22 sec	~0.20–0.37	Yes, 3–5× faster
RTX 4060 Ti 16GB	16 GB	~18 sec	~0.30	Yes, ~3× faster
RTX 3060	12 GB	~35 sec	~0.58	Borderline (faster-whisper improves this)
CPU (Intel i9)	—	~150 sec	~2.5	No — 2.5× slower than real-time

RTF (Real-Time Factor) = seconds of processing per second of audio. RTF < 1.0 means faster than real-time. RTF < 0.5 is good for live captioning. RTF < 0.1 is excellent for latency-sensitive pipelines.

An RTX 4090 with faster-whisper and Flash Attention 2 can push 70–100× real-time on short clips, and around 8× on long files. The RTX 3090 lands at 3–5× depending on VRAM pressure and compute type setting. The RTX 3060 is borderline with the stock backend but gets to ~2× real-time with faster-whisper’s INT8 quantization enabled.

If you’re buying hardware specifically for transcription workloads, a used RTX 3090 hits the sweet spot: 24 GB of VRAM means no memory pressure even at batch_size=8, and it benchmarks at 3–5× real-time for pennies on the dollar versus a new card. See Amazon for current pricing.

Not ready to buy? You can run Whisper Large-v3 on a cloud GPU for a few cents per hour while you validate the setup. RunPod offers NVIDIA A100 and H100 instances where you can benchmark the full pipeline before committing to local hardware.

CPU Fallback

Running Large-v3 on CPU is possible with faster-whisper’s INT8 path. On an Intel i9-12900K, expect RTF around 2.5 — meaning 1 second of audio takes 2.5 seconds to transcribe, and a 1-hour meeting takes 2.5 hours. That’s fine for overnight batch jobs on voice memos, but useless for any live or near-real-time use case. Downsize to the medium or small model if CPU-only is your reality.

The Backend Decision: Three Options

Option 1: faster-whisper (Recommended)

faster-whisper reimplements Whisper using CTranslate2, a C++ inference engine optimized for transformer models. It’s up to 4× faster than the stock OpenAI package at the same accuracy, uses 50–70% less VRAM, and supports INT8 quantization on both GPU and CPU. This is the backend to use for a server deployment.

Pros: Best speed-per-VRAM ratio, active maintenance, OpenAI API-compatible server wrappers available, word-level timestamps via VAD filter.
Cons: Requires CUDA 12, cuBLAS, and cuDNN 9 — the dependency chain trips up first-time installs on older CUDA setups.

Option 2: whisper.cpp

whisper.cpp (by Georgi Gerganov, the author of llama.cpp) is a pure C/C++ implementation that runs on CPU, CUDA, Metal, and OpenCL. It uses quantized GGML weights and is the most portable option — runs on a Raspberry Pi 5, a Mac Mini, or a Windows machine without Python.

Pros: No Python, no CUDA required, smallest memory footprint of the three, excellent for embedded or edge deployment.
Cons: Slower than faster-whisper on NVIDIA GPUs; hallucination rate 20% higher than faster-whisper in controlled tests; no official streaming API out of the box.

Option 3: Original OpenAI Whisper

The original package is the reference implementation, runs on PyTorch, and is the easiest to install. It’s also the slowest and most memory-hungry. If you have 12–16 GB VRAM and are doing casual single-file transcription, it works. For a server that stays running and handles concurrent requests, use faster-whisper instead.

Verdict: Use faster-whisper for any server deployment. Use whisper.cpp for resource-constrained or non-NVIDIA hardware. Use original Whisper only for quick one-off experiments.

Tutorial: Installing faster-whisper and Running a Transcription Server

The setup below uses faster-whisper + FastAPI for a lightweight HTTP endpoint that accepts audio file uploads and returns transcribed text. This stack is sufficient for personal use, meeting transcription, and family/team servers.

For a multi-user team server setup, the patterns in our Open WebUI family setup guide apply directly — replace the model backend with this transcription API and put Caddy in front.

Prerequisites

• Python 3.9+
• NVIDIA GPU with CUDA 12 installed (or CPU-only with device="cpu")
• cuBLAS and cuDNN 9 (included with recent CUDA Toolkit distributions)
• ffmpeg (for audio preprocessing; sudo apt install ffmpeg on Linux, winget install ffmpeg on Windows)

Step 1: Create a Virtual Environment

python -m venv whisper-env
# Linux/macOS
source whisper-env/bin/activate
# Windows
whisper-env\Scripts\activate

Step 2: Install Dependencies

pip install faster-whisper fastapi uvicorn python-multipart

Verify GPU access after installation:

from faster_whisper import WhisperModel
model = WhisperModel("large-v3", device="cuda", compute_type="int8_float16")
print("Model loaded successfully")

If this throws a CUDA or cuDNN error, the most common fix is reinstalling cuDNN 9: pip install nvidia-cudnn-cu12.

Step 3: Write the Server

Create transcription_server.py:

import io
from fastapi import FastAPI, UploadFile, File
from fastapi.responses import JSONResponse
from faster_whisper import WhisperModel

app = FastAPI(title="Whisper Large-v3 Transcription Server")

# Load model once at startup — large-v3 takes ~10 seconds to load
# Use device="cpu" and compute_type="int8" for CPU-only machines
model = WhisperModel(
    "large-v3",
    device="cuda",           # or "cpu"
    compute_type="int8_float16",  # ~3 GB VRAM; use "float16" for higher accuracy
    num_workers=1,
)

@app.post("/transcribe")
async def transcribe(
    file: UploadFile = File(...),
    language: str = None,   # e.g., "en", "es", "zh" — None = auto-detect
    task: str = "transcribe" # or "translate" (translates to English)
):
    audio_bytes = await file.read()
    
    segments, info = model.transcribe(
        io.BytesIO(audio_bytes),
        language=language,
        task=task,
        beam_size=5,
        vad_filter=True,           # suppress silence; reduces hallucinations
        vad_parameters=dict(min_silence_duration_ms=500),
    )
    
    transcript = " ".join(segment.text.strip() for segment in segments)
    
    return JSONResponse({
        "language": info.language,
        "language_probability": round(info.language_probability, 3),
        "duration_seconds": round(info.duration, 2),
        "text": transcript,
    })

@app.get("/health")
def health():
    return {"status": "ok", "model": "whisper-large-v3"}

Step 4: Start the Server

uvicorn transcription_server:app --host 0.0.0.0 --port 8765 --reload

The server is now running at http://localhost:8765. Test it:

curl -X POST "http://localhost:8765/transcribe" \
  -F "file=@meeting-recording.mp3" \
  -F "language=en"

Response:

{
  "language": "en",
  "language_probability": 0.998,
  "duration_seconds": 3420.5,
  "text": "Good morning everyone, let's get started with the Q2 review..."
}

Step 5: Docker Option (Recommended for Stability)

The speaches project wraps faster-whisper in an OpenAI API-compatible server with Docker support. One command brings up a fully functional transcription endpoint:

# GPU version
docker run \
  --gpus=all \
  --publish 8000:8000 \
  --volume hf-hub-cache:/home/ubuntu/.cache/huggingface/hub \
  --detach \
  ghcr.io/speaches-ai/speaches:latest-cuda

The endpoint is OpenAI-compatible, meaning any client that uses openai.Audio.transcriptions.create() works against it with a single URL change.

Step 6: Running on Windows

Windows support works but has one gotcha: CUDA and cuDNN paths need to be on PATH. After installing CUDA Toolkit 12:

$env:PATH += ";C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.x\bin"
$env:PATH += ";C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.x\libnvvp"

If you see Could not load library cudnn_ops_infer64_8.dll, download cuDNN 9 from NVIDIA’s site and place the DLLs in the CUDA bin directory.

Language Support and Accuracy

Whisper Large-v3 officially supports 99 languages. Performance is uneven:

High accuracy (WER < 10%): English, German, Spanish, French, Japanese, Mandarin Chinese, Portuguese, Dutch, Italian, Polish
Good accuracy (WER 10–20%): Arabic, Hindi, Russian, Korean, Turkish, Swedish, Norwegian
Lower accuracy (WER > 20%): Low-resource languages, heavy regional accents, code-switching between languages

On English, Large-v3 achieves 2.01% WER on clean audio (LibriSpeech) and roughly 5.6% on Community Voice 15. For a meeting with multiple accents and background noise, budget for 10–15% errors. That’s still better than most cloud providers at most price points.

The language parameter in faster-whisper matters: always specify the language when you know it. Auto-detection adds latency (the model processes the first 30 seconds before deciding) and occasionally mis-classifies quiet or multilingual audio.

Use Cases Worth Building Around

Meeting transcription: A 1-hour Zoom recording (saved as MP4) transcribes in 5–12 minutes on an RTX 3090. Feed it to a local LLM for summarization afterward — fully private, end-to-end.

Video captions: Pipe audio through the /transcribe endpoint and export with output_format="srt" using faster-whisper directly. The SRT file drops straight into Premiere, DaVinci, or VLC.

Personal voice memos: A simple iOS Shortcut or Android Tasker script can POST a recording to your home server the moment you’re on WiFi. No dictation app, no subscription, nothing leaves your network.

Medical and legal notes: This is where the privacy argument becomes a compliance argument. Cloud transcription services typically route audio through third-party infrastructure. A self-hosted Whisper deployment keeps audio on your hardware — relevant for HIPAA-adjacent workflows (clinical notes, patient calls) where sending audio to a cloud API introduces risk. Worth discussing with counsel before deploying for regulated use.

The Honest Limitations

Whisper Large-v3 is the best open-source transcription model in 2026. It’s not perfect:

No built-in speaker diarization. Whisper outputs one continuous transcript. It cannot tell you “Alice said X, Bob said Y.” For that you need WhisperX, which combines Whisper with pyannote-audio’s diarization pipeline. That adds roughly 1–2 GB of VRAM and another dependency (pyannote requires a Hugging Face token and acceptance of license terms), but it produces speaker-labeled output.

Word-level timestamps need an alignment model. The raw Whisper output gives segment-level timestamps, accurate to within a few seconds. True word-level timestamps (useful for captions sync or editing) require running a wav2vec2 alignment model as a post-processing step, which WhisperX also handles.

Hallucination on silence. If you feed Whisper a file with long silent sections, it will sometimes generate plausible-sounding text that was never spoken. The vad_filter=True setting in faster-whisper catches most of this by stripping silence before inference. Always enable it.

Not true real-time streaming. Whisper processes audio in 30-second chunks. You can build a streaming pipeline by splitting incoming audio into overlapping chunks and stitching the output, but latency is 5–10 seconds minimum on consumer hardware. For sub-1-second latency streaming, look at Whisper Streaming (experimental) or a smaller model like distil-large-v3.

CPU is impractical for Large-v3. An i9-class CPU transcribes at 2.5× slower than real-time. For CPU-only hosts, drop to medium or small.

The setup in one paragraph

The setup from this guide — faster-whisper + FastAPI — runs on any NVIDIA GPU from a GTX 1070 upward (6 GB VRAM minimum with INT8 precision). It starts in under 10 seconds after the first model download and handles audio files in any format ffmpeg understands: MP3, MP4, M4A, WAV, OGG, WebM.

For a household or small-team server that multiple people share, add Caddy basicauth in front of the uvicorn process — the same pattern from the Open WebUI family setup guide applies unchanged.

Performance ceiling: on an RTX 4090 (available on Amazon), faster-whisper with compute_type="float16" processes a 1-hour recording in under 9 minutes. On an RTX 3090 (Amazon), plan for 12–22 minutes per hour of audio depending on load. For a personal transcription server, that’s more than fast enough.

The 29 articles before this one covered GPUs, LLMs, and image generation. Transcription is where the privacy argument becomes concrete: it’s the one AI workload where the alternative — uploading audio of your meetings, your doctor’s appointments, your confidential calls — is genuinely uncomfortable. Running Whisper locally closes that gap for under $10 in electricity per month.

Sources

• Whisper Large-v3 model card — Hugging Face
• large-v3 release announcement — openai/whisper Discussion #1762
• SYSTRAN/faster-whisper GitHub — CTranslate2 benchmarks and installation
• OpenAI Whisper performance benchmarks across GPUs — 1 QuBit
• Whisper V3 Benchmarks: Performance, Accuracy, and Speed Analysis — SayToWords
• OpenAI Whisper Audio Transcription Benchmarked on 18 GPUs — Tom’s Hardware
• Choosing between Whisper variants: faster-whisper, insanely-fast-whisper, WhisperX — Modal
• speaches: OpenAI-compatible server for faster-whisper — GitHub
• WhisperX: Automatic Speech Recognition with Word-level Timestamps & Diarization — GitHub
• Whisper Large-v3 GPU requirements and VRAM discussion — Hugging Face Discussion #83

Last updated May 13, 2026. Model versions and benchmark numbers change; verify current specs before deploying in production.