TL;DR: A failed llama.cpp CUDA build almost always comes down to one of four things: an outdated CMake flag, a CUDA toolkit CMake can’t find, a GPU architecture your CUDA version doesn’t know about (hello, RTX 50-series), or a GCC that’s too new for nvcc. Read the first error in the log, not the last. Each has a one-line fix.

What you’ll be able to do after this guide:

• Read a CUDA build log and know which of the four failure classes you’re hitting.
• Compile a GPU-accelerated llama-cli on Linux and Windows, including on Blackwell (RTX 5070/5080/5090).
• Decide when building from source is worth it at all versus just running Ollama.

Honest take: Ninety percent of “llama.cpp won’t build” reports are a stale copy-pasted command (LLAMA_CUBLAS) or a CUDA toolkit that predates your GPU. Fix the flag, match the toolkit to your card, and the build goes green. If you don’t specifically need a bleeding-edge feature, don’t build at all — a prebuilt release or Ollama gets you the same kernels.

The one rule: read the first error, not the last

A CUDA build fails loudly. By the time cmake --build stops, your terminal is a wall of red, and the last line is usually a generic “Build failed” or a downstream linker complaint. That line is a symptom. The real cause is the first error nvcc or CMake printed — scroll up.

Everything below is keyed to that first error string. Match it, apply the fix, rebuild. The build directory is disposable: if a fix changes CMake flags, delete build/ and re-run cmake -B build ... from scratch so stale cache entries don’t haunt you.

30-second diagnosis

First error you see	Real cause	Jump to
LLAMA_CUBLAS is deprecated ... Use GGML_CUDA instead	Old build flag from a pre-2024 tutorial	Fix 1
Could NOT find CUDAToolkit / No CUDA toolset found	CMake can’t locate nvcc	Fix 2
nvcc fatal : Unsupported gpu architecture 'compute_120'	CUDA too old for your Blackwell GPU	Fix 3
#error -- unsupported GNU version! gcc versions later than 15 are not supported!	Host GCC newer than your CUDA allows	Fix 4
Build hangs, then Killed / c++: internal compiler error	Out of RAM compiling the CUDA kernels	Fix 5

Fix 1: LLAMA_CUBLAS is deprecated — the flag got renamed

This is the single most common failure, and it’s entirely the fault of old blog posts. For years the CUDA build flag was LLAMA_CUBLAS. It was renamed, and current llama.cpp rejects it:

CMake Error: LLAMA_CUBLAS is deprecated and will be removed in the future. Use GGML_CUDA instead

The build option now lives in ggml (the tensor library under llama.cpp), so the flag is GGML_CUDA. The correct, current build is:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

That’s it — two cmake calls. If you’re following any guide that still says make LLAMA_CUBLAS=1 or cmake -DLLAMA_CUDA=ON, it’s out of date. GGML_CUDA=ON is the flag as of 2026. For llama-cpp-python, the same rename applies:

CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python

Fix 2: Could NOT find CUDAToolkit — CMake can’t see nvcc

CMake looks for the CUDA compiler (nvcc) on your PATH and in a few standard locations. If the toolkit isn’t installed, or it’s installed somewhere CMake doesn’t check, you get:

CMake Error: Could NOT find CUDAToolkit (missing: CUDAToolkit_INCLUDE_DIR ...)

On Windows the wording is different but the cause is the same:

No CUDA toolset found.

First, confirm the toolkit is actually installed — this is separate from your GPU driver. nvidia-smi working proves the driver is fine; it says nothing about the toolkit (the thing that ships nvcc). Check:

nvcc --version

If that command isn’t found, install the CUDA Toolkit from NVIDIA. If nvcc --version does work but CMake still can’t find it, your PATH/env vars are the problem. Point CMake at the compiler explicitly:

# Linux — tell CMake exactly where nvcc lives
export CUDACXX=/usr/local/cuda/bin/nvcc
export CUDA_PATH=/usr/local/cuda
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_COMPILER=$CUDACXX

On Windows, set the env vars to your installed version’s path (adjust the version number):

set CUDACXX=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.8\bin\nvcc.exe
set CUDA_PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.8

The Windows-specific trap: if you installed Visual Studio after the CUDA Toolkit, the toolkit never registered its build integration with MSVC, and you get “No CUDA toolset found” even with a valid install (llama.cpp issue #9144). Fix: reinstall (or repair) the CUDA Toolkit after Visual Studio so the installer can drop its files into the VS extensions folder. Use the official NVIDIA installer, not a stripped-down runtime.

If you have two CUDA versions installed, CMake may grab the wrong nvcc off your PATH (issue #1832). Keep one version first on PATH, or force the right one with -DCMAKE_CUDA_COMPILER as above.

Fix 3: Unsupported gpu architecture 'compute_120' — your CUDA predates Blackwell

This one hits everyone who upgraded to an RTX 5070, 5080, or 5090 and kept an older CUDA toolkit:

nvcc fatal : Unsupported gpu architecture 'compute_120'

compute_120 / sm_120 is the Blackwell consumer architecture — compute capability 12.0. It’s what the whole RTX 50-series reports. The reason the build dies: a CUDA toolkit older than 12.8 has never heard of sm_120, so when llama.cpp auto-detects your card and tells nvcc to target it, nvcc refuses (llama.cpp issue #13271).

For reference, here’s where the common cards land:

GPU	Architecture	Compute capability	Minimum CUDA
RTX 3090 / 3080 Ti	Ampere	8.6 (sm_86)	11.1+
RTX 4090	Ada Lovelace	8.9 (sm_89)	11.8+
RTX 5090 / 5080 / 5070	Blackwell	12.0 (sm_120)	12.8

The real fix is to install CUDA Toolkit 12.8 or newer (12.9 is the smoother choice for Blackwell). CUDA 12.8 was the first release to add the sm_100/sm_101/sm_120 Blackwell targets. This is the same version wall PyTorch hit — PyTorch 2.7.0 was its first stable build with native sm_120 wheels. After upgrading the toolkit, delete build/ and re-run cmake.

If you genuinely can’t upgrade CUDA right now, there’s a workaround: force the build to target an older architecture your toolkit does support, and let Blackwell’s backward compatibility run those kernels:

cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="86"

This compiles Ampere (sm_86) kernels that a 5090 will still execute. It builds and it runs — but you’re leaving Blackwell-native performance on the table, so treat it as a stopgap until you install CUDA 12.8+. A related variant, Unsupported gpu architecture 'compute_' with an empty value (issue #18430), means the auto-detector found no GPU at build time (common in containers or headless CI) — set CMAKE_CUDA_ARCHITECTURES explicitly to the value from the table above so it stops guessing.

Fix 4: unsupported GNU version! gcc versions later than 15 — your compiler is too new

Rolling-release and bleeding-edge distros ship this one. On Fedora 44 with GCC newer than 15 and CUDA 13.2, the build dies during CMake’s compiler check (llama.cpp issue #22886, filed May 9, 2026):

#error -- unsupported GNU version! gcc versions later than 15 are not supported!

nvcc pins a maximum host-compiler version in its host_config.h header and refuses anything newer, because NVIDIA hasn’t validated it. The threshold moves with your CUDA version — older CUDA caps out at GCC 12 or 13; recent CUDA allows up to 15. Three ways out, best first:

Install an older GCC and point CUDA at it. This is the clean fix — you keep your system compiler and only use the older one for CUDA host code:

# Debian/Ubuntu example — install gcc-14 alongside your default
sudo apt install gcc-14 g++-14
cmake -B build -DGGML_CUDA=ON \
  -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/g++-14

Override the check. nvcc accepts a flag to skip the version guard. Fast, but you’re compiling with an unvalidated combination, so an obscure runtime bug is on you:

cmake -B build -DGGML_CUDA=ON \
  -DCMAKE_CUDA_FLAGS="-allow-unsupported-compiler"

Upgrade CUDA. If a newer CUDA toolkit is available that supports your GCC, that removes the conflict at the source — the preferred move if you’re on Blackwell anyway (you need 12.8+ regardless). Editing host_config.h by hand to raise the cap also works but is the dirtiest option; prefer the host-compiler pin.

Fix 5: build gets Killed — you ran out of RAM

The CUDA kernels in ggml are big translation units, and nvcc is memory-hungry. Build with too many parallel jobs on a RAM-limited box and the Linux OOM killer steps in mid-compile:

c++: fatal error: Killed signal terminated program cc1plus

This is system RAM, not VRAM — a 16GB machine running cmake --build build -j (which defaults to all cores) can spawn a dozen nvcc processes that collectively blow past available memory. Cap the parallelism:

cmake --build build --config Release -j 4

On an 8-16GB machine, -j 2 or -j 4 is safer than the default. It’s slower, but it finishes. Closing your browser and other RAM hogs during the build helps too. If you keep hitting the wall, add swap temporarily — a compile that swaps is still faster than one that dies at 90%.

Confirm the build actually uses the GPU

A green build isn’t proof of GPU acceleration — you could have quietly built a CPU-only binary. Verify:

./build/bin/llama-cli --version

Then run a model and watch the startup log. With CUDA working, you’ll see the model’s layers being offloaded to the GPU:

load_tensors: offloaded 33/33 layers to GPU
load_tensors:        CUDA0 model buffer size =  4560.96 MiB

If it says offloaded 0/33 layers or only mentions CPU, your binary isn’t CUDA-enabled (rebuild) or you forgot -ngl 99 on the command line to request offloading. Cross-check with nvidia-smi during a generation — you should see VRAM in use and the GPU pegged. If tokens are crawling despite a “GPU” build, our guide on Ollama running slow covers the same offload-verification logic (the PROCESSOR column tells you the truth), and why Ollama falls back to CPU walks the same failure from the packaged side.

Do you even need to build from source?

Here’s the part most build tutorials won’t tell you: you probably don’t need to compile llama.cpp at all.

• You just want to run models. Use Ollama or LM Studio. Both bundle prebuilt, GPU-accelerated llama.cpp/ggml kernels and update themselves. Zero build errors, because there’s no build. If a brand-new model won’t load, that’s usually a different problem entirely.
• You want the raw llama-cli/llama-server binaries but not the toolchain pain. llama.cpp ships prebuilt release binaries (including CUDA builds) on its GitHub releases page. Download, unzip, run.
• You’re in Docker. Use a CUDA base image with the toolkit already installed and matched — that sidesteps Fixes 2-4 wholesale, though you still need GPU passthrough working.

Build from source when you need a specific unreleased feature, a custom flag like -DGGML_CUDA_FORCE_MMQ=ON, or you’re developing against the library. For everything else, the prebuilt path is strictly less painful.

And if you need a working GPU right now while you fight a driver/toolkit mismatch on your own box, renting is the fast escape hatch — a cloud GPU on RunPod comes with CUDA preinstalled and matched to the card, so -DGGML_CUDA=ON just works. Building AI coding tooling instead of running models? Our sister site aicoderscope.com covers the local-coding-agent side, and aifoss.dev digs into the FOSS toolchain.

The build command that just works in 2026

For most people on a supported CUDA version, this is the whole thing:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j 4

If you’re on an RTX 50-series card, make sure nvcc --version reports 12.8 or higher first. That one check prevents the most common 2026 build failure before it happens. The card itself matters less than the toolkit match — a used RTX 3090 (24GB, ~$1,254 average on the used market as of July 2026) builds against CUDA 11.8+ without drama, while a new RTX 5090 demands the newer toolkit but rewards you with sm_120 kernels.

FAQ

Do I need the full CUDA Toolkit, or just the driver? The full toolkit. nvidia-smi working (the driver) is necessary but not sufficient — building needs nvcc, which only comes with the CUDA Toolkit. They’re separate installs.

Why does make LLAMA_CUBLAS=1 from an old tutorial fail? Two reasons: the flag was renamed to GGML_CUDA, and the project moved to a CMake-first build. Use cmake -B build -DGGML_CUDA=ON. The plain make path and LLAMA_CUBLAS are both gone.

My RTX 5090 build works with CMAKE_CUDA_ARCHITECTURES="86" — am I done? It runs, but you’re executing Ampere kernels on a Blackwell card via backward compatibility. Install CUDA 12.8+ and rebuild targeting sm_120 for native performance. The 86 trick is a stopgap, not a destination.

Can I build on Windows without Visual Studio? Not for the CUDA path — nvcc needs the MSVC host compiler. Install Visual Studio (or the Build Tools) first, then the CUDA Toolkit, so the toolkit registers its integration. Do it in that order and “No CUDA toolset found” won’t appear.

The build succeeds but generation is CPU-slow. Why? Either the binary isn’t actually CUDA-enabled (check the startup log for offloaded N/N layers to GPU) or you didn’t pass -ngl 99 to offload layers. A CUDA-capable binary still runs on CPU unless you ask it to use the GPU.

Is Metal (Mac) or Vulkan (AMD) any easier? Yes. On Apple Silicon, Metal is on by default — cmake -B build && cmake --build build --config Release needs no extra flag. For AMD, -DGGML_VULKAN=ON avoids the CUDA toolchain entirely (and often the ROCm one too). CUDA’s build is the fussiest of the three precisely because of the toolkit/driver/compiler version matrix.

Recommended Gear

• RTX 3090 — 24GB, compute capability 8.6, builds cleanly against CUDA 11.8+; ~$1,254 average used, July 2026.
• RTX 5090 — 32GB Blackwell, compute capability 12.0; requires CUDA Toolkit 12.8+ to build.

Sources

• llama.cpp build documentation — ggml-org/llama.cpp (GitHub)
• Issue #13271: nvcc fatal Unsupported gpu architecture ‘compute_120’ — ggml-org/llama.cpp
• Issue #22886: fails to build on Fedora 44 (gcc later than 15 unsupported) — ggml-org/llama.cpp
• Issue #9144: No CUDA toolset found (Windows) — ggml-org/llama.cpp
• Issue #1832: CMake Picks Up Wrong nvcc Path — ggml-org/llama.cpp
• Issue #18430: Unsupported gpu architecture ‘compute_’ (no CUDA devices found) — ggml-org/llama.cpp
• How to build with cuBLAS/GGML_CUDA support — abetlen/llama-cpp-python Discussion #871
• CUDA Toolkit 12.8: what GPU is sm_120? — NVIDIA Developer Forums
• Add official support for CUDA sm_120 (RTX 5090 / Blackwell) — pytorch/pytorch Issue #159207
• llama.cpp Build Errors: Common Fixes for Every Platform — InsiderLLM

Last updated July 14, 2026. Toolkit versions, compiler caps, and used-GPU prices change; verify nvcc --version and current listings before building or buying.