AMD Lemonade 10.7 Now Runs on NVIDIA GPUs: What the CUDA Update Changes for RTX Owners

By RunAIHome Team · 10 min read
amdnvidiacudalemonadelocal-llmollama-alternativellama-cpprtx

TL;DR: Lemonade 10.7 (June 10, 2026) added a llama.cpp CUDA backend on Windows and Linux, so the AMD-sponsored local AI server now runs on NVIDIA RTX cards — not just Ryzen AI hardware. It ships per-architecture CUDA builds for everything from RTX 20-series (sm_75) to RTX 50-series (sm_120), bundles its own CUDA runtime (no toolkit install), and exposes an OpenAI-compatible endpoint. It’s a real Ollama alternative now, but only worth switching to for the vLLM/FastFlowLM backends, built-in image gen, and Prometheus metrics.

What you’ll be able to do after this guide:

  • Run Lemonade Server on an NVIDIA RTX GPU with GPU-accelerated llama.cpp (CUDA), plus local image generation via stable-diffusion.cpp.
  • Point any OpenAI-compatible client (Open WebUI, AnythingLLM, your own scripts) at Lemonade’s endpoint.
  • Decide, honestly, whether it’s worth moving off Ollama or LM Studio for your setup.

Honest take: If you already run Ollama happily on your RTX card, Lemonade 10.7 doesn’t give you faster tokens — it’s the same llama.cpp CUDA backend underneath. Switch only if you want the extra backends (vLLM, FastFlowLM), the built-in image/speech stack in one install, or the native Prometheus endpoint. On a Ryzen AI machine, Lemonade is still the better pick; on plain NVIDIA, it’s now a legitimate option rather than the obvious winner.

What actually changed in 10.7

Until this release, Lemonade was effectively an AMD-only server. It auto-detected Ryzen AI NPUs and Radeon GPUs, routed work through ROCm or Vulkan, and left NVIDIA owners out — you could run it, but only on CPU or Vulkan, which threw away most of the point of owning an RTX card.

Version 10.7, released June 10, 2026, closed that gap. The headline items, straight from the release notes:

  • llama.cpp CUDA support added for Windows and Linux. This is the one that matters for NVIDIA owners: your RTX card now runs GGUF models with full CUDA acceleration through Lemonade.
  • stable-diffusion.cpp CUDA added for Linux, and stable-diffusion.cpp Vulkan added for Windows and Linux. Local image generation gets GPU acceleration on NVIDIA too (CUDA on Linux, Vulkan on Windows).
  • A new lemonade bench command for apples-to-apples LLM benchmarking across llama.cpp, FastFlowLM, vLLM, and Ryzen AI software.
  • LMX-Omni model support for any OpenAI-compatible app that can render multimedia output — including Open WebUI and AnythingLLM.
  • A native Prometheus endpoint for real-time stats monitoring.

There’s also a housekeeping change worth knowing: the deprecated environment variables for configuring the server were removed. Use lemonade config instead of exporting env vars, or your old scripts will silently do nothing.

Why “it runs on CUDA now” is less exciting than it sounds

Here’s the part most coverage skips. Lemonade’s LLM engine for GGUF models is llama.cpp. So is Ollama’s. So is LM Studio’s. When Lemonade added a CUDA backend, it added the same upstream CUDA kernels that Ollama has shipped for years.

Token generation (decode) on a local LLM is memory-bandwidth-bound, not framework-bound — each generated token requires reading the model’s weights out of VRAM, so your ceiling is set by the card’s memory bandwidth, not by which wrapper launched the kernel. A used RTX 3090 pushes 936 GB/s and lands around 95 tok/s on a 7B model at Q4_K_M; an RTX 4090 at ~1,008 GB/s does roughly 130–135 tok/s on the same model; a 360 GB/s RTX 3060 12GB sits near 45 tok/s. Those numbers come from our own testing across the site (see the used RTX 3090 value analysis and why local LLMs got good), and they don’t change based on whether Ollama or Lemonade calls the kernel. Same GGUF, same quant, same card, same tok/s.

So the CUDA update doesn’t make your RTX faster. What it does is let you use Lemonade’s other features on NVIDIA hardware. That’s the real value, and it’s worth being clear-eyed about.

The CUDA build matrix: which RTX cards are covered

Lemonade doesn’t ship one generic CUDA build. It ships per-architecture builds compiled for specific compute capabilities, with the CUDA runtime libraries bundled — so you do not need a system-wide CUDA Toolkit installed. That’s a genuinely nice touch; the single most common “Ollama/ComfyUI won’t use my GPU” failure is a mismatched or missing CUDA install.

The shipped compute-capability targets map to consumer GPUs like this:

Compute capabilityArchitectureConsumer/prosumer cards
sm_75TuringRTX 20-series, GTX 16-series
sm_80Ampere (data center)A100
sm_86AmpereRTX 30-series (3060 / 3080 / 3090)
sm_89AdaRTX 40-series (4070 / 4080 / 4090)
sm_90HopperH100
sm_100Blackwell (data center)B200 / GB200
sm_120BlackwellRTX 50-series (5060 / 5070 / 5080 / 5090)

If you own anything from an RTX 2060 to an RTX 5090, there’s a build for you. Note that RTX 50-series (sm_120) requires CUDA 12.8 or newer under the hood — Lemonade bundles a compatible runtime, so this is handled for you, but it’s the same reason older llama.cpp wheels throw “no kernel image available” on a 5090 when you install them by hand. If you’ve hit that error before, our Torch-not-compiled-with-CUDA fix walks through the manual version, which Lemonade sidesteps.

Installing and running Lemonade on an NVIDIA GPU

There are two install paths. Most home-lab users want the platform installer; Python users can go through pip.

Platform installer (recommended for a persistent server): Lemonade ships a Windows MSI, plus Ubuntu 24.04+, Fedora 43/44, and macOS packages, and a Docker image. This installs it as a managed service and is the least fiddly option.

Pip (recommended if you’re scripting or already live in a venv):

# Python 3.10–3.13
pip install lemonade-sdk

Once installed, start the server and pull a model. Lemonade serves an OpenAI-compatible REST API — by default on http://localhost:13305:

# start the server
lemonade-server serve

# or run a specific model directly
lemonade-server run Qwen3-8B-GGUF

To force the CUDA backend for GGUF models, select it with the --llamacpp flag (the same mechanism that already exposed vulkan and rocm):

lemonade-server run Qwen3-8B-GGUF --llamacpp cuda

Confirm your card is actually being used — if Lemonade fell back to CPU or Vulkan, you’d see the same “slow tokens” symptom Ollama users hit. Our Ollama-not-using-GPU checklist is a good mental model: watch the load log, watch VRAM allocation in nvidia-smi, and confirm the backend string says CUDA.

Point a client at it

Because the endpoint is OpenAI-compatible, any client works. From the OpenAI Python SDK:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:13305/api/v1",
    api_key="lemonade",  # any non-empty string; auth is local
)

resp = client.chat.completions.create(
    model="Qwen3-8B-GGUF",
    messages=[{"role": "user", "content": "Explain KV cache in one paragraph."}],
)
print(resp.choices[0].message.content)

For a GUI, Open WebUI and AnythingLLM both connect by pointing their OpenAI-API base URL at the same address — and with 10.7’s LMX-Omni support, those two can now render multimodal output from Lemonade directly.

Lemonade vs Ollama on NVIDIA: when each wins

This is the real question for RTX owners, and the answer isn’t “the new one.” Here’s the honest split.

Stick with Ollama if you want the biggest model library with the least friction, one-line ollama pull model management, and the deepest pool of community tutorials. For pure “run a GGUF on my RTX and chat,” Ollama is still the path of least resistance, and the tokens per second are identical.

Switch to Lemonade if you want any of these in a single install:

  • Multiple backends behind one API. Lemonade can drive llama.cpp, FastFlowLM, and vLLM. If you want to A/B a vLLM deployment against llama.cpp without standing up two servers, that’s a real convenience — and vLLM wins hard on concurrency once you’re serving more than one user.
  • Built-in image and speech. stable-diffusion.cpp (now CUDA-accelerated on Linux, Vulkan on Windows), Whisper for speech-to-text, and Kokoro for TTS all live in the same server. Ollama is LLM-only.
  • The lemonade bench command. A standardized way to compare llama.cpp vs vLLM vs FastFlowLM tok/s on your hardware, rather than trusting someone else’s numbers.
  • Native Prometheus metrics. If you run a home AI server 24/7 and already have Grafana, scraping Lemonade is trivial. Ollama needs a third-party exporter for this.

Skip both and use Vulkan/ROCm-native tools if you’re on an AMD card — though even there, Lemonade’s the better front end. See our ROCm on Ubuntu setup guide for the bare-metal AMD path.

The one real caveat

CUDA support in Lemonade is new as of 10.7. Ollama has been shipping and hardening its CUDA path for years and has a far larger community to surface edge-case bugs. If you hit a weird CUDA crash on Lemonade in the first month or two after this release, it’s more likely to be un-triaged than the equivalent Ollama bug. For a mission-critical always-on server, that argues for waiting a point release or two — or running Lemonade behind a health check.

And if your GPU can’t fit the model you want at all, no server fixes that. That’s a rent-a-bigger-card problem: a cloud RunPod instance with a 48GB or 80GB GPU will run the model Lemonade won’t, and you can point the same OpenAI-compatible client at it.

FAQ

Does Lemonade run faster than Ollama on my RTX card? No. Both use llama.cpp’s CUDA backend for GGUF models, and decode speed is set by your card’s memory bandwidth, not the wrapper. Expect the same tok/s you already get in Ollama — roughly 95 tok/s on a 7B Q4_K_M on an RTX 3090, ~130 on a 4090.

Do I need to install the CUDA Toolkit first? No. Lemonade bundles the CUDA runtime libraries with its per-architecture builds, so there’s no system-wide toolkit dependency. This avoids the most common GPU-not-detected failure mode.

Which NVIDIA cards are supported? Anything from Turing (sm_75, RTX 20-series) through Blackwell (sm_120, RTX 50-series), including Ampere (RTX 30-series) and Ada (RTX 40-series). Data center Ampere/Hopper/Blackwell (A100/H100/B200) are covered too.

Can I use Lemonade for image generation on NVIDIA? Yes. 10.7 added stable-diffusion.cpp CUDA acceleration on Linux and Vulkan on Windows and Linux. For squeezing more speed out of RTX image gen specifically, see our ComfyUI NVFP4 speed guide.

What’s the default endpoint? http://localhost:13305, OpenAI-compatible at /api/v1. Any API key string works locally.

Is it free and open source? Yes — lemonade-sdk/lemonade on GitHub (~4.7k stars), AMD-sponsored, no cloud account or API keys required.

Sources

Last updated July 1, 2026. Software versions and specs change; verify current release notes before installing.

  • RTX 3090 (used) — 24GB, 936 GB/s, still the best tok/s-per-dollar under 24GB for local LLMs.
  • RTX 4090 — ~1,008 GB/s, ~130 tok/s on a 7B Q4_K_M, the fastest 24GB consumer option.
  • RTX 5090 — 32GB Blackwell (sm_120), the only 50-series card past 16GB of VRAM.

Was this article helpful?