Jul 28, 2026

Run Claude Code on Your Own GPU in 2026: The Ollama Setup, Which Models Actually Work, and the Context Trap

By RunAIHome Team · 11 min read

claude-codeollamalocal-llmcodingtutorialgpu

TL;DR: Since Ollama v0.14.0 (January 16, 2026), Ollama exposes a native Anthropic-compatible API at /v1/messages, and since v0.15 the ollama launch claude command wires Claude Code to a local model with zero config files. No Anthropic account, no API key, no monthly bill. The catch is context: Ollama’s 4,096-token default silently truncates the agent’s memory, and fixing it properly eats 24GB-card headroom fast.

What you’ll be able to do after this guide:

Start Claude Code against a local model with one command — or with three environment variables if you want to control exactly what it talks to.
Pick the right model for your VRAM tier: GLM-4.7-Flash and Qwen3-Coder 30B-A3B on 24GB, gpt-oss:20b on 16GB, cloud tags on no GPU at all.
Diagnose the “my agent forgot the file it just read” failure and fix it with OLLAMA_CONTEXT_LENGTH before it wastes an afternoon.

Honest take: On a 24GB card this is genuinely usable for routine agent work — test writing, refactors, boilerplate — at 50–65 tok/s and $0/month. It is not July-2026 Claude quality on novel architecture problems, and nothing you run locally will be. Set it up as the free tier of a hybrid stack, not a full replacement.

What actually changed

For most of 2025, pointing Claude Code at a local model meant running a LiteLLM proxy that translated Anthropic’s Messages API into OpenAI-compatible calls, plus a config file, plus crossed fingers every time either tool updated. Two Ollama releases killed all of that.

Ollama v0.14.0 (January 16, 2026) shipped native Anthropic API compatibility. Ollama now serves the Messages format directly at http://localhost:11434/v1/messages. Claude Code — which speaks that format natively — connects to it with no translation layer at all. File editing, tool calling, multi-step agent loops: the whole protocol works, but the tokens come off your GPU instead of Anthropic’s datacenter.

Ollama v0.15 added ollama launch, which removes even the environment-variable step. It currently supports four coding agents — Claude Code, OpenCode, Codex, and Droid — detects whether the tool is installed, sets the connection up for you, and drops you into a model picker.

The integration is still actively maintained: Ollama’s release notes updated July 27, 2026 mention keeping Claude Code channels available alongside a new skills system, and the current stable release is v0.32.1 (July 16, 2026). This isn’t a community hack that will break next month; it’s a supported first-party path, with its own page in Ollama’s docs.

One licensing note worth knowing: in February 2026, Anthropic explicitly banned piping your Claude subscription token into third-party tools. Running Claude Code against Ollama is the opposite of that — the CLI is a free download, and you’re not touching Anthropic’s servers or your subscription at all.

The one-command path

Install Claude Code and Ollama, pull a model, launch:

$ ollama pull qwen3-coder
$ ollama launch claude

You get a selector listing your installed models plus Ollama’s cloud tags:

Select a model to use with Claude Code:

  > qwen3-coder            18 GB   local
    glm-4.7-flash          17 GB   local
    gpt-oss:20b            13 GB   local
    qwen3-coder:480b-cloud  —      cloud

Pick one, hit Enter, and Claude Code starts with the connection already configured. That’s the entire setup. Ollama’s own launch post names glm-4.7-flash, qwen3-coder, and gpt-oss:20b as the local picks, and glm-4.7:cloud, minimax-m2.1:cloud, gpt-oss:120b-cloud, and qwen3-coder:480b-cloud for machines that can’t hold the weights.

The manual path (three environment variables)

If you’d rather control the wiring yourself — or you’re scripting this into a dev container — the whole integration is three exports:

export ANTHROPIC_BASE_URL=http://localhost:11434
export ANTHROPIC_AUTH_TOKEN=ollama        # required by the CLI, ignored by Ollama
export ANTHROPIC_MODEL=qwen3-coder
claude

ANTHROPIC_AUTH_TOKEN can be any non-empty string; Claude Code refuses to start without one, and Ollama doesn’t check it. For finer control, Ollama’s compatibility docs also support mapping Claude Code’s internal tiers to different local models via ANTHROPIC_DEFAULT_SONNET_MODEL and ANTHROPIC_DEFAULT_HAIKU_MODEL — a nice trick for sending background/utility calls to a small fast model while the main agent loop uses your best one.

Verify what’s actually loaded while the agent runs:

$ ollama ps
NAME              SIZE     PROCESSOR    UNTIL
qwen3-coder       21 GB    100% GPU     4 minutes from now

If PROCESSOR shows a CPU split, your context setting has pushed the model past VRAM — more on that below, and our Ollama speed guide covers the general case.

Which models actually work

Claude Code is an agent, not a chatbot. It needs two things from a model: tool calling (or it can’t read files, run commands, or apply edits) and context (or it forgets what it read three steps ago). That disqualifies most of the Ollama library. These are the ones worth your disk space in July 2026:

	GLM-4.7-Flash	Qwen3-Coder 30B-A3B	gpt-oss:20b	Qwen3.6-35B-A3B
Architecture	30B MoE, 3B active	30B MoE, 3B active	21B MoE	35B MoE, 3B active
Download (Q4)	~17–18 GB	~18 GB	~13 GB	~20 GB
Native context	128K	256K	128K	256K
Comfortable card	24 GB	24 GB	16 GB	24 GB
Tool calling	Yes	Yes (built for it)	Yes	Yes

All four are mixture-of-experts models, and that’s not a coincidence: with ~3B parameters active per token, they decode at small-model speed while holding large-model knowledge — the same shift we broke down in why local LLMs got good in 2026. On a used RTX 3090, Qwen3.6-35B-A3B holds 50–65 tok/s at Q4_K_M with full 32K context, and a Hugging Face community benchmark measured ~75 tok/s at 10K context and still ~65 tok/s at 120K on the UD-Q5_K_XL quant. For an agent that streams multi-hundred-token tool calls, anything above ~40 tok/s feels responsive.

On 24GB: qwen3-coder is the default pick — it was trained specifically for agentic coding, and its Q4_K_M weights (~18GB) leave roughly 6GB for KV cache on a 24GB card, which is workable for real agent sessions. GLM-4.7-Flash is the alternative with a strong reputation on agentic benchmarks (guides circulating this month cite 59.2 on SWE-bench Verified — treat vendor-adjacent numbers with the usual salt). Unsloth’s run-it-locally docs recommend 24GB for the 4-bit quant and, if you call tools, temp 0.7, top_p 1.0 instead of the general-chat sampling.

On 16GB: gpt-oss:20b fits with room to breathe and is shockingly fast — but read our gpt-oss guide before you crank the context slider, because its 225 tok/s on an RTX 4090 collapses to ~9 tok/s at 128K context. GLM-4.7-Flash also runs here with experts offloaded to system RAM — MoE routing keeps that surprisingly tolerable, but expect a real speed haircut.

On no GPU: the cloud tags run the big models on Ollama’s metered infrastructure — we covered what :cloud tags do to your GPU (nothing) — or you can rent a 24GB+ card by the hour on RunPod and run the identical stack there. Either way you’ve left “free and private” territory; the point of this setup is owning the inference.

The context trap (the part that actually breaks)

Here’s the failure you’ll hit first. You launch Claude Code, ask it to fix a test, and it reads the file, runs the suite, then confidently re-reads the same file because it has no memory of doing so. Nothing errored. Nothing warned.

The cause: Ollama’s default context window is 4,096 tokens, and when a conversation exceeds it, Ollama silently truncates from the beginning. Claude Code’s system prompt plus tool definitions alone can swallow most of that budget before your code enters the picture. The agent isn’t dumb — it’s amnesiac. Anthropic’s own tooling assumes the ~200K window of the hosted models; Ollama’s docs recommend at least 64,000 tokens of context for coding agents, and DataCamp’s hands-on test found 10K unusable while 20K was the floor where multi-step workflows held together at reasonable speed.

Three ways to fix it, in order of preference:

1. The Ollama app setting. Newer Ollama builds expose a context-length slider in Settings. Set it to 64K if your VRAM allows; done.

2. The environment variable. Set it where the server runs (a shell export does nothing for a systemd service — same gotcha as every Ollama env var):

OLLAMA_CONTEXT_LENGTH=65536 ollama serve

3. A Modelfile, when you want it baked into one model:

FROM qwen3-coder
PARAMETER num_ctx 65536

Note the precedence trap: a num_ctx baked into a Modelfile overrides the environment variable, so if you inherited a model built with PARAMETER num_ctx 4096, the export will appear to do nothing.

Now the honest part: context is not free. KV cache scales linearly with context length, and at 64K on a 30B-class model it claims gigabytes — which is exactly the headroom a Q4 quant left on your 24GB card. If ollama ps shows the model spilling to CPU after you raise the context, you have three levers: drop to a smaller quant, settle at 32K (fine for most single-repo sessions), or enable KV-cache quantization. This trade — weights vs. memory — is the whole game of local agents, and it’s why the 24GB tier is the entry ticket rather than a luxury.

What it costs vs. what it saves

A used RTX 3090 — still the VRAM value king — averages $1,252 on eBay in July 2026, with a fair asking range of $1,200–$1,299 and clean listings starting near $1,050 per BestValueGPU’s price tracker. That’s the whole capital cost of this setup if you already own a PC that can host it. Against a $100–200/month cloud AI habit, the card pays for itself in 6–12 months; we ran the full replacement math — including electricity and where local models still lose — in Can a local LLM replace Claude for daily coding?

The realistic outcome isn’t replacement. It’s that routine agent work — the test scaffolding, the rename-across-files, the “write the obvious CRUD handler” — moves to hardware you own, and the frontier subscription drops to the cheapest tier for the problems that actually need it. If you’re choosing between this stack and an autocomplete-style setup, our Continue.dev + Ollama guide covers the editor-integrated alternative, and for tool-by-tool comparisons of the agent CLIs themselves, our sister site has the deeper AI coding tool coverage at aicoderscope.com.

FAQ

Do I need a Claude subscription or API key? No. The Claude Code CLI is a free download, ANTHROPIC_AUTH_TOKEN just needs to be non-empty, and Ollama ignores it. You only pay if you point the same CLI back at Anthropic’s API or use Ollama’s metered cloud tags.

Is this against Anthropic’s terms? Overriding ANTHROPIC_BASE_URL to a local server is a documented, first-party-supported integration on Ollama’s side and uses no Anthropic infrastructure. What Anthropic banned in February 2026 is reusing your subscription authentication in third-party tools — a different thing entirely.

Which model should I pull first on a 24GB card? qwen3-coder. It’s built for agentic tool use, its ~18GB Q4_K_M leaves usable KV-cache room, and it’s one of the three models Ollama itself names for ollama launch. Try GLM-4.7-Flash second and keep whichever fails less on your codebase.

Why does the agent keep “forgetting” files it already read? Silent context truncation — Ollama defaults to 4,096 tokens. Raise it to at least 20K (64K if VRAM allows) via the app settings, OLLAMA_CONTEXT_LENGTH, or a Modelfile num_ctx. Watch ollama ps afterward to confirm you’re still 100% GPU.

How much slower is it than hosted Claude? Decode speed is comparable or better (50–65 tok/s locally vs. typical hosted streaming), but local models think worse, not slower — expect more retries on hard multi-file reasoning. Budget quality, not latency.

Recommended Gear

RTX 3090 24GB (used) — the entry ticket: 24GB VRAM fits Qwen3-Coder Q4 plus real context, ~$1,050–$1,300 used in July 2026.
RTX 4090 — same VRAM ceiling, ~20% faster decode on 30B-class MoE models and much faster prefill for big-repo prompts.

Sources

Last updated July 28, 2026. Prices, model availability, and Ollama behavior change quickly; verify current rates and release notes before purchasing.

Was this article helpful?