How to Use LangChain with Ollama (2026 Guide)

To use LangChain with Ollama, install Ollama and pull a model that supports tool calling (for example ollama pull qwen3:8b), run pip install -U langchain langchain-ollama, and create a ChatOllama(model="qwen3:8b") object. That object works anywhere LangChain expects a chat model, so you can call it directly, bind tools to it, or hand it to create_agent to get a full agent loop that runs entirely on your machine. The one setting most people miss is the context window: raise num_ctx before you build an agent, or the model will quietly lose the start of the conversation.

This guide is written for people building agents, not just chatting with a local model. Every snippet was checked on October 15, 2026 against langchain 1.4.6, langchain-ollama 1.1.0 and langchain-mcp-adapters 0.3.2, using the official LangChain ChatOllama docs and Ollama's documentation. The latest Ollama release at that point was v0.40.5 (October 14, 2026). Both packages need Python 3.10 or newer.

LangChain vs Ollama: what each part does

People often search "LangChain vs Ollama" as if they were rivals. They sit at different layers:

• Ollama downloads open weight models and serves them from a local HTTP API, by default on 127.0.0.1 port 11434. It handles the weights, the GPU and the chat template.
• LangChain is an MIT licensed Python framework for building on top of a model: prompts, tools, structured output, retrieval and, since version 1.0, a ready made agent loop built on LangGraph.

The langchain-ollama package is the bridge. It gives you three classes: ChatOllama for chat and agents, OllamaEmbeddings for vectors, and OllamaLLM for plain text completion. For new work you will almost always want ChatOllama.

Step 1: Install Ollama and pull a tool calling model

Install Ollama for your operating system from the official download page, then pull a model. For agents the model must support tool calling. Ollama's library has a filter for this on its models page, and each model page shows a "tools" badge when it qualifies. Qwen3 is a safe starting point because it supports both tools and thinking and comes in sizes from 0.6B to 235B parameters:

ollama pull qwen3:8b
ollama list

If you pick a model without tool support and send it tools, Ollama rejects the request with an error saying the model does not support tools. That message means "choose another model", not "fix your code". Sizing a model to your GPU is its own topic; the rule of thumb is that the model plus its context has to fit in memory without spilling to the CPU.

Step 2: Install the LangChain packages

Create a virtual environment and install the two packages:

python -m venv .venv
source .venv/bin/activate
pip install -U langchain langchain-ollama

langchain pulls in langchain-core and langgraph. langchain-ollama pulls in the official ollama Python client, which is what actually talks to the server.

Step 3: Call ChatOllama and set the context window

from langchain_ollama import ChatOllama

llm = ChatOllama(
    model="qwen3:8b",
    temperature=0,
    num_ctx=32768,
    validate_model_on_init=True,
)

reply = llm.invoke("Explain what a vector database is in two sentences.")
print(reply.content)

Three parameters matter here:

• num_ctx sets the context window for this request. Ollama's context length docs say the default depends on your VRAM: 4k tokens below 24 GiB, 32k from 24 to 48 GiB, and 256k at 48 GiB or more. The same page recommends at least 64,000 tokens for agents, coding tools and web search. An agent's system prompt, tool schemas and tool results add up fast, so 4k is rarely enough. Larger values use more memory; check the CONTEXT and PROCESSOR columns of ollama ps to see what you actually got.
• validate_model_on_init=True checks that the model is pulled when the object is created, instead of failing on the first call.
• temperature=0 makes tool selection more repeatable, which helps when you are debugging an agent.

For streaming, use llm.stream(...) and print each chunk's content. For thinking models such as Qwen3, reasoning=True turns thinking on and puts the model's reasoning in the message's additional_kwargs under reasoning_content, so it stays out of the answer you show users.

Step 4: Give the model tools

A tool is a Python function with a docstring and type hints. LangChain turns it into the JSON schema that Ollama's tool calling API expects:

from langchain.tools import tool

@tool
def get_order_status(order_id: str) -> str:
    """Look up the shipping status of an order by its ID."""
    return f"Order {order_id}: shipped, arriving Friday"

llm_with_tools = llm.bind_tools([get_order_status])
msg = llm_with_tools.invoke("Where is order A-1042?")
print(msg.tool_calls)

bind_tools only asks the model which tool it wants and with what arguments. It does not run the function. You get back a list like [{'name': 'get_order_status', 'args': {'order_id': 'A-1042'}, ...}], and running the tool and sending the result back is up to you. That is exactly the loop an agent automates.

Step 5: Build a local agent with create_agent

LangChain 1.x ships create_agent, which runs the model calls tool, tool returns result loop until the model gives a final answer. The agents documentation covers the full option list. With Ollama it looks like this:

from langchain.agents import create_agent

agent = create_agent(
    model=llm,
    tools=[get_order_status],
    system_prompt="You are a support assistant. Use tools for order questions.",
)

result = agent.invoke(
    {"messages": [{"role": "user", "content": "Has order A-1042 shipped?"}]}
)
for m in result["messages"]:
    print(m.type, m.content)

The message list shows the whole trace: your question, the model's tool call, the tool's output and the final reply. Because the agent is a LangGraph graph, you can add a checkpointer for conversation memory, middleware, or human approval later without switching frameworks. If you outgrow create_agent, our LangGraph vs CrewAI vs OpenAI Agents SDK comparison covers when to drop down to a hand built graph.

You can also skip the import and let LangChain pick the provider from a string: init_chat_model("ollama:qwen3:8b") returns a ChatOllama too. That is handy when the model name comes from config and you want to switch between a local model and a hosted one without code changes.

Add MCP tools

Local models can use the same Model Context Protocol servers as Claude Code or Cursor. Install langchain-mcp-adapters, point it at a server, and pass the tools it returns to create_agent:

import asyncio
from langchain_mcp_adapters.client import MultiServerMCPClient

async def main():
    client = MultiServerMCPClient({
        "notes": {"command": "python", "args": ["notes_server.py"], "transport": "stdio"},
    })
    tools = await client.get_tools()
    agent = create_agent(llm, tools)
    out = await agent.ainvoke({"messages": [{"role": "user", "content": "List my notes"}]})
    print(out["messages"][-1].content)

asyncio.run(main())

To write the server side yourself, follow our guide on how to build an MCP server in Python. Keep the tool list short: small local models pick the right tool far more reliably from five clear tools than from fifty.

Step 6: Get structured output

When you need data rather than prose, pass a Pydantic model to with_structured_output. ChatOllama defaults to the json_schema method, which sends your schema in Ollama's format field so the server constrains the output to it (Ollama structured outputs):

from typing import Literal
from pydantic import BaseModel, Field

class Ticket(BaseModel):
    title: str = Field(description="One-line summary")
    priority: Literal["low", "medium", "high"]

triage = llm.with_structured_output(Ticket)
print(triage.invoke("The login page returns a 500 error for every user."))

You get a Ticket object back, not a string to parse. This is the most dependable way to use small local models inside a pipeline, because the output shape is enforced rather than requested.

Step 7: Add retrieval with OllamaEmbeddings

For RAG or agent memory, embed text locally with an Ollama embedding model and store the vectors in a vector database:

ollama pull embeddinggemma

from langchain_ollama import OllamaEmbeddings

emb = OllamaEmbeddings(model="embeddinggemma")
vector = emb.embed_query("How do I reset my password?")
print(len(vector))

OllamaEmbeddings plugs into any LangChain vector store, including Qdrant and Chroma. Our Qdrant how to walks through per user filters and hybrid search for agent memory. Use the same embedding model for indexing and for queries, or the vectors won't line up. Ollama also works from Node through LangChain.js and the Vercel AI SDK; see best TypeScript AI agent frameworks for those options.

Running Ollama on another machine

By default both the Ollama server and the Python client use 127.0.0.1:11434. To reach a GPU box elsewhere, pass base_url="http://gpu-box:11434" to ChatOllama or OllamaEmbeddings, or set the OLLAMA_HOST environment variable, which the Ollama client reads. On the server, the Ollama FAQ explains that OLLAMA_HOST also changes the address it binds to. Don't expose that port to the internet as is. Put a reverse proxy with authentication in front; base_url accepts credentials in the http://user:password@host:11434 form for exactly this case. Ollama also keeps a model in memory for 5 minutes after the last request by default, and the keep_alive parameter changes that.

If you want one endpoint that can fall back from a local model to a hosted one, an LLM gateway is the usual answer. See our LLM gateway vs OpenRouter guide.

Common problems and fixes

• The agent forgets the task halfway through. The context window is too small. Raise num_ctx (or OLLAMA_CONTEXT_LENGTH on the server) and check ollama ps.
• "does not support tools" error. The model has no tool calling support. Pick one from the tools filter.
• The model answers in prose instead of calling a tool. Smaller models do this. Try a larger size, a clearer tool docstring, fewer tools, or temperature=0.
• Connection refused. The server isn't running or is on another host. Start the Ollama app or ollama serve, and check base_url.
• The first call is slow, later ones are fast. That is the model loading into memory. Adjust keep_alive if it unloads too soon.

FAQ

Is LangChain free to use with Ollama?

Yes. LangChain is open source under the MIT license, and Ollama is free to run locally, so a local agent has no per token cost. You pay only for your own hardware. LangSmith, LangChain's separate tracing platform, is optional and has its own pricing. For licensing details, see is LangChain free for commercial use.

What is the difference between LangChain and Ollama?

Ollama runs the model and serves it over a local API. LangChain is the framework you write your app in: prompts, tools, retrieval and agents. You use them together through the langchain-ollama package.

Which Ollama model is best for LangChain agents?

Pick one with the "tools" capability that fits in your GPU memory with room for a large context window. Qwen3 and gpt-oss are both listed with tool support. Test a few on your own tasks, since tool calling accuracy varies a lot between small models.

Should I use ChatOllama or OllamaLLM?

Use ChatOllama. It supports tool calling, structured output, streaming and the agent loop. OllamaLLM is plain text completion (a prompt string in, a string out) with no chat messages or tool calls.

Can I use LangGraph with Ollama?

Yes. create_agent already returns a LangGraph graph, and any custom LangGraph node can call a ChatOllama model the same way. For framework choices beyond LangChain, see our best AI agent framework for Python roundup or the agent frameworks directory.