Build a local LLM agent inside product with LangChain

Introduction

“Can our product integrate AI?” and “What new business opportunities can AI bring?” For many teams, myself included, this is the first time encountering the need to integrate AI into an actual product.

Behind any AI feature, there are many factors to consider, including cost and feasibility. The nature of the product I am developing also requires operation in a strict offline environment, so this article mainly studies an MVP based as much as possible on the browser or local LLMs.

I Heard Browsers Have LLMs?

I had heard quite early on that Chrome was building LLMs into Browser APIs, such as the Prompt API🔗, but its adoption🔗 can be described as limited to trial use in the latest version of Chrome. The API looks very familiar:

// 1. Check whether the browser is supported and confirm the model has finished downloading
const capabilities = await ai.languageModel.capabilities();
if (capabilities.available === 'readily') {

  // 2. Create a session and pass in the system prompt during creation
  const session = await ai.languageModel.create({
    systemPrompt: "You are a helpful and friendly assistant."
  });

  // 3. Simulate conversation history (the Prompt API automatically maintains context within the same session)
  // First round of conversation (User)
  const response1 = await session.prompt("Can you write a short poem about coding?");
  console.log("Assistant:", response1);
  // Output: Code is the poetry of machines, Where logic weaves through screens and seams.

  // 4. If you want to inject "past conversation history" directly when creating the session (history payload)
  // You can use the initialPrompts parameter:
  const sessionWithHistory = await ai.languageModel.create({
    systemPrompt: "You are a helpful and friendly assistant.",
    initialPrompts: [
      { role: 'user', content: 'Can you write a short poem about coding?' },
      { role: 'assistant', content: 'Code is the poetry of machines, Where logic weaves through screens and seams.' }
    ]
  });

  // Continue the conversation afterward, and it will remember this history
  const response2 = await sessionWithHistory.prompt("Can you explain the first line?");
  console.log("Assistant:", response2);

  // Remember to destroy the session when it is no longer needed to free memory
  session.destroy();
  sessionWithHistory.destroy();
}

If you have build with OpenAI models API before, you must be very familiar with payloads that feel like the Chat ML (Chat Markup Language) format. This System / User / Assistant payload also appears in the Prompt API; in fact, many vendors follow a similar pattern:

<|im_start|>system
You are a helpful and friendly assistant.<|im_end|>
<|im_start|>user
Can you write a short poem about coding?<|im_end|>
<|im_start|>assistant
Code is the poetry of machines,
Where logic weaves through screens and seams.<|im_end|>

Based on the pattern above, I built an MVP project experiment using the Prompt API: local-ai-assistance🔗

But going from a simple next-token machine LLM to a capable standalone Agent is still not enough. You can use additional packages such as LangChain to build an LLM Harness.

LangChain

In the earlier example, you would need to manually manage extended issues such as different AI provider sources, long-term memory, memory retrieval (RAG, Embedding), workflow, structured responses, tool calling, and so on. If every project had to build this infrastructure from scratch, the cost would be extremely high. LangChain🔗 is a framework created to solve these pain points.

LangChain vs. LangGraph vs. Deep Agents

When you go to the LangChain getting started documentation🔗, you will see three ways to get started:

• Deep Agents: Best suited for developers who want a “fully featured” Agent from the start. It includes built-in capabilities such as automatic context compression, a virtual filesystem, and subagent-spawning. Deep Agents are built on top of LangChain agents, and you can also choose to use LangChain agents directly.
• LangChain: Suitable for scenarios that require a highly customizable framework, allowing you to easily adapt it to specific use cases and data.
• LangGraph: A low-level process orchestration framework designed for advanced requirements that need to combine “deterministic workflows” with “autonomous agent workflows.”

The relationship among the three can be understood as: LangGraph is the process engine, LangChain is the framework built on top of it, and Deep Agent is a higher-level out-of-the-box solution.

What Does LangChain Solve?

Product expectations for AI have gone beyond answering questions. The expectation is now for an intelligent entity that can solve problems. In development, the troublesome part is not the model itself, but the pile of integration work around it

Suppose you use OpenAI today and switch to Ollama tomorrow; you need to add a fixed System Message to the prompt; you need to read PDFs, split documents, and connect them to a vector database. None of these tasks is difficult on its own, but once everything is wired together, the code can easily become tedious and hard to maintain. What LangChain does is abstract these common needs into consistent interfaces.

What Does LangGraph Solve?

Managing LLM process logic with a directed graph

If LangChain solves “how components are connected,” then LangGraph solves “how the overall process runs.” If you write everything yourself, it usually becomes a large amount of if...else logic or nested function calls. As the process becomes more complex, requirements such as branching, retries, and shared state start to appear, making the code increasingly difficult to read and maintain.

Receive question → Determine whether rewriting is needed → Retrieve documents → Generate answer → Check whether the result meets expectations → Re-run a certain step if necessary

LangGraph describes this type of process using a directed graph.

• Node: Represents an execution step, such as rewriting a question, retrieving documents, or generating an answer. In essence, it is just a regular function.
• Edge: Defines the execution order between nodes.
• Conditional Edge: Determines which node to execute next based on the current state. For example, the first round of conversation retrieves directly, while subsequent conversations rewrite the question first.
• State: Uses Annotation to define shared state. Each node can read or update it, and data is automatically passed between nodes.

The flow roughly looks like this:

// 1. Define the state structure
const MyState = Annotation.Root({
  ...MessagesAnnotation.spec,
  rephrasedQuestion: Annotation<string>,
  sourceDocuments: Annotation<Document[]>,
});
// 2. Define nodes
const nodeA = async (state) => {
  return { rephrasedQuestion: "Rephrased question" };
};
// 3. Assemble the flow
const graph = new StateGraph(MyState)
  .addNode("nodeA", nodeA)
  .addNode("nodeB", nodeB)
  .addConditionalEdges("__start__", (state) => {
    return state.messages.length > 1 ? "nodeA" : "nodeB";
  })
  .addEdge("nodeA", "nodeB")
  .compile();

Breaking Down the fully-local-pdf-chatbot Example

The video Effectively Building with LLMs in the Browser with Jacob - LangChain🔗 mentions the fully-local-pdf-chatbot🔗 project, which uses LangChain to connect three local solutions and enable users to ask questions about uploaded PDF documents. Below are my notes from breaking down this project.

Tech Stack

All core logic runs inside a Web Worker🔗 (worker.ts🔗) to avoid blocking the main thread. The key point is that everything can run entirely on the client side:

Step 1: PDF Embedding

After the user uploads a PDF, the Worker executes the following flow:

import { WebPDFLoader } from "@langchain/community/document_loaders/web/pdf";
import { RecursiveCharacterTextSplitter } from "langchain/text_splitter";
import { Voy as VoyClient } from "voy-search";
import { HuggingFaceTransformersEmbeddings } from "@langchain/community/embeddings/hf_transformers";

const embeddings = new HuggingFaceTransformersEmbeddings({
  modelName: "Xenova/all-MiniLM-L6-v2",
  // Can use "nomic-ai/nomic-embed-text-v1" for more powerful but slower embeddings
  // modelName: "nomic-ai/nomic-embed-text-v1",
});

const voyClient = new VoyClient();
const vectorstore = new VoyVectorStore(voyClient, embeddings);

const embedPDF = async (pdfBlob: Blob) => {
  // 1. Use LangChain's WebPDFLoader to parse the PDF
  const pdfLoader = new WebPDFLoader(pdfBlob, { parsedItemSeparator: " " });
  const docs = await pdfLoader.load();

  // 2. Use RecursiveCharacterTextSplitter to split it into small chunks
  const splitter = new RecursiveCharacterTextSplitter({
    chunkSize: 500,
    chunkOverlap: 50,
  });
  const splitDocs = await splitter.splitDocuments(docs);

  // 3. Convert through Transformers.js Embedding and store the results in the Voy vector database
  await vectorstore.addDocuments(splitDocs);
};

The value of LangChain here is that components such as WebPDFLoader, RecursiveCharacterTextSplitter, and VoyVectorStore are all standardized interfaces. You do not need to handle PDF parsing or the edge cases of chunk splitting yourself.

Step 2: RAG Conversation Flow

When the user asks a question, the project uses LangGraph to build a three-node state graph to handle the RAG flow:

Corresponding to the code:

const graph = new StateGraph(RAGStateAnnotation)
  .addNode("rephraseQuestion", rephraseQuestion)
  .addNode("retrieveSourceDocuments", retrieveSourceDocuments)
  .addNode("generateResponse", generateResponse)
  .addConditionalEdges("__start__", async (state) => {
    // Retrieve directly for the first turn; for multi-turn conversations, rephrase the question before retrieval
    if (state.messages.length > 1) {
      return "rephraseQuestion";
    }
    return "retrieveSourceDocuments";
  })
  .addEdge("rephraseQuestion", "retrieveSourceDocuments")
  .addEdge("retrieveSourceDocuments", "generateResponse")
  .compile();

The three nodes are responsible for:

1. rephraseQuestion — If it is already a multi-turn conversation, first use the LLM to combine the latest question with the context and rewrite it as a standalone search query, improving retrieval quality (for example, the user asks “What about Chapter 2?” → rewritten as “What is the content of Chapter 2 in the document?”)
2. retrieveSourceDocuments — Use the rewritten query to perform a similarity search against the vector database and find the most relevant document fragments
3. generateResponse — Insert the retrieved document fragments into the <context> section of the System Prompt, allowing the LLM to generate an answer based on this data

Step 3: Swapping Among Three Local LLM Providers

One of the biggest benefits of LangChain is its unified interface. In this project, switching among the three models only requires replacing the class being instantiated:

let model;
if (modelProvider === "webllm") {
  model = new ChatWebLLM(modelConfig);    // In-browser WebGPU inference
} else if (modelProvider === "chrome_ai") {
  model = new ChromeAI(modelConfig);       // Chrome built-in Gemini Nano
} else {
  model = new ChatOllama(modelConfig);     // Local Ollama server
}

The same RAG Pipeline does not need any logic changes at all because of differences in the underlying model. This is the benefit brought by the framework abstraction layer. The only thing to note is that Chrome AI is currently a text-in/text-out LLM (not a Chat Model), so the Prompt format requires special handling: conversation history must be manually concatenated into a string.

Summary

After breaking down this project, we can see LangChain’s core value:

• Standardized document processing — WebPDFLoader + RecursiveCharacterTextSplitter can complete the conversion from PDF to a model-retrievable data format in just a few lines of code
• Vector retrieval abstraction — Whether the underlying system uses Voy, Pinecone, or another vector database, the API is consistent
• Swappable model providers — Ollama / WebLLM / Chrome AI share the same process
• LangGraph state machine — Describes the nodes and conditional branches in a RAG flow declaratively, making it easier to read and maintain than manually writing if-else

By breaking down the fully-local-pdf-chatbot project, we can understand the overall usage scenarios and real-world flow of LangChain.

Further Reading

• LangChain Learn Docs🔗