The problem: you’ve built a prototype that won’t survive real traffic
Your LLM responds in the console, the API call works in the notebook, and you’re thrilled until the client asks: “When can we go live?”. Then you realize your Python code knows nothing about load balancers, tokens cost a fortune, and answering product‑manual questions requires a full RAG pipeline from scratch.
We at Meteora Web have been there. That’s why we wrote this pillar guide: to give developers a clear map from ChatOpenAI to a production‑ready LLM app.
Below you’ll find the fundamentals: LangChain, RAG, vector databases, LlamaIndex, LangSmith, embeddings, streaming, fine‑tuning, deployment, and cost management. Every section ends with something you can put into production today.
LangChain fundamentals: chains, prompt templates and output parsers
LangChain is not magic: it’s an abstraction layer that turns LLM API calls into composable chains. Three core blocks.
Prompt templates: never hard‑code strings
A hard‑coded prompt is the first step towards fragile code. Templates separate logic from content.
from langchain.prompts import PromptTemplate
template = """You are a clothing store assistant.
The customer asks: {question}
Answer clearly and professionally."""
prompt = PromptTemplate.from_template(template)
print(prompt.format(question="What are your best-selling pants?"))
Why it matters: change the template without touching code. Handle dynamic variables, translations, different tones.
Output parsers: structured output, not free text
An LLM that returns JSON is far more useful than one writing paragraphs. Parsers turn text into Python objects.
from langchain.output_parsers import CommaSeparatedListOutputParser
parser = CommaSeparatedListOutputParser()
# "red, green, blue" -> ["red", "green", "blue"]
We use PydanticOutputParser for validated objects. Fewer bugs, faster iteration.
Chain: the pipe that connects everything
from langchain.chains import LLMChain
chain = LLMChain(
llm=ChatOpenAI(model="gpt-4o-mini"),
prompt=prompt,
output_parser=parser
)
response = chain.run(question="What are the three best-selling models?")
Action item: replace your raw response = client.chat(...) with an LLMChain + structured parser. In 30 minutes your code becomes maintainable and testable.
Sponsored Protocol
RAG with LangChain: retrieve real documents, don’t hallucinate
An LLM alone doesn’t know your business data. Retrieval Augmented Generation solves this: find relevant document chunks from a vector database and inject them into the prompt.
The RAG flow in 4 steps
- Load a document (PDF, HTML, text)
- Split it into meaningful chunks
- Convert each chunk into an embedding and store it in a vector store
- On a query, find the most similar chunks and attach them to the prompt
from langchain.document_loaders import TextLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import Chroma
loader = TextLoader("product_manual.txt")
docs = loader.load()
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = splitter.split_documents(docs)
vectorstore = Chroma.from_documents(chunks, OpenAIEmbeddings())
retriever = vectorstore.as_retriever()
Then in your chain:
from langchain.chains import RetrievalQA
qa_chain = RetrievalQA.from_chain_type(
llm=ChatOpenAI(model="gpt-4o-mini"),
chain_type="stuff",
retriever=retriever
)
response = qa_chain.run("How do I reset the WiFi module?")
Warning: stuff works only if chunks fit the context window. For long documents use map_reduce or refine.
Action item: pick a 10‑page PDF, build a local Chroma, and test query responses. Measure retrieval time: under 1 second is fine. Above, optimize chunk size and embedding model.
Vector Databases: when Chroma isn’t enough
Chroma is great for prototyping and small projects. But with millions of documents or sub‑50ms latency requirements, you need to level up.
- Chroma: lightweight, open source, ideal for dev and demos. Installed via pip, saves data to disk.
- Pinecone: serverless, automatic indexing, fully managed. Pay for throughput and storage. Great when you don’t want to admin a server.
- Weaviate: containerized, hybrid vector + lexical, horizontally scalable. For enterprise needs with DevOps support.
# Example with Weaviate (v3 client)
import weaviate
client = weaviate.Client("http://localhost:8080")
client.schema.create_class({
"class": "Product",
"properties": [
{"name": "name", "dataType": ["string"]},
{"name": "description", "dataType": ["text"]}
]
})
When to move from Chroma: when your dataset exceeds 100k chunks or when concurrent requests exceed 10 per second. We did it for an e‑commerce platform with 500k products – Weaviate in a Docker cluster handled it flawlessly.
Sponsored Protocol
LlamaIndex: a data framework for knowledge workers
LlamaIndex (formerly GPT Index) is an alternative or complement to LangChain. While LangChain is generic and chain‑centric, LlamaIndex specializes in data management: index, index, index.
Concrete advantages:
- native connectors for 100+ formats (Google Drive, Notion, SQL, Slack)
- automatic chunking and document hierarchy
- support for vector, keyword tree, and summary indices
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("What are the key points of the 2025 budget?")
We use it when the client has heterogeneous data (invoices, emails, manuals) and wants a single Q&A interface. Learning curve is gentle, but query customization is less flexible than LangChain.
LangSmith: debug and monitor your LLM apps
When an LLM in production gives wrong answers, you need to know why. LangSmith is LangChain’s observability platform: logs every run, shows generated prompts, token usage, response times.
We use it to:
- track regressions after a model swap
- compare GPT‑4 vs GPT‑4o‑mini responses on a test dataset
- find hallucinations by analyzing retrieved chunks
Minimal setup: export LANGCHAIN_API_KEY=... LANGCHAIN_TRACING_V2=true. Then every chain is traced automatically. Cost: nearly free (generous free tier).
Sponsored Protocol
Action item: enable tracing on your RAG chain and run 20 test queries. Inspect the dashboard: is retrieval time under control? Are the retrieved chunks relevant? If you see irrelevant chunks, change the text splitter or embedding model.
Embeddings: the foundation of semantic search
Embeddings are numerical vectors representing text meaning. The more similar two texts are, the closer their vectors (cosine similarity).
How to choose an embedding model?
- OpenAI Embeddings (text‑embedding‑3‑small/large): excellent quality and simplicity, but cost per token and data passes through external servers.
- Sentence‑Transformers (local): install with
pip install sentence-transformers. Free, fast with GPU, total privacy. - BGE‑M3 (multilingual): if your documents are in Italian, it outperforms English‑only models.
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("BAAI/bge-m3")
embedding = model.encode("The modem won't connect").tolist()
Common mistake: using a different embedding model for indexing vs querying. The vector store doesn't understand the latent space shift. Always stick to the same model.
Streaming LLM responses: UX that makes a difference
A user won’t wait 10 seconds for a complete answer. Streaming delivers tokens as they are generated: the user sees the response while it’s being written. With FastAPI and LangChain it’s straightforward.
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
llm = ChatOpenAI(
model="gpt-4o-mini",
streaming=True,
callbacks=[StreamingStdOutCallbackHandler()]
)
To integrate with a FastAPI web app: use StreamingResponse and a custom callback that writes to a generator.
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from langchain.callbacks.base import BaseCallbackHandler
class StreamHandler(BaseCallbackHandler):
def __init__(self): self.queue = []
def on_llm_new_token(self, token, **kwargs):
self.queue.append(token)
app = FastAPI()
@app.post("/chat")
async def chat(prompt: str):
handler = StreamHandler()
chain = LLMChain(
llm=ChatOpenAI(streaming=True, callbacks=[handler]),
prompt=prompt_template
)
chain.run(prompt)
async def generate():
for token in handler.queue:
yield token
return StreamingResponse(generate(), media_type="text/plain")
Action item: add streaming to your local chatbot. Measure perceived latency: if the first token arrives within 500 ms, the user stays. Otherwise optimize network or switch to a smaller model.
Sponsored Protocol
Fine‑tuning LLMs: when it makes sense and when it doesn’t
Fine‑tuning isn’t for everyone. It’s useful when you have structured input‑output pairs and want to specialize a model on a narrow domain (e.g., medical language, legal contracts).
But beware:
- fine‑tuning doesn’t fix hallucinations about facts not in the training set
- it’s expensive: GPU training + model hosting
- requires thousands of quality examples
When not needed: if prompt engineering + RAG can achieve the same result. 90% of use cases don’t need fine‑tuning.
If you decide to try it, use Unsloth or Axolotl with QLoRA – they reduce GPU memory by 70%.
# Using Unsloth
from unsloth import FastLanguageModel
model, tokenizer = FastLanguageModel.from_pretrained(
model_name="unsloth/llama-3-8b-bnb-4bit",
max_seq_length=2048,
dtype=None,
load_in_4bit=True,
)
Rule of thumb: if your dataset has fewer than 500 examples, don’t do it. Use RAG. If you have 5000+, consider QLoRA fine‑tuning on a small model (7B).
Deploying AI apps: Gradio, Streamlit, and FastAPI
The prototype works locally. Now it must go online. Three paths.
- Gradio: perfect for quick demos and prototypes. A few lines give you a web UI with sliders, chat, file uploads.
- Streamlit: more flexible, great for AI dashboards with visualizations. We use it for internal document analysis tools.
- FastAPI: the only choice for production APIs. Handles CORS, rate limiting, authentication. Combine with Docker and a reverse proxy (Nginx/Caddy).
# FastAPI RAG endpoint
from fastapi import FastAPI
from pydantic import BaseModel
app = FastAPI()
class Query(BaseModel):
text: str
@app.post("/rag")
async def rag_endpoint(query: Query):
result = qa_chain.run(query.text)
return {"response": result}
We at Meteora Web built a proprietary social‑media management platform with Laravel/Livewire, but for AI services we use FastAPI in Docker containers on Linux servers. Why? Total control and no lifetime fees. The same stack we use for our clients.
Sponsored Protocol
LLM costs: optimizing tokens and API budget
A project spending $200/month on OpenAI APIs can become $2000 if tokens aren’t monitored. Here are our levers.
- Cache responses: if the same question repeats, don’t call the API. Use Redis or an in‑memory dict for seen queries.
- Smaller models: GPT‑4o‑mini costs $0.15/M input tokens vs $2.5/M for GPT‑4o. For 90% of applications quality is sufficient.
- Optimized chunks: if every RAG call returns 4000 context tokens, cost multiplies. Try chunks of 500 tokens.
- Batch embeddings: embed hundreds of documents in one API call instead of one by one.
Monitoring tools: LangSmith tracks tokens, but you can also log to a database and calculate cost per user session. We do it on every project: the monthly report shows cost / query. If it exceeds $0.01, we take action.
In summary — what to do now
- Build a local RAG with Chroma and a product PDF. It won’t take an afternoon.
- Add streaming to your endpoint and test UX from the browser.
- Enable LangSmith tracing on an existing chain. You’ll uncover things you didn’t know.
- Calculate your cost per query with your current model. If it exceeds $0.005, start optimizing.
- Choose your deployment stack: for public APIs, FastAPI + Docker. For client demos, Gradio on a VPS.
Don’t stop at the prototype. The gap between demo and production is bridged by these practices. We do it every day for companies across Italy. And if you’d like a technical conversation, get in touch.
