The problem: you’ve built a prototype that won’t survive real traffic
Your LLM responds in the console, the API call works in the notebook, and you’re thrilled until the client asks: “When can we go live?”. Then you realize your Python code knows nothing about load balancers, tokens cost a fortune, and answering product‑manual questions requires a full RAG pipeline from scratch.
We at Meteora Web have been there. That’s why we wrote this pillar guide: to give developers a clear map from ChatOpenAI to a production‑ready LLM app.
Below you’ll find the fundamentals: LangChain, RAG, vector databases, LlamaIndex, LangSmith, embeddings, streaming, fine‑tuning, deployment, and cost management. Every section ends with something you can put into production today.
LangChain fundamentals: chains, prompt templates and output parsers
LangChain is not magic: it’s an abstraction layer that turns LLM API calls into composable chains. Three core blocks.
Prompt templates: never hard‑code strings
A hard‑coded prompt is the first step towards fragile code. Templates separate logic from content.
from langchain.prompts import PromptTemplate
template = """You are a clothing store assistant.
The customer asks: {question}
Answer clearly and professionally."""
prompt = PromptTemplate.from_template(template)
print(prompt.format(question="What are your best-selling pants?"))
Why it matters: change the template without touching code. Handle dynamic variables, translations, different tones.
Output parsers: structured output, not free text
An LLM that returns JSON is far more useful than one writing paragraphs. Parsers turn text into Python objects.
from langchain.output_parsers import CommaSeparatedListOutputParser
parser = CommaSeparatedListOutputParser()
# "red, green, blue" -> ["red", "green", "blue"]
We use PydanticOutputParser for validated objects. Fewer bugs, faster iteration.
Chain: the pipe that connects everything
from langchain.chains import LLMChain
chain = LLMChain(
llm=ChatOpenAI(model="gpt-4o-mini"),
prompt=prompt,
output_parser=parser
)
response = chain.run(question="What are the three best-selling models?")
Action item: replace your raw response = client.chat(...) with an LLMChain + structured parser. In 30 minutes your code becomes maintainable and testable.
RAG with LangChain: retrieve real documents, don’t hallucinate
An LLM alone doesn’t know your business data. Retrieval Augmented Generation solves this: find relevant document chunks from a vector database and inject them into the prompt.
The RAG flow in 4 steps
- Load a document (PDF, HTML, text)
- Split it into meaningful chunks
- Convert each chunk into an embedding and store it in a vector store
- On a query, find the most similar chunks and attach them to the prompt
from langchain.document_loaders import TextLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import Chroma
loader = TextLoader("product_manual.txt")
docs = loader.load()
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = splitter.split_documents(docs)
vectorstore = Chroma.from_documents(chunks, OpenAIEmbeddings())
retriever = vectorstore.as_retriever()
Then in your chain:
from langchain.chains import RetrievalQA
qa_chain = RetrievalQA.from_chain_type(
llm=ChatOpenAI(model="gpt-4o-mini"),
chain_type="stuff",
retriever=retriever
)
response = qa_chain.run("How do I reset the WiFi module?")
Warning: stuff works only if chunks fit the context window. For long documents use map_reduce or refine.
Action item: pick a 10‑page PDF, build a local Chroma, and test query responses. Measure retrieval time: under 1 second is fine. Above, optimize chunk size and embedding model.
Vector Databases: when Chroma isn’t enough
Chroma is great for prototyping and small projects. But with millions of documents or sub‑50ms latency requirements, you need to level up.
- Chroma: lightweight, open source, ideal for dev and demos. Installed via pip, saves data to disk.
- Pinecone: serverless, automatic indexing, fully managed. Pay for throughput and storage. Great when you don’t want to admin a server.
- Weaviate: containerized, hybrid vector + lexical, horizontally scalable. For enterprise needs with DevOps support.
# Example with Weaviate (v3 client)
import weaviate
client = weaviate.Client("http://localhost:8080")
client.schema.create_class({
"class": "Product",
"properties": [
{"name": "name", "dataType": ["string"]},
{"name": "description", "dataType": ["text"]}
]
})
When to move from Chroma: when your dataset exceeds 100k chunks or when concurrent requests exceed 10 per second. We did it for an e‑commerce platform with 500k products – Weaviate in a Docker cluster handled it flawlessly.
LlamaIndex: a data framework for knowledge workers
LlamaIndex (formerly GPT Index) is an alternative or complement to LangChain. While LangChain is generic and chain‑centric, LlamaIndex specializes in data management: index, index, index.
Concrete advantages:
- native connectors for 100+ formats (Google Drive, Notion, SQL, Slack)
- automatic chunking and document hierarchy
- support for vector, keyword tree, and summary indices
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("What are the key points of the 2025 budget?")
We use it when the client has heterogeneous data (invoices, emails, manuals) and wants a single Q&A interface. Learning curve is gentle, but query customization is less flexible than LangChain.
LangSmith: debug and monitor your LLM apps
When an LLM in production gives wrong answers, you need to know why. LangSmith is LangChain’s observability platform: logs every run, shows generated prompts, token usage, response times.
We use it to:
- track regressions after a model swap
- compare GPT‑4 vs GPT‑4o‑mini responses on a test dataset
- find hallucinations by analyzing retrieved chunks
Minimal setup: export LANGCHAIN_API_KEY=... LANGCHAIN_TRACING_V2=true. Then every chain is traced automatically. Cost: nearly free (generous free tier).
Action item: enable tracing on your RAG chain and run 20 test queries. Inspect the dashboard: is retrieval time under control? Are the retrieved chunks relevant? If you see irrelevant chunks, change the text splitter or embedding model.
Embeddings: the foundation of semantic search
Embeddings are numerical vectors representing text meaning. The more similar two texts are, the closer their vectors (cosine similarity).
How to choose an embedding model?
- OpenAI Embeddings (text‑embedding‑3‑small/large): excellent quality and simplicity, but cost per token and data passes through external servers.
- Sentence‑Transformers (local): install with
pip install sentence-transformers. Free, fast with GPU, total privacy. - BGE‑M3 (multilingual): if your documents are in Italian, it outperforms English‑only models.
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("BAAI/bge-m3")
embedding = model.encode("The modem won't connect").tolist()
Common mistake: using a different embedding model for indexing vs querying. The vector store doesn't understand the latent space shift. Always stick to the same model.
Streaming LLM responses: UX that makes a difference
A user won’t wait 10 seconds for a complete answer. Streaming delivers tokens as they are generated: the user sees the response while it’s being written. With FastAPI and LangChain it’s straightforward.
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
llm = ChatOpenAI(
model="gpt-4o-mini",
streaming=True,
callbacks=[StreamingStdOutCallbackHandler()]
)
To integrate with a FastAPI web app: use StreamingResponse and a custom callback that writes to a generator.
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from langchain.callbacks.base import BaseCallbackHandler
class StreamHandler(BaseCallbackHandler):
def __init__(self): self.queue = []
def on_llm_new_token(self, token, **kwargs):
self.queue.append(token)
app = FastAPI()
@app.post("/chat")
async def chat(prompt: str):
handler = StreamHandler()
chain = LLMChain(
llm=ChatOpenAI(streaming=True, callbacks=[handler]),
prompt=prompt_template
)
chain.run(prompt)
async def generate():
for token in handler.queue:
yield token
return StreamingResponse(generate(), media_type="text/plain")
Action item: add streaming to your local chatbot. Measure perceived latency: if the first token arrives within 500 ms, the user stays. Otherwise optimize network or switch to a smaller model.
Fine‑tuning LLMs: when it makes sense and when it doesn’t
Fine‑tuning isn’t for everyone. It’s useful when you have structured input‑output pairs and want to specialize a model on a narrow domain (e.g., medical language, legal contracts).
But beware:
- fine‑tuning doesn’t fix hallucinations about facts not in the training set
- it’s expensive: GPU training + model hosting
- requires thousands of quality examples
When not needed: if prompt engineering + RAG can achieve the same result. 90% of use cases don’t need fine‑tuning.
If you decide to try it, use Unsloth or Axolotl with QLoRA – they reduce GPU memory by 70%.
# Using Unsloth
from unsloth import FastLanguageModel
model, tokenizer = FastLanguageModel.from_pretrained(
model_name="unsloth/llama-3-8b-bnb-4bit",
max_seq_length=2048,
dtype=None,
load_in_4bit=True,
)
Rule of thumb: if your dataset has fewer than 500 examples, don’t do it. Use RAG. If you have 5000+, consider QLoRA fine‑tuning on a small model (7B).
Deploying AI apps: Gradio, Streamlit, and FastAPI
The prototype works locally. Now it must go online. Three paths.
- Gradio: perfect for quick demos and prototypes. A few lines give you a web UI with sliders, chat, file uploads.
- Streamlit: more flexible, great for AI dashboards with visualizations. We use it for internal document analysis tools.
- FastAPI: the only choice for production APIs. Handles CORS, rate limiting, authentication. Combine with Docker and a reverse proxy (Nginx/Caddy).
# FastAPI RAG endpoint
from fastapi import FastAPI
from pydantic import BaseModel
app = FastAPI()
class Query(BaseModel):
text: str
@app.post("/rag")
async def rag_endpoint(query: Query):
result = qa_chain.run(query.text)
return {"response": result}
We at Meteora Web built a proprietary social‑media management platform with Laravel/Livewire, but for AI services we use FastAPI in Docker containers on Linux servers. Why? Total control and no lifetime fees. The same stack we use for our clients.
LLM costs: optimizing tokens and API budget
A project spending $200/month on OpenAI APIs can become $2000 if tokens aren’t monitored. Here are our levers.
- Cache responses: if the same question repeats, don’t call the API. Use Redis or an in‑memory dict for seen queries.
- Smaller models: GPT‑4o‑mini costs $0.15/M input tokens vs $2.5/M for GPT‑4o. For 90% of applications quality is sufficient.
- Optimized chunks: if every RAG call returns 4000 context tokens, cost multiplies. Try chunks of 500 tokens.
- Batch embeddings: embed hundreds of documents in one API call instead of one by one.
Monitoring tools: LangSmith tracks tokens, but you can also log to a database and calculate cost per user session. We do it on every project: the monthly report shows cost / query. If it exceeds $0.01, we take action.
In summary — what to do now
- Build a local RAG with Chroma and a product PDF. It won’t take an afternoon.
- Add streaming to your endpoint and test UX from the browser.
- Enable LangSmith tracing on an existing chain. You’ll uncover things you didn’t know.
- Calculate your cost per query with your current model. If it exceeds $0.005, start optimizing.
- Choose your deployment stack: for public APIs, FastAPI + Docker. For client demos, Gradio on a VPS.
Don’t stop at the prototype. The gap between demo and production is bridged by these practices. We do it every day for companies across Italy. And if you’d like a technical conversation, get in touch.
