Notes from my talk at the AI Builders Meetup
The AI Builders Meetup was incredible. The energy in the room, the questions at the end, and the people who stayed around talking long after we shut off the projector—these are what make preparing a talk worth it. Since several ideas were still hanging in the air when we ran out of time, I want to write down the insight I was really trying to plant.
I opened with an uncomfortable premise, and I stand by it: AI can make you feel dramatically faster while making your team measurably worse. That is not an anti-AI warning. Rather, it is a warning against confusing the sensation of speed with actual throughput. If code lands in two minutes but nobody understands it, nobody can review it, and nobody knows why that particular design was chosen, you did not gain time. You borrowed it, at interest, from the next person who has to touch that module.
The architecture matters the most
If you take one thing away from the talk, make it this. An LLM application is not “a prompt and a model.” It is a layered system, and each layer addresses exactly one concern. Here is the slide I spent the longest time on:
Six layers of a modern LLM application, and the key design principles that hold them together.
Read it from the top to the bottom, because the request flows in that order and so does the debugging. Here are the layers:
— User interface layer. Web, mobile, API clients, Copilot-style UIs, streaming responses. Streaming is an architectural choice, not a cosmetic one: it changes how you handle partial output, cancellation, and error recovery.
— Orchestration layer. LangChain, LlamaIndex, custom chains, agent loops, tool dispatch. This is your control flow—retries, branching, and when the loop is allowed to stop.
— Context engineering layer. Prompt construction, memory retrieval, the RAG pipeline, token budget management. This is the starred layer, and I will expand on it in the next section.
— Model gateway layer. Routing, rate limiting, auth, cost tracking, fallback logic. Never let application code call a provider SDK directly: a gateway is what lets you swap models, degrade gracefully when a provider starts returning 429s, and attribute spend per feature.
— Model layer. GPT-4o, Claude, Gemini, Llama, fine-tuned variants, embedding models. Deliberately near the bottom, and deliberately interchangeable.
— Data and tools layer. Vector DB, SQL, APIs, file stores, external tools, function calls. Where ground truth actually lives.
The principles on the right of the slide are what keep the stack honest: separation of concerns, stateless models, context as a first-class citizen, observability at every layer, and fail-safe defaults. Observability is the one teams skip and then regret. If you cannot see the prompt that was actually sent, the chunks that were actually retrieved, and the tokens actually spent, you are guessing, not debugging.
Why context engineering is the layer that decides your product
The model is stateless. It knows nothing about your project beyond what you placed in front of it on this call. Everything it appears to “remember” is something a piece of your code decided to include. So the engineering question is not “which model should I use?” but what information goes in, in what order, and what gets dropped when it no longer fits?
That is why I treat the context window like a memory budget with named segments, roughly a system prompt, long-term memory, retrieved RAG chunks, conversation history, tool outputs, and the actual user message. Note the proportions: retrieval and history together can easily eat more than half the window, which means your most expensive tokens are usually the ones you chose least deliberately. There are three rules I would hold on to:
- Hard caps per segment, so a chatty tool response cannot silently evict your system prompt.
- An explicit priority order—system > user query > history > RAG—with the oldest history truncated first.
- Active compression: summarize long conversations, drop redundant retrieved chunks, use a sliding window over history, and rerank RAG results selectively instead of always shipping top-K.
Following this, track average tokens per call as a first-class metric: it is the closest thing you have to a leading indicator for both cost and quality regressions.
Three practical things for Monday
1. Decompose your prompts.
One giant prompt with five chained instructions almost always underperforms five simple, individually verifiable steps. This follows the same logic as small functions: each step is separately testable, and when the output is wrong, you know which stage failed instead of rewriting a paragraph and hoping. Reach for extended thinking where the reasoning is genuinely hard—multi-file bugs, migrations, architectural trade-offs, security-sensitive review—and keep it off the routine edits.
2. Write the CLAUDE.md.
Treat it as the project constitution: build and test commands, coding conventions, architectural boundaries, the explicit “never do this” list. Checked into the repo, shared by the team, read every session—the difference between an assistant that guesses your conventions and one that already knows them.
3. Package what you already know.
MCP is an open standard that connects the model to your real systems—GitHub, databases, internal APIs, browsers—through one interface, permissioned per server. Skills turn a ten-step runbook into a repeatable capability: a folder with a SKILL.md file, loaded on demand through progressive disclosure, so only the name and description sit in context until the skill is actually needed. The description is the most important field, because it is a trigger contract. For example, “Helps with documentation” is useless. On the other hand, “Update CHANGELOG.md when asked to log a change; not for release notes or commit messages” tells the model when to fire and when to stay out of the way.
Velocity with control
AI does not replace engineering judgment; it makes judgment more valuable, because you are now producing more code than you can review by reflex. Explicit layers, an engineered context window, and shared institutional memory are what stop “we are moving incredibly fast” from turning into “nobody knows how this works anymore.”
See you at the Claude Code Workshop in Guadalajara on Saturday, August 29, at Wizeline Zapopan. We will build it live.
