15 AI Engineering Concepts Senior Engineers Should Know — Not Just the Name, But How Each One Fails

I posted 15 AI engineering concepts on X this week and asked people how many they could explain clearly.

Over 1,900 people bookmarked it. The replies were honest. "I was at 11, gotta go and study." "13 for me."

But one reply was better than my post:

"Anyone can explain LLM-as-a-judge. Knowing it has position bias and when it lies to you consistently is the actual interview signal."

That's exactly right.

At Senior+ level, interviewers don't care that you know the word.
They care that you know how it breaks.

So here's the list again, with the failure mode I'd expect a strong candidate to bring up on their own:

[1] KV cache → memory grows with every token of context. Long chats get expensive and slow.

[2] Prompt caching → change one character at the start of the prompt and the cache misses.

[3] Semantic caching → "cancel my order" and "cancel my subscription" look alike to an embedding.

[4] Speculative decoding → faster only when the small model guesses right. Wrong guesses cost you.

[5] Context window vs effective context → a 1M-token window doesn't mean the model uses all of it well.

[6] Hybrid search → pure vector search misses exact matches like error codes and order IDs.

[7] Reranking → better answers, but it adds latency to every request.

[8] Structured outputs → valid JSON can still contain wrong values.

[9] Tool calling → the model can call the right tool with the wrong arguments.

[10] LLM-as-a-judge → it tends to prefer the first answer and the longer one.

[11] Eval sets → they pass if the same person wrote the prompt and the tests.

[12] Guardrails → too strict and the product becomes useless.

[13] Prompt injection → any text the model reads, including a PDF or an email, can carry instructions.

[14] Token budgets → one heavy user can eat everyone's quota.

[15] Model routing → the cheap model handles the easy 80%, which is also where nobody notices mistakes.

Notice something? Almost every one of these is an old backend problem. Caching, validation, rate limits, testing, trust.

That's why I keep telling backend engineers they're not behind on AI.

You already know the hard half.
You just have to learn the new names.

Save this for your next interview.

--
P.S. I've written up the backend half (caching, idempotency, rate limiting, failure modes) in my System Design Fundamentals vault.

Check it out: puneetpatwari.in

A list of 15 AI engineering concepts, framed around the idea that Senior+ interviews care less about whether you know the term and more about whether you know its failure mode:

1. KV cache — memory scales with every token of context; long conversations get expensive and slow.
2. Prompt caching — a cache hit depends on an unchanged prefix; altering the start of a prompt invalidates it.
3. Semantic caching — embedding similarity can conflate meaningfully different requests (e.g., "cancel my order" vs. "cancel my subscription").
4. Speculative decoding — speeds up generation only when the draft model's guesses are accepted; rejected guesses add overhead.
5. Context window vs. effective context — a large nominal window doesn't guarantee the model uses all of it equally well (a well-documented phenomenon often called "lost in the middle").
6. Hybrid search — pure vector search struggles with exact-match lookups like error codes or order IDs, which is why lexical/keyword search is paired with it.
7. Reranking — improves answer quality at the cost of added per-request latency.
8. Structured outputs — schema-valid JSON doesn't guarantee the values inside are correct.
9. Tool calling — a model can select the right tool but pass incorrect arguments.
10. LLM-as-a-judge — documented position bias (favoring earlier-presented answers) and verbosity bias (favoring longer answers) are real, though recent research shows these effects vary by model family and some newer judges are far less susceptible than the original MT-Bench-era findings suggested; style/formatting bias is now considered an even bigger factor by some researchers.
11. Eval sets — risk passing artificially when the same person designs both the prompts and the tests, inflating confidence without real coverage.
12. Guardrails — overly strict rules can make a product unusable, trading safety for utility.
13. Prompt injection — any text the model ingests (a PDF, an email, a webpage) can carry embedded instructions, not just the user's direct input.
14. Token budgets — a shared quota can be exhausted disproportionately by one heavy user, a classic noisy-neighbor problem.
15. Model routing — cheaper models often handle the easy majority of traffic, which is also where mistakes are least likely to be noticed.

The broader observation: most of these map directly onto long-standing backend engineering concerns — caching, validation, rate limiting, testing discipline, and trust boundaries — just with new vocabulary layered on top.