🚨 CHINA JUST OPEN SOURCED AN OCR THAT EATS 100 PAGE PDFS IN ONE PASSIt’s called Unlimited OCR. Only 3B params. Runs locally.Every other OCR tool chops your doc into pages and loses the thread. This one reads the whole thing in one continuous pass.→ One shot full context parsing (32K context window)→ Multilingual out of the box→ 93% on the standard parsing benchmark (6 points over baseline)→ Under 0.11 error rate past 40 pages→ Runs 100% locally on your own hardware→ Works with Transformers, vLLM, SGLang, Docker, Ollama, llama.cppTraditional cloud OCR (Textract, Google Vision, Azure Doc Intelligence) costs $1.50 to $15 per 1,000 pages.This runs on your machine. For free. Forever.Baidu built it explicitly to push DeepSeek OCR one step further. Already at 1.9M downloads on Hugging Face and most people have no idea it exists yet.100% open source.

Ollama vs vLLM vs SGLangEvery inference engine is really an answer to one question: where does the waste come from?Ollama assumes the bottleneck is access. It packages a quantized model (GGUF through llama.cpp, or MLX on Apple Silicon) into a single local runtime. By default it serves requests in order and runs entirely offline. It’s built for one person at one machine, and it does that job very well.vLLM assumes the bottleneck is GPU memory and idle time. Continuous batching lets new requests join the running batch at every decoding step instead of waiting for the slowest sequence to finish. PagedAttention borrows from operating-system virtual memory: the KV cache is split into fixed-size pages, so memory is allocated on demand rather than reserved up front. The result is high throughput across thousands of unrelated prompts.SGLang assumes the bottleneck is repetition. Agents and multi-turn conversations send the same long system prompts, tool definitions and chat history again and again. RadixAttention stores those prefixes in a radix tree so shared context is computed once and reused, and a cache-aware router sends each request to the worker that already holds its prefix. Constrained decoding then forces the output into valid JSON or a regex pattern.Three engines, three assumptions about where compute is lost:Access → OllamaConcurrency → vLLMShared context → SGLangChoosing between them is less about benchmarks and more about knowing which of these problems you actually have.Which constraint shapes your stack today?

EdgeQuake v0.34.0 is out: Graph-RAG in Rust, and an enterprise-ready Second Brain for your AI agents.One binary. One Postgres. Any LLM, cloud or fully local.Vector search finds similar text. It struggles when the answer lives in the relationships between things:"Which contract affects which product?""Who reports to whom?""What themes run across these 200 papers?"EdgeQuake implements LightRAG's Hybrid GraphRAG in Rust. Documents become a knowledge graph of entities and relationships. At query time, vector search and graph traversal work together, and every answer comes with its sources.🏢 Built for the enterprise- One datastore: PostgreSQL 16/17/18 with pgvector and Apache AGE. No separate vector or graph DB- Multi-tenant, fail-closed workspace isolation- SSO with Keycloak/OIDC, built-in auth and audit logging- Provider keys encrypted at rest (AES-256-GCM), write-only- Safe upgrades: the API never migrates your DB. Migrations are explicit and replay-tested from every release- edgequake doctor checks DB, schema, roles and security posture- PDF vision pipeline for tables, scans and complex layouts🔌 Pick your LLM, per roleOpenAI, Anthropic, Gemini, Mistral, or private on your own machine with Ollama, LM Studio, MLX or llama.cpp. Each workspace can assign a different model to extraction, query, summary, embedding and vision.- The Second Brain: EdgeQuake MCPThe MCP server plugs your knowledge graph into Claude Desktop, Cursor, VS Code or any MCP client. Your agents can:- Remember: push notes, text and PDFs into the graph🔎 Recall: multi-hop answers in hybrid mode, with sources🕸️ Explore: entities, neighbours and relationships🗂️ Organize: one workspace per project, team or clientWorkspace-scoped API keys keep each agent on least privilege. Memory persists across conversations and stays on your infrastructure.📊 Honest numbers (GraphRAG-Bench, n=200, same corpus and models)• Accuracy: 0.792 vs 0.786 for LightRAG, a statistical tie• Warm query p50: 82 ms vs 993 ms• Evidence recall: LightRAG ahead (0.949 vs 0.932)The pitch is the same retrieval quality, productionised. We publish where we tie and where we lose.🚀 Try it, Docker only:curl -fsSL https://lnkd.in/g8a6a4Ee | sh2.1k ⭐, 99 releases, 14 contributors. Thank you all.

RAG vs. Jev + RAG, clearly explained:In a standard RAG setup, documents are split into chunks, converted into embeddings, and stored in a vector DB.When a query arrives, the system embeds it and retrieves the top-k chunks with the closest vectors.Many production setups add a reranker after retrieval.The reranker compares the query with each retrieved passage, improves their ordering, and keeps the highest-scoring results.Those passages are then placed in the context window, and the LLM generates an answer from them.The problem is that ranking and answerability are different questions.A passage can be more relevant than the other candidates while still containing weak, incomplete, or merely adjacent evidence.The LLM receives it anyway and may produce a plausible answer from context that never supported one.Jev + RAG keeps the retrieval stage as is but changes what happens before generation.Instead of using a conventional reranker at this stage, Jev can judge every retrieved candidate against a typed question such as:"Does this passage help answer the query?"The query becomes the shared state, while the retrieved passages become individual candidates. Jev evaluates them together and returns a probability for each one.Application code then applies a threshold.↳ Candidates above the threshold continue to the LLM.↳ Candidates below it are removed from the context window.The same Jev request can also judge whether the remaining evidence is sufficient to answer the query.If answerability falls below the threshold, the application can skip the LLM and return "not in the documents."Jev does not replace the embedding model, vector database, or generation model. Retrieval still sets the cap because Jev cannot recover a passage that never entered the candidate set.The diagram below depicts the complete flow.- RAG retrieves a broad candidate set.- Jev reranks and gates those candidates.- The LLM generates an answer.👉 Over to you: Have you used Jev with RAG before?____Share this with your network if you found this insightful ♻️Find me → Avi Chawla. Every day, I share tutorials and insights on DS, ML, LLMs, and RAGs.

RAG vs. Jev + RAG, clearly explained:In a standard RAG setup, documents are split into chunks, converted into embeddings, and stored in a vector DB.When a query arrives, the system embeds it and retrieves the top-k chunks with the closest vectors.Many production setups add a reranker after retrieval.The reranker compares the query with each retrieved passage, improves their ordering, and keeps the highest-scoring results.Those passages are then placed in the context window, and the LLM generates an answer from them.The problem is that ranking and answerability are different questions.A passage can be more relevant than the other candidates while still containing weak, incomplete, or merely adjacent evidence.The LLM receives it anyway and may produce a plausible answer from context that never supported one.Jev + RAG keeps the retrieval stage as is but changes what happens before generation.Instead of using a conventional reranker at this stage, Jev can judge every retrieved candidate against a typed question such as:"Does this passage help answer the query?"The query becomes the shared state, while the retrieved passages become individual candidates. Jev evaluates them together and returns a probability for each one.Application code then applies a threshold.↳ Candidates above the threshold continue to the LLM.↳ Candidates below it are removed from the context window.The same Jev request can also judge whether the remaining evidence is sufficient to answer the query.If answerability falls below the threshold, the application can skip the LLM and return "not in the documents."Jev does not replace the embedding model, vector database, or generation model. Retrieval still sets the cap because Jev cannot recover a passage that never entered the candidate set.The diagram below depicts the complete flow.- RAG retrieves a broad candidate set.- Jev reranks and gates those candidates.- The LLM generates an answer.👉 Over to you: Have you used Jev with RAG before?____Share this with your network if you found this insightful ♻️Find me → Avi Chawla. Every day, I share tutorials and insights on DS, ML, LLMs, and RAGs.

Do not buy a new GPU this year.Seriously. Close the Nvidia tab and read this first.A free Python library just ran a 70B model on a 4GB graphics card.Then it ran Llama 3.1 405B on about 8GB.No quantization needed. No pruning.It's called AirLLM, and the trick behind it is smarter than it sounds.A large model is a stack of layers. Llama 70B has 80 of them. Your prompt goes into layer 1, comes out, goes into layer 2, and keeps climbing until the answer pops out the top.Normal setups load all 80 layers into your GPU at once. That's why a 70B model wants around 140GB of VRAM at full precision.AirLLM does something different.It loads one layer. Runs it. Flushes it. Loads the next one.Your GPU only ever holds a single layer at a time. So the VRAM you need depends on the size of one layer, not the size of the whole model.That one idea is why these numbers look fake:→ Llama 3.x 70B: ~4GB → Qwen3-235B: ~3GB → Llama 3.1 405B: ~8GB → DeepSeek-V3 (671B): ~12GBNow the part nobody tells you.It's slow. Every new word means streaming the model through your GPU layer by layer. Your disk becomes the bottleneck, not your graphics card. A fast NVMe SSD matters more than anything else here.It eats disk space. Before the first run, AirLLM splits the model into separate layer files. Set delete_original=True and it keeps only the split copy, which cuts disk use roughly in half.Want more speed? Pass compression='4bit' or '8bit'. The README claims up to 3x faster inference with almost no accuracy loss, because it only shrinks the weights being loaded from disk.On a Mac? Apple silicon only, and you need mlx installed.Getting started takes two steps:pip install airllmThen pass any Hugging Face model ID into AutoModel.from_pretrained(). Llama, Qwen, DeepSeek, Mistral, Gemma, and Phi all work through that same line.Where this actually shines:→ Testing a giant model before paying for cloud GPUs → Overnight batch jobs where speed doesn't matter → Running models fully offline on hardware you already own → Finally seeing how a 405B model thinks without a server rackWhere it doesn't: anyone who wants ChatGPT-speed replies.Your old gaming laptop just got a second career.35.4K stars. Apache 2.0. 100% Opensource.

Do not buy a new GPU this year.Seriously. Close the Nvidia tab and read this first.A free Python library just ran a 70B model on a 4GB graphics card.Then it ran Llama 3.1 405B on about 8GB.No quantization needed. No pruning.It's called AirLLM, and the trick behind it is smarter than it sounds.A large model is a stack of layers. Llama 70B has 80 of them. Your prompt goes into layer 1, comes out, goes into layer 2, and keeps climbing until the answer pops out the top.Normal setups load all 80 layers into your GPU at once. That's why a 70B model wants around 140GB of VRAM at full precision.AirLLM does something different.It loads one layer. Runs it. Flushes it. Loads the next one.Your GPU only ever holds a single layer at a time. So the VRAM you need depends on the size of one layer, not the size of the whole model.That one idea is why these numbers look fake:→ Llama 3.x 70B: ~4GB → Qwen3-235B: ~3GB → Llama 3.1 405B: ~8GB → DeepSeek-V3 (671B): ~12GBNow the part nobody tells you.It's slow. Every new word means streaming the model through your GPU layer by layer. Your disk becomes the bottleneck, not your graphics card. A fast NVMe SSD matters more than anything else here.It eats disk space. Before the first run, AirLLM splits the model into separate layer files. Set delete_original=True and it keeps only the split copy, which cuts disk use roughly in half.Want more speed? Pass compression='4bit' or '8bit'. The README claims up to 3x faster inference with almost no accuracy loss, because it only shrinks the weights being loaded from disk.On a Mac? Apple silicon only, and you need mlx installed.Getting started takes two steps:pip install airllmThen pass any Hugging Face model ID into AutoModel.from_pretrained(). Llama, Qwen, DeepSeek, Mistral, Gemma, and Phi all work through that same line.Where this actually shines:→ Testing a giant model before paying for cloud GPUs → Overnight batch jobs where speed doesn't matter → Running models fully offline on hardware you already own → Finally seeing how a 405B model thinks without a server rackWhere it doesn't: anyone who wants ChatGPT-speed replies.Your old gaming laptop just got a second career.35.4K stars. Apache 2.0. 100% Opensource.

𝗛𝗼𝘄 𝗱𝗼𝗲𝘀 𝗛𝘆𝗯𝗿𝗶𝗱 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝘄𝗼𝗿𝗸 𝗶𝗻 𝗥𝗔𝗚? BM25 says 12. Vector search says 0.86. So… which document is actually more relevant? In a typical RAG system, we often use one retrieval method to find relevant documents. Most commonly, that's Vector Search — which focuses on the semantic meaning of the query. 𝗕𝘂𝘁 𝘄𝗵𝗮𝘁 𝗶𝗳 𝘁𝗵𝗲 𝗲𝘅𝗮𝗰𝘁 𝗸𝗲𝘆𝘄𝗼𝗿𝗱 𝗺𝗮𝘁𝘁𝗲𝗿𝘀 𝘁𝗼𝗼? That's where Hybrid Retrieval comes in. 𝗛𝘆𝗯𝗿𝗶𝗱 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 combines keyword search (matching exact words) with semantic search (matching meaning), then merges the results to find the most relevant information. Instead of relying on just one retrieval method, it combines: 🔹 𝗩𝗲𝗰𝘁𝗼𝗿 𝗦𝗲𝗮𝗿𝗰𝗵 → Understands semantic meaning 🔹 𝗕𝗠𝟮𝟱 → Matches exact keywords So instead of choosing between meaning and keywords, 𝗛𝘆𝗯𝗿𝗶𝗱 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝘁𝗿𝗶𝗲𝘀 𝘁𝗼 𝘂𝘀𝗲 𝗯𝗼𝘁𝗵. But now we have a problem... BM25 gives one score 12, while Vector Search gives another score 0.86. So how do we combine them? That's where 𝗥𝗲𝗰𝗶𝗽𝗿𝗼𝗰𝗮𝗹 𝗥𝗮𝗻𝗸 𝗙𝘂𝘀𝗶𝗼𝗻 (𝗥𝗥𝗙) comes in. Reciprocal Rank Fusion (RRF) combines the ranked results from different retrievers 𝘄𝗶𝘁𝗵𝗼𝘂𝘁 𝗰𝗼𝗺𝗽𝗮𝗿𝗶𝗻𝗴 𝘁𝗵𝗲𝗶𝗿 𝗿𝗮𝘄 𝘀𝗰𝗼𝗿𝗲𝘀. Example: Keyword search: Doc A is #1, Doc B is #2, Doc C is #3 Vector search: Doc C is #1, Doc A is #2, Doc D is #3 Formula: Score = 1 / (60 + rank), added up across every search Doc A → 1/61 + 1/62 = 0.0325 Doc C → 1/63 + 1/61 = 0.0323 𝗔 𝘄𝗶𝗻𝘀, 𝗯𝗲𝗰𝗮𝘂𝘀𝗲 𝗶𝘁 𝗿𝗮𝗻𝗸𝗲𝗱 𝘄𝗲𝗹𝗹 𝗶𝗻 𝗯𝗼𝘁𝗵 𝗹𝗶𝘀𝘁𝘀. The idea: 𝗮 𝗱𝗼𝗰𝘂𝗺𝗲𝗻𝘁 𝘁𝗵𝗮𝘁 𝗸𝗲𝗲𝗽𝘀 𝘀𝗵𝗼𝘄𝗶𝗻𝗴 𝘂𝗽 𝗻𝗲𝗮𝗿 𝘁𝗵𝗲 𝘁𝗼𝗽 𝗶𝘀 𝗽𝗿𝗼𝗯𝗮𝗯𝗹𝘆 𝘁𝗵𝗲 𝗼𝗻𝗲 𝘆𝗼𝘂 𝘄𝗮𝗻𝘁. That's why RRF is widely used in hybrid search and RAG systems 𝗜𝗻 𝗽𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻 𝗥𝗔𝗚, 𝗯𝗲𝘁𝘁𝗲𝗿 𝗿𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝗼𝗳𝘁𝗲𝗻 𝗺𝗲𝗮𝗻𝘀 𝗯𝗲𝘁𝘁𝗲𝗿 𝗮𝗻𝘀𝘄𝗲𝗿𝘀 What retrieval strategy are you using in your RAG system — Vector, BM25, or Hybrid?

If you’re learning inference infrastructure, here’s a practical roadmap you can follow (from beginner to advanced) For Cloud, DevOps, and Platform engineers, this is a natural extension of what we already do: Make systems fast, reliable, observable, and cost-efficient. The difference? Model behavior and GPU memory now shape our infrastructure decisions. BEGINNER Stage 1: Build Your GPU & Systems Foundation → Learn CPU vs. GPU architecture and why inference workloads favor GPUs → Familiarize yourself with VRAM, memory bandwidth, CUDA cores, and Tensor Cores → Get comfortable with Linux and the nvidia-smi CLI → Learn the basics of networking(inter-node + intra-node) and storage in GPU systems Stage 2: Dive Into LLM Inference → Familiarize yourself with tokens, model weights and the inference lifecycle → Break down prefill vs. decode and their different compute patterns → See how KV cache grows with context length and impacts memory INTERMEDIATE Stage 3: Analyze Inference Performance → Identify sources of latency, throughput, and memory bottlenecks → Familiarize yourself with TTFT, TPOT, and inter-token latency → Distinguish between compute-bound and memory-bound workloads → Learn GPU profiling with nvidia-smi and Nsight Stage 4: Get Hands-On With Model Serving → Deploy models with vLLM, SGLang, or TRT-LLM (locally is fine) → Dive into batching, PagedAttention, scheduling, and request queues → Benchmark how serving strategies impact latency and throughput ADVANCED Stage 5: Dive Into GPU Networking & Parallelism → Learn when a single GPU isn’t enough for a model (multi-node inference) → Understand tensor/pipeline/data/expert parallelism techiniques → Analyze how communication overhead impacts inference Stage 7: Architect Distributed Inference → Learn how data moves across GPUs and nodes → Explore disaggregated prefill and decode → Design around network and communication bottlenecks Stage 8: Production-Ready Inference Systems → Focus on autoscaling, queueing, cold starts, and cost per token affect architecture → Learn how TTFT and ITL translate into inference SLOs → Design observability for latency, throughput, GPU utilization, and errors → Explore AI gateways for routing, rate limits, fallbacks, and traffic management The best way to learn this? Build. Benchmark. Change one variable at a time. Then document the bottleneck, the tradeoff, and the result. That write-up is portfolio proof. More importantly, it’s how you go from understanding GPUs to designing dependable inference systems.

A coding agent becomes a different kind of system the moment you give it a shell.It can run tests, install packages, edit files, call GitHub, pull dependencies and use external APIs. All of that is useful. It also means the agent is now operating with real authority on your machine.That is the problem NVIDIA’s OpenShell is trying to address.OpenShell runs the agent inside a sandbox and keeps the permissions outside it. The agent can ask to do something, but the runtime still decides whether that action is allowed.One example from the repo makes this very concrete. You can let an agent read from GitHub while blocking the operations that would change anything. It can inspect a repository without automatically gaining permission to create issues or modify resources.The same idea applies to files, network access and credentials.A process can be limited to specific paths. Network access can be restricted to particular destinations. API credentials do not have to sit inside the agent environment at all. They can remain on the trusted side and only be attached to requests that match the policy.OpenShell also has a policy prover that checks whether a proposed network rule would expand access beyond the boundary you defined.This is the part of agent infrastructure I find more interesting than another framework for planning or tool calling.Once agents can operate computers for long periods of time, the important question is not just whether they can complete the task.It is how much authority they need while doing it.GitHub Repo: https://lnkd.in/dTrtdN8G

I thought RAG was just an LLM + a vector database.I was wrong.The first time you build a real RAG system, you discover something nobody tells you in the first tutorial:The LLM is often NOT the hardest part.Bad answers usually start much earlier.Bad data.Bad chunks.Bad embeddings.Bad retrieval.Bad context.And by the time the prompt reaches the LLM,the damage is already done.That’s why a real RAG pipeline looks more like:Ingest → Chunk → Embed → Store → Retrieve → Generate → EvaluateThe interesting part?You can have the latest LLM,the best vector database,and a beautiful UI……and still build a terrible RAG system.Because RAG quality is not just about the model.It’s about the entire pipeline.That’s the part I’m focusing on while learning GenAI.If you're building RAG, don't just learn how to make it answer.Learn how to make it answer correctly.Save this architecture for your next project.🔁 Repost if you're learning AI Engineering.Follow for more AI and Tech Content Connect with[Ayushi Shukla](https://in.linkedin.com/in/ayushi-shukla-8503833a7?trk=public_post-text) [#RAG](https://www.linkedin.com/signup/cold-join?session_redirect=https%3A%2F%2Fwww.linkedin.com%2Ffeed%2Fhashtag%2Frag&trk=public_post-text) [#GenerativeAI](https://www.linkedin.com/signup/cold-join?session_redirect=https%3A%2F%2Fwww.linkedin.com%2Ffeed%2Fhashtag%2Fgenerativeai&trk=public_post-text) [#AIEngineering](https://www.linkedin.com/signup/cold-join?session_redirect=https%3A%2F%2Fwww.linkedin.com%2Ffeed%2Fhashtag%2Faiengineering&trk=public_post-text) [#LLM](https://www.linkedin.com/signup/cold-join?session_redirect=https%3A%2F%2Fwww.linkedin.com%2Ffeed%2Fhashtag%2Fllm&trk=public_post-text) [#GenAI](https://www.linkedin.com/signup/cold-join?session_redirect=https%3A%2F%2Fwww.linkedin.com%2Ffeed%2Fhashtag%2Fgenai&trk=public_post-text)

HelixDB is an open-source (Apache 2.0, core repo), YC-backed graph-vector database built from scratch in Rust, designed specifically around AI memory, knowledge graphs, and RAG use cases. Its core idea: rather than stitching together a vector database for semantic retrieval, a graph database for relationships, and a relational/metadata store — glued together by an application-layer orchestration — HelixDB combines all of it into a single engine. It supports vector search (approximate nearest-neighbor, prefilterable by graph traversal), graph traversal for relationships, BM25 full-text/keyword search, and KV/document/relational data, all queryable through one strongly typed query language (HelixQL). SDKs exist for Python, TypeScript, Rust, and Go, and it has built-in MCP support so agents can traverse the graph directly rather than generating ad-hoc queries. This combination is particularly relevant for agentic RAG and AI memory use cases — letting a single retrieval call combine semantic similarity, graph relationships, and metadata filters, which is useful for GraphRAG, multi-hop retrieval, agent memory, and enterprise knowledge systems. Two caveats are worth taking seriously: first, this is a young, fast-moving project (not a drop-in replacement for mature, battle-tested databases without evaluating it against your specific workload) backed by Y Combinator. Second, its storage model differs by deployment: the open-source local path uses in-memory/on-disk storage (LMDB), while the commercial "HelixDB Cloud" offering is built on S3-compatible object storage with auto-scaling and high availability — these are architecturally distinct tiers, not the same system. And on licensing, the main repo is Apache-2.0, but related repos in the same organization carry different, more restrictive licenses (e.g., AGPL-3.0) — so checking the specific component you'd be adopting matters before commercial use.

I posted 15 AI engineering concepts on X this week and asked people how many they could explain clearly. Over 1,900 people bookmarked it. The replies were honest. "I was at 11, gotta go and study." "13 for me." But one reply was better than my post: "Anyone can explain LLM-as-a-judge. Knowing it has position bias and when it lies to you consistently is the actual interview signal." That's exactly right. At Senior+ level, interviewers don't care that you know the word. They care that you know how it breaks. So here's the list again, with the failure mode I'd expect a strong candidate to bring up on their own: [1] KV cache → memory grows with every token of context. Long chats get expensive and slow. [2] Prompt caching → change one character at the start of the prompt and the cache misses. [3] Semantic caching → "cancel my order" and "cancel my subscription" look alike to an embedding. [4] Speculative decoding → faster only when the small model guesses right. Wrong guesses cost you. [5] Context window vs effective context → a 1M-token window doesn't mean the model uses all of it well. [6] Hybrid search → pure vector search misses exact matches like error codes and order IDs. [7] Reranking → better answers, but it adds latency to every request. [8] Structured outputs → valid JSON can still contain wrong values. [9] Tool calling → the model can call the right tool with the wrong arguments. [10] LLM-as-a-judge → it tends to prefer the first answer and the longer one. [11] Eval sets → they pass if the same person wrote the prompt and the tests. [12] Guardrails → too strict and the product becomes useless. [13] Prompt injection → any text the model reads, including a PDF or an email, can carry instructions. [14] Token budgets → one heavy user can eat everyone's quota. [15] Model routing → the cheap model handles the easy 80%, which is also where nobody notices mistakes. Notice something? Almost every one of these is an old backend problem. Caching, validation, rate limits, testing, trust. That's why I keep telling backend engineers they're not behind on AI. You already know the hard half. You just have to learn the new names. Save this for your next interview. -- P.S. I've written up the backend half (caching, idempotency, rate limiting, failure modes) in my System Design Fundamentals vault. Check it out: puneetpatwari.in

RAG Caching: How to Make AI Applications Faster, Cheaper & More ScalableIn production RAG systems, caching is not just about storing the final response. Different caches optimize different stages of the AI pipeline.Consider a simple example: “Can I work remotely from Germany?”Suppose earlier somebody asked: “Is working remotely from Germany allowed?”The wording is different:Question A:Can I work remotely from Germany?Question B:Is working remotely from Germany allowed?But their meaning is very similar.The semantic cache might find:Similarity = 0.96If the configured threshold is:>= 0.92then:CACHE HITThe system returns the previously generated answer.What did we avoid?We avoided:Embedding ↓Vector search ↓Reranking ↓Prompt construction ↓ LLMSo semantic caching can be a very powerful optimization.A production RAG flow can use multiple caching layers:🔹 1. Semantic Cache — Reuse the AnswerChecks whether a semantically similar question has already been answered.Cache HIT → return the answer without calling the LLM.🔹 2. Embedding Cache — Reuse the VectorIf the same query/document text has already been embedded, reuse the vector instead of calling the embedding model again.🔹 3. Retrieval Cache — Reuse Search ResultsCache frequently repeated vector/hybrid search results to reduce repeated retrieval work.🔹 4. Prompt Cache — Reuse Prompt ProcessingFor large, repeated system prompts, policies, instructions, or context, reuse previously processed prompt prefixes where supported.🔹 5. KV Cache — Speed Up GenerationInside the LLM, previously computed Key/Value states are reused while generating the next tokens, improving token-generation latency.The simple mental modelUser Query ↓Exact / Semantic Cache ── HIT → Answer ↓ MISSEmbedding Cache ── HIT → Reuse Vector ↓ MISSEmbedding Model ↓Vector DB / Retrieval Cache ↓Relevant Context ↓Prompt Cache ↓LLM + KV Cache ↓Answer ↓Store for Future ReuseThe key architectural principle:Don't compute the same expensive thing twice when it is safe to reuse the result.A well-designed AI caching layer can significantly improve:⚡ Latency💰 LLM & infrastructure cost📈 Throughput🔄 Scalability😊 User experienceBut production caching also requires careful handling of TTL, cache invalidation, knowledge/version changes, tenant isolation, authorization, semantic similarity thresholds, and cache stampede protection.AI caching is not one cache — it's a hierarchy of reusable computations across the RAG and LLM pipeline.

My mom keeps asking me every morning: “What is gold price today?” 😄 So instead of manually checking every day, I built a Voice AI agent for her.Every morning, it:📈 Checks the gold price and compares it with a 15-day moving average (so tiny fluctuations don’t trigger anything + you can change days as per your requirement) 🤖 Uses Gemini 3.8 via the Vercel AI SDK to analyze the movement and decide if the drop is actually worth alerting her about 🗣️ Generates a warm Hindi / Telugu audio note using Sarvam's Bulbul V3 📲 Converts it to Opus/OGG → uploads to Cloudinary → sends it to her on WhatsApp via TwilioAnd the important part: she doesn’t get a message every day. The voice note only goes out when there’s a meaningful drop in gold price.No drop = No WhatsApp spam ✅ Kept the whole thing lightweight too: no local file storage, everything runs in memory.

BREAKING: OpenAI just introduced Dots.- 24/7 always-on AI agents that have their own computer, browser, apps, context, and memory.You can give your Dot access to the tools you use, teach it how you work, and let it keep working even when you’re away. It can research, monitor things, prepare work, use your apps, and come back when it needs a decision or approval.OpenAI also introduced ChatGPT Space, where you, other people, and your Dots can work together with shared projects, files, and context.A Dot works much more like having someone working for you. You give it a computer, access to the tools it needs, context about what you’re doing, and tasks to take care of. Then it actually goes and does the work.And with Space, you basically have an office where humans and AI workers can work together.The shift from AI that helps you do the work to AI that can actually take on the work itself is going to change a lot.Pretty cool! Use it wisely!

Your coding agent burns money just finding files.jevgrep fixes the search step. Here is how it works in 3 parts:1. Ask what the code does, not where it lives↳ jg "Where is authentication checked before a request reaches a handler?" .↳ You get one stdout response: a summary, file locations, reading leads, and verbatim source with line numbers2. A model judges relevance level by level↳ Jev reads folders first, then files, then declarations↳ It follows only the branches that qualify↳ Your agent takes that evidence and does the edit, the test, and the fix3. Set it up in two steps (most people miss the second one)↳ npm install -g @dzhng/jevgrep, then jg auth↳ jg skill installs the agent skill for Claude Code, Codex, OpenCode and others↳ Without the skill, your agent does not know when to call jgThe numbers:• 10 Python SWE-bench tasks• Same result: 8/10 solved with jg and without it• Agent cost fell about 30%, from $7.62 to $5.44• With the Jev model cost included, total cost was still 25.8% lowerIt is a small sample and one run, so treat it as a strong signal, not a guarantee.Use it when:↳ You know the behavior but not the file↳ The question crosses many unfamiliar filesSkip it when:↳ You already know the symbol or path. Plain rg is enough.One warning: jg sends the source it searches to your model provider. Choose your search root with care.Save this for the next time your agent reads 30 files to change 3 lines.Repo: https://lnkd.in/dpkKwxCxI'm Shrey Shah & I build AI agents with AI coding agents.

A small professional update!! I’ve recently wrapped up my full-time role at iii. I’m incredibly proud of what we built together from working deep in the engineering stack to thinking about product, developer experience, and how developers actually build with AI infrastructure. Although I’m transitioning from my full-time role, I’ll continue rooting for the team and contributing where I can. I genuinely value the people, the work, and everything I learned while building iii. Now I’m thinking about what comes next. Over the years, I’ve realized that I enjoy working at the intersection of "Engineering, Product, and Developer Relations" building things deeply enough to understand the hard technical problems, then turning that understanding into better products, developer experiences, and communities. So, if you’re building the "next ambitious thing in AI, developer tools, infrastructure, or open source" and think I could help, I’d love to talk. Engineering. Product. DevRel. Open Source. P.S: I have one of the best open source profile on this planet. Hit me up. 🚀 Any Support, Repost, and Comments are appreciated ✨

I stopped reading about RAG and built one from scratch: FAISS + Sentence Transformers + a Cross-Encoder re-ranker + Groq + FastAPI. Every layer was mine to design, break and fix. Three things surprised me: 1️⃣ Chunking strategy matters as much as the embedding model. 2️⃣ Re-ranking did more for answer quality than anything else. 3️⃣ Your prompt is part of the system architecture, not an afterthought. Full write-up, including what broke first: https://lnkd.in/dnpVQr49 More coming. I've got follow-ups on what happened when I actually used it and measured it. #RAG #LLM #AIEngineering #Python

“Jev replaces LLMs.” “Jev optimizes LLMs.” Neither of them is true — at least not in the way these statements are usually understood. After looking into Jev, I think the better mental model is: Jev and LLMs solve different problems. An LLM is primarily a generative model. Given context, it can generate text, code, explanations, plans, and even decisions through generated output. Jev is a decision model. It takes application state/context and a focused, bounded question, then produces a structured decision such as a choice, score, or yes/no probability. For example: Support ticket ↓ Jev "Which team should handle this?" ↓ Billing → 0.91 Technical → 0.06 Sales → 0.03 An LLM could answer the same question: Support ticket ↓ LLM "This appears to be a billing issue because..." The LLM generates language containing the decision. Jev's output is the decision itself, in a form software can directly consume. So, if an application only needs: -classification -routing -scoring -model selection -tool-call gating -escalation decisions Jev can potentially handle that decision without using a generative LLM for that particular step. But if the application needs: -explanations -summaries -emails -code -conversations -open-ended reasoning a generative model is still needed. And they can work together: User request ↓ Jev "Does this need deep reasoning?" ↓ Yes / No ↓ Choose appropriate LLM ↓ Generate response Here, Jev isn't optimizing the LLM itself. It can help optimize the AI system around the LLM through routing, gating, evaluation, or other bounded decisions. So my takeaway is: LLM → Generate Jev → Decide Jev can replace an LLM for specific decision-oriented tasks, while Jev + LLM can complement each other when a system needs both structured decisions and generative output. The better question isn't “Which one replaces the other?” It's: “Does this step in my system need generation or decision?” If I’ve misunderstood any part of Jev, please correct me — and if you've worked with it, I'd genuinely love to hear your learnings from real-world use. #AI #LLM #Jev #AIAgents #MachineLearning #AIEngineering #SoftwareEngineering #GenerativeAI