Introducing First Reader, a free Agent skill that reads your draft the way an actual person would.100% Open Source. Runs fully offline with zero dependencies.There are 100+ anti-AI-slop skills out there. Every one of them checks your writing for the usual tells and flags the words you're not supposed to use. And none of them can tell you whether a real person would keep reading.First Reader simulates two people going through your draft one passage at a time, and reports how they felt. What caught them, what didn't make sense, what they skipped, and the one thing they'd still remember the next day.A third reader just skims it and tells you whether they'd have opened it at all.You point your agent at a draft and say who it's for: "review this, the audience is backend engineers who've seen ten launches this month"A few minutes later you get the read back in plain language, plus a page that lays your draft out with each reader's comments sitting next to the passage they reacted to. You see where each one leaned in or bailed, what stuck with them afterward, and a strip along the side that tracks how their attention climbed and dropped through the piece.What makes it different from those 100+ skills:• Readers get your draft one passage at a time, at human reading speed, so someone who bails at passage four genuinely never saw passage five• When they tell you what they remember, it's from their own notes, not a second pass over your text• They only report what happened to them and what would've kept them readingAfter the run you keep asking. "Ask S what would have kept her past passage 4" and she answers from her own notes, in her own words. Revise the draft, say "again," and the same readers read it fresh, with the old attention strip sitting above the new one so you can see what moved.The skill tries to tell you what your writing actually does to the person on the other end.Link to the GitHub repo in the comments.

Spotify published how they cut Claude Code token usage by ~90% on bulk reads. The interesting part isn't the number. It's that they treated it as a routing problem, not a prompting problem.The architecture is a model cascade, which anyone who has served ML in production will recognize on sight. Most of what a coding agent does is not reasoning, it's moving text. Reading a 2,000 line file into a frontier model's context bills at the top rate, then re-bills on every subsequent turn. So they set a threshold at 350 lines. Above it, the read gets intercepted, handed to a cheap worker at temperature 0.2, and only structured bullets return. The file never enters the expensive context, which makes follow-up questions free.From what I have seen, every agentic workload has a high-volume, low-judgment tier that should never touch your most expensive model. The work is drawing that boundary honestly, then enforcing it somewhere the model can't talk its way past.#GenAI #MLOps #LLMOps #InferenceCost

Found a lot of similarities in our recent meet. Village born. Born poor. Built career patiently. We first met in 2010 when I was venturing into Nueclear a PETCT “single test” company. I was flushed with cash from pathology and wanted to be a leader in PETCT, an over priced technology. My ex employer BARC (RMC) was a leader then. He was representing GE-Wipro. I asked 5 cyclotrons and 25 PETCT. What was 200 cr capex for 100 cr. He explained in pathology there are many players it works. Not in PETCT where only 1.5 player in market. We have monopoly. Also he said it may not work that way. Radiology is cut driven. But I persuaded. We had a lot of meetings. Finally he said No. I said copy your mail to Mr. Premji. He did. I got to personally meet “The man” one to one. He asked, after 30 mins of hearing me, patiently a simple Question. Blunt. “Will you give cuts?. I said no. I did not give in pathology. He said take one cyclotron and 5 PETCT at price you asked. Prove what you dream works. I will support you. I started it with a dream to do a million PETCT in 10 yrs. Cutting long story short “It finally did not work as I dreamt. I did PETCT for 6000 Rs when the market did for 25,000 Rs. Still I only burnt money. Lessons learnt. 1. In Pathology : Specimen flies. in Radiology: It is not possible. 2. Though both are diagnosis no one is successful in doing both. 3. If you are good at one, do not dream you can not do everything through both are in medical diagnosis. #Focus. Dinesh Lodha is a rare versatile employee who should have been a strong entrepreneur. He is still young. I think he is the most qualified, most versatile medical entrepreneur who can improve any loss making balance sheet to 20% EBiDTA. Happy to have met him after 15 long years. He said he missed dinner of #SumathiVelumani.

Y Combinator (YC) open-sourced the agent harness running YC. Agent frameworks are almost always built for one person. You can stretch a personal assistant to cover a whole company, but the wiring gets complicated fast. Y Combinator hit that wall directly. They started with a small agent loop in Ruby that could reach internal data, added crons and webhook triggers, then provisioned over 50 Hermes agents as personal assistants. Managing a fleet that size became its own job. QM is what they built instead, open-sourced it under MIT. The name is short for quartermaster, the person on a ship who keeps things in order belowdecks. It runs inside YC across accounting, legal, events, and engineering, including the work of building QM itself. The idea that makes it different is the scope. → A scope is the unit of isolation, and it applies the same way to a person and to a Slack channel. Each one gets its own memory, files, credential view, permissions, scheduled jobs, and sandbox. → That sandbox is durable, so tools installed during one session are still there in the next. Many execution environments get thrown away after every run, which means the agent starts from zero every time. → The core is deliberately small. It handles identity, policy, the scheduler, and the agent loop, with Postgres holding sessions, memory, and the queue. The web UI, admin panel, and Slack integration are all plugins over its HTTP API. → The harness layer is swappable. Claude Code, Codex, OpenCode, and Pi all drive the same core, so a deployment is not locked to one vendor. → Security comes in three levels. Strict pauses every tool call for human approval, Dangerous pauses nothing, and the default Auto runs a classifier over provenance-labelled external data and tool results before they reach the model. Hard denials for things like recursive deletes apply at every level. That last one is worth sitting with. Screening tool results by where they came from is a concrete answer to prompt injection, and very few open agent systems ship a named default for it.

GPT-6 Astra was all over my feed.“You don’t call enough” was all over my family conversations.So I gave Astra two very important stakeholders: Mom & Dad. I gave it a simple brief: Build “A Very Necessary Apology for Mom & Dad.”A tiny website with:→ a Family Performance Review→ carefully selected metrics→ an apology button→ and the ultimate escalation path: “Nice website. Now call us.”And then came the part I actually wanted to test:How quickly can an idea go from my head to something real?A couple of seconds later, I had a working website - with layouts, animations, interactions and button responses without writing the code myself.That’s what impressed me.Not whether this is the world’s greatest website. But how quickly you can go from “I have an idea” → “here, try it.”Would my parents prefer a normal apology?Probably.Would a normal apology have hover effects?Absolutely not.The apology has been deployed.The phone call is still on me. Go judge my apology skills here: https://lnkd.in/gnFVWpRbPowered by GPT-6 Astra. Funded by parental guilt.

Redis built a cache that cuts LLM costs by 90%!(and it also leads to 15x faster generation)Production LLM apps do not receive completely new questions every time.A customer-support assistant might receive all three of these:- "Can I get a refund after buying the monthly plan?"- "Is the monthly subscription refundable?"- "Can I cancel the plan and get my money back?"The wording is different, but the underlying question and its answer remain the same.Yet LLM apps process every version as a new request. They assemble the prompt, send it to the model, and generate an answer that may have already been generated.Prefix caching reduces part of these repeated calls.When requests begin with the same system prompt or context, the model can reuse the KV states already computed for that shared prefix.But the request still hits the LLM. The new tokens must be processed, and the complete answer must still be decoded.So even with a prefix-cache hit, there's another generation call involved.To solve this, instead of only caching computation inside the model, the application can cache the generated response outside it.When another question arrives, the system embeds it and compares it with previously answered questions. If it finds a sufficiently close match, it returns the stored response without invoking the LLM again.A cache hit removes the input tokens, output tokens, and decoding time associated with another LLM call.In practice, it is important to decide which questions can safely share an answer since a production setup needs well-tuned similarity thresholds, expiration policies, data isolation, and monitoring for incorrect matches.If you want to use this in practice, Redis already implements it as a managed service called Redis LangCache.Under the hood, it generates embeddings, searches previous responses, and returns a matching answer before another model call occurs.Redis also handles access scopes, custom filtering, TTL and eviction controls, and cache monitoring through Redis Cloud.I built an interface to compare it against direct LLM inference. The video below shows this in action, and I worked with Redis on this post to put this together.For the paraphrased question in my run, direct inference took 2.232 seconds and consumed 514 input tokens plus 250 output tokens.Redis returned the earlier response in 0.37 seconds with zero LLM input or output tokens. That was roughly 6x faster in this run.Redis reports API cost savings of up to 90% and cache-hit responses up to 15x faster. The actual result depends on how much safe repetition exists in the workload.You can try Redis LangCache here: https://fandf.co/4ikTNO8____Share this with your network if you found this insightful ♻️Find me → Avi Chawla. Every day, I share tutorials and insights on DS, ML, LLMs, and RAGs.

If you want to run powerful AI locally and cut your API bills to $0, save these 10 GitHub repos:1: // Ollama (⭐ 181k)https://lnkd.in/dbGG2xvRRun open-source LLMs locally with one command. The front door to local AI.2: // ComfyUI (⭐ 132k)https://lnkd.in/eKrkeWaANode-based canvas for building image-generation workflows.3: // llama.cpp (⭐ 128k)https://lnkd.in/dqPVimrQThe C/C++ engine that runs LLMs fast on CPUs and GPUs.4: // GPT4All (⭐ 77.4k)https://lnkd.in/eSQG3dANDesktop chat app for running LLMs locally, no GPU needed.5: // whisper.cpp (⭐ 53.5k)https://lnkd.in/dVdmUzexLocal speech-to-text using OpenAI's Whisper model.6: // LocalAI (⭐ 49k)https://lnkd.in/eSSSFkUJSelf-hosted OpenAI-compatible API for your local models.7: // text-generation-webui (⭐ 47.6k)https://lnkd.in/emiWi5BJWeb UI for loading and chatting with local text models.8: // exo (⭐ 47.3k)https://lnkd.in/dwsTMMNnLinks several devices into one cluster to run big models.9: // Jan (⭐ 44.4k)https://lnkd.in/eQAsEi2ROpen-source ChatGPT alternative that runs models right on your desktop.10: // mlx-lm (⭐ 6.9k)https://lnkd.in/eqZ6DpQZRun and fine-tune LLMs on Apple Silicon Macs.The barrier to running your own AI stack is no longer cost or compute now. It's just knowing which tools to reach for.What did I miss?--P.S. We’re building the best AI English tutor in the world. Try it to improve your speaking - it’s 15× cheaper than a human one → GetFluently.app

Code as Agent Harness!Code is no longer just something AI agents generate. It is increasingly becoming the execution layer through which they reason, act, manage state, use tools, and verify their own work.That distinction matters.A model can decide what it wants to do, but long-running agents still need a structured way to turn those decisions into actions. They need to call tools, execute code, preserve intermediate state, inspect results, recover from failures, and coordinate work across multiple steps.The paper frames code itself as that harness.Instead of treating planning, memory, tool use, and execution as separate pieces of infrastructure, code becomes the common interface connecting the model to all of them.The paper breaks this into three layers:• Harness interface: code connects the agent to reasoning, actions, and the environment it is operating in• Harness mechanisms: planning, memory, tool use, execution feedback, and optimization keep long-running tasks reliable• Multi-agent harness: shared code artifacts become a way for multiple agents to coordinate, review each other's work, and verify resultsThis applies well beyond coding agents.The paper covers GUI and OS automation, scientific discovery, embodied agents, DevOps, personalization, enterprise workflows, and other systems where agents need to interact with an environment rather than simply generate text.The interesting part is how this changes the role of code in AI systems.Code used to be primarily an output of the model. Now it can also represent the agent's plan, tools, state, interaction with the environment, and even the mechanism used to verify whether an action actually worked.That makes harness engineering a much bigger part of building reliable agents than simply wrapping an LLM with a few tool calls.I've shared the paper in the comments!

We're currently witnessing one of the most fascinating rivalries in tech: Tesla vs. Uber is starting to look a lot like iPhone vs. BlackBerry in 2007. And this isn’t about market share. At least, not yet. In 2007, BlackBerry owned enterprise email and 40%+ of the US smartphone market. The iPhone was expensive, limited, and barely a rounding error. BlackBerry kept growing after it launched. But Apple wasn’t building a better email phone. It was turning the phone into a computer. Cybercab is making a similar bet now. Uber connects riders with human drivers. Meanwhile, Tesla wants to remove the driver, and also own the vehicle, software, energy, and data loop beneath every ride. Early Austin prices show what that could mean: → $19.35 for a Cybercab vs. $34.98 for Uber Priority on comparable routes. → One rider reported $9.65 vs. $21 with tip for roughly 10 miles. → Cybercab quotes have undercut Tesla’s Model Y robotaxis by about 35% on identical routes. Tesla’s target: roughly $0.20 per mile in operating cost and a $0.30-$0.40 fare 🤯 But the model remains unproven. Yet. Its Texas fleet is tiny. Waits are long. Prices have risen with demand. Production has slipped. If Tesla cannot manufacture at scale and make unsupervised FSD reliable, Cybercab remains a novelty. But if it can, Uber may discover that its greatest asset - millions of drivers - is also the cost Tesla designed away. BlackBerry won the opening years, but the iPhone changed the game. All because Steve Jobs made BlackBerry’s greatest advantage irrelevant. Elon Musk is now betting he can do the same to Uber’s.

Strategic Research Engagement in Intelligent Systems The question is not just “Can AI predict?” — it is “Can the system act on it? The more I work on intelligent systems, the more I realise the real challenge is not the AI — it is the system around it. At Telkom University, Bandung, Indonesia, I had the opportunity to engage in a strategic research and technology interaction focused on Intelligent Systems and next-generation digital technologies. For me, this was not just about attending a session. It was an opportunity to present what I have been working on, exchange ideas with researchers, understand how intelligent systems are being approached internationally, and challenge my own technical thinking. One of the systems I presented was an AI intelligence layer for real-time water filtration monitoring. The idea goes beyond simply collecting sensor readings. A normal monitoring system tells us: “This is what is happening.” An intelligent system should move further: Why is it happening? Is this behaviour abnormal? What could happen next? Does intervention become necessary? Can the system make or recommend a decision? That is the direction I am increasingly interested in. Sensors → Data → Intelligence → Decision → Action The interaction allowed me to explore concepts across: 🧠 Artificial Intelligence & Intelligent Systems 📡 IoT and Real-Time Sensor Systems ⚙️ Cyber-Physical Systems 📊 Data-Driven Decision Making 🤖 AI-enabled Automation 🏙️ Smart Infrastructure 🌱 Sustainable Intelligent Systems What stood out to me most was the importance of looking at AI as one layer inside a complete engineered system, rather than treating the model itself as the product. A model can predict. But a real intelligent system has to sense, understand, evaluate, decide and respond. That difference is becoming central to the way I approach my projects now. I am increasingly interested in building systems where AI interacts directly with real-world data, physical infrastructure and decision workflows — whether the application is healthcare, environmental monitoring, sustainability or intelligent infrastructure. International research interactions like this are important to me for one reason: They help me test whether the systems and ideas I am working on can stand beyond a college project, beyond a presentation, and eventually in a real research and deployment environment. A valuable experience at Telkom University, Bandung, Indonesia, and another step toward the kind of engineering I want to pursue: Research → Engineer → Validate → Deploy. Not just building AI models. Building intelligent systems

Run any AI model locally with no GPU required and no cloud dependency!Most local AI setups require choosing a tool for each modality separately. One for text generation, another for image generation, another for voice, another for embeddings. Each with its own config, its own format, its own hardware requirements.LocalAI runs all of it through one OpenAI-compatible endpoint. LLMs, vision, voice, image generation, video, embeddings, and real-time audio. Every backend wraps a best-in-class engine: llama.cpp for text, whisper.cpp for transcription, stable-diffusion for images, MLX for Apple Silicon. Each backend is a separate image pulled only when a model needs it, so you install nothing you don't use.What it gives you:• 60+ backends including llama.cpp, vLLM, whisper.cpp, stable-diffusion, MLX, and more• Drop-in API compatibility with OpenAI, Anthropic, and ElevenLabs across every backend• Hardware support for NVIDIA, AMD, Intel, Apple Silicon, Vulkan, and CPU-only• Built-in agents with tool use, RAG, MCP, and skills via the Agenthub community hub• Multi-user support with API key auth, user quotas, and role-based access• WebRTC realtime audio for speech-to-speech pipelinesIn-UI fine-tuning and on-the-fly quantization• Distributed cluster mode with VRAM-aware smart routing and autoscalingGitHub repo in the comments_____♻️ Repost to share with your network➕Join 24,500+ AI/ML builders here: https://lnkd.in/gfkzKZYk

What interviewer is checking:Whether you understand that a high cache hit rate does not always mean high cost savings, and can trace where token usage and LLM costs are actually coming from.Interviewer:Your semantic cache has a 70% hit rate, yet the LLM bill barely decreases.Question: What would you investigate to determine why caching isn't translating into meaningful cost savings?Answer:1. Check whether cache hits actually skip the LLM call. 2. Measure tokens saved per cache hit, not just hit rate. 3. Check if cached queries are mostly small, low-cost requests. 4. Verify whether large requests are missing the cache. 5. Check cache similarity thresholds and false misses. 6. Check if prompt or context changes prevent cache reuse. 7. Measure cache latency, storage, and embedding costs. 8. Compare total LLM calls and token usage before and after caching.______________________________________________________________Preparing for an ML/AI interview or looking to transition into AI?Check out the resources below covering 1000+ interview Q&A, Roadmap, Projects, and practical AI topics to help you prepare with confidence.AI Interview Bundle: https://lnkd.in/dez6Ji7EAI Structured Course: https://aptai.in/learn-ai

A harnessed LLM agent. And why LLM is a very small part of the agent? Most people picture this as a model with tools bolted on. The real architecture inverts that relationship. The model itself is deliberately thin. Intelligence gets pushed outward, and the harness composes it at runtime. Three dimensions orbit the harness core: 𝗠𝗲𝗺𝗼𝗿𝘆 holds state the model shouldn't carry in weights or context. Working context, semantic knowledge, episodic experience, and personalized memory each have their own lifecycle. 𝗦𝗸𝗶𝗹𝗹𝘀 hold procedural knowledge. Operational procedures, decision heuristics, and normative constraints specialize the general model per task. 𝗣𝗿𝗼𝘁𝗼𝗰𝗼𝗹𝘀 hold the interaction contracts. Agent-to-user, agent-to-agent, and agent-to-tools are three distinct surfaces with their own failure modes. Between the core and these modules sit the mediators: sandboxing, observability, compression, evaluation, approval loops, and sub-agent orchestration. They govern how the harness reaches out and how state flows back in. The useful question this framing unlocks: for any new capability, where should it live? Stable knowledge goes to memory, learned playbooks go to skills, communication contracts go to protocols, loop governance goes to the mediators. Harness design becomes a question of what to externalize, and how to mediate it. I'm building a minimal agent harness from scratch. Didactic, easy to read, no magic. Open-sourcing it soon. Stay tuned.

Tokenization is the first boundary between human-readable text and the sequence an LLM actually computes over. A model does not receive words, sentences, or source code directly. Text passes through a tokenizer that may normalize Unicode, apply pre-tokenization rules, segment the text into learned pieces, map those pieces to integer IDs, and insert special tokens before the model ever sees an embedding. That preprocessing step has consequences across the system. It changes how much text fits into a fixed context window, how efficiently different languages and code are represented, how numbers and identifiers are split, which vocabulary entries receive useful training signal, and even whether metrics such as perplexity can be compared fairly across models. It also explains why several common simplifications are wrong. BPE is not inherently byte-level. SentencePiece is not a single tokenization algorithm. Token IDs are labels tied to learned model parameters, not semantic coordinates. And once a model has been trained around a tokenizer, replacing that tokenizer is not a harmless preprocessing swap. I put together a technical handbook that works through this from first principles => Unicode and UTF-8, normalization, pre-tokenization, BPE, byte-level BPE, Unigram, WordPiece, vocabulary construction, special tokens, chat templates, multilingual and code behavior, numerical tokenization, context length, under-trained tokens, and the failure modes that usually get skipped. The explanations are grounded in the original papers and current implementations. Sharing Understanding Tokenization here:

Voice cloning gets harder when language coverage becomes a requirement. OmniVoice is an open-source, massively multilingual zero-shot text-to-speech model for builders who need speech generation across a broad set of languages. It helps you prototype or integrate multilingual speech workflows by offering voice cloning from a short reference clip, voice design from speaker attributes, and a Python API plus command-line tools. Key features: • 600+ language support – the README lists support for more than 600 languages • Voice cloning – generate speech from reference audio and its transcript, with optional Whisper transcription when the transcript is omitted • Voice design – describe attributes such as age, pitch, accent, or whisper style without reference audio • Text-level controls – add supported non-verbal tags and override pronunciation with pinyin or English phonemes • Builder-friendly entry points – use the Python API, local Gradio demo, single-item CLI, batch CLI, Hugging Face Space, or Colab It’s open-source (Apache License 2.0 license). 🔗 GitHub: https://lnkd.in/dBmhgwrg ⸻ ♻️ Share this with your network if you found it useful or insightful. ✉️ If you’re into AI, ML, agents, and building real systems, join my newsletter (it’s free): dankornas.substack.com

What should an AI agent be allowed to do — and what computer should it get? Those are two different infrastructure problems. NVIDIA OpenShell and SmolVM approach agent sandboxing from different layers: 🔒 OpenShell focuses on policy. It controls what an agent can access: network destinations, files, credentials, processes, and model endpoints. 💻 SmolVM focuses on the computer. It gives the agent an isolated environment it can actually work inside: Linux microVMs, Windows 11, macOS desktops, browser sandboxes, and snapshots. This difference becomes concrete pretty quickly. For a coding agent with access to proprietary source code, you may want to restrict which APIs and domains it can reach. That is where OpenShell's policy model is useful. For a computer-use agent that needs to operate a Windows-only ERP or desktop application, policy alone is not enough. It needs a Windows computer. SmolVM treats that disposable computer as the sandbox boundary. So I don't think the interesting question is: OpenShell vs SmolVM — which one wins? The more useful way to look at the stack is: Agent ↓ Policy + credentials ↓ Computer runtime ↓ VM isolation OpenShell puts more emphasis on the upper layers. SmolVM puts more emphasis on the computer and VM layers. For production agents, I suspect we will need both. I wrote a deeper comparison across isolation, security policies, microVMs, persistence, Windows support, browser automation, and architecture. https://lnkd.in/evah6YMg 🔗

Connecting an agent to SaaS apps shouldn’t mean wiring OAuth provider by provider.OpenConnector is an open-source connector gateway for AI agents and apps that need access to user-authorized SaaS services.It helps you connect accounts once, then expose a shared catalog of provider Actions through SDK, CLI, MCP, HTTP, or OpenAPI interfaces.Key features:• Connector catalog – the README lists 1,000+ providers and 10,000+ prebuilt Actions• Multiple integration paths – use the Connector SDK, oo CLI, MCP, HTTP, or generated OpenAPI• Credential handling – supports API keys, OAuth 2, custom credentials, and no-auth providers• Inspectable Actions – review request/response schemas and required scopes before execution• Deployment options – run locally with Docker or Node.js, or use OOMOL’s hosted runtimeIt’s open-source (Apache License 2.0).🔗 GitHub: https://lnkd.in/d-j_PGGx⸻♻️ Share this with your network if you found it useful or insightful.✉️ If you’re into AI, ML, agents, and building real systems, join my newsletter (it’s free): dankornas.substack.com

This one-command tool that removes censorship layers from local language models. No jailbreak prompts. No prompt injection theater. It edits the model. Automatically. It's called Heretic. Once installed, point it at a supported open-weight model. It automatically searches for changes that reduce refusals while trying to preserve the model’s other abilities. - Tests different modifications. - Measures how often the model refuses. - Checks how much ordinary responses change. - Lets you save and chat with the modified model. The developer’s test on Google’s Gemma 3 12B is striking. The original refused 97 out of 100 harmful test prompts. The Heretic version refused 3. That’s one reported benchmark, not a guarantee across every model. Fewer refusals also don’t mean better or more accurate answers. The tool is free and open source. You need a working Python/PyTorch setup and hardware capable of processing your chosen model. Link: github.com/p-e-w/heretic A model saying “I can’t help with that” doesn’t necessarily mean it doesn’t know how.

2d🚀 Tamil Unmute — Voice AI that runs on CPU I’ve been working on Tamil Unmute, derived from unmute.sh, with a simple goal: Make Tamil voice AI dramatically cheaper and more accessible for India. 🇮🇳 The architecture is intentionally simple: 🎙️ STT → CPU 🔊 TTS → CPU 🧠 LLM → Gemini 2.5 Flash No GPU is required for the speech stack. The only AI inference that needs to be in the cloud is the LLM. And the economics are interesting: 💰 ₹0.26–₹0.34 per conversation minute ⏱️ ₹16–₹20 per hour of conversation 📞 ₹1,300–₹1,700 for 1,000 five-minute calls This can make voice AI practical for: 📚 Education & AI tutors for children ☎️ Customer support & voice call agents 🏫 Schools & educational institutions 🏥 Healthcare information & appointment assistants 🏢 Enterprise helpdesks 🛍️ Voice commerce & order assistance 🏛️ Government & public-service interfaces 🌾 Rural and low-connectivity services 📞 Automated outbound calls & reminders 🗣️ Tamil-first conversational assistants The bigger idea is simple: Regional-language AI shouldn't have to be expensive. By moving STT and TTS to CPU, we can significantly reduce infrastructure costs while still using a capable LLM for the intelligence layer. I’m also planning to take this further and build out the project as an open-source voice-agent stack. 🔗 GitHub: https://lnkd.in/g8QNGyUj Tamil Unmute is coming. Watch out. 🔥 #TamilUnmute #TamilAI #Tamil #TamilLanguage #TamilTech #TamilNadu #IndiaAI #AIforIndia #VoiceAI #VoiceAgent #ConversationalAI #GenerativeAI #SpeechAI #SpeechToText #STT #TextToSpeech #TTS #TamilSpeech #TamilVoice #IndianLanguages #IndicAI #IndicLanguages #RegionalAI #AI #ArtificialIntelligence #MachineLearning #DeepLearning #OpenSource #OpenSourceAI #OpenSourceProjects #GitHub #Developer #Developers #AIEngineering #AIInnovation #TechInnovation #VoiceTechnology #CustomerSupport #AIForEducation #EdTech #HealthTech #GovTech #RuralAI #DigitalIndia #MadeInIndia #BuildInPublic #Startup #TechStartup #FutureOfAI #Gemini #GeminiAI #CPUInference #EdgeAI #AffordableAI #AIforEveryone #BharatAI

SGLang isn't just another inference engine; it's a serving stack co-designed around how real LLM programs actually look — structured prompts, tool calls, and multi-turn agents. Its RadixAttention KV cache reuses prefixes across requests, which routinely delivers 1.5–5× throughput wins on agent workloads where vLLM and TGI leave the cache cold.Most "fast LLM serving" posts treat inference as a numbers problem — bigger batches, more FLOPs, lower p99. That's the vLLM story, and it's a good one. But if you've watched a production trace from a tool-using agent or a RAG pipeline, you know the real bottleneck isn't arithmetic. It's structure. The same system prompt shipped on every request. The same retrieved document chunk. The same JSON schema enforced across thousands of calls. Standard inference engines see these as unrelated requests with no shared state; SGLang sees them as the same computation, repeated.That's the through-line of this post. We'll dig into what SGLang actually is, why its frontend language and runtime belong together, how RadixAttention works under the hood, and what changes when you put it in front of real traffic next to (or instead of) vLLM.SGLang in Production: How Structured Generation Makes LLM Serving Actually FastRead the full guide: https://lnkd.in/daEuDSmY