Small Models for Fast Retrieval: What GLiNER Taught Us About Hybrid Search
Hybrid retrieval is a technique, not a product specification. You still need to agree which sources are authoritative, which business rules apply, and what users and stakeholders expect to find.
People may also value behaviours from the existing search. As with software compatibility, changing those can make sense once you understand what they rely on and why.
These decisions shape the answer, but also have to keep search interactive. Calling a large generative model for every classification or routing step can quickly consume the latency and cost budget.
GLiNER, its open-weight derivatives and other small language models have been useful building blocks here. In weekend experiments on my gaming laptop, I explored generic adaptations: extending language support and knowledge, extracting or classifying dates and locations, and detecting personally identifiable information (PII) so it can be masked or removed.
A usable dataset and fine-tuning produced useful gains over the base models within hours on the tasks tested. Performance and memory tuning, including dynamic quantization, then made them practical to run on standard CPU servers with low latency.
Since March, we’ve used some of these models and derivatives at Nearform to build software that is now in production, with encouraging customer feedback.
Laya, GLiClass, GLiFormer and open-weight alternatives for Jev-style decisions give us even more options to explore. Benchmarks such as typed-decisions and JevBench can help choose what to test, provided task-tuned results are kept separate from results on unfamiliar workflows.
For the service itself, the useful checks are decision quality, how often a model is confidently wrong, and latency and cost under real load.
Models, frameworks and evaluation links in the comments.
Chatbots grounded in retrieved content need the right source, quickly. Hybrid retrieval is a technique, not a product specification, so you still need to agree on which sources are authoritative, which business rules apply, and what users expect to find.
Existing search behaviours matter too. Users may rely on them, so changing them makes sense only once you understand what depends on them and why.
Calling a large generative model for every classification or routing step can consume your latency and cost budget quickly. Small models help keep search interactive.
GLiNER and its open-weight derivatives are useful building blocks. In weekend experiments on a consumer laptop, I explored extending language support, extracting and classifying dates and locations, and detecting personally identifiable information (PII) for masking or removal. Fine-tuning on a usable dataset improved results over the base models, and dynamic quantization made them practical on standard CPU servers.