How My AI Agent Builds Its Own Live Web Datasets

Woah...My Hermes Agent can now create LIVE web datasets on the fly 🤯

I ask for something. It fires up agents to research on the live web, verifies against real sources, and hands back a structured dataset that refreshes on a schedule.

It's using TinyFish BigSet under the hood. And it's 100% Open Source.

Most AI research still breaks at the same point: the data layer. Your agent can search, browse, and write a decent report. But when the data lives across GitHub repos, pricing pages, docs, job boards, and launch posts, you're still the one stitching it together manually.

BigSet does the boring but important work:

• Infers the schema from your sentence
• Sub-agents fan out in parallel each fetching & verifying real data
• Deduplicates and builds a structured table
• Exports CSV or XLSX, refreshes anywhere from every 30 minutes to weekly

So Hermes builds the dataset it needs before doing the analysis.

I ask my Hermes agent: "Find AI agent products and open-source projects launched or updated in the last 7 days, with category, URL, pricing, target user, and why it matters."

BigSet turns that into a real live dataset. And Hermes turns that into signal.

The research bottleneck was never the reasoning. It was getting hallucination-free data into the agent's hands. Now that layer is open-source.

BigSet is a real, recently-launched tool, and nearly every technical claim matches its actual documented architecture:

• Open source, built by TinyFish — confirmed. BigSet infers the schema automatically, sends autonomous agents to research it on the live web, verifies what they find against real sources, deduplicates, and hands you a structured dataset, downloadable as CSV or XLSX, built on TinyFish APIs. GitHub
• Schema inference from a plain sentence — confirmed. When you submit a description, an LLM infers the dataset schema — column names, data types, primary keys, and where to look for the data — before any web access happens. MarkTechPost
• Sub-agents fanning out in parallel, one per entity/row — confirmed. The orchestrator dispatches sub-agents in parallel, with each sub-agent handling exactly one entity — one row in the final table. MarkTechPost
• Deduplication + source verification — confirmed. The system applies primary key deduplication, and each row carries source attribution — a traceable link to the web page the data came from. MarkTechPost
• Refresh cadence, CSV/XLSX export — confirmed. You can set a refresh cadence — 30 minutes, 6 hours, 12 hours, daily, or weekly — and the agents re-run on schedule so the dataset never goes stale. GitHub