Unified Research Engine: One Query Across 40+ Intelligence Sources
Serious research means jumping between a dozen silos: Hacker News for engineering sentiment, GitHub for code, PubMed for papers, SEC EDGAR for filings, patent databases, government data, and more — each with its own search box, ranking, and result format. DeepLens collapses that into one unified research engine. Its Next.js front-end presents a single interface, and behind it a federated search backend (search-api) exposes 40+ source endpoints under one gateway. A DeepLens query names the backend source it wants and gets back real, freshly fetched, ranked results in a consistent shape. In the verified live run, a Hacker News query for 'rust' returned the genuinely top-ranked posts by score — 'Why Discord is switching from Go to Rust' at 1,582 points and 642 comments, followed by the real leaderboard of Rust discussions — complete with points, comment counts, authors, and both canonical and Hacker News URLs. The engine is honest about its edges too: semantic vector-recall sources are gated on Qdrant availability and degrade gracefully when the vector store is down, rather than hard-failing the whole query.
Key Metrics
Source coverage
Before
A dozen separate research tools, each searched by hand
After
40+ federated sources (hackernews, github, reddit, pubmed, patents, edgar, fda, nasa, vector recall, ...) behind one gateway
Result quality
Before
N/A — no unified engine
After
Real, live-fetched, ranked results — verified with genuine Hacker News rankings (points, comments, authors, canonical + HN URLs)
Interface
Before
One search box per source, each with its own syntax and format
After
A single DeepLens interface over every source, returning a consistent result shape
Failure handling
Before
N/A
After
Explicit validation (400 missing endpoint, 502 invalid path) and Qdrant-gated semantic recall that degrades gracefully
Watch the Walkthrough
The Challenge
Research does not live in one place. Understanding a technology means checking engineering sentiment on Hacker News, real projects on GitHub, academic evidence on PubMed, corporate filings on SEC EDGAR, patents, government datasets, and more. Each of those is a separate destination with its own query syntax, its own ranking, and its own result format — so a researcher spends as much time context-switching between tools and normalizing their output as they do actually reading. Building a custom integration for each source is worse: forty different APIs, forty rate limits, forty auth schemes, and forty result shapes to reconcile. What is missing is a single front door — one query interface that fans out to the right source and returns results in a consistent, ranked shape, while staying resilient when any individual source is unavailable.
Why This Approach?
Backend architecture
Federated gateway — 40+ source endpoints behind one search-api service
Each source (Hacker News, GitHub, PubMed, EDGAR, patents, FDA, NASA) has its own upstream API and result shape. Wrapping each as an endpoint behind one gateway lets the front-end treat them uniformly while keeping each integration isolated — a change to one source never destabilizes the others.
Alternatives considered: One monolithic scraper, per-source client apps, a single third-party meta-search SaaS
Front-end proxy pattern
DeepLens UI proxies an ?endpoint=<backend-path> parameter to search-api
The UI names which backend source it wants via the endpoint parameter, strips it, and forwards the rest to search-api. This keeps the backend URL server-side, gives one /api/search route for every source, and makes adding a source a config change rather than a new page. The route validates input — a missing endpoint returns 400, an invalid path 502 — so failures are explicit.
Alternatives considered: Direct browser-to-backend calls, hard-coded per-source pages, GraphQL federation layer
Vector-recall availability
Gate semantic/recall endpoints on Qdrant, degrade gracefully when it is down
Semantic recall depends on the Qdrant vector store. Rather than fail every query when Qdrant is unavailable, the backend disables only the semantic endpoints and keeps the other 40+ sources fully operational — partial capability beats total outage.
Alternatives considered: Hard-require Qdrant for all queries, silently return empty recall results
Build Process
Federated source backend
search-api exposes 40+ source endpoints under a single service — hackernews, reddit, github, duckduckgo, pubmed, patents, edgar, fda, nasa, semantic/recall, reputation, shopping, tmdb, and more. Each wraps a distinct upstream and returns results in a consistent shape, so the front-end never has to know the peculiarities of any single source.
Unified DeepLens interface
The DeepLens Next.js UI ('DeepLens — Unified Research Engine') presents one search experience over all of them. Its /api/search route takes an endpoint parameter naming the backend source, forwards the query, and returns ranked results — one front door for forty-plus back rooms.
Real ranked retrieval
Queries return live, freshly fetched, ranked results — not cached stubs. The verified Hacker News run for 'rust' surfaced the genuine top posts by score with points, comment counts, authors, and both canonical and Hacker News URLs, demonstrating live upstream integration.
Resilient degradation
The gateway validates and isolates failures: a missing endpoint parameter returns 400, an invalid backend path returns 502, and semantic vector-recall endpoints are disabled when Qdrant is down while the rest of the engine keeps serving — graceful degradation instead of a hard outage.
Challenges & Solutions
Impact
A researcher context-switches between Hacker News, GitHub, PubMed, EDGAR, and more — each with its own search, ranking, and result format — spending as much effort normalizing as reading.
Solution
A federated backend exposing 40+ sources behind one gateway, fronted by a single DeepLens search interface that returns results in a consistent shape.
Result
One query interface reaches news, code, academic, government, and vector-recall sources — verified live returning real ranked Hacker News results for 'rust'.
How AI Powers This
AI Capability
Unified Federated Search
Search 40+ intelligence sources — news, code, academic papers, patents, corporate filings, government data — from a single interface instead of forty separate tools
Business outcome
Verified: a Hacker News query for 'rust' returned the genuine top posts by score with points, comments, authors, and URLs
Under the hood
DeepLens UI /api/search takes an endpoint param naming the backend source -> strips it and forwards to search-api (:3500) -> the matching source endpoint fetches and ranks live results -> consistent shape returned
How success is measured: Source coverage, result freshness, ranking fidelity vs the native source
AI Capability
Semantic Vector Recall
Go beyond keyword matching with vector-based semantic recall over indexed content — surfacing conceptually related results, not just exact matches
Business outcome
Adds meaning-based retrieval alongside keyword sources, while failing safe when the vector store is unavailable
Under the hood
semantic/recall endpoints backed by a Qdrant vector store; gated on Qdrant availability and disabled gracefully when the store is down
How success is measured: Recall relevance, availability handling when Qdrant is offline
AI Capability
Isolated Source Integrations
Every source is wrapped independently, so adding a new one — or one going down — never destabilizes the rest of the engine
Business outcome
A single source failing degrades only that source; the unified engine keeps serving everything else
Under the hood
each of the 40+ sources is a distinct search-api endpoint with its own upstream client; the gateway validates the requested path (400 on missing, 502 on invalid)
How success is measured: Fault isolation, ease of adding a source, correctness of routing validation
Results & Metrics
Source coverage
ImprovedBefore
A dozen separate research tools, each searched by hand
After
40+ federated sources (hackernews, github, reddit, pubmed, patents, edgar, fda, nasa, vector recall, ...) behind one gateway
Result quality
ImprovedBefore
N/A — no unified engine
After
Real, live-fetched, ranked results — verified with genuine Hacker News rankings (points, comments, authors, canonical + HN URLs)
Interface
ImprovedBefore
One search box per source, each with its own syntax and format
After
A single DeepLens interface over every source, returning a consistent result shape
Failure handling
ImprovedBefore
N/A
After
Explicit validation (400 missing endpoint, 502 invalid path) and Qdrant-gated semantic recall that degrades gracefully
System Overview
The architecture below is the technical foundation behind every business outcome on this page — built for reliability, low running cost, and the speed your customers feel.
A federated search backend (search-api, Rust, :3500) exposes 40+ source endpoints — hackernews, reddit, github, duckduckgo, pubmed, patents, edgar, fda, nasa, semantic/recall (Qdrant vector store), reputation, shopping, tmdb, and more — each wrapping a distinct upstream in a consistent result shape. The DeepLens Next.js front-end ('DeepLens — Unified Research Engine') proxies queries via /api/search, which takes an endpoint parameter naming the backend source, validates it (400 on missing, 502 on invalid path), and forwards the rest. Semantic recall endpoints are gated on Qdrant availability and degrade gracefully when it is down. No auth on the verified route.Technologies Used
Rust (search-api), Next.js, TypeScript, vector search, federated source integrations