Methodology
The Wider Lens is a deterministic aggregation pipeline. There is no LLM in the loop, no editorial curation, and no personalization. Here is exactly what happens on every refresh.
1.Collection
Server-side collectors fetch public RSS/Atom feeds, the arXiv API, the GitHub REST API, and the Hugging Face Hub API. Every request has a timeout (10–20s). Sources are fetched withPromise.allSettled — if one feed is down, slow, or returns malformed data, it is logged and skipped; the page renders with the rest.
2.Normalization
Every item becomes a common Article record: title, short description, URL, source, author, publication date, and image when the feed provides one. Missing fields are handled gracefully — an item is never dropped just because it lacks an image or author.
3.Relevance filtering
General tech feeds (e.g. TechCrunch) cover more than AI. Items must match AI keywords — model names, techniques, companies, or research terms — to survive. Obvious junk (job posts, sponsored content) is removed.
4.Categorization
A maintainable keyword-rule system assigns each story to one category: Models, Open Source, Research, Tools, Agents, Robotics, Coding, Image & Video, Business, or AI Hardware. Rules are scored (longer phrases weigh more) and the highest-scoring category wins; ties resolve by a fixed priority order. Companies are detected with an alias list (e.g. “DeepMind”, “Gemini” → Google).
5.Deduplication & clustering
Titles are normalized (lowercased, punctuation and stopwords removed) and compared with Jaccard token similarity. Near-identical items (≥ 0.88) from the same source are dropped. Items about the same event (≥ 0.38 similarity, or ≥ 0.25 with a shared company) are clustered, not deleted: the cluster shows one primary story and a “View N sources” expander listing every outlet covering it. When an official company announcement exists in the cluster, it becomes the primary.
6.Importance ranking (0–100)
The score is a transparent heuristic:
- Source authority: official 40 · research 30 · journalism 25 · models 20 · GitHub/Hugging Face/tools 15
- Freshness: <24h +25 · <72h +15 · <7d +5
- Corroboration: +8 per additional independent source in the cluster (max +24)
- Event signals: model-release keywords +10 · official launch announcement +8 · benchmark/breakthrough +5
“Today in AI” shows the top-ranked stories published in the last 24 hours; the trending list shows the top-ranked stories overall.
7.Refresh frequency
- News feeds: every 15 minutes
- GitHub: every 30 minutes
- Hugging Face models & arXiv: every 60 minutes
- Learn pages: static, rebuilt on deploy
Implemented with Next.js data-cache revalidation (fetch(..., { next: { revalidate } })), so upstream APIs are never hit once per visitor.
8.Limitations
- We link to original reporting; The Wider Lens hosts only headlines and excerpts.
- Automated clustering can occasionally group loosely related stories — use the source list to verify.
- Model metadata reflects what the Hugging Face API reports; open-weight status is shown only when the license reliably indicates it.
- GitHub “interesting repos” come from curated topic searches, not a global trending calculation.