Source Scrapers

Horizon fetches content from multiple source types. All scrapers inherit from BaseScraper, share an async HTTP client, and implement a fetch(since) method that returns a list of ContentItem objects. Sources are fetched concurrently via asyncio.gather.

Hacker News

File: src/scrapers/hackernews.py

Uses the Firebase HN API:

GET /topstories.json — fetches top story IDs
GET /item/{id}.json — fetches story/comment details

Stories and their comments are fetched concurrently. For each story, the top 5 comments are included (deleted/dead comments excluded, HTML stripped, truncated at 500 chars).

Config (sources.hackernews):

{
  "enabled": true,
  "fetch_top_stories": 30,
  "min_score": 100,
  "category": "tech"
}

fetch_top_stories — number of top story IDs to fetch
min_score — minimum HN points to include a story
category — optional tag for balanced digest grouping

Extracted data: title, URL (falls back to HN discussion URL), author, score, comment count, top comment text, and category.

GitHub

File: src/scrapers/github.py

Uses the GitHub REST API:

GET /users/{username}/events/public — user activity events
GET /repos/{owner}/{repo}/releases — repository releases

Two source types are supported:

user_events — tracks push, create, release, public, and watch events for a user
repo_releases — tracks new releases for a specific repository

Config (sources.github, list of entries):

{
  "type": "user_events",
  "username": "torvalds",
  "enabled": true,
  "category": "oss"
}

{
  "type": "repo_releases",
  "owner": "golang",
  "repo": "go",
  "enabled": true,
  "category": "oss"
}

category — optional tag for balanced digest grouping; set per source entry

Authentication: Set GITHUB_TOKEN in your environment for higher rate limits (5000 req/hr vs 60 without).

RSS

File: src/scrapers/rss.py

Fetches any Atom/RSS feed using the feedparser library. Tries multiple date fields (published, updated, created) with fallback parsing.

Config (sources.rss, list of entries):

{
  "name": "Simon Willison",
  "url": "https://simonwillison.net/atom/everything/",
  "enabled": true,
  "category": "ai-tools",
  "content_extractor": "trafilatura"
}

category — optional tag for grouping (e.g., "programming", "microblog")
content_extractor — optional name of an extractor defined in extractors config; when set, the full article text replaces the feed-provided excerpt (see Extractors)

Extracted data: title, URL, author, content (from summary/description/content fields, or full article text if an extractor is configured), feed name, category, and entry tags.

HTTP 429 responses are retried at most once after respecting Retry-After (seconds or an HTTP date). When that header is absent or invalid, the scraper waits 60 seconds. Cooldowns longer than five minutes are left to a future scheduled run rather than retried early. A feed that still fails is skipped while other configured feeds continue.

File: src/scrapers/reddit.py

Supports approved app-only OAuth access and the original public endpoint mode.

Reddit’s October 8, 2026 changelog announces that new public Data API access requests stop on October 31, 2026, RSS support ends on November 13, 2026, and remaining public API access ends in March 2027. The integrations below therefore depend on Reddit’s applicable approval and migration arrangements. They are not a permanent substitute for the Developer Platform migration. These Reddit changes do not disable this Horizon reader’s own generated RSS feed.

For a confidential Reddit client with approved Data API access, set these fields inside sources.reddit:

{
  "enabled": true,
  "mode": "oauth",
  "client_id_env": "REDDIT_CLIENT_ID",
  "client_secret_env": "REDDIT_CLIENT_SECRET",
  "user_agent_env": "REDDIT_USER_AGENT"
}

Retain the configured subreddits, users, and fetch_comments fields. Put the three values in the server’s .env, not in JSON. Use an identifying User-Agent such as linux:horizon-reader:1.0 (by /u/YOUR_REDDIT_USERNAME). This app-only flow does not require a Reddit account password. The client obtains a short-lived token from Reddit and reads listings and comments at oauth.reddit.com.

Reddit currently requires prior approval for Data API access. Apply using the official developer request form and accurately describe the communities, daily request volume, AI summarization, external model service, and website display. Follow the approved application’s registration instructions; creating a client ID by itself does not establish approved access. See the Responsible Builder Policy.

OAuth mode skips Reddit if credentials are missing. It stops further Reddit requests for the run on authentication errors, HTTP 403, or HTTP 429 and does not fall back to anonymous endpoints. Other sources can still complete normally.

The default mode: "public" retains the original no-key behavior. Subreddit listings and comments prefer old.reddit.com HTML; these endpoints may be blocked:

GET https://old.reddit.com/r/{subreddit}/{sort}/ — subreddit posts
GET https://old.reddit.com/r/{subreddit}/comments/{post_id}/ — post comments
GET /r/{subreddit}/{sort}.json — subreddit posts fallback
GET /user/{username}/submitted.json — user submissions
GET /r/{subreddit}/comments/{post_id}.json — post comments fallback
GET /r/{subreddit}/{sort}/.rss — subreddit posts fallback when JSON is blocked

Subreddits and users are fetched concurrently. Comments are sorted by score, limited to the configured count, and exclude moderator-distinguished comments. Self-text is truncated at 1500 chars, comments at 500 chars.

Config (sources.reddit):

{
  "enabled": true,
  "fetch_comments": 5,
  "subreddits": [
    {
      "subreddit": "MachineLearning",
      "sort": "hot",
      "fetch_limit": 25,
      "min_score": 10,
      "category": "ai-ml"
    }
  ],
  "users": [
    {
      "username": "spez",
      "sort": "new",
      "fetch_limit": 10,
      "category": "social"
    }
  ]
}

sort — hot, new, top, or rising (subreddits); hot or new (users)
time_filter — for top/rising sorts: hour, day, week, month, year, all
min_score — minimum post score (subreddits only)
category — optional tag for balanced digest grouping; set per subreddit or per user entry

Rate limiting: Detects HTTP 429 responses on JSON requests, reads the Retry-After header, waits, and retries once. Uses browser-like request headers for no-key public access.

Extracted data: title, URL, author, score, upvote ratio, comment count, subreddit, flair, self-text, top comments, and category.

OpenBB

File: src/scrapers/openbb.py

Uses the OpenBB Platform Python SDK via obb.news.company() to fetch company news for one or more ticker watchlists.

The scraper imports openbb lazily. If the optional dependency is not installed, Horizon logs a warning and skips the source instead of failing the whole run.

Config (sources.openbb):

{
  "enabled": true,
  "watchlists": [
    {
      "name": "megacaps",
      "symbols": ["AAPL", "MSFT", "NVDA"],
      "enabled": true,
      "provider": "yfinance",
      "fetch_limit": 20,
      "category": "equities"
    }
  ]
}

watchlists — each enabled watchlist triggers one news.company() call per run
provider — OpenBB provider name for that watchlist
symbols — tickers fetched together for the same provider
fetch_limit — maximum rows requested from the provider
category — optional metadata tag stored on each item

Behavior:

Wraps the synchronous OpenBB SDK in asyncio.to_thread so the event loop stays responsive
Deduplicates duplicate news across watchlists by article URL
Skips malformed rows, rows without URL/title/date, and items older than the current time window
Keeps fetching other watchlists if one provider call fails

Credentials: provider-specific secrets are resolved by the OpenBB SDK from its own environment variables or settings file. Horizon does not pass those values directly.

Extracted data: title, URL, author, published time, article body/excerpt, watchlist name, provider, category, and symbol list.

Twitter

File: src/scrapers/twitter.py

Uses the Apify platform to bypass Twitter’s anti-scraping measures. The actor altimis~scweet is called via the Apify REST API.

Flow:

POST to /v2/acts/{actor_id}/runs to trigger a run
Poll /v2/actor-runs/{run_id} until status is SUCCEEDED or a terminal failure
GET /v2/datasets/{dataset_id}/items to retrieve results

Profile timelines use source_mode: "profiles". Keyword discovery and reply expansion reuse source_mode: "search" with search_query.

When users are configured, a single profile run fetches their timelines together. Each non-empty keyword query starts a separate search run, independently of the configured users. Either path can be used on its own. Results are filtered to the current time window and merged with deduplication by tweet ID across all runs.

Config (sources.twitter):

{
  "enabled": true,
  "users": ["karpathy", "ylecun"],
  "keywords": ["LLM", "open source"],
  "fetch_limit": 10,
  "fetch_reply_text": false,
  "max_replies_per_tweet": 3,
  "max_tweets_to_expand": 10,
  "reply_min_likes": 5,
  "actor_id": "altimis~scweet",
  "apify_token_env": "APIFY_TOKEN"
}

users — Twitter screen names to monitor, without the @ prefix
keywords — independent Apify search queries (source_mode: "search"), not filters on the configured users’ timelines. Not supported in Playwright mode.
fetch_limit — in Apify mode, each profile or keyword-search actor run requests up to max(100, fetch_limit) tweets. This is not a combined limit: the example above starts three discovery runs with a limit of 100 tweets each, before time filtering and deduplication. Each keyword adds an actor run and associated Apify usage.
category — optional tag for balanced digest grouping (applies to all tweets from this source)
fetch_reply_text — when true, a second Apify run fetches reply bodies for each important tweet and appends them under --- Top Comments --- for AI analysis
max_replies_per_tweet — maximum reply lines per tweet (sorted by engagement score)
max_tweets_to_expand — cap on reply expansion runs per pipeline cycle, to control Apify credit usage
reply_min_likes — minimum likes required for a reply to be included
actor_id — Apify actor ID (default: altimis~scweet)
apify_token_env — environment variable name containing the Apify API token

Authentication: Set APIFY_TOKEN in your .env. Get a token at console.apify.com.

Extracted data: tweet text, URL, author, publish time, likes, retweets, replies, views, category, and (optionally) reply-thread text appended under --- Top Comments ---.