AI engineering
Reddit Idea Miner
A daily pipeline that scans job-search subreddits and Hacker News, extracts actionable product ideas with Claude in a single structured call, and emails a ranked digest.
Role — Sole author — pipeline, prompt design, fetch strategy, scheduling.
Highlights
- One model call per run handles classification, theme deduplication, effort/confidence scoring and suggestion drafting as structured output — everything upstream is cheap filtering, so cost stays flat no matter how many posts are scanned.
- Two fetch modes chosen automatically from the environment: authenticated OAuth JSON for true engagement filtering, and an anonymous Atom-feed path that falls back to feed position for ranking.
- The RSS path exists because anonymous Reddit JSON started returning 403 under the 2025 Responsible Builder Policy — it respects the x-ratelimit headers at roughly one request per minute, which is irrelevant for a once-daily job.
- Persistent state (seen posts, dated digests) plus a feedback loop that tunes what gets surfaced on later runs.
Overview
A daily bot that mines job-search subreddits and Hacker News for product-opportunity signal, then emails a digest.
The pipeline is fetch → filter → analyze → digest → email → persist. The design constraint was cost: rather than asking the model about each post, everything cheap happens first — engagement thresholds, noise rules, already-seen checks — and a single Claude call with structured output does classification, deduplication, scoring and drafting for the whole batch at once. Spend per run is therefore flat in the number of posts scanned.
Built to feed the roadmap of a CV-tailoring product, but the pipeline is domain-agnostic: point it at different subreddits and prompts and it mines whatever signal you care about. Runs on launchd locally or GitHub Actions in CI.