The primary way I keep up with news around the internet is by keeping track of a bunch of blogs (the whole list is at https://feedreader.co/arpith). I do this with an RSS/Atom feed reader that I built, https://feedreader.co. This is the story of how it works.
At its core, a feed reader has a list of feeds that it fetches and stores. Typically, a feed will have ten or twenty articles, so you’re going to have to store a list of articles (for when you want to infinite scroll all the way back). This is done using Dan MacTough’s excellent Feedparser module. The articles are stored on S3 using the AWS SDK and the article ids are added to a sorted set in Redis (in ElastiCache).
There are three things that Feedreader supports:
Folders are stored as sets of feed ids (the feed url). Creating a feed for a folder is easy — once you’ve got the feed ids, Redis’s zunionstore will give you a union of the sorted sets (of article ids).
Labels are similar to feeds — they are sorted sets of article ids. When articles are read, they are added to the label “read”. This makes it easy to fetch a list of articles that are unread:
First zunionstore is used to combine the feeds as well as the “read” label (which is assigned a negative weight). zrangebyscore is then used to filter out the read articles.
What’s the bigger picture? I’m glad you asked!
I’ve built a REST based API that lets you fetch, add and organize your feeds.This is an Express app running on EC2 behind a load balancer (which lets me use AWS Certificate Manager). As mentioned earlier, it uses Redis (in ElastiCache) as a backend and stores articles as static files on S3.
You can subscribe to feeds by using the bookmarklet when you’re on a website you like. The bookmarklet looks through the webpage you’re on to find the RSS/Atom feed url, and takes you to the Feedreader page for that feed.
This brings us to the actual website. It’s a tiny Express app that runs on Heroku behind CloudFlare. All it does is serve a HTML file that fetches and presents data from the API. You can do fun things like adding feeds to different folders (yupp, a feed can be in more than one folder!) or add labels to articles. And there’s infinite scroll :)
Sign up if this sounds interesting!