[](http://rubygems.org/gems/html2rss/) [](https://www.rubydoc.info/gems/html2rss) 
`html2rss` is a Ruby gem that generates RSS 2.0 feeds from websites automatically, and as a fallback via _feed config_.
With the _feed config_, you provide a URL to scrape and CSS selectors for extracting information (like title, URL, etc.). The gem builds the RSS feed accordingly. [Extractors](#using-extractors) and chainable [post processors](#using-post-processors) make information extraction, processing, and sanitizing a breeze. The gem also supports [scraping JSON](#scraping-and-handling-json-responses) responses and [setting HTTP request headers](#set-any-http-header-in-the-request).
**Looking for a ready-to-use app to serve generated feeds via HTTP?** [Check out `html2rss-web`](https://github.com/html2rss/html2rss-web)!
Support the development by sponsoring this project on GitHub. Thank you! 💓
## Generating a feed on the CLI
[Install Ruby](https://www.ruby-lang.org/en/documentation/installation/) (latest version is recommended) on your machine and run `gem install html2rss` in your terminal.
After the installation has finished, `html2rss help` will print usage information.
### using automatic generation
html2rss offers an automatic RSS generation feature. Try it with:
`html2rss auto https://unmatchedstyle.com/`
### creating a feed config file and using it
If the results are not to your satisfaction, you can create a feed config file.
Create a file called `my_config_file.yml` with this sample content:
```yml
channel:
url: https://unmatchedstyle.com
selectors:
items:
selector: "article[id^='post-']"
title:
selector: h2
link:
selector: a
extractor: href
description:
selector: ".post-content"
post_process:
- name: sanitize_html
```
Build the feed from this config with: `html2rss feed ./my_config_file.yml`.
## Generating a feed with Ruby
You can also install it as a dependency in your Ruby project:
| 🤩 Like it? | Star it! ⭐️ |
| -------------------------------: | -------------------- |
| Add this line to your `Gemfile`: | `gem 'html2rss'` |
| Then execute: | `bundle` |
| In your code: | `require 'html2rss'` |
Here's a minimal working example using Ruby:
```ruby
require 'html2rss'
rss =
Html2rss.feed(
channel: { url: 'https://stackoverflow.com/questions' },
selectors: {
items: { selector: '#hot-network-questions > ul > li' },
title: { selector: 'a' },
link: { selector: 'a', extractor: 'href' }
}
)
puts rss
```
## The _feed config_ and its options
A _feed config_ consists of a `channel` and a `selectors` hash. The contents of both hashes are explained below.
Good to know:
- You'll find extensive example feed configs at [`spec/*.test.yml`](https://github.com/html2rss/html2rss/tree/master/spec).
- See [`html2rss-configs`](https://github.com/html2rss/html2rss-configs) for ready-made feed configs!
- If you've created feed configs, you're invited to send a PR to [`html2rss-configs`](https://github.com/html2rss/html2rss-configs) to make your config available to the public.
Alright, let's move on.
### The `channel`
| attribute | | type | default | remark |
| ------------- | ------------ | ------- | -------------- | ------------------------------------------ |
| `url` | **required** | String | | |
| `title` | optional | String | auto-generated | |
| `description` | optional | String | auto-generated | |
| `ttl` | optional | Integer | `360` | TTL in _minutes_ |
| `time_zone` | optional | String | `'UTC'` | TimeZone name |
| `language` | optional | String | `'en'` | Language code |
| `author` | optional | String | | Format: `email (Name)` |
| `headers` | optional | Hash | `{}` | Set HTTP request headers. See notes below. |
| `json` | optional | Boolean | `false` | Handle JSON response. See notes below. |
#### Dynamic parameters in `channel` attributes
Sometimes there are structurally similar pages with different URLs. In such cases, you can add _dynamic parameters_ to the channel's attributes.
Example of a dynamic `id` parameter in the channel URLs:
```yml
channel:
url: "http://domainname.tld/whatever/%s.html"
```
Command line usage example:
```sh
html2rss feed the_feed_config.yml id=42
```
See a Ruby example
```ruby
config = Html2rss::Config.new({ channel: { url: 'http://domainname.tld/whatever/%s.html' } }, {}, { id: 42 })
Html2rss.feed(config)
```
See a Ruby example
```ruby
Html2rss.feed(
channel: {}, selectors: { link: { selector: 'a', extractor: 'href' } }
)
```
See a YAML feed config example
```yml
channel:
# ... omitted
selectors:
# ... omitted
link:
selector: "a"
extractor: "href"
```
YAML example: build the description from a template String (in Markdown) and convert that Markdown to HTML
```yml
channel:
# ... omitted
selectors:
# ... omitted
price:
selector: '.price'
description:
selector: '.section'
post_process:
- name: template
string: |
# %{self}
Price: %{price}
- name: markdown_to_html
```
See a Ruby example
```ruby
Html2rss.feed(
channel: {},
selectors: {
title: { selector: 'a', post_process: [{ name: 'gsub', pattern: 'foo', replacement: 'bar' }] }
}
)
```
See a YAML feed config example
```yml
channel:
# ... omitted
selectors:
# ... omitted
title:
selector: "a"
post_process:
- name: "gsub"
pattern: "foo"
replacement: "bar"
```
See a Ruby example
```ruby
Html2rss.feed(
channel: {},
selectors: {
genre: {
# ... omitted
selector: '.genre'
},
branch: { selector: '.branch' },
categories: %i[genre branch]
}
)
```
See a YAML feed config example
```yml
channel:
# ... omitted
selectors:
# ... omitted
genre:
selector: ".genre"
branch:
selector: ".branch"
categories:
- genre
- branch
```
#### Custom item GUID
By default, html2rss generates a GUID from the `title` or `description`.
If this does not work well, you can choose other attributes from which the GUID is build.
The principle is the same as for the categories: pass an array of selectors names.
In all cases, the GUID is a SHA1-encoded string.
See a Ruby example
```ruby
Html2rss.feed(
channel: {},
selectors: {
title: {
# ... omitted
selector: 'h1'
},
link: { selector: 'a', extractor: 'href' },
guid: %i[link]
}
)
```
See a YAML feed config example
```yml
channel:
# ... omitted
selectors:
# ... omitted
title:
selector: "h1"
link:
selector: "a"
extractor: "href"
guid:
- link
```
See a Ruby example
```ruby
Html2rss.feed(
channel: {},
selectors: {
enclosure: {
selector: 'audio',
extractor: 'attribute',
attribute: 'src',
content_type: 'audio/mp3'
}
}
)
```
See a YAML feed config example
```yml
channel:
# ... omitted
selectors:
# ... omitted
enclosure:
selector: "audio"
extractor: "attribute"
attribute: "src"
content_type: "audio/mp3"
```
## Scraping and handling JSON responses
By default, `html2rss` assumes the URL responds with HTML. However, it can also handle JSON responses. The JSON response must be an Array or Hash.
The JSON is converted to XML which you can query using CSS selectors.
See a Ruby example
```ruby
Html2rss.feed(
channel: { url: 'http://domainname.tld/whatever.json', json: true },
selectors: { title: { selector: 'foo' } }
)
```
See a YAML feed config example
```yml
channel:
url: "http://domainname.tld/whatever.json"
json: true
selectors:
title:
selector: "foo"
```
See a YAML feed config example
```yml
channel:
url: https://www.imdb.com/user/ur67728460/ratings
time_zone: UTC
ttl: 1440
strategy: browserless
headers:
User-Agent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
selectors:
items:
selector: "li.ipc-metadata-list-summary-item"
title:
selector: ".ipc-title__text"
post_process:
- name: gsub
pattern: "/^(\\d+.)\\s/"
replacement: ""
- name: template
string: "%{self} rated with: %{user_rating}"
link:
selector: "a.ipc-title-link-wrapper"
extractor: "href"
user_rating:
selector: "[data-testid='ratingGroup--other-user-rating'] > .ipc-rating-star--rating"
```
Build feeds in Ruby
```ruby
require 'html2rss'
myfeed = Html2rss.feed_from_yaml_config('feeds.yml', 'myfeed')
myotherfeed = Html2rss.feed_from_yaml_config('feeds.yml', 'myotherfeed')
```
Build feeds on the command line
```sh
html2rss feed feeds.yml myfeed
html2rss feed feeds.yml myotherfeed
```
## Display the RSS feed nicely in a web browser
To display RSS feeds nicely in a web browser, you can:
- add a plain old CSS stylesheet, or
- use XSLT (e**X**tensible **S**tylesheet **L**anguage **T**ransformations).
A web browser will apply these stylesheets and show the contents as described.
In a CSS stylesheet, you'd use `element` selectors to apply styles.
If you want to do more, then you need to create a XSLT. XSLT allows you
to use a HTML template and to freely design the information of the RSS,
including using JavaScript and external resources.
You can add as many stylesheets and types as you like. Just add them to your global configuration.
Ruby: a stylesheet config example
```ruby
config = Html2rss::Config.new(
{ channel: {}, selectors: {} }, # omitted
{
stylesheets: [
{
href: '/relative/base/path/to/style.xls',
media: :all,
type: 'text/xsl'
},
{
href: 'http://example.com/rss.css',
media: :all,
type: 'text/css'
}
]
}
)
Html2rss.feed(config)
```
YAML: a stylesheet config example
```yml
stylesheets:
- href: "/relative/base/path/to/style.xls"
media: "all"
type: "text/xsl"
- href: "http://example.com/rss.css"
media: "all"
type: "text/css"
feeds:
# ... omitted
```
Recommended further readings:
- [How to format RSS with CSS on lifewire.com](https://www.lifewire.com/how-to-format-rss-3469302)
- [XSLT: Extensible Stylesheet Language Transformations on MDN](https://developer.mozilla.org/en-US/docs/Web/XSLT)
- [The XSLT used by html2rss-web](https://github.com/html2rss/html2rss-web/blob/master/public/rss.xsl)
## Gotchas and tips & tricks
- Check that the channel URL does not redirect to a mobile page with a different markup structure.
- Do not rely on your web browser's developer console. `html2rss` does not execute JavaScript.
- Fiddling with [`curl`](https://github.com/curl/curl) and [`pup`](https://github.com/ericchiang/pup) to find the selectors seems efficient (`curl URL | pup`).
- [CSS selectors are versatile. Here's an overview.](https://www.w3.org/TR/selectors-4/#overview)
## Contributing
Find ideas what to contribute in:
1.
2. the issues tracker:
To submit changes:
1. Fork this repo ( )
2. Create your feature branch (`git checkout -b my-new-feature`)
3. Implement a commit your changes (`git commit -am 'feat: add XYZ'`)
4. Push to the branch (`git push origin my-new-feature`)
5. Create a new Pull Request using the Github web UI
## Development Helpers
1. `bin/setup`: installs dependencies and sets up the development environment.
2. `bin/guard`: automatically runs rspec, rubocop and reek when a file changes.
3. for a modern Ruby development experience: install [`ruby-lsp`](https://github.com/Shopify/ruby-lsp) and integrate it to your IDE.
For example: [Ruby in Visual Studio Code](https://code.visualstudio.com/docs/languages/ruby).