| title | description | layout | toc | ||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Subscribers |
Superfeedr allows you to subscribe to content on the web, and receive push notifications in real-time when new content is published. Here's how to set it up. |
page |
|
Superfeedr allows you to subscribe to content on the web, and receive push notifications in real-time when new content is published. It also allows you to retrieve past content, and we’ve also got a range of normalization options that make for easier consumption.
You can use the Superfeedr feed API to subscribe to anything that has a URL. The key caveat to this is that the content needs to be publicly accessible by our servers – the content can’t be in a private network or behind a firewall.
While URLs can include an authentication element, but Superfeedr will not treat these URLs with any specific security concern. For this reason, we strongly discourage including URLs with authentication mechanisms.
Our historical use case is to allow you to subscribe to RSS or Atom feeds and then have that content pushed to you in real-time. These are also normalized, to make consumption easier for you (see the schema section). Superfeedr supports RSS, Atom and RDF, as well as the most popular namespaces.
An update is detected when one or more item is added to the feed you are subscribed to. The real-time push notification includes the new item(s) and some key attributes from the feed, like the title.
Each entry in a feed is mapped with a unique identifier - the id element in our schema. We will send you a notification for every new unique identifier we detect, whatever the rest of the entry is.We don’t rely on the title, links or even the signature of the entry’s content to detect updates.
This behaviour is compliant with both RSS and Atom specifications. If we’re not able to find a unique ID, Superfeedr generates one, using complex feed-specific rules. (We indicate when the <id> has been generated by Superfeedr using the generated_ids field in the status).
If an entry is updated, they are not propagated by default. This is because we want to avoid creating numerous false positives.
There is, however, one exception: if a new entry contains a valid, updated element, and the update is very recent (within three minutes), you will receive a notification. This means that we will usually propagate updates for feeds if we receive a ping from the publisher.
If a feed goes into an error state (say, if it’s putting up HTTP error codes, or we’ve been unable to parse content successfully), you’ll also receive a notification.
This will allow you to monitor the situation, and decide whether you want to maintain the subscription to that feed. These notifications will only include the status part of the schema.
It is obviously possible to subscribe to any kind of JSON document. In order to identify new content, we compute a hash signature of the whole document. If that signature changes between two fetches, Superfeedr propagates the change by sending the full JSON document to your endpoint.
You will also receive a notification if a resource goes into an HTTP error state. These notifications will only include the status part of the schema.
Superfeedr also allows you to subscribe to HTML fragments within an HTML page. A fragment is a portion of an HTML document, defined by a CSS path. You can subscribe to DOM elements inside a page as long as there is a CSS path that leads to them. You will append the URL-escaped CSS path to the document URL, using the fragment part of the URL. If there are multiple elements matching your CSS path, Superfeedr will concatenate each of them.
For example, if you want to subscribe to the .h-entry element of http://blog.superfeedr.com, you will need to subscribe to http://blog.superfeedr.com#.h-entry.
Much like subscribing to HTML fragments, Superfeedr also allows you to subscribe to parts of a JSON document, using the fragments API. To do this, Superfeedr uses the JSONPath syntax. To build the topic url, you append the URL-escaped JSONPath to the document URL, using the fragment part of the URL.
We have the following document at http://for.sale/inventory.json:
{% prism javascript %} { "store": { "book": [ { "category": "reference", "author": "Nigel Rees", "title": "Sayings of the Century", "price": 8.95 }, { "category": "fiction", "author": "Evelyn Waugh", "title": "Sword of Honour", "price": 12.99 }, { "category": "fiction", "author": "Herman Melville", "title": "Moby Dick", "isbn": "0-553-21311-3", "price": 8.99 }, { "category": "fiction", "author": "J. R. R. Tolkien", "title": "The Lord of the Rings", "isbn": "0-395-19395-8", "price": 22.99 } ], "bicycle": { "color": "red", "price": 19.95 } } } {% endprism %}
If you wanted to keep an eye on the price of the bicycle, you would subscribe to: http://for.sale/inventory.json#%24.store.bicycle.price. Want more examples? Click here.
When you are subscribing to any other type of HTML resource, Superfeedr will compute a signature from the bytes included in the document. If the signature has changed when we next fetch the resource, you will receive the whole document again. Note that timestamps, changes in tracking codes and so on can create false positives here.
When fetching resources, Superfeedr will follow up to five redirects. These can be permanent or temporary, and you will be notified of the content at the end of the redirect chain. For this reason, we strongly recommend subscribing to the canonical URL of each resource.
Our API is based on the PubSubHubbub protocol with a couple simplifications. However, you can use your subscriber code with any other hub. You can also use any other library that supports and implements PubSubHubbub.
Our PubSubHubbub endpoint is at https://push.superfeedr.com/. The key difference is that our endpoint uses HTTP Basic Auth to authenticate your PubSubHubbub calls, making all the verification steps of each request optional.
If you want to manually specify an HTTP method different to the one used in the request, we also support the use of the X-HTTP-Method-Override HTTP header.
Authentication using the Webhooks API is performed through HTTP Basic Authentication. Most HTTP libraries will allow for an easy configuration with this. To get started with an authentication, use your Superfeedr login, then pick from one of these options for your password:
- A password token that you can generate.
- Your main Superfeedr password (though we recommend using a token for security purposes).
If your HTTP library does not support HTTP headers, you should submit a base64 encoded string of login:token as a query string parameter named authorization.
You can create an unlimited number of tokens, with different combinations of rights associated with them:
- Subscribe
- Unsubscribe
- List
- Retrieve (used for Streaming as well)
The tokens cannot be used to log into the main Superfeedr site.
Tokens can be made public – just keep in mind that any call made using them will be associated with your account. This means that if someone else makes a call with your token, it will be your account that is billed.
If you want your tokens to be private along with the rest of your account details, please remember to use https when sending authentication against our endpoint.
https://push.superfeedr.com
| Parameter Name | Note | Value |
|---|---|---|
| hub.mode | required | subscribe |
| hub.topic | required | The URL of the HTTP resource to which you want to subscribe. It cannot be more than 2048 characters long. |
| hub.callback | required | The webhook: it's the URL to which notifications will be sent. Make sure you it's web-accessible, ie not behind a firewall. Its size is currently limited to 250 characters. |
| hub.secret | optional, recommended | A unique secret string which will be used by us to compute a signature. You should check this signature when getting notifications. |
| hub.verify | optional | sync or async: will perform a PubSubHubbub verification of intent synschronously or asynschronously. |
| format | optional |
|
| retrieve | optional | If set to true, the response will include the current representation of the feed as stored in Superfeedr, in the format desired. Please check our Schema for more details. |
Subscriptions at Superfeedr are a unique combination of a resource URL and a callback URL. If you resubscribe with both URLs, we will only keep one. If you use a different callback URL to the resource URL, we will keep both.
{% prism markup %} curl https://push.superfeedr.com/ -X POST -u demo:demo -d'hub.mode=subscribe' -d'hub.topic=http://push-pub.appspot.com/feed' -d'hub.callback=http://mycallback.tld/ok' {% endprism %}
204: the subscription was performed and you should expect notifications202: the subscription has yet to be verified (only if you supplied ahub.verify=asyncparameter).200: you used theretrieveparameter. The content of the feed is in the body of the response.422: please check the body, as it will include the reason for the subscription failure.- Other HTTP response codes are outlined in the HTTP spec.
When subscribing, the callback url can also be https://push.superfeedr.com/dev/null which is our null device. This is particularly useful if you're not interested in realtime updates to feeds but want to retrieve their statuses from our servers. This way, you're able to subscribe to tell Superfeedr to not notify you but still let you poll the content of these feeds.
Please note that even though we won't notify you for the changes in the feed, these notifications will still be accounted for when it comes to billing.
https://push.superfeedr.com
<th>Parameter Name</th>
<th>Note</th>
<th>Value</th>
</tr>
</thead>
<tbody>
<tr>
<td>hub.mode</td>
<td>required</td>
<td><code>unsubscribe</code></td>
</tr>
<tr>
<td>hub.topic</td>
<td>required</td>
<td>The URL of the HTTP resource to which you want to subscribe.</td>
</tr>
<tr>
<td>hub.callback</td>
<td>optional</td>
<td>The URL to which notifications will be sent. It is optional if you are only subscribed to the feed 'once', with a single <code>hub.callback</code>. If you have multiple subscriptions, you will need to supply the <code>hub.callback</code> parameter. It is also required if you use the <code>hub.verify</code> param. (see below). </td>
</tr>
<tr>
<td>hub.verify</td>
<td>optional</td>
<td><code>sync</code> or <code>async</code>. We will perform a <a href="http://pubsubhubbub.github.io/PubSubHubbub/pubsubhubbub-core-0.4.html#rfc.section.5.3">PubSubHubbub verification</a> of intent synschronously or asynschronously.</td>
</tr>
</tbody>
{% prism markup %} curl https://push.superfeedr.com/ -X POST -u demo:demo -d'hub.mode=unsubscribe' -d'hub.topic=http://push-pub.appspot.com/feed' -d'hub.callback=http://mycallback.tld/ok' {% endprism %}
204: the unsubscribe was successful202: the unsubscribe has yet to be verified (only if you supplied ahub.verify=asyncparameter).422: Check the body, as it will include the reason for the unsubscribe failure.- Other HTTP response codes are outlined in the HTTP spec.
https://push.superfeedr.com
<tr>
<th>Parameter Name</th>
<th>Note</th>
<th>Value</th>
</tr>
</thead>
<tbody>
<tr>
<td>hub.mode</td>
<td>required</td>
<td><code>list</code></td>
</tr>
<tr>
<td>page</td>
<td>optional</td>
<td>If there are more than 20 matching subscriptions, you may want to paginate over them. First page (default) is 1.</td>
</tr>
<tr>
<td>by_page</td>
<td>optional</td>
<td>Number of items per page. Max is 500. Min is 1.</td>
</tr>
<tr>
<td>search</td>
<td>optional</td>
<td>A search query. Please <a href="#search-queries">see below</a> for the various fields and values to use.</td>
</tr>
<tr>
<td>detailed</td>
<td>optional</td>
<td>Get feed details along with the subscriptions. Check the <a href="/schema.html">Schema section</a> for more details.</td>
</tr>
</tbody>
By default, subscriptions are listed in order of creation. The oldest subscriptions are listed first, while the most recent is listed last.
200: the list of matching subscriptions is in the body in a JSON format (or jsonp if applicable).- Other HTTP response codes are outlined in the HTTP spec.
{% prism markup %} curl https://push.superfeedr.com/ -G -u demo:demo -d'hub.mode=list' -d'page=2' {% endprism %} {% prism javascript %} [{ "subscription": { "format": "atom", "endpoint": "http://mycallback.tld/ok", "secret": null, "feed": { "title": "Publisher example", "url": "http://push-pub.appspot.com/feed" } } }] {% endprism %}
Search queries are nested string parameters. We use the following keys:
<tr>
<th>Query</th>
<th>Note</th>
<th>Example</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>format</code></td>
<td>can either be <code>JSON</code> or <code>ATOM</code> and will return all subscription made using that format</td>
<td><em>search[format]=json</em></td>
</tr>
<tr>
<td><code>feed url</code></td>
<td>an exact match of the feed url.</td>
<td><em>search[feed][url]=http://blog.superfeedr.com/atom.xml</em> or <em>search[feed.url]=http://blog.superfeedr.com/atom.xml</em></td>
</tr>
<tr>
<td><code>feed inurl</code></td>
<td>string (sequence of characters) included in the URL. The match is done using n-grams so approchaing sequences will also match</td>
<td><em>search[feed][inurl]=superfeedr</em> or <em>search[feed.inurl]=superfeedr</em></td>
</tr>
<tr>
<td><code>feed hostname</code></td>
<td>an exact match of the feed's URL hostname.</td>
<td><em>search[feed][hostname]=blog.superfeedr.com</em> or <em>search[feed.hostname]=blog.superfeedr.com</em></td>
</tr>
<tr>
<td><code>feed velocity</code></td>
<td>a number or range for the <a href="/schema.html#velocity">feed's velocity</a>. Ranges start with >, >=, <=, < </td>
<td><em>search[feed][velocity]=>10</em> or <em>search[feed][velocity]=<=1</em></td>
</tr>
<tr>
<td><code>feed bozo rank</code></td>
<td>a number or range for the <a href="/schema.html#bozo_rank">feed's bozo rank</a>. Ranges start with >, >=, <=, < </td>
<td><em>search[feed][bozo]=>0</em></td>
</tr>
<tr>
<td><code>feed porn rank</code></td>
<td>a number or range for the <a href="/schema.html#porn_rank">feed's porn rank</a>. Ranges start with >, >=, <=, < </td>
<td><em>search[feed][porn]=>0</em></td>
</tr>
<tr>
<td><code>endpoint url</code></td>
<td>an exact match of pubsubhubbub <code>hub.callback</code> URL.</td>
<td><em>search[endpoint][url]=http://my.domain.tld/feed/1</em> or <em>search[endpoint.url]=http://my.domain.tld/feed/1</em></td>
</tr>
<tr>
<td><code>endpoint inurl</code></td>
<td>string (sequence of characters) included in the <code>hub.callback</code> URL. The match is done using n-grams so approchaing sequences will also match</td>
<td><em>search[endpoint][inurl]=domain</em> or <em>search[endpoint.inurl]=domain</em></td>
</tr>
<tr>
<td><code>endpoint hostname</code></td>
<td>an exact match of the <code>hub.callback</code>'s URL hostname.</td>
<td><em>search[endpoint][hostname]=domain.tld</em> or <em>search[endpoint.hostname]=domain.tld</em></td>
</tr>
</tbody>
https://push.superfeedr.com
<tr>
<th>Parameter Name</th>
<th>Note</th>
<th>Value</th>
</tr>
</thead>
<tbody>
<tr>
<td>hub.mode</td>
<td>required</td>
<td><code>retrieve</code></td>
</tr>
<tr>
<td>hub.topic</td>
<td>optional</td>
<td>The URL of the HTTP resource for which you want the past entries. Make sure you have previously subscribed to the resource.</td>
</tr>
<tr>
<td>hub.callback</td>
<td>optional</td>
<td>The value can either be a callback with which you are subscribed to one or more feeds or a search query that should match one or more callback urls used to subscribed to several feeds. Please, use the query syntax used to <a href="#search-queries">search for subscriptions</a> (see example below) . In both cases, make sure there are less than 200 matching feeds. </td>
</tr>
<tr>
<td>count</td>
<td>optional</td>
<td>Optional number of items you want to retrieve. Current max is 50 and default is 10.</td>
</tr>
<tr>
<td>before</td>
<td>optional</td>
<td>The <code>id</code> of an entry in the feed. The response will only include entries published before this one.</td>
</tr>
<tr>
<td>after</td>
<td>optional</td>
<td>The <code>id</code> of an entry in the feed. The response will only include entries published after this one.</td>
</tr>
<tr>
<td>format</td>
<td>optional</td>
<td><code>json</code> if you want to retrieve entries in json format (for feeds only!). You can also use an <code>Accept</code> HTTP header like this: <code>Accept: application/json</code></td>
</tr>
<tr>
<td>callback</td>
<td>optional, only if you're using the JSON format</td>
<td>This will render the entries as a <a href="http://en.wikipedia.org/wiki/JSONP">JSONP</a>. </td>
</tr>
</tbody>
{% prism markup %} curl https://push.superfeedr.com/ -H 'Accept: application/json' -G -u demo:demo -d'hub.mode=retrieve' -d'hub.topic=http://push-pub.appspot.com/feed' {% endprism %}
Response:
{% prism javascript %} Content-Type: application/json; charset=utf-8 Connection: keep-alive Status: 200 OK ETag: "c58f3a54cfa565539672a6f2f3276ddc" X-Runtime: 275 Content-Length: 3550
{ "status": { "code": 200, "feed": "http://push-pub.appspot.com/feed", "http": "Fetched (ring) 200 121 and parsed 0/20 entries", "lastParse": 1377845845, "period": 43200, "lastMaintenanceAt": 1377842368, "nextFetch": 1377889045, "lastFetch": 1377845845 }, "title": "Publisher example", "items": [ { "title": "mhmmm", "published": 1377845723000, "id": "http://push-pub.appspot.com/feed/1104006", "links": { "http://push-pub.appspot.com/entry/1104006": { "href": "http://push-pub.appspot.com/entry/1104006", "title": "mhmmm", "rel": "alternate", "mime_type": "text/html", "id": "mhmmm" } }, "content": "mmhmhjmm", "updated": 1377845723000 }, { "title": "test 2 - bis", "published": 1377758355000, "id": "http://push-pub.appspot.com/feed/1109002", "links": { "http://push-pub.appspot.com/entry/1109002": { "href": "http://push-pub.appspot.com/entry/1109002", "title": "test 2 - bis", "rel": "alternate", "mime_type": "text/html", "id": "test-2-bis" } }, "content": "recieved - bis", "updated": 1377758355000 }, { "title": "test", "published": 1377756592000, "id": "http://push-pub.appspot.com/feed/1111001", "links": { "http://push-pub.appspot.com/entry/1111001": { "href": "http://push-pub.appspot.com/entry/1111001", "title": "test", "rel": "alternate", "mime_type": "text/html", "id": "test" } }, "content": "not recieving from google hub", "updated": 1377756592000 }, ... { "title": "test 2", "published": 1377758355000, "id": "http://push-pub.appspot.com/feed/1108004", "links": { "http://push-pub.appspot.com/entry/1108004": { "href": "http://push-pub.appspot.com/entry/1108004", "title": "test 2", "rel": "alternate", "mime_type": "text/html", "id": "test-2" } }, "content": "recieved", "updated": 1377758355000 } ] } {% endprism %}
{% prism markup %} curl https://push.superfeedr.com/ -H 'Accept: application/json' -G -u demo:demo -d'hub.mode=retrieve' -d'hub.callback=http://my.domain.com/webhook/1' {% endprism %}
{% prism markup %} curl https://push.superfeedr.com/ -H 'Accept: application/json' -G -u demo:demo -d'hub.mode=retrieve' -d'hub.callback[endpoint][hostname]=my.domain.com' {% endprism %}
200: the body includes the feed's status and past items.404: you are not subscribed to the feed.422: please check the body as it will include the reason for the failure.- Other HTTP response codes are outlined in the HTTP spec.
When retrieving past RSS content, it is possible to keep the HTTP connection to Superfeedr alive. This ensures new entries are sent directly to your client. To do this, you need to use the stream.superfeedr.com endpoint:
https://stream.superfeedr.com
You will then perform a retrieve call, as outlined above, with the following extra parameter:
| Parameter Name | Note | Value |
|---|---|---|
| wait | required. stream or poll. |
With stream, the past entries will be returned and the HTTP connections will be kept open to show any future entries as well. When using poll, there are two potential responses:
|
{% prism bash %}
curl -G 'https://stream.superfeedr.com?wait=stream&hub.mode=retrieve&hub.callback=http://my.webhook.com/path&format=json'
-u'demo:demo'
{% endprism %}
Superfeedr also supports Server Sent Events (or EventSource). This W3C specification defines a browser-side Javascript API to receive content from a server in the form of events.
{% prism javascript %} var url = "https://stream.superfeedr.com/"; url += "&hub.mode=retrieve"; url += "&hub.topic="; url += "&authorization=";
var source = new EventSource(url); source.addEventListener("notification", function(e) { var notification = JSON.parse(e.data); }); {% endprism %}
Notifications are POST requests that are sent to the callback you specified for the subscription. The body of the response includes the notification, in the format you specified upon subscription. Please check the schema if you need more detail on this.
We consider notifications successful if we can reach your callback and it returns a 200 code. If the notification fails, we will retry three times after five, ten and 15 seconds (these times may vary slightly). If we are not able to notify you after these additional attempts, we will drop the notification and you can use the our retrieve feature, discussed above, to get the missing data.
Additionally, notifications will include the following headers for you to inspect:
<tr>
<th>Header Name</th>
<th>Note</th>
<th>Value</th>
</tr>
</thead>
<tbody>
<tr>
<td>X-Superfeedr-Credits</td>
<td></td>
<td>Your credit balance. It should always decrease by 1 between 2 notifications. If it doesn't, it means you missed a notification.</td>
</tr>
<tr>
<td>X-PubSubHubbub-Callback</td>
<td></td>
<td>Your callback url.</td>
</tr>
<tr>
<td>X-PubSubHubbub-Topic</td>
<td></td>
<td>The topic to which you subscribed. Very useful to easily unsubscribe if you haven't kept track of that subscription.</td>
</tr>
<tr>
<td>Content-Type</td>
<td></td>
<td>Very useful to see how to parse the body.</td>
</tr>
<tr>
<td>X-Superfeedr-Retried</td>
<td>optional</td>
<td>If this header is present, it means we had trouble notification you on previous attempts. You should make sure your app doesn't encounter health issues.</td>
</tr>
</tbody>
You should always use the https endpoints when sending requests to Superfeedr. We also recommend using https for your own endpoints. This guarantees privacy and the integrity of the complete notification (including the headers).
When subscribing to a feed, you should use hub.secret, unless you are using https for your callback URLs. This secret will be used to compute a signature for each notification. You should always make sure these signatures match. You can read more about that here .
Your callback urls should be hard to guess. More importantly, you should use different callbacks for each of your subscriptions. This way, you can quickly identify which feed is involved in each notification, without having to parse the content. It also makes it easier to identify problems using the access logs of your HTTP servers.
This call is mostly used as a debugging tool. It allows you to replay past notifications. The token used to perform this call must have the retrieve right set to true.
https://push.superfeedr.com
<tr>
<th>Parameter Name</th>
<th>Note</th>
<th>Value</th>
</tr>
</thead>
<tbody>
<tr>
<td>hub.mode</td>
<td>required</td>
<td><code>replay</code></td>
</tr>
<tr>
<td>hub.topic</td>
<td>required</td>
<td>The URL of the HTTP resource for which you want the past entries.</td>
</tr>
<tr>
<td>hub.callback</td>
<td>required</td>
<td>The hub.callback parameter of the subscription.</td>
</tr>
<tr>
<td>count</td>
<td>optional</td>
<td>Optional number of items you want to retrieve. Current max is 50 and default is 1.</td>
</tr>
<tr>
<td>async</td>
<td>optional</td>
<td>If set to true, Superfeedr will respond to this very request and issue the notification right <em>after</em>.</td>
</tr>
</tbody>
{% prism markup %} curl https://push.superfeedr.com/ -G -u demo:demo -d'hub.mode=replay' -d'hub.topic=http://push-pub.appspot.com/feed' -d'hub.callback=http://mycallback.tld/ok' {% endprism %}
204: the notification was performed successfully.404: you are not subscribed to the feed.422: please check the body as it will include the reason for the failure.- Other HTTP response codes are outlined in the HTTP spec.
Multiple PubSubHubbub wrappers exist in a variety of languages. All of these should work fine with the Superfeedr endpoint.
You can also check our Github repository for some platform-specific libraries, like our Ruby Rack Gem.