Sep 7 2012

A Live Feed with Drupal, node.js and socket.io

Shortly I needed to create a real time live feed for the redbullstormchase.com page. While there where realtime solutions around before node.js arrived, I like the idea to use javascript and the light-weight approach of it. Furthermore it works great with the socket.io library, why you do not have to worry about cross-browser pains. While there is even already a Drupal module, which probably can do everything and even more, I do not like using such modules, because they are just too much and then often do not do what you want unless you start fixing it. So I started from scratch, which is also more fun in my opinion.

So what needed to be done was:

Twitter Streaming Api to get live updates from several twitter accounts
If Tweet comes from Instagram, fetch the image
Fallback to Twitter Search Api if live tweets are missed for any reason
Vimeo Updates
Refresh clients if any new Content arrives

The node.js community is probably not as big as other communities that have been around for a longer time, but there are many good modules that are really helpful. What I used was:

Immortal ntwitter (Ntwitter that reconnects following twitter stream concepts.)
Request (easy http requests + oauth and more)
Socket.io (painless socket connections)
Cheerio (light-weight dom parsing)

So for the twitter streaming api you can easily fetch new tweets like this: /** * Twitter */ var itwitter = require(‘immortal-ntwitter’);

var twit = itwitter.create({ consumerkey: ‘CONSUMERKEY’, consumersecret: ‘CONSUMERSECRET’, accesstokenkey: ‘ANOTHERKEY’, accesstokensecret: ‘ANOTHERSECRET’ });

twit.immortalStream(‘statuses/filter’, {‘follow’:[‘911’]}, function(stream) { stream.on(‘data’, function (data) { processTweet(data); }); });

For the Twitter Fallback via search api I used something like this: function twitterFallback () { var follow = [‘britneyspears’]; var search = “”; for (var i=0; i < follow.length; i++) { follow[i] = ‘from:’ + follow[i]; };
search = follow.join(“ OR ”);

twit.search(search, {"include_entities":"true", "result_type":"recent", "rpp":"50"}, function(err, data) {

    if(err > 200) return;
    if(data == undefined) return;

    var timeout = 100; //timeout because of innodb lock trouble
    for (var i = data.results.length - 1; i >= 0; i--){
        setTimeout(processTweet(data.results[i]), i * timeout);
    };
});

}

For the fetching of instagram images I wrote a small module that does this, if the Twitter source is instagram:

var http = require(“http”); var request = require(“request”);

module.exports = { getInsta: getInsta }

As you can see parsing the dom is really easy, because you can use jQuery like syntax. No RegEx needed here..

With vimeo, you have to generate all access tokens for your app step by step. I also wrote a module for that, if someone is interested, just post a comment…

So now on to the next part, posting the data to the Drupal endpoint: var options = { headers: {‘content-type’ : ‘application/x-www-form-urlencoded’}, uri: baseurl + ‘/your/endpoint’, body:“tweet=” + sTweet + “&token=” + token, method:‘POST’ }

request(options, function(error, response, body) {                         
    body = JSON.parse(body);
    if(body != undefined && body.refresh && socket) refresh();
});

This makes a POST Request to your Drupal endpoint. It is important to set the request headers right. Drupal checks if the data is already in the system (no refresh) or if the content is new (refresh all clients) and sends the answer as response.

So what you would have to is write a small Drupal module that defines an endpoint that you can post new content to (remember: safety first…). What’s left is the js part that refreshes the Drupal content if necessary. I won’t get into detail on this one, but if you have set up socket.io and are familiar with the Drupal Ajax System it should work out for you.

This should not be a step by step guide, just a rough overview of a possible solution.

A Live Feed with Drupal, node.js and socket.io

Let's connect 👋