Author: masnun

UPDATE: Download the improved version of the application. I have made some code changes after the blog post got posted to reddit. The source code is now more readable and the app easily configurable.

---

I am amazed to see the implementation of cron jobs on Google App Engine. I was hosting my twitter bots on my paid shared hosting. My scripts were written in php and they were run via cron jobs set from my cPanel. Of course the cPanel cron UI is now quite familiar to me, to us who use it very often. But as a matter of fact, I can’t run per minute cron on my host. I should have a minimum interval of 15 minutes between two execution of the same cron job. I was amazed when I first discovered that Google App Engine lets me run cron every minute. Since they calculate my usage in different types of quotas, they hardly care about how often you schedule your cron. So, I decided to port my twitter bots to GAE and Python.

The idea was pretty simple. I already had a Yahoo! Pipe setup which mashes up multiple RSS feeds into one. My app will be collecting data from the pipe in JSON format. Then it’ll check the last tweeted entry and tweet the newer entries. Pretty straightforward algorithm. I registered a new application for my bots and started developing it on my local machine.

I integrated the u.nu url shortening service with their API quite easily. Though Python 2.6.2 has a built in module for JSON handling, Python 2.5 doesn’t have it. So, I had to install simplejson for JSON decoding. Later, I moved on to finding a good twitter client for my app. I first tried python-twitter. I have previously used it and it worked fine. But the application was generating some error messages when I integrated into GAE. I was puzzled and tried to find if there was something wrong in my coding. After about an hour, I concluded that my code was okay! There’s something wrong with the client. A bit of googling revealed that “python-twitter” doesn’t work on GAE without any hacks :'(. I had to find an alternative and I chose tweepy for that. It was working fine with the app.

Finally after 3 hours, the app was ready and it’s working! GAE doesn’t permit long execution period for a process, so I had to decrease the amount of tweets per request. But I covered that up by setting a more frequent cron job :). I would try the new taskqueue API soon to see if that has any help for my app.

My twitter bots have proved handy in the past and it feels great that I can host them on a scalable and efficient platform like Google App Engine — without paying a single dollar per month! 😀

Download Source : http://masnun.googlecode.com/files/Twitter_BOT_GAE.zip 🙂

To develop a Jabber / Google Talk Bot, we first need to enable the XMPP inbound service for the application. Just add the lines:

to your app.yaml file. That would activate the service for our application. Have a look at my application’s app.yaml file:

Now, we have the app configured, we need to do the actual coding. Every GAE app has it’s own IM id: [application_id]@appspot.com 🙂 There are other extensive IDs also available but for this task we will use this format. So, my app has the ID: masnun-test@appspot.com as it’s Jabber / Google Talk ID. If the user adds this ID to his/her contact list, it’ll auto accept the friend invitation and appear online. Now when he/she sends a message, the message is sent as a POST request to the url: /_ah/xmpp/message/chat/ . This URL is not publicly accessible without admin rights. Our app can handle this URL like any other URL. So we need to setup a request handler for any POST request to this URL.

GAE has a nice API for it’s XMPP service which lets us build a Message object directly from the POSTed data. We will do that. Extract the message parts and send a response back. Here is source code for “masnun.py”:

On php, I just use json_encode() and json_decode() to handle JSON data with ease. Python has a built in module named “json” for the same purpose. Here’s the link for the official documentation: http://docs.python.org/library/json.html 🙂 The documentation has pretty examples. Still, I wanted to try things out by myself. I used Python to parse the JSON output of a certain Yahoo! Pipes feed :

Output: