Category: Python

Categories
Python

Docker Workflow of Iron.io: Python Workers

If you’re planning to use Iron.io with Python, their new docker workflow is pretty cool. In this blog post, I shall walk you through the setup.

Setup Docker

The first step would be to install Docker. Here’s a pretty nice installation guide from Digital Ocean: https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-getting-started 🙂

If you’re not using Ubuntu (eg. Windows or OS X), you might consider using Ubuntu in a virtualbox. I usually use Vagrant to do that.

If you’re using another Linux distro, please go through the official docker docs, I’m sure it’s quite easy 🙂

Once the setup is complete, run the following command to confirm that Docker has successfully installed and currently running:

You would see something like this:

Installing PIP Dependencies

Now it’s the time to install the dependencies for our projects. Change directory to our project root and use the requirements.txt file to install the packages. Please note we shall install them into a separate directory than the usual system path.

Here we have asked docker to run a new container based on the “iron/images:python-2.7” image. Inside the container we run the pip command. The “-t” flag would install packages in a separate target directory.

Since this is the first time we’re using this image, docker will need to pull it first. Then it will continue to install the dependencies. Once it’s done, feel free to examine the packages directory.

Locally Running Workers

The command is similar, except we have the packages installed in “packages” directory. So we need to set the PYTHONPATH env variable.

That would work 🙂

Bundling workers

To bundle the workers, we need to first zip everything:

You must remember to zip everything in, including the packages directory.

Now, we can start using the iron cli. In case you don’t have it installed, it’s easy:

Now let’s upload the zip file:

I think the command is pretty self explanatory. However, for the env argument, you need a file named “iron.json” which would contain the project id and tokens for your different environments. Here’s a sample:

Also note, we’re using PYTHONPATH=”packages” before invoking Python.

If everything goes right, your worker will be packaged and uploaded. Try running it to see how it goes.

Categories
Django Python

Python/Django: Running multiple commands in subprocesses

Here’s the scneario:

I have 4-5 different commands which I need to run every time I want to test my app. So I decided to write one command to rule them all. I run this one command and it runs the others using the subprocess module. It stores the processes created in a list. Then it goes to sleep inside a while loop waiting for a KeyboardInterrupt exception (which is fired when you press Ctrl + C – that is you send an interrupt signal from your keyboard). When this exception occurs, we iterate through the process ids we stored and kill them all. Simple, No?

Talk is cheap, here’s the code:

So this is how it looks on my machine:

Screen Shot 2015-09-29 at 5.16.44 PM

🙂

Categories
Linux Python

Deploying a Flask app on Ubuntu: uWSGI, nginx and Upstart

Let’s get straight to the point. I have a Flask app that I want to run on Ubuntu. I shall use uwsgi and nginx. Upstart will be used to monitor the process and restart it if it’s killed or dies an untimely death.

Running the uWSGI app with nginx

It’s very easy to run Flask apps with uwsgi, just cd into the directory where your main app file exists and pass the uwsgi module name. If your main app file is called “main.py” and if your Flask app instance is called “app” inside main.py, then your uWSGI module name would be “main:app” from the directory where main.py resides.

Now, we need to configure nginx:

Here, we first need to define an upstream. We define the “flaskapp” upstream for our app. Then inside the location block, we use it as the source of our uwsgi_pass directive. It would tell nginx to connect to the uwsgi server we defined and serve as a bridge between the internet and the flask uwsgi app.

Upstarting the uWSGI process

Now, what happens if our uWSGI process dies? nginx would no longer be able to connect to the app and throw a “bad gateway” error. So we need to make sure that if the process dies from an error or somehow gets killed, it is restarted automatically. There are a few tools which can do this. Supervisord is the most popular one, Circus is also very nice. However, I like to use Upstart for cases like this. Upstart comes with Ubuntu and used by the OS for many of it’s own services. It’s easy to use, already available, reliable and most importantly – it allows me to create a service too!

To create our upstart config, we cd into “/etc/init” and create a new file named “flaskapp.conf”. Then we put the following contents in there:

Here, we’re telling Upstart to launch our app on startup and respawn whenever the process dies. We put our commands in a script block. We have defined our service, now time to fire it up. If you have any uwsgi process run with the same config, first kill them. Then run this command to start the service:

Right, you can use the usual “start|stop|restart|status” service commands with it. The service name is derived from the file name we used in “/etc/init”. The log of the process would be found in “/var/log/upstart/flaskapp.log”.

From Upstart to Systemd

Ubuntu is moving away from Upstart to systemd. Most of my apps use Ubuntu 14.04 LTS so I can still use Upstart fine. However, if your version of Ubuntu doesn’t ship Upstart, you can use systemd for similar tasks. Here’s the link: systemd for upstart users