Year: 2015

Python/Django: Running multiple commands in subprocesses

Post author By masnun
Post date September 29, 2015
4 Comments on Python/Django: Running multiple commands in subprocesses

Here’s the scneario:

I have 4-5 different commands which I need to run every time I want to test my app. So I decided to write one command to rule them all. I run this one command and it runs the others using the subprocess module. It stores the processes created in a list. Then it goes to sleep inside a while loop waiting for a KeyboardInterrupt exception (which is fired when you press Ctrl + C – that is you send an interrupt signal from your keyboard). When this exception occurs, we iterate through the process ids we stored and kill them all. Simple, No?

Talk is cheap, here’s the code:

from django.core.management.base import BaseCommand

from subprocess import Popen
from sys import stdout, stdin, stderr
import time, os, signal


class Command(BaseCommand):
    help = 'Run all commands'
    commands = [
        'redis-server',
        'python manage.py spider',
        'python manage.py schedule',
        'python manage.py postprocess',
        'python manage.py runserver'

    ]

    def handle(self, *args, **options):
        proc_list = []

        for command in self.commands:
            print "$ " + command
            proc = Popen(command, shell=True, stdin=stdin, stdout=stdout, stderr=stderr)
            proc_list.append(proc)

        try:
            while True:
                time.sleep(10)
        except KeyboardInterrupt:
            for proc in proc_list:
                os.kill(proc.pid, signal.SIGKILL)

from django.core.management.base import BaseCommand

from subprocess import Popen

from sys import stdout, stdin, stderr

import time, os, signal

class Command(BaseCommand):

help = 'Run all commands'

commands = [

'redis-server',

'python manage.py spider',

'python manage.py schedule',

'python manage.py postprocess',

'python manage.py runserver'

]

def handle(self, *args, **options):

proc_list = []

for command in self.commands:

print "$ " + command

proc = Popen(command, shell=True, stdin=stdin, stdout=stdout, stderr=stderr)

proc_list.append(proc)

try:

while True:

time.sleep(10)

except KeyboardInterrupt:

for proc in proc_list:

os.kill(proc.pid, signal.SIGKILL)

So this is how it looks on my machine:

Screen Shot 2015-09-29 at 5.16.44 PM

🙂

Linux Python

Deploying a Flask app on Ubuntu: uWSGI, nginx and Upstart

Post author By masnun
Post date July 31, 2015

Let’s get straight to the point. I have a Flask app that I want to run on Ubuntu. I shall use uwsgi and nginx. Upstart will be used to monitor the process and restart it if it’s killed or dies an untimely death.

Running the uWSGI app with nginx

It’s very easy to run Flask apps with uwsgi, just cd into the directory where your main app file exists and pass the uwsgi module name. If your main app file is called “main.py” and if your Flask app instance is called “app” inside main.py, then your uWSGI module name would be “main:app” from the directory where main.py resides.

cd /home/dev/awesome-flask-app
uwsgi --socket 127.0.0.1:8080 -w main:app

1 2	cd /home/dev/awesome-flask-app uwsgi --socket 127.0.0.1:8080 -w main:app

Now, we need to configure nginx:

upstream flaskapp {
        server 127.0.0.1:8080;
 }


server {
    listen 80;

    location / {

            include            uwsgi_params;
            uwsgi_pass         flaskapp;

            proxy_redirect     off;
            proxy_set_header   Host $host;
            proxy_set_header   X-Real-IP $remote_addr;
            proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header   X-Forwarded-Host $server_name;

        }

}

upstream flaskapp {

server 127.0.0.1:8080;

}

server {

listen 80;

location / {

include uwsgi_params;

uwsgi_pass flaskapp;

proxy_redirect off;

proxy_set_header Host $host;

proxy_set_header X-Real-IP $remote_addr;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

proxy_set_header X-Forwarded-Host $server_name;

}

Here, we first need to define an upstream. We define the “flaskapp” upstream for our app. Then inside the location block, we use it as the source of our uwsgi_pass directive. It would tell nginx to connect to the uwsgi server we defined and serve as a bridge between the internet and the flask uwsgi app.

Upstarting the uWSGI process

Now, what happens if our uWSGI process dies? nginx would no longer be able to connect to the app and throw a “bad gateway” error. So we need to make sure that if the process dies from an error or somehow gets killed, it is restarted automatically. There are a few tools which can do this. Supervisord is the most popular one, Circus is also very nice. However, I like to use Upstart for cases like this. Upstart comes with Ubuntu and used by the OS for many of it’s own services. It’s easy to use, already available, reliable and most importantly – it allows me to create a service too!

To create our upstart config, we cd into “/etc/init” and create a new file named “flaskapp.conf”. Then we put the following contents in there:

start on runlevel [2345]
stop on runlevel [016]

respawn

script
    cd /home/dev/awesome-flask-app
    exec uwsgi --socket 127.0.0.1:8080 -w main:app
end script

start on runlevel [2345]

stop on runlevel [016]

respawn

script

cd /home/dev/awesome-flask-app

exec uwsgi --socket 127.0.0.1:8080 -w main:app

end script

Here, we’re telling Upstart to launch our app on startup and respawn whenever the process dies. We put our commands in a script block. We have defined our service, now time to fire it up. If you have any uwsgi process run with the same config, first kill them. Then run this command to start the service:

sudo service flaskapp start

1	sudo service flaskapp start

Right, you can use the usual “start|stop|restart|status” service commands with it. The service name is derived from the file name we used in “/etc/init”. The log of the process would be found in “/var/log/upstart/flaskapp.log”.

From Upstart to Systemd

Ubuntu is moving away from Upstart to systemd. Most of my apps use Ubuntu 14.04 LTS so I can still use Upstart fine. However, if your version of Ubuntu doesn’t ship Upstart, you can use systemd for similar tasks. Here’s the link: systemd for upstart users

Python

Iron Worker: Processing SendGrid Inbound Parse Data

Post author By masnun
Post date July 28, 2015

If you use SendGrid, you know that they provide this awesome Inbound Parse APIs which allows users to catch emails sent to their domains. When there’s a new email, Sendgrid would make a POST request to your webhook endpoint. On the other hand, I love Iron Worker webhooks. When you POST data to their webhooks, the data is passed to a worker and processed in queue. It helps keep things simpler since I don’t need to setup a separate web server for accepting the data; however, keep in mind that I do make sure to use data protection services like the Venyu cybersecurity services at all times.

By now, you can probably guess my intention. I would like to make Sendgrid POST incoming emails to a Iron Worker webhook and then grab the data. But there’s one catch – Iron.io stores the incoming data in a text file and expect us to parse the text file to collect our data. Since very often the data is JSON, most client libraries take the payload, decode it from JSON and provide us with native data structures. I use Python, so I usually get Python dictionary. However, in case of Sendgrid Inbound APIs, the data is not JSON encoded. It’s simple multi-part form data. So the Python client I use, it fails to process the data and I don’t get anything for the payload.

So I went ahead and modified the payload processing part from the client library. Now this is what I have for getting the payload:

def get_iron_args():
    arguments = {}
    for i in range(len(sys.argv)):
        if sys.argv[i] == "-id":
            arguments['task_id'] = sys.argv[i + 1]
        if sys.argv[i] == "-d":
            arguments['dir'] = sys.argv[i + 1]
        if sys.argv[i] == "-payload":
            arguments['payload_file'] = sys.argv[i + 1]
        if sys.argv[i] == "-config":
            arguments['config_file'] = sys.argv[i + 1]

        if os.getenv('TASK_ID'): arguments['task_id'] = os.getenv('TASK_ID')
        if os.getenv('TASK_DIR'): arguments['dir'] = os.getenv('TASK_DIR')
        if os.getenv('PAYLOAD_FILE'): arguments['payload_file'] = os.getenv('PAYLOAD_FILE')
        if os.getenv('CONFIG_FILE'): arguments['config_file'] = os.getenv('CONFIG_FILE')

        if 'payload_file' in arguments and file_exists(arguments['payload_file']):
            f = open(arguments['payload_file'], "r")
            try:
                content = f.read()
                f.close()
                arguments['payload'] = content
            except Exception, e:
                print "Couldn't parse IronWorker payload  %s" % e

    return arguments


def get_raw_payload():
    args = get_iron_args()
    return args['payload']

def get_iron_args():

arguments = {}

for i in range(len(sys.argv)):

if sys.argv[i] == "-id":

arguments['task_id'] = sys.argv[i + 1]

if sys.argv[i] == "-d":

arguments['dir'] = sys.argv[i + 1]

if sys.argv[i] == "-payload":

arguments['payload_file'] = sys.argv[i + 1]

if sys.argv[i] == "-config":

arguments['config_file'] = sys.argv[i + 1]

if os.getenv('TASK_ID'): arguments['task_id'] = os.getenv('TASK_ID')

if os.getenv('TASK_DIR'): arguments['dir'] = os.getenv('TASK_DIR')

if os.getenv('PAYLOAD_FILE'): arguments['payload_file'] = os.getenv('PAYLOAD_FILE')

if os.getenv('CONFIG_FILE'): arguments['config_file'] = os.getenv('CONFIG_FILE')

if 'payload_file' in arguments and file_exists(arguments['payload_file']):

f = open(arguments['payload_file'], "r")

try:

content = f.read()

f.close()

arguments['payload'] = content

except Exception, e:

print "Couldn't parse IronWorker payload %s" % e

return arguments

def get_raw_payload():

args = get_iron_args()

return args['payload']

Now, we have the payload. But it’s still just plain text. We need to process it to get the different POST fields. Luckily, we can do that easily with the cgi library. This is what I use:

def parse_post_data(payload):
    """
    Transforms raw text into POST dictionary

    :param payload: the raw text
    :return dictionary: parsed data
    """
    body_as_file = StringIO.StringIO(payload)
    pdict = {'boundary': 'xYzZY'}
    httpbody = cgi.parse_multipart(body_as_file, pdict)

    return httpbody

def parse_post_data(payload):

"""

Transforms raw text into POST dictionary

:param payload: the raw text

:return dictionary: parsed data

"""

body_as_file = StringIO.StringIO(payload)

pdict = {'boundary': 'xYzZY'}

httpbody = cgi.parse_multipart(body_as_file, pdict)

return httpbody

Finally, now this is what the worker looks like:

from utils import parse_post_data, get_raw_payload

data = parse_post_data(payload)
print data

from utils import parse_post_data, get_raw_payload

data = parse_post_data(payload)

print data

Recent Posts

Recent Comments

Archives

Categories