Category: Python

Django: Running management commands inside a Docker container

Post author By masnun
Post date April 23, 2016
3 Comments on Django: Running management commands inside a Docker container

Okay, so we have dockerized our django app and we need to run a manage.py command for some task. How do we do that? Simple, we have to locate the container that runs the django app, login and then run the command.

Locate The Container

It’s very likely that our app uses multiple containers to compose the entire system. For exmaple, I have one container running MySQL, one container running Redis and another running the actual Django app. If we want to run manage.py commands, we have to login to the one that runs Django.

While our app is running, we can find the running docker containers using the docker ps command like this:

$ docker ps
CONTAINER ID        IMAGE                COMMAND                  CREATED             STATUS              PORTS                    NAMES
308f40bba888        crawler_testscript   "/sbin/my_init"          31 hours ago        Up 3 seconds        5000/tcp                 crawler_testscript_1
3a5ccc872215        crawler_web          "bash run_web.sh"        31 hours ago        Up 4 seconds        0.0.0.0:8000-&gt;8000/tcp   crawler_web_1
14f0e260fb2c        redis:latest         "/entrypoint.sh redis"   31 hours ago        Up 4 seconds        0.0.0.0:6379-&gt;6379/tcp   crawler_redis_1
252a7092870d        mysql:latest         "/entrypoint.sh mysql"   31 hours ago        Up 4 seconds        0.0.0.0:3306-&gt;3306/tcp   crawler_mysql_1

$ docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

308f40bba888 crawler_testscript "/sbin/my_init" 31 hours ago Up 3 seconds 5000/tcp crawler_testscript_1

3a5ccc872215 crawler_web "bash run_web.sh" 31 hours ago Up 4 seconds 0.0.0.0:8000->8000/tcp crawler_web_1

14f0e260fb2c redis:latest "/entrypoint.sh redis" 31 hours ago Up 4 seconds 0.0.0.0:6379->6379/tcp crawler_redis_1

252a7092870d mysql:latest "/entrypoint.sh mysql" 31 hours ago Up 4 seconds 0.0.0.0:3306->3306/tcp crawler_mysql_1

In my case, I am using Docker Compose and I know my Django app runs using the crawler_web image. So we note the name of the container. In the above example, that is – crawler_web_1.

Nice, now we know which container we have to login to.

Logging Into The Container

We use the name of the container to login to it, like this:

docker exec -it crawler_web_1 bash

1	docker exec -it crawler_web_1 bash

The command above will connect us to the container and land us on a bash shell. Now we’re ready to run our command.

Running the command

We cd into the directory if necessary and then run the management command.

cd /project
python manage.py <command>

1 2	cd /project python manage.py <command>

Summary

docker ps to list running containers and locate the one
docker exec -it [container_name] bash to login to the bash shell on that container
cd to the django project and run python manage.py [command]

Django Python

Django REST Framework: Remember to disable Web Browsable API in Production

Post author By masnun
Post date April 20, 2016
7 Comments on Django REST Framework: Remember to disable Web Browsable API in Production

So this is what happened – I built an url shortening service at work for internal use. It’s a very basic app – shortens urls and tracks clicks. Two models – URL and URLVisit. URL model contains the full url, slug for the short url, created time etc. URLVisit has information related to the click, like user IP, browser data, click time etc and a ForeignKey to URL as expected.

Two different apps were using this service, one from me, another from a different team. I kept the Web Browsable API so the developers from other teams can try it out easily and they were very happy about it. The only job of this app was url shortening so I didn’t bother building a different home page. When people requested the / page on the domain, I would redirect them directly to /api/.

Things were going really great initially. There was not very heavy load on the service. Roughly 50-100 requests per second. I would call that minimal load. The server also had decent hardware and was running on an EC2 instance from AWS. nginx was on the front while the app was run with uwsgi. Everything was so smooth until it happened. After a month and half, we started noticing very poor performance of the server. Sometimes it was taking up to 40 seconds to respond. I started investigating.

It took me some time to find out what actually happened. By the time it happened, we have shortened more than a million urls. So when someone was visiting /api/url-visit/ – the web browsable api was trying to render the html form. The form allows the user to choose one of the entries from the URL model inside a select (dropdown). Rendering that page was causing usages of 100% cpu and blocking / slowing down other requests. It’s not really DRF’s fault. If I tried to load a million of entries into a select like that, it would crash the app too.

Even worse – remember I added a redirect from the home page, directly to the /api/ url? Search engines (bots) started crawling the urls. As a result the app became extremely slow and often unavailable to nginx. I initially thought, I could stop the search engine crawls by adding some robots.txt or simply by adding authentication to the API. But developers from other teams would still time to time visit the API to try out things and then make the app non responsive. So I did what I had to – I disabled the web browsable API and added a separate documentation demonstrating the use of the API with curl, PHP and Python.

I added the following snippet in my production settings file to only enable JSONRenderer for the API:

REST_FRAMEWORK = {
    'DEFAULT_RENDERER_CLASSES': (
        'rest_framework.renderers.JSONRenderer',
    )
}

REST_FRAMEWORK = {

'DEFAULT_RENDERER_CLASSES': (

'rest_framework.renderers.JSONRenderer',

)

}

Things have become pretty smooth afterwards. I can still enjoy the nice HTML interface locally where there are much fewer items. While on my production servers, there is no web browsable APIs to cause any bottlenecks.

Python

Composition over Inheritance

Post author By masnun
Post date April 19, 2016
7 Comments on Composition over Inheritance

Inheritance

If you know basic OOP, you know what Inheritance is. When one class extends another, the child class inherits the parent class and thus the child class has access to all the variables and methods on the parent class.

class Duck:
    speed = 30

    def fly(self):
        return "Flying at {} kmph".format(self.speed)


class MallardDuck(Duck):
    speed = 20


if __name__ == "__main__":
    duck = Duck()
    print(duck.fly())

    mallard = MallardDuck()
    print(mallard.fly())

class Duck:

speed = 30

def fly(self):

return "Flying at {} kmph".format(self.speed)

class MallardDuck(Duck):

speed = 20

if __name__ == "__main__":

duck = Duck()

print(duck.fly())

mallard = MallardDuck()

print(mallard.fly())

Here the MallardDuck extends Duck and inherits the speed class variable along with the fly method. We override the speedin the child class to suit our needs. When we call fly on the mallard duck, it uses the fly method inherited from the parent. If we run the above code, we will see the following output:

Flying at 30 kmph
Flying at 20 kmph

1 2	Flying at 30 kmph Flying at 20 kmph

This is inheritance in a nutshell.

Composition

Let’s first see an example:

class GmailProvider:
    def send(self, msg):
        return "Sending `{}` using Gmail".format(msg)


class YahooMailProvider:
    def send(self, msg):
        return "Sending `{}` using Yahoo Mail!".format(msg)


class EmailClient:
    email_provider = GmailProvider()
    
    def setup(self):
        return "Initialization and configurations"

    def set_provider(self, provider):
        self.email_provider = provider

    def send_email(self, msg):
        print(self.email_provider.send(msg))


client = EmailClient()
client.setup()

client.send_email("Hello World!")

client.set_provider(YahooMailProvider())
client.send_email("Hello World!")

class GmailProvider:

def send(self, msg):

return "Sending `{}` using Gmail".format(msg)

class YahooMailProvider:

def send(self, msg):

return "Sending `{}` using Yahoo Mail!".format(msg)

class EmailClient:

email_provider = GmailProvider()

def setup(self):

return "Initialization and configurations"

def set_provider(self, provider):

self.email_provider = provider

def send_email(self, msg):

print(self.email_provider.send(msg))

client = EmailClient()

client.setup()

client.send_email("Hello World!")

client.set_provider(YahooMailProvider())

client.send_email("Hello World!")

Here we’re not implementing the email sending functionality directly inside the EmailClient. Rather, we’re storing a type of email provider in the email_provider variable and delegating the responsibility of sending the email to this provider. When we have to send_email, we call the send method on the email_provider. Thus we’re composing the functionality of the EmailClient by sticking composable objects together. We can also swap out the email provider any time we want, by passing it a new provider to the set_provider method.

Composition over Inheritance

Let’s implement the above EmailClient using inheritance.

class EmailClient:
    def setup(self):
        return "Initializions and configurations!"

    def send_email(self, msg):
        raise NotImplementedError("Use a subclass!")


class GmailClient(EmailClient):
    def send_email(self, msg):
        return "Sending `{}` from Gmail Client".format(msg)


class YahooMailClient(EmailClient):
    def send_email(self, msg):
        return "Sending `{}` from YMail! Client".format(msg)


client = GmailClient()
client.setup()

client.send_email("Hello!")

# If we want to send using Yahoo, we have to construct a new client

yahoo_client = YahooMailClient()
yahoo_client.setup()

yahoo_client.send_email("Hello!")

class EmailClient:

def setup(self):

return "Initializions and configurations!"

def send_email(self, msg):

raise NotImplementedError("Use a subclass!")

class GmailClient(EmailClient):

def send_email(self, msg):

return "Sending `{}` from Gmail Client".format(msg)

class YahooMailClient(EmailClient):

def send_email(self, msg):

return "Sending `{}` from YMail! Client".format(msg)

client = GmailClient()

client.setup()

client.send_email("Hello!")

# If we want to send using Yahoo, we have to construct a new client

yahoo_client = YahooMailClient()

yahoo_client.setup()

yahoo_client.send_email("Hello!")

Here, we created a base class EmailClient which has the setup method. Then we extended the class to create GmailClient and YahooMailClient. Things got interesting when we wanted to start sending emails using Yahoo instead of Gmail. We had to create a new instance of YahooMailClient for that purpose. The initially created client was no longer useful for us since it only knows how to send emails through Gmail.

This is why composition is often favoured over inheritance. By delegating the responsibility to the different composable parts, we form loose coupling. We can swap out those components easily when needed. We can also inject them as dependencies using dependency injection. But with inheritance, things get tightly coupled and not easily swappable.

Recent Posts

Recent Comments

Archives

Categories