Author: masnun

Django REST Framework: Remember to disable Web Browsable API in Production

Post author By masnun
Post date April 20, 2016
7 Comments on Django REST Framework: Remember to disable Web Browsable API in Production

So this is what happened – I built an url shortening service at work for internal use. It’s a very basic app – shortens urls and tracks clicks. Two models – URL and URLVisit. URL model contains the full url, slug for the short url, created time etc. URLVisit has information related to the click, like user IP, browser data, click time etc and a ForeignKey to URL as expected.

Two different apps were using this service, one from me, another from a different team. I kept the Web Browsable API so the developers from other teams can try it out easily and they were very happy about it. The only job of this app was url shortening so I didn’t bother building a different home page. When people requested the / page on the domain, I would redirect them directly to /api/.

Things were going really great initially. There was not very heavy load on the service. Roughly 50-100 requests per second. I would call that minimal load. The server also had decent hardware and was running on an EC2 instance from AWS. nginx was on the front while the app was run with uwsgi. Everything was so smooth until it happened. After a month and half, we started noticing very poor performance of the server. Sometimes it was taking up to 40 seconds to respond. I started investigating.

It took me some time to find out what actually happened. By the time it happened, we have shortened more than a million urls. So when someone was visiting /api/url-visit/ – the web browsable api was trying to render the html form. The form allows the user to choose one of the entries from the URL model inside a select (dropdown). Rendering that page was causing usages of 100% cpu and blocking / slowing down other requests. It’s not really DRF’s fault. If I tried to load a million of entries into a select like that, it would crash the app too.

Even worse – remember I added a redirect from the home page, directly to the /api/ url? Search engines (bots) started crawling the urls. As a result the app became extremely slow and often unavailable to nginx. I initially thought, I could stop the search engine crawls by adding some robots.txt or simply by adding authentication to the API. But developers from other teams would still time to time visit the API to try out things and then make the app non responsive. So I did what I had to – I disabled the web browsable API and added a separate documentation demonstrating the use of the API with curl, PHP and Python.

I added the following snippet in my production settings file to only enable JSONRenderer for the API:

REST_FRAMEWORK = {
    'DEFAULT_RENDERER_CLASSES': (
        'rest_framework.renderers.JSONRenderer',
    )
}

REST_FRAMEWORK = {

'DEFAULT_RENDERER_CLASSES': (

'rest_framework.renderers.JSONRenderer',

)

}

Things have become pretty smooth afterwards. I can still enjoy the nice HTML interface locally where there are much fewer items. While on my production servers, there is no web browsable APIs to cause any bottlenecks.

Python

Composition over Inheritance

Post author By masnun
Post date April 19, 2016
7 Comments on Composition over Inheritance

Inheritance

If you know basic OOP, you know what Inheritance is. When one class extends another, the child class inherits the parent class and thus the child class has access to all the variables and methods on the parent class.

class Duck:
    speed = 30

    def fly(self):
        return "Flying at {} kmph".format(self.speed)


class MallardDuck(Duck):
    speed = 20


if __name__ == "__main__":
    duck = Duck()
    print(duck.fly())

    mallard = MallardDuck()
    print(mallard.fly())

class Duck:

speed = 30

def fly(self):

return "Flying at {} kmph".format(self.speed)

class MallardDuck(Duck):

speed = 20

if __name__ == "__main__":

duck = Duck()

print(duck.fly())

mallard = MallardDuck()

print(mallard.fly())

Here the MallardDuck extends Duck and inherits the speed class variable along with the fly method. We override the speedin the child class to suit our needs. When we call fly on the mallard duck, it uses the fly method inherited from the parent. If we run the above code, we will see the following output:

Flying at 30 kmph
Flying at 20 kmph

1 2	Flying at 30 kmph Flying at 20 kmph

This is inheritance in a nutshell.

Composition

Let’s first see an example:

class GmailProvider:
    def send(self, msg):
        return "Sending `{}` using Gmail".format(msg)


class YahooMailProvider:
    def send(self, msg):
        return "Sending `{}` using Yahoo Mail!".format(msg)


class EmailClient:
    email_provider = GmailProvider()
    
    def setup(self):
        return "Initialization and configurations"

    def set_provider(self, provider):
        self.email_provider = provider

    def send_email(self, msg):
        print(self.email_provider.send(msg))


client = EmailClient()
client.setup()

client.send_email("Hello World!")

client.set_provider(YahooMailProvider())
client.send_email("Hello World!")

class GmailProvider:

def send(self, msg):

return "Sending `{}` using Gmail".format(msg)

class YahooMailProvider:

def send(self, msg):

return "Sending `{}` using Yahoo Mail!".format(msg)

class EmailClient:

email_provider = GmailProvider()

def setup(self):

return "Initialization and configurations"

def set_provider(self, provider):

self.email_provider = provider

def send_email(self, msg):

print(self.email_provider.send(msg))

client = EmailClient()

client.setup()

client.send_email("Hello World!")

client.set_provider(YahooMailProvider())

client.send_email("Hello World!")

Here we’re not implementing the email sending functionality directly inside the EmailClient. Rather, we’re storing a type of email provider in the email_provider variable and delegating the responsibility of sending the email to this provider. When we have to send_email, we call the send method on the email_provider. Thus we’re composing the functionality of the EmailClient by sticking composable objects together. We can also swap out the email provider any time we want, by passing it a new provider to the set_provider method.

Composition over Inheritance

Let’s implement the above EmailClient using inheritance.

class EmailClient:
    def setup(self):
        return "Initializions and configurations!"

    def send_email(self, msg):
        raise NotImplementedError("Use a subclass!")


class GmailClient(EmailClient):
    def send_email(self, msg):
        return "Sending `{}` from Gmail Client".format(msg)


class YahooMailClient(EmailClient):
    def send_email(self, msg):
        return "Sending `{}` from YMail! Client".format(msg)


client = GmailClient()
client.setup()

client.send_email("Hello!")

# If we want to send using Yahoo, we have to construct a new client

yahoo_client = YahooMailClient()
yahoo_client.setup()

yahoo_client.send_email("Hello!")

class EmailClient:

def setup(self):

return "Initializions and configurations!"

def send_email(self, msg):

raise NotImplementedError("Use a subclass!")

class GmailClient(EmailClient):

def send_email(self, msg):

return "Sending `{}` from Gmail Client".format(msg)

class YahooMailClient(EmailClient):

def send_email(self, msg):

return "Sending `{}` from YMail! Client".format(msg)

client = GmailClient()

client.setup()

client.send_email("Hello!")

# If we want to send using Yahoo, we have to construct a new client

yahoo_client = YahooMailClient()

yahoo_client.setup()

yahoo_client.send_email("Hello!")

Here, we created a base class EmailClient which has the setup method. Then we extended the class to create GmailClient and YahooMailClient. Things got interesting when we wanted to start sending emails using Yahoo instead of Gmail. We had to create a new instance of YahooMailClient for that purpose. The initially created client was no longer useful for us since it only knows how to send emails through Gmail.

This is why composition is often favoured over inheritance. By delegating the responsibility to the different composable parts, we form loose coupling. We can swap out those components easily when needed. We can also inject them as dependencies using dependency injection. But with inheritance, things get tightly coupled and not easily swappable.

Python

Python: pyenv, pyvenv, virtualenv – What’s the difference?

Post author By masnun
Post date April 10, 2016
7 Comments on Python: pyenv, pyvenv, virtualenv – What’s the difference?

So I see questions around these terms very often in our growing Python Bangladesh community. Most of the times beginners are confused about what is what. I hope I can refer to this blog post to explain the similarities and differences.

pyenv

Have you ever wanted to test your code against multiple versions of Python? Or just wanted to install a newer version of Python without affecting your existing version? May be you heard about PyPy a lot and want to install it on your machine?

If you did, then pyenv is the perfect tool for you. It allows you to easily install multiple copies and multiple flavors of the Python interpreter. So you can not only install different versions of CPython, you can also install PyPy, Jython, Stackless Python and their different versions.

The tool provides a nice command line tool to easily swap out the global python interpreter. It also allows to define per application python version. You can use it’s local command or directly mention a python version in a file named .python-version under a directory and for that directory and it’s children, the mentioned version will be used.

Trust me, this project is awesome. I use it to switch between Python 2 and 3 on my local machine. I also use it often on servers to quickly install any flavor/version of Python. Do check out their docs, you will love it.

pyvenv & virtualenv

pyvenv and virtualenv allow you to create virtual environments so we can isolate our project dependencies. Why are they helpful? Say for example, you have one project which uses Django 1.6 still while your newer projects start with 1.9. When you install one version of Django, it replaces the other one, right? Virtual environments can rescue us from such situation. From the official docs:

A virtual environment (also called a venv) is a Python environment such that the Python interpreter, libraries and scripts installed into it are isolated from those installed in other virtual environments, and (by default) any libraries installed in a “system” Python, i.e. one which is installed as part of your operating system.

When we create a new virtual environment, it creates an isolated environment with it’s own local interepreter linked to it’s own libraries/scripts paths. So when we use this local interpreter, it loads the libraries from the local environment. If it can’t find one locally, then tries to locate that library in the parent/system environment.

Please note, these tools do not compile/install new Python interpreters. They simply create “virtual environments” on top of an installed Python version. Say, I have Python 3.5 installed on my machine and created virtual environments for this version. Then these environments would also have local copies of Python 3.5, except their environment paths would point to different locations. It’s like we’re copying the main interpreter to a new location and then making it use a different path to load libraries and packages.

virtualenv is often the most popular choice for creating the virtual environments. It has been around for a long period of time, it supports Python versions from 2.6 up to the latest 3.5. But it’s not something built into the standard Python distribution. You have to install it from the PyPi.

pyvenv comes with Python standard distribution from version 3.4. There is also a venv module in the standard library which allows us to access this functionality programmatically. We can find more details here: https://docs.python.org/3/library/venv.html.

Summary

pyenv – A Python version manager. Installs different versions and flavors of Python interpreters.

pyvenv – A tool to create isolated virtual environments from a Python interpreter. Ships with Python from 3.4.

virtualenv – Creates virtual environments, available in PyPi.

So pyvenv is comparable to virtualenv while pyenv is a totally different kind of tool.

Recent Posts

Recent Comments

Archives

Categories