Year: 2015

Python

Homebrew & Pyenv: Installing PyQT5 with Python3 on OSX

Post author By masnun
Post date December 26, 2015
3 Comments on Homebrew & Pyenv: Installing PyQT5 with Python3 on OSX

There was a time back in 2014 and earlier when PyQT5 installation was not straightforward and needed manual compilation. When searching on Google, still those posts come up on top results. But nothing to worry about, things have changed – it’s now quite simple.

Installation

If you are not already using Homebrew, you should start using it. Once Homebrew is installed, let’s install PyQT5 with this single command:

brew install pyqt5

1	brew install pyqt5

Verifying Installation

Let’s take a sample PyQT5 code as example and run it. For examples, I usually pick one up from the excellent PyQT tutorials on zetcode.com. Here’s one:

#!/usr/bin/python3
# -*- coding: utf-8 -*-

"""
ZetCode PyQt5 tutorial

In this example, we create a simple
window in PyQt5.

author: Jan Bodnar
website: zetcode.com
last edited: January 2015
"""

import sys
from PyQt5.QtWidgets import QApplication, QWidget


if __name__ == '__main__':

    app = QApplication(sys.argv)

    w = QWidget()
    w.resize(250, 150)
    w.move(300, 300)
    w.setWindowTitle('Simple')
    w.show()

    sys.exit(app.exec_())

#!/usr/bin/python3

# -*- coding: utf-8 -*-

"""

ZetCode PyQt5 tutorial

In this example, we create a simple

window in PyQt5.

author: Jan Bodnar

website: zetcode.com

last edited: January 2015

"""

import sys

from PyQt5.QtWidgets import QApplication, QWidget

if __name__ == '__main__':

app = QApplication(sys.argv)

w = QWidget()

w.resize(250, 150)

w.move(300, 300)

w.setWindowTitle('Simple')

w.show()

sys.exit(app.exec_())

Run it using:

python3 filename.py

1	python3 filename.py

If you get a nice looking small window – it worked!

Integrating with Pyenv

I am a big fan of pyenv and use it for running different versions and flavours of Python. If you use pyenv too, chances are you have your own version of Python installed through it. However, the brew formula that installs PyQT5 depends on another formula python3 – homebrew’s own Python 3 installation. When we install PyQT5, this formula is used to install the bindings, so the bindings are only available to this particular Python 3 installation and unavailable to our pyenv versions.

We will discuss two potential solutions to this issue.

Switching to `system`

One simple work around is to use the Python 3 version installed by Homebrew. We can ask pyenv to switch to the system version whenever we’re doing PyQT5 development.

pyenv global system
python3 filename.py

1 2	pyenv global system python3 filename.py

We can create an alias to quickly switch between Python versions. I have this in my .zshrc:

alias py2="pyenv global 2.7.8"
alias py3="pyenv global 3.5.0"
alias pysys="pyenv global system"

alias py2="pyenv global 2.7.8"

alias py3="pyenv global 3.5.0"

alias pysys="pyenv global system"

This way is very quick and simple but we miss the benefits of using pyenv.

Adding Site Packages

Alternatively, we can add the site-packages for this homebrew installed python 3 to our pyenv installation of python 3. Since both installations were built on the same machine and OS, the bindings should work correctly. We would be using .pth files to do this.

Let’s first find out the site-packages for the homebrew installation:

brew info python3

1	brew info python3

We would notice a message like:

They will install into the site-package directory
  /usr/local/lib/python3.5/site-packages

1 2	They will install into the site-package directory /usr/local/lib/python3.5/site-packages

That is the site-packages for this version.

Now let’s find our pyenv python3’s local site directory:

# Switch to pyenv python first
python -m site --user-site

1 2	# Switch to pyenv python first python -m site --user-site

Now create a homebrew.pth file in that directory and put the previously found site packages path there.

Let’s create the file:

# In my local site directory for pyenv

vim /Users/masnun/.local/lib/python3.5/site-packages/homebrew.pth

# In my local site directory for pyenv

vim /Users/masnun/.local/lib/python3.5/site-packages/homebrew.pth

And put these contents:

/usr/local/lib/python3.5/site-packages

1	/usr/local/lib/python3.5/site-packages

Save and exit. Now you should be able to just use:

python filename.py

1	python filename.py

Django Python

Dockerizing a Django Application

Post author By masnun
Post date December 19, 2015
2 Comments on Dockerizing a Django Application

I assume you are already familiar with Docker and it’s use cases. If you haven’t yet started using Docker, I strongly recommend you do soon.

I have a Django application that I want to dockerize it for local development. I am also new to Docker, so everything I do in this post might not be suitable for your production environment. So please do check Docker best practices for production apps. This tutorial is meant to be a basic introduction to Docker. In this post, I am going to use Docker Machine and Docker Compose. You can get them by installing the awesome Docker Toolbox.

Components Breakdown

Before we start, we need to break down our requirements so we can individually build the required components. For my particular application, we need these:

Django App Server
MySQL Database Server
Redis Server

We will build images for these separately so we can create individual containers and link them together to compose our ultimate application. We shall build our Django App server and use pre-built images for MySQL and Redis.

Building the Django App Server

Before we begin, let’s talk Dockerfiles. Dockerfiles are scripts to customize our docker builds. It allows us control and flexibility over how we build the images for our applications. We will use our custom Dockerfile to build the Django app server.

To build an image for a Django application we need to go through these following steps:

Select a Linux image, we choose Ubuntu
Install required packages for the distro.
Install Python packages which are required for the app
Provide a default command to run and ports to expose

Here’s the Dockerfile we shall use:

FROM phusion/baseimage
MAINTAINER masnun

ENV DEBIAN_FRONTEND noninteractive

RUN apt-get update
RUN apt-get install -y python python-pip python-dev 
RUN apt-get install -y libxml2-dev libxslt-dev libffi-dev libssl-dev 
RUN apt-get install -y libmysqlclient-dev

ADD requirements.txt /app/src/requirements.txt
WORKDIR /app/src
RUN pip install -r requirements.txt

WORKDIR /app/src/lisp
CMD [ "python", "manage.py", "runall"]

EXPOSE 8000

FROM phusion/baseimage

MAINTAINER masnun

ENV DEBIAN_FRONTEND noninteractive

RUN apt-get update

RUN apt-get install -y python python-pip python-dev

RUN apt-get install -y libxml2-dev libxslt-dev libffi-dev libssl-dev

RUN apt-get install -y libmysqlclient-dev

ADD requirements.txt /app/src/requirements.txt

WORKDIR /app/src

RUN pip install -r requirements.txt

WORKDIR /app/src/lisp

CMD [ "python", "manage.py", "runall"]

EXPOSE 8000

So what are we doing here:

We’re choosing phusion/baseimage as our base image. It’s a barebone image based on Ubuntu. Ubuntu by default comes with many packages which we don’t need to run inside docker. This base image gets rid of those and provides a very lean and clean image to start with.
We just provide a Maintainer name
We set DEBIAN_FRONTEND to be non interactive. This will not display any interactive prompts during the build process. Since the docker build process is automated, we really don’t have any way to interact during it. So we disable interaction. And as you might have guessed already ENV sets an environment variable.
We install some packages we shall need.
We copy our requirements.txt file to /app/src/requirements.txt, change the work directory and install the packages using pip. ADD is used to copy any files or directories to the container while it builds. You might wonder why we didn’t copy over our entire project – that’s because we want to use docker for our development. We will use a nice featire of Docker which would allow us to mount our local directories directly inside the container. Doing this, we would not need to copy files every time they change. More on this will come later.
We change directory to /app/src/lisp and run the runall management command. This command runs the Django default server along with some other services my application needs. But usually we would want to just do runserver
We EXPOSE port 8000

If you go through the Dockerfile References you will notice – we can do a lot more with Dockerfiles.

Docker Compose and Linking Services

As we mentioned earlier, we shall use pre-built images for MySQL and Redis. We could build them ourselves too but why not take advantage of the well maintained images from the generous folks in the docker community?

We can link multiple docker containers to compose a final application. We can do that using the docker command manually. But Docker Compose is a very nice tool which allows us to define the services we need in a very easy to read syntax. With docker compose, we don’t need to run them manually, we can just use simple commands to do complex docker magic! Here’s our docker-compose.yml file:

web:
  build: .
  restart: always
  volumes:
    - .:/app/src
  ports:
    - "8000:8000"
  links:
    - redis
    - mysql

redis:
  image: redis:latest
  volumes:
    - /var/lib/redis
  ports:
    - "6379"

mysql:
  image: mysql:latest
  volumes:
    - /var/lib/mysql
  ports:
    - "3306:3306"
  environment:
    MYSQL_ROOT_PASSWORD: su53RsEc53T!

web:

build: .

restart: always

volumes:

- .:/app/src

ports:

- "8000:8000"

links:

- redis

- mysql

redis:

image: redis:latest

volumes:

- /var/lib/redis

ports:

- "6379"

mysql:

image: mysql:latest

volumes:

- /var/lib/mysql

ports:

- "3306:3306"

environment:

MYSQL_ROOT_PASSWORD: su53RsEc53T!

In our docker-compose file, we define 3 components:

For the web, we pass the path to Dockerfile to build key. We ask to restart always and define volumes to mount. .:/app/src means – mount the current directory on my OS X as /app/src/ on the container. We also define which ports to expose and which containers should be linked with it
We also define the mysql and redis components with respective configurations. Note that we define the pre-built image name in the image key. Please make sure the volume paths exist and are accessible.

You can consult the Compose File Reference for more details.

Running The Services

To run the application, we can do:

docker-compose up

1	docker-compose up

Please note, the Django server might throw errors if the MySQL / Redis server takes time to initialize. So I usually run them separately:

docker-compose start mysql
docker-compose start redis

# After some time

docker-compose start web

docker-compose start mysql

docker-compose start redis

# After some time

docker-compose start web

Database Configuration for Django

Our MySQL server is running on the IP of the Docker Machine. You need to use this IP address in your Django settings file. To get the IP of a docker machine, type in:

# Here `default` is the machine name
docker-machine ip default

1 2	# Here `default` is the machine name docker-machine ip default

Creating Initial Databases

We can pass a MYSQL_DATABASE environment value to the mysql image so the database is created when creating the service. Or we can also connect to the docker machine manually and create our databases.

Python

Extracting links and their page title from your Twitter Archive

Post author By masnun
Post date December 18, 2015

Twitter allows us to download our Tweets from the account settings page. Once we request our archive, Twitter will take some time to prepare it and send us an email once this is ready. We will get a download link in the email. After unpacking the archive, we shall find a csv file that contains our tweets – tweets.csv. The archive also contains a html page (index.html) that displays our tweets on a nice UI. While this is nice to look at, our primary objective is to extract the links from our tweets.

If we look at the CSV file closely, we shall find a field named expanded_urls which generally contains the urls we use in our tweets. We will work with the values in this field. With the url, we also want to fetch their title. For this we will use Python 3 (I am using 3.5) and we need the requests and beautifulsoup4 packages to download and parse the pages. Let’s install them:

pip install requests
pip install beautifulsoup4

1 2	pip install requests pip install beautifulsoup4

We will follow these steps to extract links and their page titles from the tweets:

Open the csv file and read row by row
Each row contains a tweet, we take the expanded_urls field
This field can contain multiple urls, separated by a comma. We need to iterate over them all
We will skip some domains, for example, we don’t want to visit links to twitter status updates
We fetch the html content using the requests library. If the page doesn’t return a HTTP 200, we ignore the response
We extract the title using beautiful soup and display it

Now let’s convert these steps to codes. Here’s the final script I came up with:

import csv
import requests
from bs4 import BeautifulSoup

DOMAINS_TO_SKIP = ['twitter.com']

with open('tweets.csv', 'r') as csvfile:
    reader = csv.DictReader(csvfile)

    # each row is a tweet
    for row in reader:
        url_string = row.get('expanded_urls')
        urls = url_string.split(",")
        for url in urls:

            # Skip the domains
            skip = False
            for domain in DOMAINS_TO_SKIP:
                if domain in url:
                    skip = True
                    break

            # fetch the title
            if url and not skip:
                print("Crawling: {}".format(url))
                resp = requests.get(url)
                if resp.status_code == 200:
                    soup = BeautifulSoup(resp.content, "html.parser")
                    if soup.title:
                        print("Title: {}".format(soup.title.string))

import csv

import requests

from bs4 import BeautifulSoup

DOMAINS_TO_SKIP = ['twitter.com']

with open('tweets.csv', 'r') as csvfile:

reader = csv.DictReader(csvfile)

# each row is a tweet

for row in reader:

url_string = row.get('expanded_urls')

urls = url_string.split(",")

for url in urls:

# Skip the domains

skip = False

for domain in DOMAINS_TO_SKIP:

if domain in url:

skip = True

break

# fetch the title

if url and not skip:

print("Crawling: {}".format(url))

resp = requests.get(url)

if resp.status_code == 200:

soup = BeautifulSoup(resp.content, "html.parser")

if soup.title:

print("Title: {}".format(soup.title.string))

I am actually using this for a personal project I am doing here – https://github.com/masnun/bookmarks – it’s basically a bare bone django admin app where I intend to store the links I visit/share. I come across a lot of interesting projects, articles, videos and then later lose track of them. Hope this app will remedy that. This piece of code is part of a twitter import functionality of the mentioned app.

Recent Posts

Recent Comments

Archives

Categories

Installation

Verifying Installation

Integrating with Pyenv

Switching to system

Adding Site Packages

Components Breakdown

Building the Django App Server

Docker Compose and Linking Services

Running The Services

Database Configuration for Django

Creating Initial Databases

Switching to `system`