Author: masnun

Python

Creating a Twitter Retweet Bot in Python

Post author By masnun
Post date December 5, 2015

We want to create a bot that will track specific topics and retweet them. We shall use the Twitter Streaming API to track topics. We will use the popular tweepy package to interact with Twitter.

Let’s first install Tweepy

pip install tweepy

1	pip install tweepy

We need to create a Twitter app and get the tokens. We can do that from : https://apps.twitter.com/.

Now let’s see the codes:

from tweepy.streaming import StreamListener
from tweepy import OAuthHandler, API
from tweepy import Stream
import json
import logging
import warnings
from pprint import pprint

warnings.filterwarnings("ignore")

access_token = "4464294380-GHJ3PY...lEqzZ8FikULPnkxaS4huT"
access_token_secret = "L033NEsDOA...tZVoaU8prpFbiREWhcVROw2"
consumer_key = "D1G0bn...9Y88p"
consumer_secret = "rGJjCU...FRISsiMURYlqCXJvOP"

auth_handler = OAuthHandler(consumer_key, consumer_secret)
auth_handler.set_access_token(access_token, access_token_secret)

twitter_client = API(auth_handler)

logging.getLogger("main").setLevel(logging.INFO)

AVOID = ["monty", "leather", "skin", "bag", "blood", "bite"]


class PyStreamListener(StreamListener):
    def on_data(self, data):
        tweet = json.loads(data)
        try:
            publish = True
            for word in AVOID:
                if word in tweet['text'].lower():
                    logging.info("SKIPPED FOR {}".format(word))
                    publish = False

            if tweet.get('lang') and tweet.get('lang') != 'en':
                publish = False

            if publish:
                twitter_client.retweet(tweet['id'])
                logging.debug("RT: {}".format(tweet['text']))

        except Exception as ex:
            logging.error(ex)

        return True

    def on_error(self, status):
        print status


if __name__ == '__main__':
    listener = PyStreamListener()
    stream = Stream(auth_handler, listener)
    stream.filter(track=['python', 'django', 'kivy', 'scrapy'])

from tweepy.streaming import StreamListener

from tweepy import OAuthHandler, API

from tweepy import Stream

import json

import logging

import warnings

from pprint import pprint

warnings.filterwarnings("ignore")

access_token = "4464294380-GHJ3PY...lEqzZ8FikULPnkxaS4huT"

access_token_secret = "L033NEsDOA...tZVoaU8prpFbiREWhcVROw2"

consumer_key = "D1G0bn...9Y88p"

consumer_secret = "rGJjCU...FRISsiMURYlqCXJvOP"

auth_handler = OAuthHandler(consumer_key, consumer_secret)

auth_handler.set_access_token(access_token, access_token_secret)

twitter_client = API(auth_handler)

logging.getLogger("main").setLevel(logging.INFO)

AVOID = ["monty", "leather", "skin", "bag", "blood", "bite"]

class PyStreamListener(StreamListener):

def on_data(self, data):

tweet = json.loads(data)

try:

publish = True

for word in AVOID:

if word in tweet['text'].lower():

logging.info("SKIPPED FOR {}".format(word))

publish = False

if tweet.get('lang') and tweet.get('lang') != 'en':

publish = False

if publish:

twitter_client.retweet(tweet['id'])

logging.debug("RT: {}".format(tweet['text']))

except Exception as ex:

logging.error(ex)

return True

def on_error(self, status):

print status

if __name__ == '__main__':

listener = PyStreamListener()

stream = Stream(auth_handler, listener)

stream.filter(track=['python', 'django', 'kivy', 'scrapy'])

The code is pretty much self explanatory:

We create a Twitter API client using the oAuth details we got earlier
We subclass StreamListener to implement our own on_data method
We create an instance of this class, then create a new Stream by passing the auth handler and the listener
We use the track method to track a number of topics we are interested in
When we start to track the topics, it will pass the data to on_data method where we parse the tweet, check some common words to avoid, check language and then retweet it.

Python

Scrapy: Scraping each type of Item to it’s own collection in mongodb

Post author By masnun
Post date December 4, 2015

I am using Scrapy and I have two different Items. I want to store entries for each specific item to it’s own mongo collection. For example, let’s assume this is what I have in the items.py file:

import scrapy

class Student(scrapy.Item):
    name = scrapy.Field()
    email = scrapy.Field()
    phone = scrapy.Field()


class Course(scrapy.Item):
    name = scrapy.Field()
    teacher_name = scrapy.Field()

import scrapy

class Student(scrapy.Item):

name = scrapy.Field()

email = scrapy.Field()

phone = scrapy.Field()

class Course(scrapy.Item):

name = scrapy.Field()

teacher_name = scrapy.Field()

I want to store Student items to student collection and Course items to course collection. How do we do that?

If you have used Scrapy before, you already know that for storing data, we use Pipelines. Here’s our own MongoPipeline that stores items to their own collection:

from pymongo import MongoClient
from scrapy.conf import settings
import logging


class MongoPipeline(object):
    def __init__(self):
        connection = MongoClient(settings['MONGODB_HOST'], settings['MONGODB_PORT'])
        self.db = connection[settings['MONGODB_DATABASE']]

    def process_item(self, item, spider):
        collection = self.db[type(item).__name__.lower()]
        logging.info(collection.insert(dict(item)))
        return item

from pymongo import MongoClient

from scrapy.conf import settings

import logging

class MongoPipeline(object):

def __init__(self):

connection = MongoClient(settings['MONGODB_HOST'], settings['MONGODB_PORT'])

self.db = connection[settings['MONGODB_DATABASE']]

def process_item(self, item, spider):

collection = self.db[type(item).__name__.lower()]

logging.info(collection.insert(dict(item)))

return item

So this is what’s happening:

We’re using PyMongo as the mongodb driver
I have the MongoDB related configurations to settings. I am getting them and constructing a mongodb client. I am also selecting the database based on a setting
In the process_item function, we are getting the type of the item and lowering it’s name. This type name would serve as the mongodb collection name for us.
We are inserting the item. We’re calling dict() on the item to get a dictionary representation which we can directly save using PyMongo.

That’s it. Now if you run your spiders, items of each type will go to it’s own collection on mongodb.

Mac Python

Fixing fatal error: ‘openssl/aes.h’ file not found on OS X

Post author By masnun
Post date December 1, 2015
5 Comments on Fixing fatal error: ‘openssl/aes.h’ file not found on OS X

OS X recently started using their own library instead of OpenSSL. So new installations for anything that depends on OpenSSL might fail. If you already have OpenSSL installed on your system, for example using Homebrew, you just need to point to the library while compiling your program.

For example, if you’re trying to install the popular cryptography package from PyPI, you can do these:

env LDFLAGS="-L$(brew --prefix openssl)/lib" CFLAGS="-I$(brew --prefix openssl)/include" pip install cryptography

1	env LDFLAGS="-L$(brew --prefix openssl)/lib" CFLAGS="-I$(brew --prefix openssl)/include" pip install cryptography

The above mentioned package is a common dependency of many other packages, for example Scrapy. So if you encounter an issue like this, try installing that single dependency first and then the dependent package. In this case, first use the above command to install the cryptography package and later install Scrapy.

Recent Posts

Recent Comments

Archives

Categories