Category: Python

Python

Iron Worker: Processing SendGrid Inbound Parse Data

Post author By masnun
Post date July 28, 2015

If you use SendGrid, you know that they provide this awesome Inbound Parse APIs which allows users to catch emails sent to their domains. When there’s a new email, Sendgrid would make a POST request to your webhook endpoint. On the other hand, I love Iron Worker webhooks. When you POST data to their webhooks, the data is passed to a worker and processed in queue. It helps keep things simpler since I don’t need to setup a separate web server for accepting the data; however, keep in mind that I do make sure to use data protection services like the Venyu cybersecurity services at all times.

By now, you can probably guess my intention. I would like to make Sendgrid POST incoming emails to a Iron Worker webhook and then grab the data. But there’s one catch – Iron.io stores the incoming data in a text file and expect us to parse the text file to collect our data. Since very often the data is JSON, most client libraries take the payload, decode it from JSON and provide us with native data structures. I use Python, so I usually get Python dictionary. However, in case of Sendgrid Inbound APIs, the data is not JSON encoded. It’s simple multi-part form data. So the Python client I use, it fails to process the data and I don’t get anything for the payload.

So I went ahead and modified the payload processing part from the client library. Now this is what I have for getting the payload:

def get_iron_args():
    arguments = {}
    for i in range(len(sys.argv)):
        if sys.argv[i] == "-id":
            arguments['task_id'] = sys.argv[i + 1]
        if sys.argv[i] == "-d":
            arguments['dir'] = sys.argv[i + 1]
        if sys.argv[i] == "-payload":
            arguments['payload_file'] = sys.argv[i + 1]
        if sys.argv[i] == "-config":
            arguments['config_file'] = sys.argv[i + 1]

        if os.getenv('TASK_ID'): arguments['task_id'] = os.getenv('TASK_ID')
        if os.getenv('TASK_DIR'): arguments['dir'] = os.getenv('TASK_DIR')
        if os.getenv('PAYLOAD_FILE'): arguments['payload_file'] = os.getenv('PAYLOAD_FILE')
        if os.getenv('CONFIG_FILE'): arguments['config_file'] = os.getenv('CONFIG_FILE')

        if 'payload_file' in arguments and file_exists(arguments['payload_file']):
            f = open(arguments['payload_file'], "r")
            try:
                content = f.read()
                f.close()
                arguments['payload'] = content
            except Exception, e:
                print "Couldn't parse IronWorker payload  %s" % e

    return arguments


def get_raw_payload():
    args = get_iron_args()
    return args['payload']

def get_iron_args():

arguments = {}

for i in range(len(sys.argv)):

if sys.argv[i] == "-id":

arguments['task_id'] = sys.argv[i + 1]

if sys.argv[i] == "-d":

arguments['dir'] = sys.argv[i + 1]

if sys.argv[i] == "-payload":

arguments['payload_file'] = sys.argv[i + 1]

if sys.argv[i] == "-config":

arguments['config_file'] = sys.argv[i + 1]

if os.getenv('TASK_ID'): arguments['task_id'] = os.getenv('TASK_ID')

if os.getenv('TASK_DIR'): arguments['dir'] = os.getenv('TASK_DIR')

if os.getenv('PAYLOAD_FILE'): arguments['payload_file'] = os.getenv('PAYLOAD_FILE')

if os.getenv('CONFIG_FILE'): arguments['config_file'] = os.getenv('CONFIG_FILE')

if 'payload_file' in arguments and file_exists(arguments['payload_file']):

f = open(arguments['payload_file'], "r")

try:

content = f.read()

f.close()

arguments['payload'] = content

except Exception, e:

print "Couldn't parse IronWorker payload %s" % e

return arguments

def get_raw_payload():

args = get_iron_args()

return args['payload']

Now, we have the payload. But it’s still just plain text. We need to process it to get the different POST fields. Luckily, we can do that easily with the cgi library. This is what I use:

def parse_post_data(payload):
    """
    Transforms raw text into POST dictionary

    :param payload: the raw text
    :return dictionary: parsed data
    """
    body_as_file = StringIO.StringIO(payload)
    pdict = {'boundary': 'xYzZY'}
    httpbody = cgi.parse_multipart(body_as_file, pdict)

    return httpbody

def parse_post_data(payload):

"""

Transforms raw text into POST dictionary

:param payload: the raw text

:return dictionary: parsed data

"""

body_as_file = StringIO.StringIO(payload)

pdict = {'boundary': 'xYzZY'}

httpbody = cgi.parse_multipart(body_as_file, pdict)

return httpbody

Finally, now this is what the worker looks like:

from utils import parse_post_data, get_raw_payload

data = parse_post_data(payload)
print data

from utils import parse_post_data, get_raw_payload

data = parse_post_data(payload)

print data

Python

Upwork recommended jobs to OS X notifications center

Post author By masnun
Post date July 14, 2015
1 Comment on Upwork recommended jobs to OS X notifications center

If you use Upwork and you’d love to get recommended job notifications on your OS X notification center, this is for you. Grab the RSS feed of the recommended jobs from your account and place it inside the URL variable.

To run the script, we need to install some required packages first:

pip install requests --upgrade
pip install pync --upgrade
pip install beautifulsoup4 --upgrade
pip install lxml --upgrade

pip install requests --upgrade

pip install pync --upgrade

pip install beautifulsoup4 --upgrade

pip install lxml --upgrade

Then save the following code as “upwork.py”

import requests
from bs4 import BeautifulSoup
from pync import Notifier
from urllib.parse import unquote

URL = 'https://www.upwork.com/find-work-home/rss?securityToken=66da1df2fc9020ca1631a0b7775962a521e5761b14de30418e92c4ce67a89b28~6179f50'

response = requests.get(URL)
soup = BeautifulSoup(response.content, 'xml')
items = soup.find_all('item')

for job_posting in items:
    Notifier.notify(job_posting.title.text, open=unquote(job_posting.link.text))

import requests

from bs4 import BeautifulSoup

from pync import Notifier

from urllib.parse import unquote

URL = 'https://www.upwork.com/find-work-home/rss?securityToken=66da1df2fc9020ca1631a0b7775962a521e5761b14de30418e92c4ce67a89b28~6179f50'

response = requests.get(URL)

soup = BeautifulSoup(response.content, 'xml')

items = soup.find_all('item')

for job_posting in items:

Notifier.notify(job_posting.title.text, open=unquote(job_posting.link.text))

Now you can manually run it or add it to cron job. Every time the script is executed, it would fetch all the recommended jobs for you and display in the notifications cenyer. You can afterwards click an item from the notification center to visit the actual job post.

Python

Fixing lxml installation on OS X

Post author By masnun
Post date July 14, 2015

I have recently upgraded to OS X Yosemite. And somehow my lxml installation broke. When I tried to do pip install lxml, I was getting an error: ‘libxml/xmlversion.h’ file not found. The error message is obvious. While trying to compile lxml, it can’t find the libxml header files. What was not very obvious to me is how to fix this. I am using Homebrew. I upgraded both libxml2 and libxslt. Didn’t work. After quite some googling and checking out some stack overflow answers, I found out my mistake. By default, brew wouldn’t link the libxml and libxslt to avoid conflicts. So I have to force them. In short, this is what I needed to do:

brew link libxml2 --force
brew link libxslt --force

1 2	brew link libxml2 --force brew link libxslt --force

Of course, if you didn’t have them installed before, you need to install them first:

brew install libxml2
brew install libxslt

1 2	brew install libxml2 brew install libxslt

If it doesn’t work even after force linking, you may want to first unlink previous versions.

brew unlink libxml2
brew unlink libxslt

1 2	brew unlink libxml2 brew unlink libxslt

Then I tried to install lxml from pip and it worked like a charm! 🙂

Recent Posts

Recent Comments

Archives

Categories