If you use SendGrid, you know that they provide this awesome Inbound Parse APIs which allows users to catch emails sent to their domains. When there’s a new email, Sendgrid would make a POST request to your webhook endpoint. On the other hand, I love Iron Worker webhooks. When you POST data to their webhooks, the data is passed to a worker and processed in queue. It helps keep things simpler since I don’t need to setup a separate web server for accepting the data; however, keep in mind that I do make sure to use data protection services like the Venyu cybersecurity services at all times.
By now, you can probably guess my intention. I would like to make Sendgrid POST incoming emails to a Iron Worker webhook and then grab the data. But there’s one catch – Iron.io stores the incoming data in a text file and expect us to parse the text file to collect our data. Since very often the data is JSON, most client libraries take the payload, decode it from JSON and provide us with native data structures. I use Python, so I usually get Python dictionary. However, in case of Sendgrid Inbound APIs, the data is not JSON encoded. It’s simple multi-part form data. So the Python client I use, it fails to process the data and I don’t get anything for the payload.
So I went ahead and modified the payload processing part from the client library. Now this is what I have for getting the payload:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
def get_iron_args(): arguments = {} for i in range(len(sys.argv)): if sys.argv[i] == "-id": arguments['task_id'] = sys.argv[i + 1] if sys.argv[i] == "-d": arguments['dir'] = sys.argv[i + 1] if sys.argv[i] == "-payload": arguments['payload_file'] = sys.argv[i + 1] if sys.argv[i] == "-config": arguments['config_file'] = sys.argv[i + 1] if os.getenv('TASK_ID'): arguments['task_id'] = os.getenv('TASK_ID') if os.getenv('TASK_DIR'): arguments['dir'] = os.getenv('TASK_DIR') if os.getenv('PAYLOAD_FILE'): arguments['payload_file'] = os.getenv('PAYLOAD_FILE') if os.getenv('CONFIG_FILE'): arguments['config_file'] = os.getenv('CONFIG_FILE') if 'payload_file' in arguments and file_exists(arguments['payload_file']): f = open(arguments['payload_file'], "r") try: content = f.read() f.close() arguments['payload'] = content except Exception, e: print "Couldn't parse IronWorker payload %s" % e return arguments def get_raw_payload(): args = get_iron_args() return args['payload'] |
Now, we have the payload. But it’s still just plain text. We need to process it to get the different POST fields. Luckily, we can do that easily with the cgi library. This is what I use:
|
1 2 3 4 5 6 7 8 9 10 11 12 |
def parse_post_data(payload): """ Transforms raw text into POST dictionary :param payload: the raw text :return dictionary: parsed data """ body_as_file = StringIO.StringIO(payload) pdict = {'boundary': 'xYzZY'} httpbody = cgi.parse_multipart(body_as_file, pdict) return httpbody |
Finally, now this is what the worker looks like:
|
1 2 3 4 |
from utils import parse_post_data, get_raw_payload data = parse_post_data(payload) print data |