Django, Python, Slack, Software Development

How to handle I/O Blocking Processes with Advanced Logging

72622893-e3b23600-3922-11ea-88ed-0d107445c47c

This article is a follow-up to our previous tutorial on how to create a custom Log Handler for sending specific logs as Slack messages.

Let’s say you have implemented a SlackHandler, and at some point in your code you end up performing an HTTP request against the slack API. Usually, this kind of operation is I/O blocking, so the execution thread will wait for this call to finish to continue its own execution, i.e. your code will be blocked until it gets a response from Slack.

The drawback here is that if something goes wrong with the external service such as the request hanging for a while, it may trigger a timeout exception on your server as the original request is taking as much time or more than your request against that external service. 

This means that any of your processes could potentially be killed due to these kinds of errors. Since this shutdown mechanism may occur on higher layers of your architecture, i.e. Nginx, Apache, etc, you might not notice this behavior fast enough.

So, a way to handle this is to separate the logging thread from the main process to ensure any error or delay during logging will not affect the execution of your code. As an example, I’ll highlight how to avoid this in a clean manner, using Python’s QueueHandler and QueueListener classes.

QueueHandler

According to Python’s documentation, this handler is “located in the logging.handlers module, and supports sending logging messages to a queue, such as those implemented in the queue or multiprocessing modules.”

We’ll use this handler to “collect” any log messages we want to send to Slack and then store them in a queue. To achieve this, the first modification we are going to do is in the settings file of your Django project. See here how to do so.

This change means that instead of using our SlackHandler directly, we want to enqueue the messages on SLACK_QUEUE. These are all the modifications we need regarding the QueueHandler. Next, we need a way to grab those enqueued messages and send them to our SlackHandler so they actually reach Slack, we need to create and start a QueueListener.

QueueListener

Again, according to Python: “The QueueListener class, located in the logging.handlers module supports receiving logging messages from a queue, such as those implemented in the queue or multiprocessing modules. The messages are received from a queue in an internal thread and passed, on the same thread, to one or more handlers for processing.

To do this, we need to instantiate a QueueListener, configuring it with the queue we want to use, and the handlers we want to redirect the messages to.

For this example, I chose to put this startup code in two places. I defined a method to make the actual startup, and then I put a call on my_app/apps.py inside the ready method, to ensure the listener initialization occurs just once in the application’s execution.

So, in the slack.py file, you will need to follow these steps. Then, on my_app/apps.py, you will need to follow this method

And, that’s it, your log messages will now go directly into the queue we defined before, and then, on a separate thread, will be processed by QueueListener, which will put them in the hands of our SlackHandler.

You can test this by intentionally putting a sleep() call on SlackHandler and see how your code will continue its execution. After the sleep ends, you will see the messages coming to slack.

Summary

In this breakdown, we described how to use a separate thread to handle heavy-load log operations using a queue, how to use a QueueHandler to put items into the queue, and how to use a QueueListener to get items from that queue to send into other handlers, i.e.the SlackHandler.

Juan Sanchez

Want to work with Juan Sanchez? We are hiring :)

Other articles you might like