How to handle I/O Blocking Processes with Advanced Logging

This article is a follow-up to our previous tutorial on how to create a custom Log Handler for sending specific logs as Slack messages.

Let’s say you have implemented a SlackHandler, and at some point in your code you end up performing an HTTP request against the slack API. Usually, this kind of operation is I/O blocking, so the execution thread will wait for this call to finish to continue its own execution, i.e. your code will be blocked until it gets a response from Slack.

The drawback here is that if something goes wrong with the external service such as the request hanging for a while, it may trigger a timeout exception on your server as the original request is taking as much time or more than your request against that external service. 

This means that any of your processes could potentially be killed due to these kinds of errors. Since this shutdown mechanism may occur on higher layers of your architecture, i.e. Nginx, Apache, etc, you might not notice this behavior fast enough.

So, a way to handle this is to separate the logging thread from the main process to ensure any error or delay during logging will not affect the execution of your code. As an example, I’ll highlight how to avoid this in a clean manner, using Python’s QueueHandler and QueueListener classes.


According to Python’s documentation, this handler is “located in the logging.handlers module, and supports sending logging messages to a queue, such as those implemented in the queue or multiprocessing modules.”

We’ll use this handler to “collect” any log messages we want to send to Slack and then store them in a queue. To achieve this, the first modification we are going to do is in the settings file of your Django project. See here how to do so.

This change means that instead of using our SlackHandler directly, we want to enqueue the messages on SLACK_QUEUE. These are all the modifications we need regarding the QueueHandler. Next, we need a way to grab those enqueued messages and send them to our SlackHandler so they actually reach Slack, we need to create and start a QueueListener.


Again, according to Python: “The QueueListener class, located in the logging.handlers module supports receiving logging messages from a queue, such as those implemented in the queue or multiprocessing modules. The messages are received from a queue in an internal thread and passed, on the same thread, to one or more handlers for processing.

To do this, we need to instantiate a QueueListener, configuring it with the queue we want to use, and the handlers we want to redirect the messages to.

For this example, I chose to put this startup code in two places. I defined a method to make the actual startup, and then I put a call on my_app/ inside the ready method, to ensure the listener initialization occurs just once in the application’s execution.

So, in the file, you will need to follow these steps. Then, on my_app/, you will need to follow this method

And, that’s it, your log messages will now go directly into the queue we defined before, and then, on a separate thread, will be processed by QueueListener, which will put them in the hands of our SlackHandler.

You can test this by intentionally putting a sleep() call on SlackHandler and see how your code will continue its execution. After the sleep ends, you will see the messages coming to slack.


In this breakdown, we described how to use a separate thread to handle heavy-load log operations using a queue, how to use a QueueHandler to put items into the queue, and how to use a QueueListener to get items from that queue to send into other handlers, i.e.the SlackHandler.

Want to work with ?
We are hiring :)

Thanks to the creation of blockchain technology in 2008, a lot of developments have occurred since. For example, the creation […]
June 15
5 min read
One of the most important western philosophers of all time, Aristotle once said – “Man is by nature a social […]
December 7
5 min read
Month-over-month growth is a key metric for measuring the growth of your business.
June 18
5 min read
DevOps describes the integration of software production, development, and operations, with a focus on increasing the speed of software delivery. […]
August 4
5 min read
For those of you looking to transition from graphic design to product design, I’d like to start out by saying […]
August 3
5 min read
Any contemporary financial system must include channels for people to lend and borrow valuable assets. Using these financial mediums, asset […]
August 2
5 min read
Marketing today depends heavily on digital advertising to reach consumers and many digital businesses rely upon revenue from personalized ads. […]
August 1
5 min read
Quality assurance (QA) serves an essential role in product development, improving the software for the customers, and reducing defect repair […]
July 26
5 min read