Log Breadcrumbs: only show Logs leading up to an Error

In this article, we'll explore a way to efficiently log breadcrumbs, showing only the logs that lead up to an error. We'll only use Python's standard logging library to create an efficient logging setup that capture debug logs only when an exception occurs. This approach provides a detailed view of the steps leading up to a problem while reducing clutter and minimizing I/O. Let's code!

---

Why Log Breadcrumbs?

When an error occurs you want to have as much information as possible to lead you to the problem in your code. Logging a lot of information is very useful in this respect.

The downside is that all of these logs need to be processed. Then need to be written to a file or sent to an endpoint over HTTP, which might impact the performance of your app or server. Additionally it might clutter up your logs, making it harder to find relevant information when an error occurred.

The breadcrumbs-approach "ignores" e.g. all debug logs unless an error occurs. This allows you to both log a lot of detail information about your error and keep performance and overview at level.

---

Setting up the breadcrumb trail

Below is a simple function, divide, with debug logs that help track its internal behavior. Ideally we don't want to see the log every time, just when an error occurs so that we can see wich two numbers the function tried to divide.

def divide(a, b):
  logger.debug(f"Dividing [{a}] by [{b}]")
  return a / b

for value in [1, 2, 3, 4, 5, 6, 7, 8, 9, 'not a number', 0]:
  try:
    logger.debug(f"start dividing..")
    res = divide(a=10, b=value)
  except Exception as e:
    logger.error(f"❌An exception occurred: {e}")

The first few values (1 till 9) divide successfully and don't necessarily need the debug logs in these cases.

For faulty inputs like not a number and 0, however, capturing the debug logs would provide valuable context. Our. How can we go back in time and retrieve the logs?

Turn Your Python Function into a Decorator with One Line of Code

---

Solution overview

To create breadcrumbs we'll configure our logger with two handlers:

• StreamHandler: Display only INFO-level messages and above
• MemoryHandler: Stores DEBUG messages temporarily and passes them to the StreamHandler only when an error occurs

This setup will not display DEBUG logs since the StreamHandler is set to INFO. We can store DEBUG messages in the MemoryHandler temporarily and, when we detect an error, flush the messages to the StreamHandler. This will then display the stored DEBUG messages.

---

Configuring the logger

Here's how we configure the logger with a StreamHandler for regular output and a MemoryHandler for buffered DEBUG logs:

import logging
from logging.handlers import MemoryHandler

# Create logger and formatter for a structured log message
logger = logging.getLogger("my_logger")
formatter = logging.Formatter(
    fmt="%(levelname)-7s ⏱️%(asctime)s           
Tags:          Coding Data Science Programming Python Software Development          
          

		  

		  Add Fav