7.4. Logging and Debuggingยถ

Collections of tools for logging and debugging Python code.

7.4.1. rich.inspect: Produce a Beautiful Report on any Python Objectยถ

!pip install rich 

If you want to quickly see which attributes and methods of a Python object are available, use richโ€™s inspect method.

richโ€™s inspect method allows you to create a beautiful report for any Python object, including a string.

from rich import inspect

print(inspect('hello', methods=True))
โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ <class 'str'> โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ
โ”‚ str(object='') -> str                                                                     โ”‚
โ”‚ str(bytes_or_buffer[, encoding[, errors]]) -> str                                         โ”‚
โ”‚                                                                                           โ”‚
โ”‚ โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚
โ”‚ โ”‚ 'hello'                                                                               โ”‚ โ”‚
โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ โ”‚
โ”‚                                                                                           โ”‚
โ”‚   capitalize = def capitalize(): Return a capitalized version of the string.              โ”‚
โ”‚     casefold = def casefold(): Return a version of the string suitable for caseless       โ”‚
โ”‚                comparisons.                                                               โ”‚
โ”‚       center = def center(width, fillchar=' ', /): Return a centered string of length     โ”‚
โ”‚                width.                                                                     โ”‚
โ”‚        count = def count(...) S.count(sub[, start[, end]]) -> int                         โ”‚
โ”‚       encode = def encode(encoding='utf-8', errors='strict'): Encode the string using the โ”‚
โ”‚                codec registered for encoding.                                             โ”‚
โ”‚     endswith = def endswith(...) S.endswith(suffix[, start[, end]]) -> bool               โ”‚
โ”‚   expandtabs = def expandtabs(tabsize=8): Return a copy where all tab characters are      โ”‚
โ”‚                expanded using spaces.                                                     โ”‚
โ”‚         find = def find(...) S.find(sub[, start[, end]]) -> int                           โ”‚
โ”‚       format = def format(...) S.format(*args, **kwargs) -> str                           โ”‚
โ”‚   format_map = def format_map(...) S.format_map(mapping) -> str                           โ”‚
โ”‚        index = def index(...) S.index(sub[, start[, end]]) -> int                         โ”‚
โ”‚      isalnum = def isalnum(): Return True if the string is an alpha-numeric string, False โ”‚
โ”‚                otherwise.                                                                 โ”‚
โ”‚      isalpha = def isalpha(): Return True if the string is an alphabetic string, False    โ”‚
โ”‚                otherwise.                                                                 โ”‚
โ”‚      isascii = def isascii(): Return True if all characters in the string are ASCII,      โ”‚
โ”‚                False otherwise.                                                           โ”‚
โ”‚    isdecimal = def isdecimal(): Return True if the string is a decimal string, False      โ”‚
โ”‚                otherwise.                                                                 โ”‚
โ”‚      isdigit = def isdigit(): Return True if the string is a digit string, False          โ”‚
โ”‚                otherwise.                                                                 โ”‚
โ”‚ isidentifier = def isidentifier(): Return True if the string is a valid Python            โ”‚
โ”‚                identifier, False otherwise.                                               โ”‚
โ”‚      islower = def islower(): Return True if the string is a lowercase string, False      โ”‚
โ”‚                otherwise.                                                                 โ”‚
โ”‚    isnumeric = def isnumeric(): Return True if the string is a numeric string, False      โ”‚
โ”‚                otherwise.                                                                 โ”‚
โ”‚  isprintable = def isprintable(): Return True if the string is printable, False           โ”‚
โ”‚                otherwise.                                                                 โ”‚
โ”‚      isspace = def isspace(): Return True if the string is a whitespace string, False     โ”‚
โ”‚                otherwise.                                                                 โ”‚
โ”‚      istitle = def istitle(): Return True if the string is a title-cased string, False    โ”‚
โ”‚                otherwise.                                                                 โ”‚
โ”‚      isupper = def isupper(): Return True if the string is an uppercase string, False     โ”‚
โ”‚                otherwise.                                                                 โ”‚
โ”‚         join = def join(iterable, /): Concatenate any number of strings.                  โ”‚
โ”‚        ljust = def ljust(width, fillchar=' ', /): Return a left-justified string of       โ”‚
โ”‚                length width.                                                              โ”‚
โ”‚        lower = def lower(): Return a copy of the string converted to lowercase.           โ”‚
โ”‚       lstrip = def lstrip(chars=None, /): Return a copy of the string with leading        โ”‚
โ”‚                whitespace removed.                                                        โ”‚
โ”‚    maketrans = def maketrans(...) Return a translation table usable for str.translate().  โ”‚
โ”‚    partition = def partition(sep, /): Partition the string into three parts using the     โ”‚
โ”‚                given separator.                                                           โ”‚
โ”‚      replace = def replace(old, new, count=-1, /): Return a copy with all occurrences of  โ”‚
โ”‚                substring old replaced by new.                                             โ”‚
โ”‚        rfind = def rfind(...) S.rfind(sub[, start[, end]]) -> int                         โ”‚
โ”‚       rindex = def rindex(...) S.rindex(sub[, start[, end]]) -> int                       โ”‚
โ”‚        rjust = def rjust(width, fillchar=' ', /): Return a right-justified string of      โ”‚
โ”‚                length width.                                                              โ”‚
โ”‚   rpartition = def rpartition(sep, /): Partition the string into three parts using the    โ”‚
โ”‚                given separator.                                                           โ”‚
โ”‚       rsplit = def rsplit(sep=None, maxsplit=-1): Return a list of the words in the       โ”‚
โ”‚                string, using sep as the delimiter string.                                 โ”‚
โ”‚       rstrip = def rstrip(chars=None, /): Return a copy of the string with trailing       โ”‚
โ”‚                whitespace removed.                                                        โ”‚
โ”‚        split = def split(sep=None, maxsplit=-1): Return a list of the words in the        โ”‚
โ”‚                string, using sep as the delimiter string.                                 โ”‚
โ”‚   splitlines = def splitlines(keepends=False): Return a list of the lines in the string,  โ”‚
โ”‚                breaking at line boundaries.                                               โ”‚
โ”‚   startswith = def startswith(...) S.startswith(prefix[, start[, end]]) -> bool           โ”‚
โ”‚        strip = def strip(chars=None, /): Return a copy of the string with leading and     โ”‚
โ”‚                trailing whitespace removed.                                               โ”‚
โ”‚     swapcase = def swapcase(): Convert uppercase characters to lowercase and lowercase    โ”‚
โ”‚                characters to uppercase.                                                   โ”‚
โ”‚        title = def title(): Return a version of the string where each word is titlecased. โ”‚
โ”‚    translate = def translate(table, /): Replace each character in the string using the    โ”‚
โ”‚                given translation table.                                                   โ”‚
โ”‚        upper = def upper(): Return a copy of the string converted to uppercase.           โ”‚
โ”‚        zfill = def zfill(width, /): Pad a numeric string with zeros on the left, to fill  โ”‚
โ”‚                a field of the given width.                                                โ”‚
โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ
None

7.4.2. Richโ€™s Console: Debug your Python Function in One Line of Codeยถ

!pip install rich 

Sometimes, you might want to know which elements in the function created a certain output. Instead of printing every variable in the function, you can simply use Richโ€™s Console object to print both the output and all the variables in the function.

from rich import console
from rich.console import Console 
import pandas as pd 

console = Console()

data = pd.DataFrame({'a': [1, 2, 3], 'b': [4, 5, 6]})

def edit_data(data):
    var_1 = 45
    var_2 = 30
    var_3 = var_1 + var_2
    data['a'] = [var_1, var_2, var_3]
    console.log(data, log_locals=True)

edit_data(data)
[08:12:24]     a  b                                                          1165738010.py:14
           0  45  4                                                                          
           1  30  5                                                                          
           2  75  6                                                                          
           โ•ญโ”€โ”€โ”€โ”€โ”€ locals โ”€โ”€โ”€โ”€โ”€โ•ฎ                                                              
           โ”‚  data =     a  b โ”‚                                                              
           โ”‚         0  45  4 โ”‚                                                              
           โ”‚         1  30  5 โ”‚                                                              
           โ”‚         2  75  6 โ”‚                                                              
           โ”‚ var_1 = 45       โ”‚                                                              
           โ”‚ var_2 = 30       โ”‚                                                              
           โ”‚ var_3 = 75       โ”‚                                                              
           โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ                                                              

Link to my article about rich.

Link to rich.

7.4.3. loguru: Print Readable Traceback in Pythonยถ

!pip install loguru 

Sometimes, it is difficult to understand the traceback and to know which inputs cause the error. Is there a way that you can print a more readable traceback?

That is when loguru comes in handy. By adding decorator logger.catch to a function, loguru logger will print a more readable trackback and save the traceback to a separate file like below

from sklearn.metrics import mean_squared_error
import numpy as np
from loguru import logger

logger.add("file_{time}.log", format="{time} {level} {message}")

@logger.catch
def evaluate_result(y_true: np.array, y_pred: np.array):
    mean_square_err = mean_squared_error(y_true, y_pred)
    root_mean_square_err = mean_square_err ** 0.5

y_true = np.array([1, 2, 3])
y_pred = np.array([1.5, 2.2])
evaluate_result(y_true, y_pred)
> File "/tmp/ipykernel_174022/1865479429.py", line 14, in <module>
    evaluate_result(y_true, y_pred)
    โ”‚               โ”‚       โ”” array([1.5, 2.2])
    โ”‚               โ”” array([1, 2, 3])
    โ”” <function evaluate_result at 0x7f279588f430>

  File "/tmp/ipykernel_174022/1865479429.py", line 9, in evaluate_result
    mean_square_err = mean_squared_error(y_true, y_pred)
                      โ”‚                  โ”‚       โ”” array([1.5, 2.2])
                      โ”‚                  โ”” array([1, 2, 3])
                      โ”” <function mean_squared_error at 0x7f27958bfca0>

  File "/home/khuyen/book/venv/lib/python3.8/site-packages/sklearn/utils/validation.py", line 63, in inner_f
    return f(*args, **kwargs)
           โ”‚  โ”‚       โ”” {}
           โ”‚  โ”” (array([1, 2, 3]), array([1.5, 2.2]))
           โ”” <function mean_squared_error at 0x7f27958bfb80>
  File "/home/khuyen/book/venv/lib/python3.8/site-packages/sklearn/metrics/_regression.py", line 335, in mean_squared_error
    y_type, y_true, y_pred, multioutput = _check_reg_targets(
            โ”‚       โ”‚                     โ”” <function _check_reg_targets at 0x7f27958b7af0>
            โ”‚       โ”” array([1.5, 2.2])
            โ”” array([1, 2, 3])
  File "/home/khuyen/book/venv/lib/python3.8/site-packages/sklearn/metrics/_regression.py", line 88, in _check_reg_targets
    check_consistent_length(y_true, y_pred)
    โ”‚                       โ”‚       โ”” array([1.5, 2.2])
    โ”‚                       โ”” array([1, 2, 3])
    โ”” <function check_consistent_length at 0x7f279676e040>
  File "/home/khuyen/book/venv/lib/python3.8/site-packages/sklearn/utils/validation.py", line 319, in check_consistent_length
    raise ValueError("Found input variables with inconsistent numbers of"

ValueError: Found input variables with inconsistent numbers of samples: [3, 2]

Link to loguru.

7.4.4. Icecream: Never use print() to debug againยถ

!pip install icecream

If you use print or log to debug your code, you might be confused about which line of code creates the output, especially when there are many outputs.

You might insert text to make it less confusing, but it is time-consuming.

from icecream import ic

def plus_one(num):
    return num + 1

print('output of plus_on with num = 1:', plus_one(1))
print('output of plus_on with num = 2:', plus_one(2))
output of plus_on with num = 1: 2
output of plus_on with num = 2: 3

Try icecream instead. Icrecream inspects itself and prints both its own arguments and the values of those arguments like below.

ic(plus_one(1))
ic(plus_one(2))
ic| plus_one(1): 2
ic| plus_one(2): 3
3

Output:

ic| plus_one(1): 2
ic| plus_one(2): 3

Link to icecream

Link to my article about icecream

7.4.5. heartrate โ€” Visualize the Execution of a Python Program in Real-Timeยถ

!pip install heartrate 

If you want to visualize which lines are executed and how many times they are executed, try heartrate.

You only need to add two lines of code to use heartrate.

import heartrate 
heartrate.trace(browser=True)

def factorial(x):
    if x == 1:
        return 1
    else:
        return (x * factorial(x-1))


if __name__ == "__main__":
    num = 5
    print(f"The factorial of {num} is {factorial(num)}")
 * Serving Flask app 'heartrate.core' (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
The factorial of 5 is 120
Opening in existing browser session.

You should see something similar to the below when opening the browser:

image

Link to heartrate.

7.4.6. snoop : Smart Print to Debug your Python Functionยถ

!pip install snoop

If you want to figure out what is happening in your code without adding many print statements, try snoop.

To use snoop, simply add the @snoop decorator to a function you want to understand.

import snoop 

@snoop
def factorial(x):
    if x == 1:
        return 1
    else:
        return (x * factorial(x-1))


if __name__ == "__main__":
    num = 5
    print(f"The factorial of {num} is {factorial(num)}")
10:19:00.73 >>> Call to factorial in File "<ipython-input-2-57aff36d5f6d>", line 4
10:19:00.73 ...... x = 5
10:19:00.73    4 | def factorial(x):
10:19:00.73    5 |     if x == 1:
10:19:00.73    8 |         return (x * factorial(x-1))
    10:19:00.74 >>> Call to factorial in File "<ipython-input-2-57aff36d5f6d>", line 4
    10:19:00.74 ...... x = 4
    10:19:00.74    4 | def factorial(x):
    10:19:00.74    5 |     if x == 1:
    10:19:00.74    8 |         return (x * factorial(x-1))
        10:19:00.74 >>> Call to factorial in File "<ipython-input-2-57aff36d5f6d>", line 4
        10:19:00.74 ...... x = 3
        10:19:00.74    4 | def factorial(x):
        10:19:00.74    5 |     if x == 1:
        10:19:00.75    8 |         return (x * factorial(x-1))
            10:19:00.75 >>> Call to factorial in File "<ipython-input-2-57aff36d5f6d>", line 4
            10:19:00.75 ...... x = 2
            10:19:00.75    4 | def factorial(x):
            10:19:00.75    5 |     if x == 1:
            10:19:00.75    8 |         return (x * factorial(x-1))
                10:19:00.75 >>> Call to factorial in File "<ipython-input-2-57aff36d5f6d>", line 4
                10:19:00.75 ...... x = 1
                10:19:00.75    4 | def factorial(x):
                10:19:00.76    5 |     if x == 1:
                10:19:00.76    6 |         return 1
                10:19:00.76 <<< Return value from factorial: 1
            10:19:00.76    8 |         return (x * factorial(x-1))
            10:19:00.77 <<< Return value from factorial: 2
        10:19:00.77    8 |         return (x * factorial(x-1))
        10:19:00.77 <<< Return value from factorial: 6
    10:19:00.77    8 |         return (x * factorial(x-1))
    10:19:00.77 <<< Return value from factorial: 24
10:19:00.78    8 |         return (x * factorial(x-1))
10:19:00.78 <<< Return value from factorial: 120
The factorial of 5 is 120

7.4.7. Logging in Pandas Pipelinesยถ

!pip install scikit-lego

When using pandas pipe, you might want to check whether each pipeline transforms your pandas DataFrame correctly. To automatically log the information of a pandas DataFrame after each pipeline, use the decorator sklego.pandas_utils.log_step.

import pandas as pd 
from sklego.pandas_utils import log_step 
import logging 
df = pd.DataFrame({"col1": [1, 2, 3], "col2": ["a", "b", "c"]})

To use log_step, simply use it as a decorator for functions being applied to your DataFrame.

@log_step(print_fn=logging.info)
def make_copy(df: pd.DataFrame):
    return df.copy()


@log_step(print_fn=logging.info)
def drop_column(df: pd.DataFrame):
    return df[["col2"]]


@log_step(print_fn=logging.info)
def encode_cat_variables(df: pd.DataFrame):
    df["col2"] = df["col2"].map({"a": 1, "b": 2, "c": 3})
    return df
df = df.pipe(make_copy).pipe(drop_column).pipe(encode_cat_variables)
INFO:root:[make_copy(df)] time=0:00:00.000239 n_obs=3, n_col=2
INFO:root:[drop_column(df)] time=0:00:00.002117 n_obs=3, n_col=1
INFO:root:[encode_cat_variables(df)] time=0:00:00.003217 n_obs=3, n_col=1

Find more ways to customize your logging here

7.4.8. Add Progress Bar to Your List Comprehensionยถ

!pip install tqdm

If your for loop or list comprehension takes a long time to run, you might want to know which element is being processed. You can add clarity to your for-loop by using tqdm. Using tqdm with an iterable will show a progress bar.

from tqdm.notebook import tqdm
from time import sleep


def lower(word):
    sleep(1)
    print(f"Processing {word}")
    return word.lower()


words = tqdm(["Duck", "dog", "Flower", "fan"])

[lower(word) for word in words]
Processing Duck
Processing dog
Processing Flower
Processing fan
['duck', 'dog', 'flower', 'fan']

Link to tqdm.