6.11. Tools for Best Python Practices

This section cover tools that encourage best Python practices.

6.11.1. Don’t Hard-Code. Use Hydra Instead

Show code cell content Hide code cell content

!pip install hydra-core

When writing code, it is a good practice to put the values that you might change in a separate file from your original script.

This practice not only saves you from wasting time searching for a specific variable in your scripts but also makes your scripts more reproducible.

My favorite tool to handle config files is Hydra. The code below shows how to get values from a config file using Hydra.

All parameters are specified in a configuration file named config.yaml:

# config.yaml
data: data1 
variables: 
  drop_features: ['iid', 'id', 'idg', 'wave']
  categorical_vars: ['undergra', 'zipcode']

In seperate file named main.py, the parameters in the config.yaml file are called using Hydra:

# main.py
import hydra 

@hydra.main(config_name='config.yaml')
def main(config):
    print(f'Process {config.data}')
    print(f'Drop features: {config.variables.drop_features}')

if __name__ == '__main__':
    main()

On your terminal, type:

$ python main.py

Output:

!python hydra_examples/main.py

Show code cell output Hide code cell output

hydra_examples/main.py:3: UserWarning: 
config_path is not specified in @hydra.main().
See https://hydra.cc/docs/next/upgrades/1.0_to_1.1/changes_to_hydra_main_config_path for more information.
  @hydra.main(config_name='config.yaml')
Process data1
Drop features: ['iid', 'id', 'idg', 'wave']

Link to my article about Hydra.

Link to Hydra.

6.11.2. Store Sensitive Information Securely in Python with .env Files

Show code cell content Hide code cell content

!pip install python-dotenv

It’s a good practice to store sensitive information in a .env file and load it into your Python script using python-dotenv.

This helps protect the information from being exposed in your codebase or version control systems, reducing the risk of unauthorized access.

%%writefile .env
PASSWORD=123
USERNAME=myusername

Overwriting .env

from dotenv import load_dotenv
import os 

load_dotenv()
PASSWORD = os.getenv('PASSWORD')
USERNAME = os.getenv('USERNAME')
print(PASSWORD)
print(USERNAME)

123
myusername

Link to python-dotenv

6.11.3. docopt: Create Beautiful Command-line Interfaces for Documentation in Python

Show code cell content Hide code cell content

!pip install docopt 

Writing documentation for your Python script helps others understand how to use your script. However, instead of making them spend some time to find the documentation in your script, wouldn’t it be nice if they can view the documentation in the terminal?

That is when docopt comes in handy. docopt allows you to create beautiful command-line interfaces by passing a Python string.

To understand how docopt works, we can add a docstring at the beginning of the file named docopt_example.py.

%%writefile docopt_example.py
"""Extract keywords of an input file
Usage:
    docopt_example.py --data-dir=<data-directory> [--input-path=<path>]
Options:
    --data-dir=<path>    Directory of the data
    --input-path=<path>  Name of the input file [default: input_text.txt]
"""

from docopt import docopt 

if __name__ == '__main__':
    args = docopt(__doc__, argv=None, help=True)
    data_dir = args['--data-dir']
    input_path = args['--input-path']

    if data_dir:
        print(f"Extracting keywords from {data_dir}/{input_path}")

Running the file docopt_example.py should give us the output like below:

$ python docopt_example.py

Show code cell source Hide code cell source

!python docopt_example.py

Usage:
    docopt_example.py --data-dir=<data-directory> [--input-path=<path>]
Options:
    --data-dir=<path>    Directory of the data
    --input-path=<path>  Name of the input file [default: input_text.txt]

Link to docopt.