4.2. Get Certain Values From a DataFrame

This section contains some methods to get specific values of a pandas DataFrame or a pandas Series.

4.2.1. pd.Series.between: Select Rows in a Pandas Series Containing Values Between 2 Numbers

To get the values that are smaller than the upper bound and larger than the lower bound, use the pandas.Series.between method.

In the code below, I obtained the values between 0 and 10 using between.

import pandas as pd 

s = pd.Series([5, 2, 15, 13, 6, 10])

s[s.between(0, 10)]
0     5
1     2
4     6
5    10
dtype: int64

4.2.2. DataFrame rolling: Find The Average of The Previous n Datapoints Using pandas

If you want to find the average of the previous n data points (simple moving average) with pandas, use df.rolling(time_period).mean().

The code below shows how to find the simple moving average of the previous 3 data-points.

from datetime import date
import pandas as pd 

df = pd.DataFrame(
    {
        "date": [
            date(2021, 1, 20),
            date(2021, 1, 21),
            date(2021, 1, 22),
            date(2021, 1, 23),
            date(2021, 1, 24),
        ],
        "value": [1, 2, 3, 4, 5],
    }
).set_index("date")

df
value
date
2021-01-20 1
2021-01-21 2
2021-01-22 3
2021-01-23 4
2021-01-24 5
df.rolling(3).mean()
value
date
2021-01-20 NaN
2021-01-21 NaN
2021-01-22 2.0
2021-01-23 3.0
2021-01-24 4.0

4.2.3. pandas.Series.pct_change: Find The Percentage Change Between The Current and a Prior Element in a pandas Series

If you want to find the percentage change between the current and a prior element in a pandas Series, use the pct_change method.

In the example below, 35 is 75% larger than 20, and 10 is 71.4% smaller than 35.

import pandas as pd 

df = pd.DataFrame({"a": [20, 35, 10], "b": [1, 2, 3]})
df
a b
0 20 1
1 35 2
2 10 3
df.a.pct_change()
0         NaN
1    0.750000
2   -0.714286
Name: a, dtype: float64

4.2.4. DataFrame.diff and DataFrame.shift: Take the Difference Between Rows Within a Column in pandas

If you want to get the difference between rows within a column, use DataFrame.diff().

import pandas as pd 

df = pd.DataFrame({"a": [1, 2, 3, 4], "b": [2, 3, 4, 6]})
diff = df.diff()
diff
a b
0 NaN NaN
1 1.0 1.0
2 1.0 1.0
3 1.0 2.0

This will leave the first index null. You can shift the rows up to match the first difference with the first index using DataFrame.shift(-1).

shift = diff.shift(-1)
shift
a b
0 1.0 1.0
1 1.0 1.0
2 1.0 2.0
3 NaN NaN
processed_df = shift.dropna()
processed_df
a b
0 1.0 1.0
1 1.0 1.0
2 1.0 2.0

4.2.5. df.to_dict: Turn a DataFrame into a Dictionary

To turn a DataFrame into a Python dictionary, use df.to_dict().

import pandas as pd

df = pd.DataFrame({"fruits": ["apple", "orange", "grape"], "price": [1, 2, 3]})
print(df)
   fruits  price
0   apple      1
1  orange      2
2   grape      3

This will return a dictionary whose keys are columns and values are rows.

df.to_dict()
{'fruits': {0: 'apple', 1: 'orange', 2: 'grape'}, 'price': {0: 1, 1: 2, 2: 3}}

However, if you prefer to get a list of dictionaries whose elements are rows, use df.to_dict(orient='records') instead.

df.to_dict(orient="records")
[{'fruits': 'apple', 'price': 1},
 {'fruits': 'orange', 'price': 2},
 {'fruits': 'grape', 'price': 3}]

4.2.6. Get Count and Percentage of a Value in a Column

If you want to get the count of each value in a column, use value_counts.

import pandas as pd 

size = pd.Series(['S', 'S', 'M', 'L', 'M', 'M'])
size.value_counts()
M    3
S    2
L    1
dtype: int64

However, If you want to get the percentage of each value in a column, use value_counts(normalize=True).

size.value_counts(normalize=True)
M    0.500000
S    0.333333
L    0.166667
dtype: float64