How to process a column from a dataframe in pandas

Question

I am writing a python program to calculate the chi-square value for a set of observed and expected frequencies. The program that I have constructed is written like so

# Author: Evan Gertis
# Date  : 10/25
# program : quantile decile calculator
import csv
import pandas as pd
import numpy as np 
from scipy.stats import chi2_contingency

import seaborn as sns
import matplotlib.pyplot as plt
import logging 
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s - %(levelname)s - %(message)s')

# Step 1: read csv
dicerollsCSV       = open('dice_rolls.csv')
df      = pd.read_csv(dicerollsCSV) 
logging.debug(df['Observed'])
logging.debug(df['Expected'])


# Step 2: Convert the data into a contingency table
logging.debug('Step 2: Convert the data into a contingency tables')
# Compute a simple cross tabulation of two (or more) factors. By default computes a frequency table of the factors unless an array of values and an aggregation function are passed.
# Implement steps from: https://predictivehacks.com/how-to-run-chi-square-test-in-python/
contingency = pd.crosstab(df['Observed'], df['Expected'])
logging.debug(f'contingency:{contingency}')

# Step 3; calculate the percentages by Observed(row)
logging.debug('Step 3; calculate the percentages by Observed(row)')
# add normalize='index'
contingency_pct = pd.crosstab(df['Observed'],df['Expected'],normalize='index')
logging.debug(f'contingency_pct:{contingency_pct}')


# Step 4; calculate the chi-square test
logging.debug('Step 4: calculate the chi-square test')
c, p, dof, expected = chi2_contingency(contingency)
# c: The test statistic
# p: The p-value of the test
# dof: Degrees of freedom
# expected: The expected frequencies, based on the marginal sums of the table
logging.debug(f'c: The statistic test  {c}')
logging.debug(f'p: The p-value of the test {p}')
logging.debug(f'dof: Degrees of freedom {dof}')
logging.debug(f'expected: The expected frequencies, based on the marginal sums of the table {expected}')

I am using https://predictivehacks.com/how-to-run-chi-square-test-in-python/ as a guide for completing this task. The specific dataset that I am using is

Observed, Expected
15, 13.9
35, 27.8
49, 41.7
58, 55.6
65, 69.5
76, 83.4
72, 69.5
60, 55.6
35, 41.7
29, 27.8
6, 13.9

Expected:
chi-square value from the observed and expected frequencies. The p-value should be 0.411.

Actual

2022-10-31 06:57:07,338 - DEBUG - c: The statistic test  49.499999999999986
2022-10-31 06:57:07,338 - DEBUG - p: The p-value of the test 0.2983423936107591
2022-10-31 06:57:07,338 - DEBUG - dof: Degrees of freedom 45
2022-10-31 06:57:07,339 - DEBUG - expected: The expected frequencies, based on the marginal sums of the table [[0.18181818 0.18181818 0.18181818 0.18181818 0.18181818 0.09090909]

What can I try next?

Asked By: Evan Gertis

||

Source

Answer 1

I believe your DF does not contains ‘Expected’ columns.

You can test it with the below code.

import pandas as pd
df = pd.DataFrame(columns = ['a','b'], data=[[1,2],[2,2]])
df['Expected']

You can observe the error is the same as yours.

Answered By: Joshua

Answer 2

Expected column name has a space in the beginning, so use df[' Expected'] or correct your csv.
And also you can read a csv into a pandas df just by giving the path
Ex: pd.read_csv(‘./test.csv’)
If you want to see the column names, run df.columns

Answered By: Vinay G

How to process a column from a dataframe in pandas

Question:

Answers: