Fill missing values based on the condition of other columns Question: I have this large dataframe, illustrated below is for simplicity purposes. pd.DataFrame(df.groupby([‘Pclass’, ‘Sex’])[‘Age’].median()) Groupby results: And it have this data that needs to be imputed Missing Data: How can I impute these values based on the median of the grouped statistic The result that …
Pandas resample based on string like PeriodIndex Question: I have a dataframe like as below df = pd.DataFrame({‘subject_id’:[1,1,1,1,1,2,2,2,2,2], ‘qtr_info’ :[‘2017Q1′,’2017Q3′,’2017Q4′,’2018Q1′,’2018Q4′,’2017Q1′,’2017Q4′,’2018Q2′,’2018Q4′,’2019Q1’], ‘val’ :[5,5,5,5,1,6,5,5,8,3], ‘Prod_id’:[‘A’,’B’,’C’,’A’,’E’,’Q’,’G’,’F’,’G’,’H’]}) I would like to do the below a) Fill all the missing quarters of a subject b) fillna for other columns using the mean value for respective columns (for the same subject). …
How can i use .fillna with specific values? Question: df["load_weight"] = df.loc[(df["dropoff_site"] == "HORNSBY BEND") & (df[‘load_type’] == "BRUSH")].fillna(1000, inplace=True) i want to change the NaN value in "load_weight" column, but only for the rows that contain "HORNSBY BEND" and "BRUSH", but above code gave me "none" to the whole "load_weight" column, what did i …
find in datfarame outliers and fill with nan python Question: I am trying to make a function to spot the columns with "100" in the header and replace all values in these columns that are above 100 with nan values : import pandas as pd data = {‘first_100′: [’25’, ‘1568200’, ‘5’], ‘second_column’: [‘first_value’, ‘second_value’, ‘third_value’], …
A dataset with Int64, Float64 and datetime64[ns] gets converted to object after applying Pandas fillna method Question: I am using Kaggle’s dataset (https://www.kaggle.com/datasets/claytonmiller/lbnl-automated-fault-detection-for-buildings-data) I have A dataset with Int64, Float64, and datetime64[ns] datatypes; after using the pandas fillna method, however, all of my data type changes to object datatype. Could anyone assist me with what …
Pandas Dataframe – Fillna with Mean by Month Question: I have a dataframe with som "NaN" and Outlier values which I want to fill with the mean value of the specific month. df[["arrival_day", "ib_units", "month","year"]] arrival_day ib_units month year 37 2020-01-01 262 1 2020 235 2020-01-02 2301 1 2020 290 2020-01-02 145 1 2020 476 …
How to efficiently fill a column of a dataframe based on a dictionary Question: I have a dataframe and dictionary like this import pandas as pd import numpy as np df = pd.DataFrame({ ‘A’: [1, 1, 1, 2, 2, 3, 3, 3, 3], ‘ignore_me’: range(9), ‘fill_me’: [np.nan] * 9 }) di = { 1: [‘a’, …
How to shift a dataframe element-wise to fill NaNs? Question: I have a DataFrame like this: >>> df = pd.DataFrame({‘a’: list(‘ABCD’), ‘b’: [‘E’,np.nan,np.nan,’F’]}) a b 0 A E 1 B NaN 2 C NaN 3 D F I am trying to fill NaN with values of the previous column in the next row and dropping …
How to fillna limited by date in a groupby Question: I am working with the following Dataframe that has some NaN values inside. df = pd.DataFrame({‘day’:[pd.datetime(2020,1,1),pd.datetime(2020,1,3),pd.datetime(2020,1,4),pd.datetime(2020,1,5),pd.datetime(2020,1,6),pd.datetime(2020,1,7),pd.datetime(2020,1,8),pd.datetime(2020,1,8),pd.datetime(2020,6,9)], ‘TradeID’:[’01’,’02’,’03’,’04’,’05’,’06’,’07’,’08’,’09’], ‘Security’: [‘GOOGLE’, ‘GOOGLE’, ‘APPLE’, ‘GOOGLE’, ‘GOOGLE’,’GOOGLE’,’GOOGLE’,’GOOGLE’,’GOOGLE’], ‘ID’: [‘ID001’, ‘ID001’, ‘ID001’, ‘ID001’, ‘ID001′,’ID001′,’ID001′,’ID001′,’ID001’], ‘BSType’: [‘B’, ‘S’, ‘B’, ‘B’, ‘B’,’S’,’S’,’S’,’B’], ‘Price’:[105.901,106.969,np.nan,107.037,107.038,107.136,np.nan,107.25,np.nan], ‘Quantity’:[1000000,-300000,np.nan,7500000,100000,-100000,np.nan,-7800000,np.nan] }) Out[318]: day TradeID Security ID BSType Price …
How to use the value of one column as part of a string to fill NaNs in another column? Question: Let’s say I have the following df: year date_until 1 2010 – 2 2011 30.06.13 3 2011 NaN 4 2015 30.06.18 5 2020 – I’d like to fill all – and NaNs in the date_until …
