Outlier detection of time-series data Question: I have a pandas dataframe where I want to detect outliers on a single column. Please bear in mind that I am not experienced when it comes to data handling/cleaning. The dataframe looks like this: Time MW 2019-01-01 00:00:00 1234.0 2019-01-01 01:00:00 1234.5 2019-01-01 02:00:00 1235.2 2019-01-01 03:00:00 1235.1 …
Error message 'Data frame name not defined' when plotting histogram in python Question: I want to plot a histogram for all numerical columns in my data frame. An error message keeps on coming up saying my data frame name (Cab_Data) is not defined. I have installed and imported matplotlib as plt, I used the code …
Why do these different outlier methods fail to detect outliers? Question: I am trying to find the outliers by group for my dataframe. I have two groups: Group1 and Group2, and I am trying to find the best way to implement an outlier method data = {‘Group1’:[‘A’, ‘A’, ‘A’, ‘B’, ‘B’, ‘B’,’A’,’A’,’B’,’B’,’B’,’A’,’A’,’A’,’B’,’B’,’B’,’A’,’A’,’A’,’B’,’B’,’B’,’A’,’A’,’A’,’A’,’A’,’B’,’B’], ‘Group2’:[‘C’, ‘C’, ‘C’, …
Laplace correction with conditions for smoothing Question: I have a data (user_data) that represent the number of examples in each class (here we have 5 classes), for example in first row, 16 represent 16 samples in class 1 for user1, 15 represent that there is 15 samples belong to class 2 for user 1, ect. …
Unable to assign different values in each cell of a column in dataframe, containing 99,000 records Question: I want to change values greater than 70 in column CT_feat7 but it only changes till 59000. After that, I have to run the iteration again, with a different index value. Please, explain why this happens. Is there …
find in datfarame outliers and fill with nan python Question: I am trying to make a function to spot the columns with "100" in the header and replace all values in these columns that are above 100 with nan values : import pandas as pd data = {‘first_100′: [’25’, ‘1568200’, ‘5’], ‘second_column’: [‘first_value’, ‘second_value’, ‘third_value’], …
save outlier removed data back to new csv file Question: I have a pandas dataframe and I am experimenting with sci-kit learn Novelty and Outlier Detection. I am trying figure out how to save my good dataset back to new a new CSV file after the outlier detector flags outliers. import numpy as np import …
Using k-Means Clustering to try to identify a 2D outlier shows no outliers at all (instead of one) Question: I was working my way through An Introduction to Outlier Analysis by Charu Aggarwal and doing Exercise 7 from Chapter 1. I am trying to use k-Means Clustering to identify an outlier in my data. What …
How to detect outliers in a timeseries dataframe and write the "clean" ones in a new dataframe Question: I’m really new to Python (and programming in general, hihi) and I’m analyzing 2 years of metereological data measured every 10s, in total I have 12 metereological parameters and I’ve created my dataframe df with the time …
Outliers in Categorical Data? Question: I am unable to find a solution to find outliers in categorical data. My data consists of combinations of rows. I want to mark outliers that differ in certain combinations. In the above question as specified, I cannot cluster the data as a nonoutlier data row and the outlier row …
