How should I generate outliers randomly?

Question

I’m generating a random dataset. My dataset is sequential, and has upper and under limits. At some random points, I want my dataset to have outliers above and under limits. Here’s my code.

generated_data = (12) * np.random.rand(100) + 630
outlier_data = (12) * np.random.rand(20) + (*HERE'S THE PROBLEM)
merged_data = np.concatenate((generated_data, outlier_data))

After this, I think I will shuffle the merged_data. But I don’t know how to generate outliers properly.

Asked By: 1a2a3a 4a5a6a

||

Source

Answer 1

Just generate three parts of the data independently: first non-outliers, then lower- and upper outliers, merge them together, and finally shuffle them:

def generate(median=630, err=12, outlier_err=100, size=80, outlier_size=10):
    errs = err * np.random.rand(size) * np.random.choice((-1, 1), size)
    data = median + errs

    lower_errs = outlier_err * np.random.rand(outlier_size)
    lower_outliers = median - err - lower_errs

    upper_errs = outlier_err * np.random.rand(outlier_size)
    upper_outliers = median + err + upper_errs

    data = np.concatenate((data, lower_outliers, upper_outliers))
    np.random.shuffle(data)

    return data

You’ll get something like this:

>>> data = generate()
>>> data.shape
(100,)
>>> data.min()
518.1635764484727
>>> data.max()
729.9467630423616
>>> np.median(data)
629.9427184256936

Answered By: constt

Answer 2

def generate_outlier(data,perc):
   perc/=100
   lower_outlier=np.random.randint(data.min()-300,data.min()-100,size= (int(data.size/2),1))
   upper_outlier=np.random.randint(data.max()+100,data.max()+300,size=(int(data.size/2),1))
   outlier=np.concatenate((lower_outlier,upper_outlier))
   np.random.shuffle(outlier)
   outlier=pd.DataFrame(np.reshape(outlier,data.shape))
   outlier=outlier.mask(np.random.random(data.shape)>perc)
   result=outlier.fillna(data)
return result

Answered By: Ankit

How should I generate outliers randomly?

Question:

Answers: