利用熊猫清除数据中的本地异常点

Question

有谁能建议如何从数据中删除本地的离群点？我有检测本地异常值的代码，但我需要帮助删除它们(将这些值设置为零)。如有任何建议，将不胜感激。

检测本地异常值的代码如下：

def printOutliers(series, window, scale= 1.96, print_outliers=False):

rolling_mean = series.rolling(window=window).mean()

#Print indices of outliers
if print_outliers:
    mae = mean_absolute_error(series[window:], rolling_mean[window:])#mean absolute error is a measure of difference between two continuous variables. 
    deviation = 3*np.std(series[window:] - rolling_mean[window:])
    lower_bound = rolling_mean - (mae + scale * deviation)
    upper_bound = rolling_mean + (mae + scale * deviation)
    outliers_lower = series[series<lower_bound]
    outliers_upper = series[series>upper_bound]
    print("values beyond lower bound are: " +  "\n"  + str(outliers_lower))
    print("values beyond lower bound are: " + "\n" + str(outliers_upper))  

printOutliers(df['Column1'].dropna(how='any'), 10, print_outliers=True)

Answer 1

您可以在这里使用这种方法：

#------------------------------------------------------------------------------
# accept a dataframe, remove outliers, return cleaned data in a new dataframe
# see http://www.itl.nist.gov/div898/handbook/prc/section1/prc16.htm
#------------------------------------------------------------------------------
def remove_outlier(df_in, col_name):
    q1 = df_in[col_name].quantile(0.25)
    q3 = df_in[col_name].quantile(0.75)
    iqr = q3-q1 #Interquartile range
    fence_low  = q1-1.5*iqr
    fence_high = q3+1.5*iqr
    df_out = df_in.loc[(df_in[col_name] > fence_low) & (df_in[col_name] < fence_high)]
    return df_out