获取pandas中每列的差异

Question

我想计算保存在Train和Test文件中的特征的方差a：

col1  Feature0  Feature1     Feature2   Feature3  Feature4  Feature5  Feature6  Feature7     Feature8     Feature9
col2     26658     40253.5  3.22115e+09  0.0277727   5.95939    266.56   734.248   307.364   0.000566779  0.000520574
col3     2658   4053.5     3.25e+09  0.0277   5.95939    266.56   734.248   307.364  0.000566779  0.000520574 
....

为此，我写了以下内容：

import numpy as np
from sklearn.decomposition import PCA
import pandas as pd
#from sklearn.preprocessing import StandardScaler
from sklearn import preprocessing
from matplotlib import pyplot as plt

# Reading csv file
training_file = 'Training.csv'
testing_file  = 'Test.csv'
Training_Frame = pd.read_csv(training_file)
Testing_Frame  = pd.read_csv(testing_file)
Training_Frame.shape
# Now we have the feature values saved we start
# with the standardisation of the those values
stdsc = preprocessing.MinMaxScaler()
np_scaled_train = stdsc.fit_transform(Training_Frame.iloc[:,:-2])

sel = VarianceThreshold(threshold=(.2 * (1 - .2)))
sel.fit_transform(np_scaled_train)
pd_scaled_train = pd.DataFrame(data=np_scaled_train)
pd_scaled_train.to_csv('variance_result.csv',header=False, index=False)

这显然不起作用。 variance_result.csv的结果只是火车矩阵归一化。所以我的问题是如何获得具有20％的差异的列（特征）的索引。提前致谢 ！

更新

我用这种方式解决了方差问题：

    import numpy as np
from sklearn.decomposition import PCA
import pandas as pd
#from sklearn.preprocessing import StandardScaler
from sklearn import preprocessing
from matplotlib import pyplot as plt
from sklearn.feature_selection import VarianceThreshold

# Reading csv file
training_file = 'Training.csv'
testing_file  = 'Test.csv'
Training_Frame = pd.read_csv(training_file)
Testing_Frame  = pd.read_csv(testing_file)

Training_Frame.shape
# Now we have the feature values saved we start
# with the standardisation of the those values
stdsc = preprocessing.MinMaxScaler()
np_scaled_train = stdsc.fit_transform(Training_Frame.iloc[:,:-2])
pd_scaled_train = pd.DataFrame(data=np_scaled_train)
variance =pd_scaled_train.apply(np.var,axis=0) 
pd_scaled_train.to_csv('variance_result.csv',header=False, index=False)
temp_df = pd.DataFrame(variance.values,Training_Frame.columns.values[:-2])
temp_df.T.to_csv('Training_features_variance.csv',index=False)

不，我仍然不知道如何得到一个方差的特征，比0.2更大的variance，其他感谢运行循环！

Answer 1

只需将阈值设置为0.0，然后使用VarianceThreshold对象的variances_属性来获取所有要素的差异，然后就可以确定哪些属性具有较低的方差。

from sklearn.feature_selection import VarianceThreshold
X = [[0, 2, 0, 3], [0, 1, 4, 3], [0, 1, 1, 3]]
selector = VarianceThreshold()
selector.fit_transform(X)

selector.variances_
#Output: array([ 0.        ,  0.22222222,  2.88888889,  0.        ])