我有一个名为

Question

repo_name

）组成，每个存储库都有多个文件（

file_name

），并且定义了每个文件的超参数配置。 我正在尝试找到超参数之间的相关性，但是我不确定如何将

hyperparam_name

列分解为单独的列？我还需要将分类超级参数转换为数值。我以前从未处理过这样的情况，所以不确定如何解决这个问题。关于我如何做到这一点的任何建议或想法将不胜感激！
    

1st）展开HyperParam_name列

hyperparam_name

2nd）句柄分类变量

import pandas as pd

# Assuming hyperparam_df is your dataframe
expanded_df = pd.json_normalize(hyperparam_df['hyperparam_name'])

# Concatenate the expanded columns with the original dataframe
hyperparam_df = pd.concat([hyperparam_df.drop(columns=['hyperparam_name']), expanded_df], axis=1)

3rd）处理缺失值

from sklearn.preprocessing import LabelEncoder

# Identify categorical columns
categorical_columns = hyperparam_df.select_dtypes(include=['object']).columns

# Apply one-hot encoding or label encoding
for col in categorical_columns:
    if hyperparam_df[col].nunique() > 10:  # Example threshold for using label encoding
        le = LabelEncoder()
        hyperparam_df[col] = le.fit_transform(hyperparam_df[col])
    else:
        hyperparam_df = pd.get_dummies(hyperparam_df, columns=[col], prefix=[col])

Answer 1

4th）计算相关性

# Fill missing values with a default value, e.g., 0
hyperparam_df = hyperparam_df.fillna(0)

# Alternatively, drop rows with missing values
# hyperparam_df = hyperparam_df.dropna()

例如：

correlation_matrix = hyperparam_df.corr()

# Optionally, visualize the correlation matrix using a heatmap
import seaborn as sns
import matplotlib.pyplot as plt

plt.figure(figsize=(12, 8))
sns.heatmap(correlation_matrix, annot=True, cmap='coolwarm', fmt='.2f')
plt.show()