Databricks中的Shap渲染

Question

我正在尝试将Shap指数可视化，以解释xgboost机器学习模型。我可以在google collab中做到这一点，但我在Databricks中很难做到这一点。

import shap
import pandas as pd
import numpy as np
import matplotlib
import xgboost as xgb

df = pd.read_csv("/dbfs/databricks-datasets/Rdatasets/data-001/csv/ggplot2/diamonds.csv").iloc[:,1:]

#label encode the categorical features:
df = pandasData.copy() 
df["clarity"] = df["clarity"].astype('category')
df["color"] = df["color"].astype('category')
df["cut"] = df["cut"].astype('category')

df["clarity_cat"] = df["clarity"].cat.codes
df["color_cat"] = df["color"].cat.codes
df["cut_cat"] = df["cut"].cat.codes
df= df.drop(["clarity","color","cut"],axis = 1)

Y = df['price']
X = df.drop(['price'], axis=1)

X_traintest, X_valid, Y_traintest, Y_valid = train_test_split(X, Y, test_size=0.3, random_state=7)

model = xgb.XGBRegressor(learning_rate=0.02, n_estimators=3161, max_depth=3)
model.fit(X_traintest, Y_traintest, eval_metric="rmse", verbose = False)

在这个时刻，这是可行的

shap.initjs() # load JS visualization code to notebook

explainer = shap.TreeExplainer(model) # explain the model's predictions using SHAP values

shap_values = explainer.shap_values(X_traintest)
    shap_explain = shap.force_plot(explainer.expected_value, shap_values[0,:], X.iloc[0,:]) # visualize the first prediction's explanation
    display(shap_display)

并产生

​

这不起作用(但在Colab中起作用了)：

shap.initjs()
shap_display = shap.dependence_plot('y', shap_values, X_traintest)
display(shap_display) #tried matplotlib=True/False)

最后看起来像这样

​

​

当它看起来像这样的时候

​

​

Answer 1

你能看看这个能不能用？

displayHTML(shap_display.to_html()