Abstract: There is a growing demand for explainable and interpretable model predictions in high-stake decision-making machine learning applications to ensure transparency, trust, and accountability.