Consistent Estimators for Learning to Defer to an Expert

Part of Proceedings of the International Conference on Machine Learning 1 pre-proceedings (ICML 2020)

Bibtex »Metadata »Paper »Supplemental »

Bibtek download is not availble in the pre-proceeding


Authors

Hussein Mozannar, David Sontag

Abstract

<p>Learning algorithms are often used in conjunction with expert decision makers in practical scenarios, however this fact is largely ignored when designing these algorithms. In this paper we explore how to learn predictors that can either predict or choose to defer the decision to a downstream expert. Given samples of the expert's decisions, we give a procedure based on learning a classifier and a rejector and analyze it theoretically. Our approach is based on a reduction to cost sensitive learning where we give a novel calibrated surrogate loss that resolves the open problem of (Ni et al., 2019) for multiclass rejection learning. We show the effectiveness of the new surrogate loss and approach on image and text classification tasks.</p>