Model Fusion with Kullback--Leibler Divergence

Part of Proceedings of the International Conference on Machine Learning 1 pre-proceedings (ICML 2020)

Bibtex »Metadata »Paper »Supplemental »

Bibtek download is not availble in the pre-proceeding


Authors

Sebastian Claici, Mikhail Yurochkin, Soumya Ghosh, Justin Solomon

Abstract

<p>We propose a method to fuse posterior distributions learned from heterogeneous datasets. Our algorithm relies on a mean field assumption for both the fused model and the individual dataset posteriors, and proceeds in a simple assign-and-average approach. The components of the dataset posteriors are assigned to the proposed global model components by solving a regularized variant of the assignment problem. The global components are then updated based on these assignments by their mean under a KL divergence. For exponential family variational distributions, our formulation leads to an efficient non-parametric algorithm for computing the fused model. Our algorithm is easy to describe and implement, efficient, and performs competitive with state-of-the-art when tested on motion capture analysis, topic modeling, and federated learning of Bayesian neural networks.</p>