Information-Theoretic Segmentation by Inpainting Error Maximization

Illustration of our Inpainting Error Maximization (IEM) framework for completely unsupervised segmentation, applied to flowers, birds, and cars. Segmentation masks maximize the error of inpainting foreground given background and vice-versa.

Abstract

We study image segmentation from an information-theoretic perspective, proposing a novel adversarial method that performs unsupervised segmentation by partitioning images into maximally independent sets. More specifically, we group image pixels into foreground and background, with the goal of minimizing predictability of one set from the other. An easily computed loss drives a greedy search process to maximize inpainting error over these partitions. Our method does not involve training deep networks, is computationally cheap, class-agnostic, and even applicable in isolation to a single unlabeled image. Experiments demonstrate that it achieves a new state-of-the-art in unsupervised segmentation quality, while being substantially faster and more general than competing approaches.

Information-theoretic segmentation

Our Inpainting Error Maximization (IEM) framework is motivated by the intuition that a segmentation into objects minimizes the mutual information between the pixels in the segments and hence makes inpainting of one segment given the others difficult. This gives a natural adversarial objective where a segmenter tries to maxmize, while an inpainter tries to minimize, inpainting error. However, rather than adopt an adversarial training objective we found it more effective to fix a basic inpainter and directly maximize inpainting error through a form of gradient descent on the segmentation. IEM is learning-free and can be applied directly to any image in any domain. Given an unlabeled image X, a mask generator module first produces segmentation masks (e.g., foreground M and background M). Each mask selects a subset of pixels from the original image by performing an element-wise product between the mask and the image, hence partitioning the image into regions. Inpainting modules try to reconstruct each region given all others in the partition, and the IEM loss is defined by a weighted sum of inpainting errors. Foreground and background inpainting results using ground-truth, eroded (smaller), and dilated (bigger) masks. We see that the ground-truth mask incurs high inpainting error for both the foreground and the background, while the eroded mask allows reasonable inpainting of the foreground and the dilated mask allows reasonable inpainting of the background. Hence we expect IEM, which maximizes the inpainting error of each partition given the others, to yield a segmentation mask close to the ground-truth.

Results

Unsupervised segmentation results on Flowers, CUB-200-2011, and LSUN Car. Segmentation masks used for evaluation are publicly available ground-truth (Flowers, CUB-200-2011) or automatically generated with Mask R-CNN (LSUN Car).

Related Work

           

Contact