Covering CelebA Bias
With Markov Blankets

Motivation: Attractive Barack Obama $\neq$ White Blone Barack Obama

CelebA has been a popular choice for deep generative models, randing from Progressive Growing of GANs for Improved Quality, Stability, and Variation from NVIDIA, to Glow: Generative Flow with Invertible 1x1 Convolutions from OpenAI. These generative models produce realistic human portraits that are indistinguishable from real portraits to the human eye.

Such generative models directly learn the distribution of the data, $p(x)$. However, CelebA has 30,000 images, each with 40 binary semantic attributes such as blonde hair, mustache, and smiling. By directly modeling the distribution of pixels of CelebA without considering the co-occurences of these attributes, we face inter-attribute spill-over effects which reveal the inherent bias of the dataset.

We use Glow as an example and manipulate across the attractive attribute, which reveals the inherent bias: by increasing 'attractiveness', we lose information about the existing profile and generalize to a young, blonde female.

Where does this bias come from?

Liu et. al. claimed that CelebA was annotated by a professional labeling company. While no parties have been identitified, we draw similarities to Amazon Mechanical Turk and examine the workers labelling datasets.

CelebA-HQ which was generated from CelebA also inherits the bias associated with it.

(We) discard all but the best 30000 images. We use a frequency-based quality metric that favors images whose power spectrum contains a broad range of frequencies and is approximately radially symmetric. This penalizes blurry images as well as images that have conspicuous directional features ... We selected the cutoff point of 30000 images as a practical sweet spot between variation and image quality, because it appeared to yield the best results. - Karras et. al., Appendix C

Modeling Existing Biases: Diving Into Ising Models

Now that we know these biases exist, how do we explicitly model them and deploy corrective countermeasures? Here we introduce a variant called Ising Model, which borrows from physics.

Results

# Train flow model on large, unlabelled dataset X
m = train(X_unlabelled)

# Split labelled dataset based on attribute, say blonde hair
X_positive, X_negative = split(X_labelled)

# Obtain average encodings of positive and negative inputs
z_positive = average([m.encode(x) for x in X_positive])
z_negative = average([m.encode(x) for x in X_negative])

# Get manipulation vector by taking difference
z_manipulate = z_positive - z_negative

# Manipulate new x_input along z_manipulate, by a scalar alpha \in [-1,1]
z_input = m.encode(x_input)
x_manipulated = m.decode(z_input + alpha * z_manipulate)

Corrective Bias in Action: Glow

Using Glow as an example, we manipulate the latent vector across the positive direction for different attributes, and illustrate the effect of applying corrective measures that preserve the inherent priors and prevents the image from converging to a generalized learnt bias.

Universality of Existing Bias in CelebA/CelebA-HQ beyond GLOW

We emphasize that this bias is present in the inherent dataset, and not constrained to any particular model. Though in our example we interpolate along the latent attributes using GLOW as a flow model, this phenomena is present across other generative models including VAEs and GANs (citation here) that are trained on CelebA that explicitly learn to model p(x) instead of p(x|prior). For example, the inherent bias are present in Generative Adversarial Interpolative Autoencoding (GAIA) and DNA-GAN: Learning Disentangled Representations from Multi-Attribute Images.