GANs: Generative Adversarial Networks
Two networks in competition: one creates, one critiques. This adversarial training produces stunning results—and notoriously unstable training.
The GAN Game
Generator (G): Creates fake images from random noise Discriminator (D): Distinguishes real images from fakes
Noise z → Generator → Fake Image → Discriminator → Real or Fake?
↑
Real Image ────────┘
The generator wins when it fools the discriminator. The discriminator wins when it correctly classifies.
The Minimax Objective
$$\min_G \max_D \mathbb{E}{x \sim data}[\log D(x)] + \mathbb{E}{z \sim noise}[\log(1 - D(G(z)))]$$
In practice, we alternate:
- Train D to maximize: classify real as real, fake as fake
- Train G to maximize: fool D into classifying fake as real
The Training Loop
for real_images in dataloader:
# === Train Discriminator ===
optimizer_D.zero_grad()
# Real images should be classified as real
real_labels = t.ones(batch_size, 1)
real_output = discriminator(real_images)
loss_real = criterion(real_output, real_labels)
# Fake images should be classified as fake
noise = t.randn(batch_size, latent_dim)
fake_images = generator(noise)
fake_labels = t.zeros(batch_size, 1)
fake_output = discriminator(fake_images.detach())
loss_fake = criterion(fake_output, fake_labels)
loss_D = loss_real + loss_fake
loss_D.backward()
optimizer_D.step()
# === Train Generator ===
optimizer_G.zero_grad()
# Generator wants discriminator to think fakes are real
fake_output = discriminator(fake_images)
loss_G = criterion(fake_output, real_labels) # Fool D!
loss_G.backward()
optimizer_G.step()
The DCGAN Architecture
Deep Convolutional GAN with specific architectural choices:
Generator:
class Generator(nn.Module):
def __init__(self, latent_dim=100, img_channels=3):
super().__init__()
self.main = nn.Sequential(
# latent_dim → 512×4×4
nn.ConvTranspose2d(latent_dim, 512, 4, 1, 0, bias=False),
nn.BatchNorm2d(512),
nn.ReLU(True),
# 512×4×4 → 256×8×8
nn.ConvTranspose2d(512, 256, 4, 2, 1, bias=False),
nn.BatchNorm2d(256),
nn.ReLU(True),
# 256×8×8 → 128×16×16
nn.ConvTranspose2d(256, 128, 4, 2, 1, bias=False),
nn.BatchNorm2d(128),
nn.ReLU(True),
# 128×16×16 → 64×32×32
nn.ConvTranspose2d(128, 64, 4, 2, 1, bias=False),
nn.BatchNorm2d(64),
nn.ReLU(True),
# 64×32×32 → 3×64×64
nn.ConvTranspose2d(64, img_channels, 4, 2, 1, bias=False),
nn.Tanh(),
)
def forward(self, z):
return self.main(z.view(-1, z.size(1), 1, 1))
Discriminator:
class Discriminator(nn.Module):
def __init__(self, img_channels=3):
super().__init__()
self.main = nn.Sequential(
# 3×64×64 → 64×32×32
nn.Conv2d(img_channels, 64, 4, 2, 1, bias=False),
nn.LeakyReLU(0.2, inplace=True),
# 64×32×32 → 128×16×16
nn.Conv2d(64, 128, 4, 2, 1, bias=False),
nn.BatchNorm2d(128),
nn.LeakyReLU(0.2, inplace=True),
# 128×16×16 → 256×8×8
nn.Conv2d(128, 256, 4, 2, 1, bias=False),
nn.BatchNorm2d(256),
nn.LeakyReLU(0.2, inplace=True),
# 256×8×8 → 512×4×4
nn.Conv2d(256, 512, 4, 2, 1, bias=False),
nn.BatchNorm2d(512),
nn.LeakyReLU(0.2, inplace=True),
# 512×4×4 → 1×1×1
nn.Conv2d(512, 1, 4, 1, 0, bias=False),
nn.Sigmoid(),
)
def forward(self, img):
return self.main(img).view(-1, 1)
DCGAN Best Practices
From the original paper:
- Use strided convolutions instead of pooling
- Use BatchNorm in both G and D (except D input, G output)
- Use ReLU in G, LeakyReLU in D
- Use Tanh in G output, Sigmoid in D output
- Adam with lr=0.0002, β₁=0.5
These aren't arbitrary—they emerged from extensive experimentation.
Training Instabilities
GANs are notoriously hard to train:
Mode collapse: Generator produces only one type of image Vanishing gradients: Discriminator too good, no signal for G Oscillation: Neither network converges
Symptoms:
- D loss → 0, G loss → infinity (D winning)
- G produces all similar images (mode collapse)
- Losses oscillate wildly (unstable training)
Debugging Tips
Check discriminator accuracy:
# Should be ~50% when balanced
# If >90%, discriminator is too strong
real_acc = (real_output > 0.5).float().mean()
fake_acc = (fake_output < 0.5).float().mean()
Label smoothing:
# Instead of 1.0 for real
real_labels = t.full((batch_size,), 0.9)
Two-timescale update:
# Train D more often than G
for _ in range(n_critic):
train_discriminator_step()
train_generator_step()
Transposed Convolutions
How generators upsample:
Regular conv: Kernel slides inside input → smaller output Transposed conv: Kernel "stamps" onto output → larger output
# 4×4 → 8×8 with stride 2
nn.ConvTranspose2d(in_channels, out_channels, kernel_size=4, stride=2, padding=1)
Output size: $(H-1) \times stride - 2 \times padding + kernel_size$
Capstone Connection
Adversarial training and AI safety:
The GAN framework is fundamentally adversarial:
- One model tries to deceive
- Another tries to detect deception
This mirrors AI safety concerns:
- AI systems might learn to "fool" evaluators
- Sycophancy IS a form of fooling: tell humans what they want to hear
When we evaluate models for sycophancy, we're playing discriminator. The model might learn to pass our tests without being genuinely honest.
🎓 Tyla's Exercise
Derive why the generator gradient vanishes when D is too strong. (Hint: What's $\nabla_G \log(1 - D(G(z)))$ when $D(G(z)) \to 0$?)
The minimax objective has a Nash equilibrium at $D(x) = 0.5$ for all x. Prove this.
Why does DCGAN use LeakyReLU in the discriminator but ReLU in the generator?
💻 Aaliyah's Exercise
Train a DCGAN on CelebA faces:
def train_dcgan(epochs=20):
"""
1. Load CelebA (64×64 faces)
2. Train DCGAN
3. Log to wandb: D loss, G loss, sample images every epoch
4. Save model checkpoints
Target: Recognizable faces by epoch 10
"""
pass
def explore_latent_space():
"""
1. Generate two random z vectors
2. Interpolate between them
3. Generate images along the path
4. Make a GIF of the morph
"""
pass
📚 Maneesha's Reflection
The GAN training dynamic is often described as a "game." What kind of game is it? (zero-sum? cooperative?)
Mode collapse can be seen as the generator "gaming the metric." How does this relate to Goodhart's Law?
If you were explaining GANs to an artist, how would you describe what the generator and discriminator "learn"?