Standard FFN: FFN(x) = W2 · GELU(W1 · x). Two weight matrices, one activation. SwiGLU FFN: SwiGLU(x) = W2 · (SiLU(W1 · x) ⊗ W3 · x). Three weight matrices, a gating mechanism. The gate (W3 · x) controls what passes through, letting the network selectively suppress or amplify different features. To keep parameter count constant, the intermediate dimension is typically reduced from 4×model_dim to (8/3)×model_dim.
Gating gives the network a multiplicative interaction that standard activations lack. Standard activations apply a fixed non-linearity. Gating applies a learned, input-dependent non-linearity. This additional expressiveness helps the network learn more complex functions per layer, which means you need fewer layers (or smaller layers) for equivalent performance. Shazeer (2020) showed that GLU variants consistently outperform standard FFN across model sizes.
SwiGLU is one of several GLU variants: GeGLU (uses GELU instead of SiLU), ReGLU (uses ReLU), and the original GLU (uses sigmoid). SwiGLU and GeGLU perform similarly and both outperform ReGLU. The choice between them is mostly empirical — SwiGLU has become the default through convention (LLaMA adopted it, others followed) rather than clear theoretical superiority over GeGLU.