The standard FFN: FFN(x) = W2 · activation(W1 · x + b1) + b2, where W1 projects from the model dimension to a larger intermediate dimension (typically 4x), the activation function introduces non-linearity, and W2 projects back to the model dimension. Each position (token) passes through this independently — the FFN doesn't see other tokens, only the attention layer does.
Modern LLMs (LLaMA, Mistral, etc.) use SwiGLU instead of the standard FFN: SwiGLU(x) = (W1 · x · SiLU) ⊗ (W3 · x). This adds a third weight matrix (W3) and a gating mechanism that lets the network control what information passes through. Despite the extra parameters, it performs better at equivalent compute, so the intermediate dimension is adjusted down to compensate. This is a case where a slightly more complex component improves the whole system.
Research suggests that FFN layers function like key-value memories: the first linear layer (W1) detects patterns in the input (keys), and the second linear layer (W2) maps those patterns to output updates (values). "The Eiffel Tower is in" activates specific neurons in W1, which through W2 promote the token "Paris." This key-value interpretation explains why FFN layers store factual knowledge and why knowledge editing techniques can modify specific facts by updating specific FFN weights.