In a Transformer encoder (BERT, the left half of T5), every token attends to every other token bidirectionally. This means the representation of the word "bank" incorporates information from both "river" (left context) and "fishing" (right context) simultaneously. This bidirectional attention is why encoder representations are richer than decoder (left-to-right only) representations for understanding tasks.
The key distinction: encoders process input (understanding), decoders generate output (creation). Encoders see everything at once (bidirectional). Decoders see only past tokens (causal/left-to-right). This is why encoder models (BERT) are better for classification and search, while decoder models (GPT, Claude) are better for generation. Encoder-decoder models (T5, BART) use an encoder for input understanding and a decoder for output generation, connected by cross-attention.
Multimodal systems typically use separate encoders for each modality: a vision encoder (ViT) for images, a text encoder (BERT/CLIP) for text, and potentially audio encoders for speech. These produce embeddings in a shared space, enabling cross-modal understanding. The quality of each encoder determines how well the system understands that modality. This is why CLIP's training (aligning image and text encoders) was so impactful — it created a bridge between vision and language understanding.