The attention weight matrix is (seq_len × seq_len) for each head and layer. To visualize: pick a layer and head, display the matrix as a heatmap where row i shows which tokens token i attends to. Bright cells mean high attention. For multi-head attention, you can visualize individual heads (each specializes in different patterns) or average across heads (overall attention distribution).
Attention maps show which tokens a head considers when computing its output, but they don't directly show what the model "understands" or why it made a decision. High attention doesn't mean "important" — some heads attend to punctuation or positional patterns without semantic meaning. Attention maps are descriptive (what the model looked at) not explanatory (why it made its decision). They're a useful debugging tool, not a complete explanation.
BertViz provides interactive attention visualizations for Transformer models. Ecco and Captum offer attention-based interpretability for PyTorch models. For LLMs accessed via API, some providers return attention weights or log-probabilities that enable partial visualization. In image generation, cross-attention maps show which image regions correspond to which prompt words — useful for understanding why the model placed objects where it did.