The typical architecture: images are processed by a vision encoder (usually a Vision Transformer or CLIP variant) that converts image pixels into a sequence of visual tokens. These tokens are projected into the same embedding space as text tokens and concatenated with the text input. The language model then processes both visual and text tokens together through its standard attention layers, enabling cross-modal reasoning.
Current vision models excel at: describing image content, reading text in images (OCR), understanding charts and diagrams, identifying objects and people (when appropriate), and reasoning about spatial relationships. They struggle with: precise counting (especially in cluttered scenes), fine-grained spatial reasoning ("is A above or below B?"), reading small or stylized text, and understanding images that require domain expertise (medical scans, specialized equipment).
Higher resolution images produce more visual tokens, consuming more context window and costing more. Most providers automatically resize or tile images to balance quality and cost. A typical image might produce 500–2000 tokens. Understanding this helps you optimize: don't send a 4K screenshot when a 1080p crop of the relevant area would work better and cost less.