The term "feature" has different meanings depending on context. In classical ML, features are hand-engineered input variables (height, weight, age). In deep learning, features are learned representations in hidden layers — the model discovers useful patterns on its own. This shift from hand-engineered to learned features is the core innovation of deep learning and why it outperforms classical ML on complex tasks like vision and language.
Deep networks learn hierarchical features: each layer builds on the previous one. In a vision model: layer 1 detects edges, layer 2 combines edges into textures and corners, layer 3 combines textures into object parts (eyes, wheels), layer 4 combines parts into objects (faces, cars). This hierarchy emerges automatically from training — no one programs it. The same hierarchical feature learning happens in language models, from character patterns to syntax to semantics to reasoning.
Researchers visualize features to understand what models learn. For vision models, you can generate images that maximally activate a specific neuron or direction, revealing what pattern it detects. For language models, you can find the text examples that most activate a specific feature direction. Anthropic's research has visualized features in Claude, finding interpretable concepts like "Golden Gate Bridge," "code bugs," "deception," and "French language" encoded as specific directions in the model's activation space.