Skip to main content
Zubnet AI › Learn › Wiki › PyTorch
Tools

PyTorch

Also known as: Torch
The open-source deep learning framework that most AI research and most open models are built with.

Why it matters

PyTorch is the default language of modern AI. The overwhelming majority of research papers ship PyTorch code, and the models you download from Hugging Face, such as Llama, Qwen, Mistral and DeepSeek, are typically defined in PyTorch. Hugging Face's Transformers library, vLLM, SGLang and most fine-tuning tools are built on it.

Even if you never write a training loop, PyTorch is underneath much of what you run. When you see a CUDA out of memory error, a .pt file or an installation step that asks for a specific PyTorch and CUDA version, that's PyTorch.

Deep Dive

PyTorch is a Python library for building and training neural networks. It gives you tensors (multi-dimensional arrays that can live on a GPU), automatic differentiation (it works out gradients for you, so backpropagation is one line of code), and building blocks for layers, optimizers and data loading. It was released by Facebook's AI research lab (now part of Meta) in early 2017, building on the older Lua-based Torch library.

Since 2022, PyTorch has been governed by the PyTorch Foundation under the Linux Foundation, with members including Meta, Microsoft, Google, NVIDIA, AMD and AWS.

Why It Won

When PyTorch arrived, the leading framework was Google's TensorFlow, which at the time required you to define a whole computation graph first and then run it. PyTorch ran code eagerly: each line executed immediately, like normal Python. You could print a tensor mid-model, use regular if statements and loops, and debug with standard tools. Researchers loved it, and by around 2019 to 2020 it dominated new research. Industry followed, because the research models were in PyTorch.

Key Pieces

  • torch.Tensor: the core data type, on CPU or GPU (.to("cuda")).
  • Autograd: records operations on tensors and computes gradients with loss.backward().
  • torch.nn: layers, activation functions and loss functions to compose models.
  • torch.optim: optimizers such as SGD and AdamW.
  • torch.compile: added in PyTorch 2.0 (2023), it compiles models into faster kernels without rewriting your code.
  • Distributed tools: DDP and FSDP for training across many GPUs.

Hardware Support

PyTorch runs best on NVIDIA GPUs through CUDA, which is the most tested path. It also supports AMD GPUs through ROCm, Apple Silicon through the MPS backend, Intel GPUs and Google TPUs (via PyTorch/XLA). Support outside NVIDIA has improved a lot but can still lag on new features and custom kernels.

The Ecosystem Around It

  • Hugging Face Transformers, Diffusers, PEFT, TRL: model libraries and fine-tuning tools.
  • PyTorch Lightning: structure for training code.
  • vLLM and SGLang: high-throughput inference servers.
  • ExecuTorch: running PyTorch models on phones and edge devices.

Its main alternative today is Google's JAX, used to train Gemini and popular in some research labs. On Macs, Apple's MLX is a lighter option for local work.

← All Terms
ESC