Skip to main content
Zubnet AI › Learn › Wiki › Gradient Boosting
Fundamentals

Gradient Boosting

Also known as: Decision Trees, XGBoost, LightGBM, GBDT
A classical machine learning method that builds many small decision trees in sequence, each one correcting the errors of the ones before it, and still the top choice for spreadsheet-style data.

Why it matters

Not every AI problem needs a neural network. For tabular data, meaning rows and columns such as customer records, transactions, sensor readings or medical measurements, gradient-boosted trees remain the default, and they regularly match or beat deep learning in benchmarks and competitions. They train in seconds or minutes on a CPU, need little data preparation, handle missing values and mixed column types, and work with a few thousand rows.

A huge share of production machine learning is exactly this: fraud detection, credit scoring, churn prediction, demand forecasting, ad ranking. If you have a spreadsheet and a column you want to predict, start here, not with an LLM.

Deep Dive

A decision tree is the most readable model in machine learning: a flowchart of yes/no questions ("is income above 50,000?", "is the account older than two years?") that ends in a prediction. One tree alone is weak and overfits easily. The trick is to combine many of them.

Gradient boosting builds trees one after another. The first tree makes a rough prediction. The second is trained on the first one's errors, the third on what's still wrong, and so on for hundreds or thousands of rounds. Each tree is small, but their sum is very accurate. The method was formalized by Jerome Friedman around 2001, and libraries such as XGBoost (2014), LightGBM (2016) and CatBoost (2017) made it fast and easy to use.

Two Ways to Combine Trees

  • Random forests (bagging): train many deep trees independently, each on a random sample of the data and features, and average them. Averaging cancels out the individual trees' noise. Robust and hard to get wrong.
  • Gradient boosting: train shallow trees sequentially, each fitting the gradient of the loss, that is, the direction in which the current predictions are most wrong. Usually more accurate than a random forest, but more sensitive to settings.

The Knobs That Matter

  • Number of trees and learning rate: each tree's contribution is scaled down by the learning rate. A lower rate with more trees usually generalizes better. Use early stopping on a validation set to pick the count.
  • Tree depth or number of leaves: controls how complex each tree is. Shallow trees (depth 3 to 8) are typical.
  • Subsampling: using a random fraction of rows and columns per tree adds regularization.

Why Trees Beat Neural Networks Here

Tabular columns have very different meanings and scales, the important patterns are often sharp thresholds, and many columns are irrelevant. Trees handle all three naturally. Neural networks prefer smooth patterns and large homogeneous inputs such as pixels or tokens. Newer approaches, such as the TabPFN foundation model for small tables, are closing the gap in some settings, but boosted trees remain the baseline to beat.

Seeing Inside

Trees offer useful interpretability: feature importance scores, and exact SHAP values that explain how much each column pushed a single prediction up or down. That matters in regulated uses such as lending.

Where LLMs Fit

LLMs and boosted trees complement each other. A common pattern is to use an LLM or embedding model to turn messy text into features (a category, a sentiment score, an embedding) and feed those into a boosted model alongside the numeric columns.

← All Terms
ESC