Falcon
Why it matters
Deep Dive
Falcon is a family of decoder-only large language models developed by the Technology Innovation Institute, an Abu Dhabi research lab backed by the UAE government. The first releases landed in 2023: a 7B and a 40B model trained on the lab's own RefinedWeb dataset, followed later that year by Falcon 180B, a 180-billion-parameter model trained on about 3.5 trillion tokens that sat at the top of the Hugging Face Open LLM Leaderboard at release. Crucially, TII published the weights under permissive terms — the 40B moved to Apache 2.0 shortly after launch — which made Falcon the strongest genuinely open-weight option for much of 2023. Later generations kept coming: Falcon 2 in 2024, Falcon 3 later that year in sizes from 1B to 10B, and experimental branches like Falcon-Mamba and the hybrid Falcon-H1 that move beyond the standard transformer recipe.
The RefinedWeb Bet
Most models of that era trained on carefully curated mixtures of books, code, academic papers, and filtered web text. TII's bet with Falcon was different: take raw Common Crawl web data, apply aggressive deduplication and quality filtering at scale, and show that sheer volume of well-cleaned web text can match fancy curation. The result was RefinedWeb, a multi-trillion-token dataset that TII released publicly alongside the models. Falcon 40B was trained on roughly one trillion tokens of it, while the 180B model scaled that to about 3.5 trillion tokens with some curated corpora mixed in. Pre-training almost entirely on public web data was unusual at that scale, and RefinedWeb itself became a widely reused asset for other open-model efforts long after the models themselves were superseded.
Licensing as a Competitive Weapon
Timing made Falcon's license as important as its weights. When Falcon 40B appeared in mid-2023, Meta's first Llama was research-only and most other strong models sat behind APIs, so an Apache 2.0 release — adopted shortly after launch, replacing a custom license with a royalty clause — made Falcon the default choice for teams building commercial products. That permissiveness helped force the broader open-versus-closed conversation: within months, Llama 2 arrived with commercial use allowed under its community license, and permissive licensing became an expectation rather than a differentiator. Falcon 3 and later releases moved to a TII Falcon License that stays Apache-style but adds its own conditions, which is a reminder that 'open weights' covers a spectrum of terms and every license is worth actually reading.
The Leaderboard Didn't Buy Mindshare
A common reading of Falcon's history is that it won the open-model race in 2023 because it topped the Hugging Face leaderboard. It didn't. Benchmark position translated poorly into ecosystem: Mistral and Llama captured the community of fine-tuners, tooling authors, and tutorial writers, and their models accumulated the quantized builds, merge recipes, and deployment guides that drive real adoption. Falcon 180B in particular was more a prestige artifact than a practical one — at hundreds of gigabytes of VRAM before quantization, almost nobody could run it, and its fine-tuning costs ruled it out for most teams. The lesson generalizes: in open models, distribution, hardware fit, and tooling compound, while leaderboard points decay quickly.
Beyond the Dense Transformer
Later Falcon releases show TII treating the family as an architecture research program rather than a single product line. Falcon 2 arrived in 2024 as an 11B model with a vision-language sibling that could process images, and Falcon 3 followed later that year, targeting the practical end of the market with 1B, 3B, 7B, and 10B sizes and multilingual training. The more interesting moves are the side branches: Falcon-Mamba 7B swapped attention entirely for a Mamba state-space architecture, giving constant-memory inference over long sequences, and the Falcon-H1 series followed in 2025 with hybrid designs that mix attention layers and state-space layers inside one model. These experiments have kept TII visible in the architecture conversation even as the mainstream Falcon line competes against far larger open-weights ecosystems.