MIT Technology Review's Eileen Guo published a substantive privacy investigation Wednesday on AI chatbots surfacing real, working phone numbers in their outputs. The named victim is Daniel Abraham, a 28-year-old software engineer in Israel who started receiving WhatsApp messages from strangers asking for PayBox customer support — Google Gemini had been hallucinating his personal number as PayBox's WhatsApp service line. Abraham does not work for PayBox; PayBox does not offer WhatsApp customer service; PayBox confirmed this to MIT Tech Review. When the reporter independently queried Gemini, it returned a different person's number — an Israeli credit-card-company employee. Two other documented cases: a Reddit user who was deluged for a month by callers looking for "a lawyer, a product designer, a locksmith" (apparently misdirected by Google AI), and a University of Washington PhD candidate who casually got Gemini to surface a colleague's personal cell number.

The hard data point worth pinning is from DeleteMe, the personal-information-removal company: customer queries about generative AI have jumped 400% over the last seven months, into the low thousands. The breakdown of which chatbot users complain about: 55% ChatGPT, 20% Gemini, 15% Claude, 10% other. DeleteMe's CEO Rob Shavell describes two failure modes: (1) the user asks the chatbot something about themselves and gets back accurate home address, phone, family members, or employer; (2) the user reports the chatbot surfacing someone else's contact info, sometimes "plausible-but-wrong." Both modes are showing up across the major chatbots. California's data broker registry adds another data point: 31 of 578 registered brokers self-reported selling or sharing consumer data with a GenAI developer in the past year — a small fraction of the registry but a measurable formal channel for PII into training sets.

The mechanism is the part builders need to internalize. LLM training crawls include hundreds of millions of PII instances — Abraham's number, for instance, appears to have been posted once on a 2015 Quora-like site, and that single 11-year-old post is the most plausible source for Gemini reproducing it now. The DataComp CommonPool dataset documented last summer included scanned résumés, driver's licenses, and credit cards. Models memorize and reproduce data verbatim from training sets, and recent research suggests it's not just high-frequency content that gets memorized — rare items can be retained too. Standard guardrails (PII content filters, Claude's "least personal information" instruction) catch some of this but visibly fail when the model treats the PII as authoritative customer-service data rather than as private content. The structural fix is at the training-data preprocessing layer, not at output filtering — but no major lab has published a comprehensive PII-redaction audit that would let third parties verify their training corpora are clean.

For builders: if you ship a product that wraps a foundation model and exposes outputs to end users, PII leakage is now a real surface area. Three concrete actions: (1) add output-side phone/email/address regex + named-entity detection before passing model responses to users, treating them as content-policy violations to suppress or rewrite; (2) ask your model vendor for documentation on training-data PII redaction practices and memorization-mitigation techniques (differential privacy, deduplication, k-anonymity at scale) — most will dodge but the question creates a paper trail; (3) build an end-user reporting mechanism for "this output exposed my information" with a fast-path retraction process. For end users and policymakers: the DeleteMe 400% spike is the leading indicator that this category of harm is no longer theoretical, and the structural fix sits at the lab level, which means regulator pressure is the most likely lever. Abraham's quote — "what if I asked for money to solve that customer service issue?" — names the harm shape: not just nuisance calls, but a new attack surface where a bad actor can monetize Gemini-misdirected traffic.