Tailscale Traces Backup Corruption to a 16-Year-Old SQLite Bug
A corruption incident that could not be reproduced from the code or recovered at the time turned into a shared hunt with the database's own developers, and the replay pipeline built for recovery is what narrowed the search.
Reporting from 1 source: GIGAZINE.
Tailscale used SQLite as its primary database starting in 2022. From August 2025, backups upload to S3 began showing corruption, occurring 19 times over six months, each time forcing a control plane process to stop for repair or restore. Recent code changes were ruled out. Tailscale signed a support contract with SQLite developers and built a transaction log pipeline, which helped lead the SQLite team to a bug latent in the database for 16 years.
Tailscale ran SQLite as its primary database since 2022, with each shard holding a single Go process that accessed the database exclusively. Backups took a snapshot every few minutes and uploaded the whole file to an S3 bucket. That setup held from early 2023 until August 2025, when corruption started showing up in the backups.
Corruption hit 19 times over six months, limited to control plane configuration data but each time requiring the process to stop for repair or restore, taking down the control plane on the affected shard. Code review and the absence of recent related changes left Tailscale without a reproducible trigger, so it contracted SQLite developers for joint verification. A transaction log pipeline that streamed every modifying SQL statement to a log file supported both recovery and the eventual path to the defect.
Synthesized by Yomimono from the 1 cited source below, including Japanese-language reporting where cited, then editorially reviewed before publishing.
Sources
- GIGAZINE 16年間放置されていたSQLiteのバグを突き止めた方法