Patching Linux is easy. Except when you need to patch tens of thousands of servers without downtime. Here’s how Meta does it.
Anyone with a tech clue can patch a Linux server. But, patching thousands of them without any downtime, that’s not easy.
At the Linux Plumbers Conference, the invite-only conference of top Linux kernel developers earlier this month, Meta Linux kernel engineer Breno Leitao explained how Facebook pulls the trick off with its millions of servers around the world.
If you were to use ordinary techniques, Leitao said it would take more than 45 days to roll out a new kernel to all machines. As he put it, “Draining and un-draining hosts is hard.” You can say that again.
That may be fine if it’s a minor update, but if it’s a security patch, that won’t work.
So, Meta uses Kernel Live Patching (KLP) with Red Hat‘s Kpatch, to deliver fast patches. In KLP, you can apply the latest security updates to Linux kernels without rebooting. This maximizes system uptime and availability.
