So, at Google I’ve been working for several years on things called “Prodspec & Annealing”. While we started those for one service, it is now used by a very large chunk of Google production. Prodspec takes care of taking any service configuration and massage it into a clean “Prodspec intent”, representing what is the production supposed to look like from the infrastructure perspective. And those bits are then consumed by Annealing, which takes care of updating production to match the intent. The overall pipeline provides us with a safe continuous delivery (CD) system, deploying things from binary versions to switch firmwares.
And I’ve just written about it in Usenix, thanks to the help of Betsy Beyer, and the good folks at ;login: - Laura Nolan and others.
The article is … a bit long :) It goes in depth into what we did, what works for us and the overall architecture and its philosophy.