John Micco is a Cloud Transformation Architect at VMWare, where he works on CI/CD systems. He’s worked in the CI/CD space for nearly twenty years at various companies like Netflix and Google, he’s authored several research papers, and was a keynote speaker at the International Conference of Software Testing and Verification (ICST). Our collaboration helped shape my work on Athena at Dropbox.
This episode is extremely technical. We talk about the management of CI servers and systems in large companies, common problems faced, and themes in emerging solutions. Large-scale CI/CD management is far from being commoditized due to the custom configuration of every company’s build tooling.
Apple Podcasts | Spotify | Google Podcasts
Highlights
0:00 - What is a “Cloud Transformation Architect”?
6:00 - Sharing knowledge in the CI/CD space
7:00 - A comparison between starting a car company and starting a software company, and why standardization on tools is so much trickier in software
11:30 - The scale of testing systems. Managing a system that handled 10k RPS of test result data. The quadratic growth in compute resources required to manage a testing system
18:44 - “Only 7% of code changes had problems discovered by tests”
23:36 - Is unit testing overrated?
30:00 - At what point can companies no longer afford to run all tests for every code change
35:00 - What is Culprit Finding / Regression ID? What is auto-rollback / test quarantine?
41:00 - Culprit Finding at Dropbox
45:00 - Dealing with flakiness at the test layer vs. the platform layer. An ongoing challenge for VMWare
60:00 - Increase in demand for CI compute when developers don’t have any meetings
65:00 - Bazel migration at VMWare
73:00 - Developing an interest in CI/CD systems. And why it’s an exciting space where there’s a lot of innovation, and why more innovation needs to happen
80:00 - How can someone tell whether they’re investing enough in developer productivity / their CI/CD experience for engineers? The trade-offs that companies should consider when deciding to run tests.
Software at Scale 12 - John Micco: Cloud Transformation Architect, VMWare