Code Reviews at Google are lightweight and fast

Code reviews at Google play an important role as an engineering practice and have been adopted already in the early days of Google. Still today, they are used to keep the code base clean, coherent and to ensure no arbitrary code is committed. Even though the code review process looks similar to code reviews at Microsoft, there are some Google specifics that allow for a particular lightweight code review process.

So let me show you how code reviews at Google look like and what sets them apart from code reviews at Microsoft. In particular, I’ll show you what allows the 25,000 engineers at Google to review their code much faster than at other companies of this size. So, let’s get started.

Code Review Study at Google

Caitlin Sadowski, a researcher at Google, and others have performed a study to understand Google’s internal code review processes. This study is similar to the code review study at Microsoft, which makes it interesting to compare the code review processes at the two companies.

A typical code review process at Google

A typical code review at Google looks very much like a typical code review at Microsoft. Let’s look at an example, by imagining a Googler’s code review process. Let’s call him Mark.

Preparing code for review

It all starts after Mark has made some changes to the code and wants those code changes to be merged with the shared codebase. At Google, code reviews are, similar to Microsoft, done with the help of a tool. So before Mark sends his code changes out for review, he uses the tool to look through the code one last time.

Critique, Google’s internal code review tool, offers some diffing capabilities that make it easy for Mark to spot errors and see what has changed in this new version of the code.

Before sending the code out for review Mark needs to perform another step. Running the code through a static analysis tool. So, Mark runs, for example, Tricorder, a tool widely used at Google, and reviews the results of the static analysis tool. When he is happy with his changes, he sends the changes to at least one code reviewer.

Reviewers give feedback

The code reviewer carefully looks through the code and leaves comments if she sees a problem or needs some clarification. Mark then addresses each comment either by changing the code or replying to the comment. If Mark made some changes to the code under review, he uploads the new version for reviewers to check again. If a reviewer is satisfied, she can approve the change by marking it as “LGTM” (looks good to me). To be able to commit the code to the shared codebase, at least one reviewer must approve the code.

Looking at this code review lifecycle from a distance it looks like a carbon copy of a code review at Microsoft. But, there are profound differences that I’ll show you now.

Company-wide policies for Approval

First of all, Google requires each code change to be reviewed. No exceptions. Period.

At Microsoft, on the other hand, code reviews, as well as how and what needs to be reviewed is at the discretion of the divisions or teams. Some teams skip, for example, code reviews for small and trivial changes. In general, there aren’t any company-wide policies around code reviews. Teams and divisions decide on how many code reviewers are needed, or how code reviews are linked with testing and static analysis activities and more.

At Google, each code change is reviewed. Period. Click To Tweet

Code reviews at Google address ownership and readability

Also contrary to Microsoft, Google has some company-wide requirements that must be fulfilled by the code reviewers in order to be able to approve the code change. One has to do with Google’s strong code ownership. Each directory of the codebase is explicitly owned by a set of people. To be able to get the code change approved, at least one reviewer must be an owner of the code under review. That person acts as a gatekeeper. Only if this person gives his or her okay, code can be checked in.

Another strict requirement is that at least one person on the review must be trained in code “readability”. What this means is that this person must have obtained a readability certification. This certification shows they’ve demonstrated they know how readable and maintainable code looks.
The readability certification must be obtained per language. Having such criteria is a great practice to ensure consistency in style and design.

To get decision power, reviewers at Google must obtain a readability certificate that shows they understand what readable code looks like. Click To Tweet

Requirements for reviewers aren’t focused on seniority or status

So, while many other companies, including several divisions at Microsoft, rather look at the seniority of the reviewer, the area of expertise or the hierarchy to grant decision power, Google looks at ownership and readability certification.

This addresses some common code review pitfalls. Requiring senior developers to approve code can easily lead to work overload and in turn, create bottlenecks.

On the other hand, it is also important that enough people have such a readability certificate. Otherwise, it also creates bottlenecks for reviews. And we all know, waiting on code review feedback is one of the main pitfalls during code reviews. But while it takes quite some effort to obtain readability certification, it is easier than changing hierarchy or seniority.

How to get a readability certification

To demonstrate their ability to review code for readability, developers at Google go through a “review of their code review practices”. Therefore, the developer submits code changes to a team of readability experts. Those will inspect the code. But this inspection is not like a normal code review. No, readability experts look at the code with much more scrutiny.  The purpose of such reviews is to point out every little mistake and every potential for improvement especially in terms of coding conventions and coding style. Also, nitpicking issues such as indentation or extra spaces are part of this learning process. For the interested, you can find Google style guides for various languages here.

Once the experts are convinced that the developer learned and is able to apply Google’s coding style and conventions, they issue the readability certification.

What it takes to get code approved

So to recap, to get your code approved at Google, you need at least one person on the code review that has ownership over the code and the right readability certification for the used language. If those two criteria are met, you are good to go.

No rule without exceptions. Also at Google teams exist where more than one developer must approve or where different criteria for reviewers are enforced. But, the general rule is one developer’s approval is enough.

Code reviews at Google are light-weight and fast

Google explicitly wants its code review practices to be light-weight and fast. And even though Google enforces the ownership and readability criteria for approval, the code review process is – with 4 hours on average – really fast. Small changes are reviewed within 1 hour, larger ones within 5 hours. Other companies report average turnaround times of over 15 hours.

At Google, code review are on average completed within 4 hours. Click To Tweet

So, how does Google do code reviews so fast?

Well, looking at the data reported, we can see that there are two important factors: the number of review participants and change size.

Requiring only one reviewer speeds up code review times

One of the most interesting findings from the study is that more than 75% of the code reviews have just one reviewer. That’s unusual. Especially because studies have shown that two reviewers tend to give more valuable feedback.

Requiring only one reviewer seems like a conscious decision at Goggle and trades review rigor for speed. Only that way, Google can realize fast turn-around times. Skipping the need to wait for another person reduces a lot of complexity. But it hurts review rigor, as also the study mentions. How much this costs in terms of quality is unknown.  Still, Google seems to have great outcomes with this set-up.

Small change sizes are essential for fast and high-quality code reviews

Another crucial insight from this study is the size of the change. Can you imagine, 90% of the code reviews have fewer than 10 files changed? That’s truly impressive and also explains why code reviews at Google are lightning-fast. Most of the changes also have only around 24 lines of code changed. That’s a much smaller change size than reported by studies of other companies, including Microsoft.

Reviewing small, coherent changes is a proven code review best practice. First, it increases the review speed. But, as we have seen in our study on valuable code review feedback, it also improves the value of the code review feedback. The usefulness of the code review feedback decreases as the size of the code reviews increase. Googlers know that and submit frequently and small code changes. How smart!

Code Reviews at Google are fast for two main reasons. First, 90% of code reviews at Google comprise less than 10 files. Second, 75% of the reviews have only one reviewer Click To Tweet

Are code reviews worth it?

While analyzing code review practices and tools at Microsoft, I often thought about what it actually means to provide value during code reviews. When are code reviews worth the time a team spends on them?

To answer that question, I turned to the developers and ask them why they are doing code reviews and when they get value out of them.

Well, turns out code reviews must lead to changes to provide value. Even though it is okay if some code reviews don’t lead to any changes, it is important that a majority of them actually have an impact on the code. Otherwise, we could just skip them, right?

Code reviews must lead to change to unfold their true value. Click To Tweet

So, coming back to the Google study, I found it interesting that the researchers also had the premise that if there is no action taken, the code review could have been skipped. The good news is 80% of the code reviews at Google do require the developer to take action. This clearly shows that code reviews have an active impact on the codebase. But what about the 20%? Is that time wasted?

Well, things are not that simple. Fortunately, code reviews provide a broad set of benefits.

Motivations and benefits for code reviews at Google

Even though code reviews are often associated with finding bugs, several studies on code reviews show that benefits and motivations for doing code reviews go far beyond that. Also, Googlers are aware that the benefits of code reviews are manifold, especially if code review best practices are followed. The original vision of the employee that introduced code reviews at Google was to force developers to write code that other developers understand.

In the study, Googlers report the following motivations for doing code reviews:

• Education (mentoring, learning for developers, knowledge dissemination)
• Maintaining norms (such as having adequate tests, consistency in style and design)
• Gatekeeping (ensuring security, and having an additional safety net so that a single developer can not commit arbitrary code) and
• Accident prevention (this includes making sure bugs and defects are prevented as good as possible, and that the source code is of high quality).
• Tracing and tracking decisions (understanding the evolution of the code and why and how changes have happened)

Motivations change per role

Another interesting finding from the code review study at Google was that the motivations and expectations for doing code reviews depend on the person’s role and responsibilities. A manager, for example, is more interested in the benefit of creating a coherent coding style at the codebase than a single developer. On the other hand, developers are more concerned with finding defects or bugs.

The reasons for code reviews reported by Googlers are aligned with the reasons reported by Microsoft engineers, except that Microsofties do not describe code reviews as a way of “gatekeeping”. Security checks, for example, aren’t part of the normal code review process at Microsoft.

Google’s code review tooling

At Google, code reviews are done with the help of tooling. Two main code review systems are predominant at Google. For open-source code and code shared with collaborators outside, like Go, Chromium, Android Googlers use the Gerrit code review tool. Gerrit is an open-source code review tool that integrates with Git.

For the internal code, on the other hand, Googlers use an internal code review tool called  Critique. Critique is a rewrite of Mondrian. There aren’t any detailed description of the functionality of Critique, but Googlers seem to be pretty happy with the workflow and functionality.

A lot of code reviews at Microsoft are also performed via tooling. But at Microsoft, other forms of code reviews, such as over the shoulder reviews, have their fair and justified warrant. Sometimes, nothing can beat a face-to-face conversation.

Google’s uniformed process is optimized for speed

To sum it up, Google has clear guidelines on what it takes to get a code review approved. What stands between you and the commit to the shared codebase is a review approval from at least one person with code ownership and readability certification. That most reviews only have one reviewer takes also a lot of complexity out of the code review process. Company-wide code styles, make it crystal clear how readable code must look like. This combined with the small code change sizes allows Googlers to expect code review feedback in 1-5 hours.

Similar to Microsofties, Googlers are very satisfied with the code review process and find it a valuable engineering practices.

If you and your team want to boost your code review practices, book a code review workshop with me. I have already worked with many product teams around the world.

Credit where credit is due

This blog post is based on the wonderful study of code reviews at Google done by Caitlin Sadowski, Emma Söderberg, Luke Church, Michal Sipko from Google and Alberto Bacchelli from the University of Zurich. I also want to thank Robert Göritzer and Leif Singer for taking the time to read my drafts and to provide feedback.

Updated on January 22nd, 2022