Build your own “intelligent” code review reminder

In this post, I show you how the intelligent code review reminder Microsoft Nudge works, so that you can build or enhance your own and speed up code review completion time by 60%.

---

Without code review reminders, developers might wait for a long time to receive code review feedback. This is a huge code review pitfall, and one of the reasons some developers rightfully dread code reviews. Waiting on feedback means that the whole development process is slowed down, and code velocity decreases.

Developers can take a couple of countermeasures such as submitting small, and coherent code reviews, integrating code reviews into their daily routine, giving peers a head’s up that a code review is coming, or reminding their colleagues when a review is overdue.

Especially for the latter case, many companies have developed automated systems that check the status of code reviews and remind developers through email or group chats. For example, at Automatic, this kind of developer productivity app is called the “stick”. But also other companies, that I’ve worked with, have similar automation.

Nudge – An intelligent code review reminder at Microsoft

At Microsoft, little automation sending reminders exist for a long time. But over the last year, Microsoft went a step further and made a code review reminder that possesses some “intelligence” to increase its effectiveness.

This code review reminder service is called “Nudge” and has been developed by researchers at Microsoft and the University of Delft. Over the last year, the impact of this intelligent reminder tool on code velocity, development speed, and pull request completion has been studied.

Apart from the positive feedback reported by developers that used the service, Nudge has helped to reduce the code review completion time by over 60%. That means that the average time it takes to complete a pull request decreased from 197.2 hours to 77.65 hours at Microsoft.

So, now you are definitely curious how Nudge works, right?

Let me give you a small summary of the most important facts about Nudge so that you can implement, or enhance your own code review reminder.

You can find all the nitty-gritty of which signals Nudge uses to predict code review completion time, or how the effectiveness has been evaluated here in this paper.

Using code review characteristics to estimate completion time

Nudge is super interesting as it predicts the completion time of a code review, whereby it fully relies on data that is already available during the creation time of the pull request. Nudge uses effort estimation models, that have been studied extensively in software engineering research. Those models are used to make accurate estimations about how long a software project will take.

What determines code review completion time?

Using linear regression, the researchers investigated how different factors, such as day of the week, number of reviewers, number of distinct file types, etc., impact pull request completion time.

The following correlations and relationships could be observed:

• The later in the week – and thus closer to the weekend – a PR is created, the longer the code review takes.
• Also, if PRs created by a particular dev took longer to complete in the past, future PRs will also take longer. This can be explained through experience level and coding style. A developer who’s new to the project or has less experience will need more feedback, thus leading to a longer PR completion time.
• The more reviewers are commenting on a PR, the longer the PR takes to complete. Researchers explain that is due to the increase in comments. I personally see it as a sign that the PR was either complex, had some fundamental problems, or is associated with high risks. All of those factors lead to more reviewers being added and actively engaging with a PR.
• If the project file has been changed, the PR also takes longer. Here, the project file is a proxy for a more complex change, as it indicates that files have been added or deleted, or dependencies changed.

In the paper, the researchers evaluate and discuss several other metrics, that also showed correlations – even if less strong ones – with code review completion time.

Increasing user satisfaction by considering code review activities

To increase developer satisfaction with the code review reminder service, Nudge also considers interactions with the pull request. Those interactions can be state changes (like active, draft, closed), or comments from reviewers and authors, or new git commits/updates to the code.

Many of those interactions can either come from reviewers or the author. To not pester responsive folks, it is important to understand who “blocks” or slows down a review.

Based on a manual investigation of 70 pull request, the researchers designed an algorithm that determines if one of the reviewers or the author should be “nudged”.

In the flow chart above, you can see how the algorithm determines who is responsible for blocking the pull request.

Putting it all together: How the code review reminder works

The code review reminder is built as a service on top of Microsoft Azure and uses the Azure DevOps REST API. But you can implement a similar service for example on top of GitHub, by using GitHub action, and the GitHub Developer API to gather the data.

Here are the main steps you must consider when implementing an intelligent code review reminder:  

1. When a pull request is created, the code review reminder service starts it is work by estimating how long the code review is going to take based on the pull request data available and the effort estimation model. This expected completion time is saved in a DB.
2. As soon as the pull request exceeds it expected completion time, the code review reminder service determines who should be notified (reviewers or authors) based on the activities that happened on the PR.
3. If the algorithm determines that there is “enough” activity, no notification is sent. If the code review notification service detects that there is a blocker, the service reminds the appropriate developers about the code review.
4. Rinse and repeat. The reminder service will be triggered periodically to check on active PRs, recalculate the code review estimation – if needed – and send reminders to the right people.

Summary: Build your own code review reminder

Well, apart from the significant decrease in review completion time, the study reports plenty of positive feedbacks from developers. All in all, this study shows great results that I believe can be useful for a lot of companies and teams.

When I work with teams to increase their code review happiness, code review reminders are a low-hanging fruit that can improve developers’ satisfaction and speed up review time.

Many code review reminders that I have seen implemented by companies and teams can be enhanced by just a few tweaks here and there.

The few key factors, such as day of the week, previous PR completion time, review engagement, and that I mentioned in this short summary are probably enough to drastically increase the notification effectiveness if teams haven’t used them before.

Also, building some smartness based on the activity of the reviewers and authors might also be a nice boost for your notification automation.

But, what about you? Do you use your own reminders? Do they already consider some factors? Leave a comment. I’d love to hear from you.

---

Updated on January 5th, 2021