Measuring Productivity of Distributed Teams

Measuring productivity is hard, especially for knowledge workers and craftspeople who are building software products. 

People are typically the most expensive asset for technology organizations, yet it feels like the internal systems for defining and measuring productivity are archaic and don’t match the methodical approach we use for external product development (iterative, data+gut) in customer facing products. 

As more of our work happens in the cloud (and distributed) we are now able to actually record contribution and interaction in a way that was impossible even just a few years ago. 

This post is not meant to be an ‘answer’ but a set of early thoughts (from someone who runs a distributed team at scale). I’m excited to see how our work evolves, and how our tools evolve to power more productive global and distributed work. 


Defining productivity

  • How do we define “productive”?
  • How do we think about this at both an individual and a team level?
  • How do popularity and personal (and cultural) bias factor into this?
  • How frequently / over what time period should we measure productivity? (Creative work happens in phases)

These are hard questions, and I don’t have clear answers, and I think there is room for improvement across the industry for in-person, hybrid, and fully distributed teams to define and measure all of these better.

In my mind, productivity is a function of volume, quality, and impact. It’s easy to say this, but these are all hard to define, track, and attribute which is why this is still an unsolved problem.

The most productive teams can have no outstandingly productive individuals, and the least productive teams could have exceptionally productive individuals. It’s hard to actually parse individual productivity when running a larger organization.

As managers of managers, we don’t have sufficient tools to be able to understand most of this and most of our information is driven by sentiment and infrequent interactions with individuals on our teams.

For in person teams, productivity is often a function of time spent in the office combined with their team’s sentiment/perception of their work (and likeability) which is imprecise, and polluted by biases and proxies for actual productivity.


Measuring Productivity

To measure productivity of teams and individuals, we should look to both quantitative and qualitative measures. I prefer measures that apply to individuals but can also roll up to both teams and projects.

Quantitative metrics

Quantitative metrics to measure productivity can be hard to pick, measure and implement in the organization. These types of metrics may make employees feel ‘watched’, don’t tell the full story, and could encourage suboptimal behavior to gamify metrics.

Defining quantitative metrics that can roll up from individuals to team level and project level metrics are best, because it allows us to have the most flexible view of contribution and productivity, but these metrics are hard to select.

Here are a few ideas:

  • Volume: A measure of activity across different tools (e.g. Slack messages, GitHub commits, words on internal Wikis). These metrics will vary by team and role, and need to be measurable across tools. I don’t think more activity necessarily implies more productivity but an absence of activity likely implies a lack of productivity.
  • Quality: Quality is hard to measure. There are some heuristics we could use like rejected pull requests, bugs, live issues, comments/likes on internal memos. I don’t have a clear view if these would actually work at an atomic level without running some experiments to figure out if they are actually predictive of quality.
  • Impact: First we need to define impact (e.g. OKRs, trackable metrics). What business or user goal are we trying to achieve? How do we attribute work directly to these goals, especially at the individual level? It’s easier to do at a project or team level, but still difficult to map work to impact accurately.
  • Micro-contracts: I generally work using a series of micro contracts. I commit to a specific scope and timeframe and it’s usually an agreement between me and one other person. It would be valuable to track these micro contracts in a transparent, two player view (vs. separate todo lists) that integrates nicely with my existing workflow (e.g. a Slackbot). This is a feature, not a product, but it would be useful to capture data (e.g. missed, delayed, complete) on this over time across the organization.
  • Engagement: We have an internal tool called P2 at Automattic (replaces email) which we use to write and store long form content and share it transparently with our colleagues. Much of my work as a manager is engaging with these P2 posts, leaving likes, comments and asking questions. We track some of this data internally, but don’t have a great view for managers to look at aggregate views for individuals, teams or projects (which would be helpful). 
  • Project Iteration: I strongly believe that speed of iteration is a sustainable advantage for companies, and many companies lose this speed as they grow. Something which captures progress towards stated project goals, as well as the project state over time, would be very helpful especially when combined with some of the other individual metrics listed above.

I’d start by tracking all these metrics individually (and creating a time series for each) and then worry about creating compound metrics later, for the metrics that are most accepted by the organization.

Qualitative Metrics

Qualitative feedback is important both as a source of positive and developmental feedback for individuals and teams. This feedback contains the most bias, but is the most widely used and accepted form of assessment across our industry.

  • Peer Review: This provides a good view of the person’s capabilities as a team player, and a measure of their ‘popularity’ with their peers. It’s also a good measure of team fit as different teams can have different microcultures.
  • Manager Review: Managers typically write regular (e.g. annual) reviews which are a useful point in time synthesis but have recency bias (something of which I’m certainly guilty). Managers don’t record all the ‘micro feedback’ and ‘micro improvements’ in a transparent way to the organization (e.g. logged in a database) which I think could be interesting to try.
  • Direct Report Review: This is similar to manager reviews, except it’s many to one for the manager. This is useful to check for consistency of feedback, which can be a stronger signal than an isolated piece of feedback which we could over-react to if particularly positive or negative.
  • Self Review: This is an opportunity for individuals to synthesize what they think is most important and impactful retroactively. I personally find it useful, and think it’s also useful to kick off a conversation with a direct report (when there is discrepancy in perception). It’s also a good measure of self awareness (if self view is very different from the others).

In general, storing micro feedback (e.g. positive/developmental plus a string), and periodic summaries (discrepancies, sentiment by group, change over time) would be a good place to start, and then HR teams could perform analysis on the usefulness and predictiveness of each of these inputs over time.


Both this qualitative and quantitative data would be helpful to both the individual and the manager to track their development and journey during their tenure at the organization. It would also make manager handoffs (team switches or people leaving the company) cleaner and easier without losing valuable organizational data.

Open Questions

A couple of additional open questions that I did not have good answers to:

  • Meetings: I almost added meeting time/number of meetings to the quantitative metrics section, but I’m in two minds about them. I think recurring meetings are a ‘lazy’ way to have individuals and groups get together and make up an agenda. Large meetings with lots of passive meetings are also very ‘expensive’ for the organization, and likely better handled via written, asynchronous alternatives. I do think there are a few exceptions (1x1s for direct reports, project kick off, strategy/direction changes) which have their place to drive alignment and build relationships. I personally like short interactions about specific problems spun up in real time but these are hard to coordinate if teams are over-scheduled.
  • Real Time vs. Synthesized: For all of these metrics, how much should available in real time vs. point in time? Real time plus history allows for better trend analysis, but point in time allows for synthesis. Point in time reviews suffer from recency bias and most creative work happens in phases (with natural spikes and troughs).
  • Transparency: For all the quantitative and qualitative metrics how transparent should this be to the individuals affected and the organization? If totally transparent, I worry about gamification but if totally hidden it feels like spying. I think it should be run as a contained experiment to see the impact on the culture.
  • Popularity vs. Objective Metrics: As humans, how do we eliminate bias? We are often biased towards people who we perceive to be similar to ourselves. We are more likely to excuse poor performance for someone we like v.s. someone we don’t. Measuring and accounting for bias is extremely difficult.

After writing this, I was left with more questions than answers, but feel confident that most organizations (distributed or hybrid) are not good at measuring and storing performance over time at the individual, team, or project level. I also think that the appropriate solution(s) will vary by company and its culture. Building tools to track performance and creating a culture of internal experimentation is necessary for organizations to get to better solutions across all levels. 

I’m personally excited about innovation in this space as a manager, and an increased focus on output over facetime and popularity as companies work in a more distributed or hybrid work environment.


As a side note, this was an interesting view on how the folks at GitLab think about this — I appreciate them being open with sharing a lot of their practices in general.

Performance Management

This post will summarize my personal learnings for managing performance in both a distributed and non-distributed environment. I recently hosted a discussion on ‘Remote Performance Management’ with engineering and product leads at other companies (through Enrich) and these were some of the topics we covered.

Performance management in a distributed environment is very similar to working in person, except you need to rely more on measuring actual contribution, and more written communication. 

It can be very draining for a team, and for managers in particular, to deal with performance issues on their teams. If these are not dealt with quickly, they can fester and affect the entire team. It is important to have a clear path to gather data, diagnose and solve for the productivity and for the ‘health’ of your team.

1. GATHER INFORMATION

Start by gathering data to figure out if there are performance issues.

Your Gut: Most of the time, you know if someone is performing well. Trust your gut and use it as the starting point. Write down examples of issues you observe in a document so you can spot repeated patterns. In a distributed environment, it can take longer to calibrate your gut – you can’t ‘feel’ the energy of a person or a team as easily, so you need to rely on the output of teams. The more teams work out in the open (public by default), the easier it is to understand their output.

Team: Ask for feedback from folks who interact with the person closely – peers, direct reports, other functions. This can be more casual or part of a broader discussion if you don’t want to cause ‘alarm’. You can also ask your HR rep to help gather feedback for you as well. 360 degree feedback tools are also really valuable for managers and teammates to give feedback on each other.

Data: Try and figure out objective measures of output – communication metrics (e.g. Slack stats, public posts and comments), projects delivered, GitHub commits can all help paint a picture of productivity. It’s important not to use these metrics as the starting point for performance management – they are simply a useful tool to help validate or invalidate hypotheses. If people feel like they are being ‘watched’ they will not like it, or try to game the system which is not where you want them to focus.

2. DIAGNOSE ISSUE

Once you have established there is a performance issue, the next step is to unpack the why behind the issue.

Capability vs. Effort : I start by trying to understand if there is a capability problem or a motivation problem and use this simple matrix. Folks who are high capability and effort should be rewarded, and folks who are low capability and effort should be transitioned out of the company quickly.

This image has an empty alt attribute; its file name is impact_effort_matrix_brainstorm_-_brainstorm.png

Manager / Team Fit: The individual may just not fit in well with the team culture or have a good rapport with their manager. Once a manager has ‘lost faith’ in someone on their team, it’s very hard to regain faith without a team switch.

Project Fit: The individual may not enjoy or be well suited to the type of work they are doing. This person might have a capability or effort issue, and in most cases this requires a move to a different role or to a different project. 

3. IMPLEMENT CHANGES

Once you’ve diagnosed the issue, the next step is to figure out a path forward. Many new managers simply avoid having these hard conversations because they are awkward and can be difficult.

Communicate clearly: Communicate performance issues clearly with the individual and then lay out a clear path forward with areas for improvement and timelines. This step comes before a formal performance improvement plan (PIP) which is more serious to developmental feedback.

Termination: If you have reached the point of termination, then be clear, direct and kind. Schedule a short in person meeting or video call, and get straight to the point. Often HR is involved in this call, and at some companies they are responsible for this meeting.

Team / Role / Project switch: If the individual is new to the company and has performance issues, and you suspect they are in the wrong role, wrong project or have fit issues with their current manager or team then you should allow one switch to give the person another shot. If the performance issues are persistent, then they should be let go from the company.

Permission to leave: Often, an individual was the right fit for the company or for a role at a point in time but given the stage of the company’s growth, or a shift in the nature of the work this person may not be a good fit any more. As a manager, you can give this person the ‘permission to leave’ and they will be able to find a place outside your team or company where their skills are better suited. It’ll be better both for the individual and the company.


Managing Distributed Teams

At Automattic, I lead a fully distributed product development and engineering team. This post will cover some of my personal practices for managing teams and if/how this is different in a distributed environment. These practices are probably more useful to newer managers running distributed teams for the first time.

I recently listened to Matt (Automattic’s CEO) and Raj Choudhury’s (Prof at Harvard Business School) discussion about the future of distributed work and the ‘Work from Anywhere’ movement which were the inspirations for writing this post.

DISTRIBUTED PRACTICES

The principles of managing a distributed team are the same as managing a team in person, but a few of the practices are different. People are still people, whether they are sitting right next to you or halfway around the world.

Here are a few practices that I’ve found helpful:

  • Trust: Start from a place of trust. Assume positive intent in written communication, and assume your team is working and trying their best regardless if they are sitting right next to you or they are working from home.
  • Expect Asynchronous Communication: Don’t expect a response immediately, even over chat tools like Slack. Learn how to use Slack asynchronously, and set the same expectation on your teams. I deleted Slack from my phone (because I would miss things), and close Slack on my computer when I want to remove distractions. I respond to messages in batches, and use the reminder feature if I need to come back to something later.
  • Focus on Output: Don’t falsely assume someone is more productive because they work longer hours (even when working in person). Focus on the quality and quantity of the work produced by an individual vs. the number of hours worked.
  • Clear Goals, Roles, Expectations: Develop clear goals and a shared understanding of the ‘why’ behind these goals, roles and responsibilities and what is expected of managers (and their teams) in terms of output. Extreme clarity here leads to more empowerment, not less, in my experience (one of my takeaways from Essentialism, by Greg McKeowen, which I recommend).
  • Project Kick Off: For new projects, with new groups of people working together or working across different teams it’s good to get alignment right at the start. I suggest experimenting with a kick off call with project stakeholders and participants followed by a written summary. The call may be difficult to schedule, and less conducive to working across time zones. but project kick offs are infrequent enough that I think these calls are worth it.
  • “Grab a Room”: If you sense a real time conversation is going off the rails in Slack and if it was in person you would ‘grab a room’ to chat it through, do the same over Zoom for 10 minutes. It helps if your team is not inundated with regular meetings so this can happen more seamlessly. I personally also leverage ‘office hours’ to skip level meetings a few times a quarter. 
  • Hiring: When hiring folks who are distributed, put extra weight on the quality of their written communication and their ability to work in a self directed manner. Documentation becomes even more important in a distributed environment.
  • Feedback: Give frequent, specific feedback both positive and developmental over Slack or in your regular 1x1s (both personal and project related). Write up more thoughtful feedback every 6-12 months. We all have recency bias in the longer reviews, so I keep a record of the small pieces of feedback in a running document. At Automattic, we have a tool called ‘Kudos’ which allows folks to send public thank you messages to a few colleagues a month. It’s a nice way to show appreciation.

MANAGEMENT DURING A PANDEMIC

Managing a distributed team during a global pandemic (Covid-19) requires greater care and empathy. Many folks who are working from home had it forced on them and it may have felt jarring. They may have additional responsibilities of looking after their children, caring for sick/old folks or dealing with loss either directly or indirectly. There is also a psychological toil that is hard to quantify, and simply not knowing when we will return to “normal” can weigh on people. As a manager, simply recognizing these issues explicitly and then being empathetic to their circumstance can go a long way.

I would encourage your teams to take the time they need for self care, and be accommodating to more flexible hours. If individuals or teams are going to experience a productivity hit, adjust goals accordingly (and publicly) as long as your business can afford it. It will pay off in the long term with improved happiness, productivity which will translate to better employee retention.

I’ve noticed extra output from some folks who are now simply working more to fill the extra time they have, and less output from others who are more affected. Teams realize and recognize this asymmetric contribution and much like any small community there are times where we need to contribute more to help out our colleagues. That’s ok, as long as it’s not permanent.


For more on this area check out the companion post around managing performance.

This series from Greylock is excellent as well – https://greylock.com/workfromanywhere-podcast/