GitHub, with its unique global reach, has a significant impact on the developer experience. So, how does GitHub measure developer efficiency?We spoke with Akshaya Aradhya, Senior Director of Engineering at GitHub, to find out how her team is improving development efficiency across tech companies.
Translated from GitHub Developer Productivity at 30 Billion Messages Per Day by Jennifer Riggins is a journalist, author, and host of events and podcasts that tell stories of technology culture, help share stories of cultural and technology collisions, and explain the impact of the technology we're building. She's been there. If GitHub is used by 85% of software engineers, then GitHub's developer experience becomes a matryoshka doll for developer productivity. As the world's largest repository, Copilot, a six-month-old AI pair programmer, has 14.2 billion GitHub users, which gives it the unique ability to impact the developer experience globally. This scale also complicates things. GitHub has 999% service-level agreement serving users in nearly 400 million repositories. This makes the world's largest software development platform support an incredible 40 billion messages per day – yes, tens of billions indeed. As we continue to unleash developer productivity, The New Stack caught up with Akshaya Aradhya, Senior Director of Engineering at GitHub, to find out how her team is increasing productivity across GitHub's entire company – which in turn should increase productivity for the vast majority of tech organizations. Her team is part of GitHub's infrastructure and platform engineering organization – supporting the entire company horizontally, including the application team.
*Search. github actions
copilot
GitHub Next R&D team.
Partnering with the future of the Azure cloud.
The platform team consists of nearly 300 GitHub employees, with about one-third of them led by Aradhya. The platform team also works closely with the other major horizontal entity – the security team – in everything they do.
Her organization, formerly known as Data Services, is responsible for what she calls traditional data-related issues, including data pipelines, data aspects of operations, and message queues.
We have Hydra and Aqueduct, which process billions of messages every day — about 30 billion messages," she explains, "and the numbers are even higher if you include the developer ecosystem at the top." Her team of about 120 people also solves all of GitHub's internal and external data storage problems – object storage, cold storage, blob storage – as well as data patterns, next-generation scalability issues, productivity issues, and more. Aradhya told The New Stack that her team is focused on: "How do we really listen to developers and then talk to our own applications, test with our own data, Xi, and then come up with a product?."”
And then, of course, they measure everything for continuous improvement.
How do you improve developer efficiency and productivity while ensuring seamless deployment, maximizing ease of interpretation, and automating what people love, especially if you're running distributed teams in different time zonesIt all fascinates Aradhya's team by enabling their highly global team of developers to overcome the most frustrating bottlenecks. "Why do I need to wait until people in Germany and Amsterdam wake up?I'm stuck. ”
Because GitHub was already a remote-first company before the pandemic, its thousands of engineers became great first customers and constant feedback**, such as onboarding people. "When you get into the software industry, you're told, 'Oh, xyz is a team leader' or 'This is a senior engineer, if you don't understand, ask them.'" Then there's the waiting time to find that person, to explain to you for the sake of knowledge transfer, and the embarrassment that comes from forgetting the content if you want to ask the question again. It's embarrassing for anyone, especially newbies, but it's simply intolerant for many, Aradhya said.
There's a lot of neurodiversity communities that will say, 'This is a problem with my executive function skills.'" I hope I won't be judged for that. ’”
With this feedback – and probably from the personal experiences of 30% of GitHub employees who self-identify as neurodiverse – the team built a feature where Copilot's chat feature (imagine Clippy but more powerful) will explain how it works when you select a segment.
Works with any ID. No matter what you're working on. Your copilot will suggest ways and tell you how it works. For example, she explains, make sure you don't write the wrong method name from a different class. This helps increase developer confidence while also increasing the speed of writing and reducing the likelihood of bugs being introduced.
This innovation, which targets feedback from internal developers, is part of the reason why Copilot adoption has exceeded all expectations, with about 92% of GitHub users in the U.S. using it since its initial public release last June. It's also the result of GitHub's comprehensive approach to everything, acting as the first customers and testers.
Before we push anything to GA – even alpha, beta versions of anything – we want to make sure it's developer-ready. She said. This is reflected in GitHub's values, which all revolve around fostering collaboration and belonging, which is driven by continuous feedback. "When we're collaborating, we always ask productivity-related questions, like: What impact do you think AI has on your developer experience?We use this information to relate it to what businesses would think. How can we help enterprise companies – whether it's Apple, Walmart, Tesla, or Google – understand and support engineering teams?”
Like many other organizations we've talked about developer productivity, a lot of this feedback is collected through developer surveys.
When we talk about developer experience, people tend to associate it traditionally with a product or experience, like a UI. But it's much more than that. Aradhya explained. "Mathematically, it's developer productivity plus developer impact plus developer satisfaction. If you add it all up and multiply it by an exponent, the developer experience is the sum of all of them and more. GitHub's research team then published internal blogs, and early on, their DevEx insights were involved in product-market fit discussions and executive meetings.
One of the early pain points that GitHub research unanimously identified was waiting for tests to finish or builds to end — which Aradhya calls dying waiting for collaboration: "How do you talk to people or understand what everyone is doing?".How do you automate the review?How do you speed up the build?How do you deal with bugs?”
That's where AI comes into play. In fact, while many are boasting early gains in individual developer productivity, faster incident responses, and fewer errors, one of the most surprising results of the 2023 GitHub Octoverse report, which surveys all people with a GitHub account each year, found that 81% of respondents expect AI to increase team-based collaboration.
This echoes the internal survey, where GitHub's developers are demanding more measurement of their collaboration – because you can't improve what you can't measure. But how do you measure the interaction between people?Aradhya says this goes far beyond basic requirements like Slack, Jira, pull requests, and documentation. In the world of developer productivity, this examination of collaboration improvement factors takes into account the breadth of synchronous and asynchronous communication. They can derive important quantitative metrics from these tools and message queues. Her team also focuses on platform engineering goals for discoverability:
How do you know how to spot something?
How do you measure the flow of information between two people or teams?
Are you dealing with the right questions?
Did you attend the right meeting?
Can you protect your time to reach the developer flow state?
In this remote-first company, individuals have a great responsibility to understand their best way of working and to be able to self-organize their focused chunks of time. It's not surprising that GitHub has a higher percentage of neurodiverse teammates, as it's a very inclusive way of working. GitHub also provides unconscious bias, privilege, and ally training for all employees. GitHub's collaboration metrics intentionally include diversity, equity, inclusion, and belonging. Like most technology companies, Aradhya mentioned that they have problems mentoring women and underrepresented populations in other tech fields.
We have challenges in mentoring sponsorship [and] mentor-mentee relationships. She said. Not only in terms of establishing formal mentoring programs, but "we also have the problem that people think they are mentors to others, but they don't add enough value to the mentee, or mentees don't recognize that they can mentor them." "This negative experience disproportionately affects those from underrepresented groups.
In response to this feedback, they established a more formal mentoring process and training.
The massive Octoverse report is just one annual survey of many internal and external data**. While teams are able to obtain qualitative metrics, such as the well-known DORA Core Four Metrics, there is often a combination of quantitative and qualitative surveys in any developer productivity engineering journey, including the Space framework and DevEx metrics. Aradhya says that GitHub's R&D department quickly tests proofs of concept every week, with almost instant feedback. The platform and infrastructure teams run different developer experience surveys every few weeks to quarters. She assured me that it would not be a three-page survey that no one wanted to answer, but a combination of shorter surveys with higher response rates.
Not everything needs to be investigated. She continued. For example, the product team is constantly taking action from customer conversations and making quick rating system feedback during the event.
I even remember at an engineering leadership meeting where I was waiting in line for a food truck during my lunch break and somebody saw my github jacket and they said, 'What do you do?'I said we support XYZ and Copilot. As soon as I said that, they started asking me for features," she recalls, "and by the time the food truck line was over, I had received 18 different ideas for different products that we could enhance or improve," and she immediately fed them back to their respective product teams.
GitHub is a feedback-driven company. Measurement of developer productivity and developer experience is an ongoing topic in executive meetings, Aradhya said. On GitHub, these productivity metrics include:
How long does it take to get user feedback?
What does asynchronous communication look like?
How long can you work uninterrupted, without meetings or interruptions?
How much time do you spend building solutions to new problems?
How much time do you spend writing or fixing bugs?
How often do you encounter security or vulnerability issues?
How much time do you spend on self-improvement during a new product launch or update?How much time is spent mentoring and improving others?
How much time do you spend writing automated tests?
How often do you switch contexts?
How productive do you feel?
She explained that many of these smaller internal metrics are associated with larger ones. For example, when Aradhya was an engineer, she realized that it would be faster to run a production test herself in the short term, but in the long run, not writing repeatable automated tests would be a waste of time. Automating repetitive tasks is an important goal of developer productivity engineering.
Together with her team leader, she says, "I take stock of the number of human touchpoints involved in certain things." It's also important to measure the amount of time developers can spend innovating to solve unique problems, which she says is critical for retention. "It's about designing novel metrics – novel communication, new APIs, for all the latest issues. Basically, do you have a 10x mindset, do you understand the customer's problem and what approach you want to use to solve it?”
These are what Aradhya calls the "unconventional developer productivity" stuff that her team struggles to find a solution to. GitHub uses these metrics to foster a 10x developer mindset, which she defines as "a developer who has the vision and technical ability to anticipate problems they may encounter in the short and long term, and who thinks in terms of an extended user base, beyond the boundaries defined by the product or defined by the engineering lead." ”
Consideration of size was critical to Copilot's success, as they expected it to start with maybe 15 million users, but it quickly became clear that it was about 11.5 billion would be a much closer estimate – by then it will be too late to consider availability, reliability, and security.
Anything that GitHub puts out has to be extensible.
A 10x developer can foresee this, and then provide backups, provide part b, help with the technical strategy so that we can understand the unknown of the unknown. Then find a way to scale based on consumer demand without letting the quality drop. Aradhya said.
Overall, at Microsoft, which owns GitHub, she says that anyone has the opportunity to become a 10x developer, regardless of level, brings extraordinary thinking, and an emphasis on company culture and collaboration. She gives an example like this: "How did you scale up the teams around you?".How do you affect them?Have you changed your team or company culture in a positive way?”
Another area of developer productivity metrics that GitHub is focusing on is, of course, tools, such as finding answers to the following questions:
What does it mean to have a fully configured environment?
How easy is it to build on top of that platform environment?
GitHub also collects a lot of automated feedback and is always looking for ways to tighten the feedback loop, from validation and compliance checks to helping developers understand where they're least productive, especially communicating customer feedback to developers to make sure they're incorporating feature requests.
All of this is measured through ongoing anonymous and certified surveys, questions, pull requests, etc.
If you flood people with surveys, they'll stop answering, so we're going to be creative here," she said, "and we also look at our own analytics and channels that run internally." ”
All these results are communicated to the rest of the organization, explaining:
What percentage of time you spend on x. We improved it with y. Then ask for feedback again. She explained that anyone can see this information internally, and all engineers around the world can.