News
How Good Is GitHub Copilot? Depends on Who You Ask
Since leveraging generative AI breakthroughs to introduce the original "AI pair programmer" called GitHub Copilot, the company has been on a mission to publish research to showcase its positive impact on developers and organizations.
The latest such research, published just this week, was presented in a post titled "Research: Quantifying GitHub Copilot's impact in the enterprise with Accenture." As the title suggests, Microsoft-owned GitHub focused on Accenture developers for the report, studying how that company's developers integrated GitHub Copilot into their daily workflows.
In a randomized controlled trial, developers were assigned to either a group with access to GitHub Copilot or a group without. The study aimed to evaluate how GitHub Copilot impacted developers' experience in an enterprise setting, where they work on complex projects. DevOps telemetry provided insights into their regular coding activity.
Along with that experiment, the GitHub team examined company-wide adoption of GitHub Copilot, investigating installation rates, code acceptance rates, and the time it took developers to accept the first coding suggestion from Copilot. Success was measured by whether developers accepted a suggestion.
What's more, GitHub Copilot users were also asked about their experiences and perceptions in using the tool to understand their perceived impacts on their workflows.
Findings were summarized thusly:
- Improved developer satisfaction. 90 percent of developers found they were more fulfilled with their job when using GitHub Copilot, and 95 percent said they enjoyed coding more with Copilot's help.
- Quickly adopted by developers. Over 80 percent of Accenture participants successfully adopted GitHub Copilot with a 96 percent success rate among initial users. 43 percent found it "extremely easy to use." Additionally, 67 percent of total participants used GitHub Copilot at least 5 days per week, averaging 3.4 days of usage weekly.
[Click on image for larger view.] How Easy GitHub Copilot Is to Use (source: GitHub).
Other data points pulled from the narrative report include:
- 70 percent of respondents relied on GitHub Copilot for coding tasks in a familiar programming language
- 81.4 percent of developers installed the GitHub Copilot IDE extension on the same day that they received a license
- 96 percent of those who installed the IDE extension started receiving and accepting suggestions on the same day
- 51 percent rated it as "extremely useful"
[Click on image for larger view.] How Useful GitHub Copilot Is (source: GitHub).
- Accenture developers saw an 8.69 percent increase in pull requests
- Accenture saw a 15 percent increase to the pull request merge rate
- At Accenture, we saw an 84 percent increase in successful builds
- Developers accepted around 30 percent of GitHub Copilot's suggestions
- 90 percent of the developers reported that they committed code suggested by GitHub Copilot
- 91 percent of the developers reported that their teams had merged pull requests containing code suggested by GitHub Copilot
[Click on image for larger view.] GitHub Copilot's Impact on Production Code for Accenture Developers (source: GitHub).
- Developers retained 88 percent of GitHub Copilot-generated characters in their editor
- 70 percent of developers reported quite a bit less mental effort was expended on repetitive tasks
- 54 percent spent less time searching for information or examples when utilizing GitHub Copilot
"With this study, we have uncovered compelling evidence that GitHub Copilot significantly enhances developer experience, satisfaction, and overall job fulfillment in real-world enterprise settings," the May 13 post said. "With GitHub Copilot in their toolkits, developers can also enhance their skill sets and gain greater proficiency in their organization's codebase, which ultimately leads to heightened contribution levels across teams, all without sacrificing the quality of code."
As noted, the new report follows similar GitHub efforts including:
From the Non-GitHub Camp: It's Not All Sunshine and Roses
GitHub Copilot research and third-party opinions haven't always been so positive, however, as explained in our article earlier this, year, "New GitHub Copilot Research Finds 'Downward Pressure on Code Quality'"
The article explains a "Coding on Copilot" whitepaper from GitClear sought to investigate the quality and maintainability of AI-assisted code compared to what would have been written by a human. In other words: "Is it more similar to the careful, refined contributions of a Senior Developer, or more akin to the disjointed work of a short-term contractor?"
The answer to that is summarized in this paragraph from the whitepaper's abstract:
"We find disconcerting trends for maintainability. Code churn -- the percentage of lines that are reverted or updated less than two weeks after being authored -- is projected to double in 2024 compared to its 2021, pre-AI baseline. We further find that the percentage of 'added code' and 'copy/pasted code' is increasing in proportion to 'updated,' 'deleted,' and 'moved 'code. In this regard, AI-generated code resembles an itinerant contributor, prone to violate the DRY-ness [don't repeat yourself] of the repos visited."
Other counterpoint studies include Practices and Challenges of Using GitHub Copilot: An Empirical Study, which said: "Our results suggest that using Copilot is like a double-edged sword, which requires developers to carefully consider various aspects when deciding whether or not to use it."
A study titled Is GitHub Copilot a Substitute for Human Pair-programming? An Empirical Study, said: "The results suggest that although Copilot increases productivity as measured by lines of code added, the quality of code produced is inferior by having more lines of code deleted in the subsequent trial."
Plenty more research has been published on the topic of GitHub Copilot and AI-assisted software development in general to help devs make up their own minds, including:
While not scholarly, empirical research, other sources provide insights into the topic, such as:
From this reporter's own investigations and from just keeping tabs on the landscape, it's clear GitHub Copilot is a game-changing technology that is only going to improve with time (I use it often every single day, but just for writing, with hits and misses).
In the meantime, as with any technology, it's not a panacea and has its own set of challenges and limitations (we haven't even touched on the ethical, security, copyright/legal and other issues). The key is to understand those challenges and limitations and use the technology judiciously to maximize its benefits while minimizing its drawbacks.
About the Author
David Ramel is an editor and writer at Converge 360.