News

New GitHub Copilot Research Finds 'Downward Pressure on Code Quality'

New research on the effect of AI-powered GitHub Copilot on software development cites some adverse results.

The "Coding on Copilot" whitepaper from GitClear sought to investigate the quality and maintainability of AI-assisted code compared to what would have been written by a human. In other words: "Is it more similar to the careful, refined contributions of a Senior Developer, or more akin to the disjointed work of a short-term contractor?"

The answer to that is summarized in this paragraph from the whitepaper's abstract:

"We find disconcerting trends for maintainability. Code churn -- the percentage of lines that are reverted or updated less than two weeks after being authored -- is projected to double in 2024 compared to its 2021, pre-AI baseline. We further find that the percentage of 'added code' and 'copy/pasted code' is increasing in proportion to 'updated,' 'deleted,' and 'moved 'code. In this regard, AI-generated code resembles an itinerant contributor, prone to violate the DRY-ness [don't repeat yourself] of the repos visited."

That serves as a counterpoint to findings of some other studies, including one from GitHub in 2022 that found, for one thing: "developers who used GitHub Copilot completed the task significantly faster -- 55 percent faster than the developers who didn't use GitHub Copilot." That study was noted in the new whitepaper from GitClear, which sells a cloud-based code review tool. In addition to productivity, the GitHub study also measured positive effects in developer satisfaction and conserving mental energy.

GitClear's research, however, investigated "how the composition of code changes when AI is used." GitClear said its report sheds light on:

  • What are the three significant changes since Copilot's introduction?
  • What do Technical Leaders need to be on the lookout for 2024?
  • How can you measure the impact of AI on your team's code quality?

Regarding that first item, the paper indicated the three most significant changes associated with Copilot's rise concerned "Churn" and "Moved" and "Copy/Pasted" code:

  • Burgeoning Churn: "The bottom line is that 'using Copilot' is strongly correlated with 'mistake code' being pushed to the repo."
     Code Churn by Year
    [Click on image for larger view.] Code Churn by Year (source: GitClear).
  • Less Moved Code Implies Less Refactoring, Less Reuse: "Combined with the growth in code labeled 'Copy/Pasted,' there is little room to doubt that the current implementation of AI Assistants discourages code reuse. Instead of refactoring and working to DRY ('Don't Repeat Yourself') code, these Assistants offer a one-keystroke temptation to repeat existing code."
  • More Copy/Pasted Code Implies Future Headaches: "There is perhaps no greater scourge to long-term code maintainability than copy/pasted code. In effect, when a non-keyword line of code is repeated, the code author is admitting 'I didn't have the time to evaluate the previous implementation.' By re-adding code instead of reusing it, the chore is left to future maintainers to figure out how to consolidate parallel code paths that implement repeatedly-needed functionality."

The paper concludes: "How will Copilot transform what it means to be a developer? There's no question that, as AI has surged in popularity, we have entered an era where code lines are being added faster than ever before. The better question for 2024: who's on the hook to clean up the mess afterward?"

Some other studies on the topic include:

For its study, GitClear collected and analyzed 153 million changed lines of code, authored between January 2020 and December 2023.

About the Author

David Ramel is an editor and writer at Converge 360.

comments powered by Disqus

Featured

  • Cloud-Focused .NET Aspire 9.1 Released

    Along with .NET 10 Preview 1, Microsoft released.NET Aspire 9.1, the latest update to its opinionated, cloud-ready stack for building resilient, observable, and configurable cloud-native applications with .NET.

  • Microsoft Ships First .NET 10 Preview

    Microsoft shipped .NET 10 Preview 1, introducing a raft of improvements and fixes across performance, libraries, and the developer experience.

  • C# Dev Kit Previews .NET Aspire Orchestration

    Microsoft's dev team has been busy updating the C# Dev Kit, a Visual Studio Code extension that enhances the C# development experience by providing tools for managing, debugging, and editing C# projects.

  • Hands On: New VS Code Insiders Build Creates Web Page from Image in Seconds

    New Vision support with GitHub Copilot in the latest Visual Studio Code Insiders build takes a user-supplied mockup image and creates a web page from it in seconds, handling all the HTML and CSS.

  • Naive Bayes Regression Using C#

    Dr. James McCaffrey from Microsoft Research presents a complete end-to-end demonstration of the naive Bayes regression technique, where the goal is to predict a single numeric value. Compared to other machine learning regression techniques, naive Bayes regression is usually less accurate, but is simple, easy to implement and customize, works on both large and small datasets, is highly interpretable, and doesn't require tuning any hyperparameters.

Subscribe on YouTube

Upcoming Training Events