News

GitHub Copilot Tops Research Report on AI Code Assistants

• By David Ramel
• 08/26/2024

GitHub topped research firm's Gartner's inaugural Magic Quadrant report on vendors of AI code assistants, leading in both completeness of vision and ability to execute.

That's perhaps unsurprising as GitHub Copilot was the first AI coding assistant to break out of machine learning restraints some three years ago to help kickstart the GenAI era with deep learning and natural language processing (NLP) capabilities.

Gartner now describes the market like this:

Gartner defines AI code assistants as tools that assist in generating and analyzing software code and configuration. The assistants use foundation models such as large language models (LLMs) that have been optionally fine-tuned for code, or program-understanding technologies, or a combination of both. Software developers prompt the code assistants to generate, analyze, debug, fix, and refactor code, to create documentation, and to translate code between languages. Code assistants integrate into developer tools like code editors, command-line terminals and chat interfaces. Some can be customized to an organization's specific codebase and documentation.

Generally, these tools have advanced quickly since the debut of Copilot, moving beyond early, simple code-completion suggestions. Now, Gartner said, AI code assistants work across a range of use cases.

• Code generation: Developers use the code editors in AI code assistants to autocomplete code and generate features, which helps them to complete programming tasks faster.
• Code debugging: Developers use AI code assistants to detect and fix bugs in code, which helps them to resolve errors without the need to ask peers or search the internet for solutions.
• Code modernization: Developers use AI code assistants to understand complex dependencies across many programs, which helps them to reduce technical debt and modernize code.
• Artifact building and testing: Developers use AI code assistants to generate acceptance tests from user stories (for example, in Gherkin format) and to generate unit tests.
• Code explanation: Developers use AI code assistants to get natural-language explanations for code, which helps them understand complex and unfamiliar code.

Noting the AI-powered code suggestions and contextual assistance provided by GitHub Copilot, the first-ever report said: "Its operations are geographically diversified, and its clients tend to be large organizations across various sectors. GitHub provides GitHub Copilot to active maintainers in the open-source community, as well as teachers and students free of charge. GitHub is expanding Copilot with features like Copilot Workspace for a collaborative AI-native developer environment, Copilot Extensions for seamless tool integration, and enhanced security and compliance."

Joining GitHub Copilot in the Leaders quadrant were Google Cloud, Amazon Web Services (AWS) and GitLab.

Mandatory features for this nascent market as listed by Gartner include:

• Code completion from natural language (e.g., comment).
• Multiline, fill-in-the-middle code completion with the ability to plug integrations for multiple code editors.
• Ability to use the code assistant in more than one vendor ecosystem.
• Guarantee that base models will not be trained on customer code or documentation (excluding approved fine-tuning).
• Conversational chat interface integrated into the development environment.

Other common features range from on-premises or private cloud instances to filters for biased code, explicit language and images.

"The vision behind GitHub Copilot is simple: augment the innate human creativity of every developer with a boost from generative AI," the Microsoft-owned GitHub said in a blog post last week publicizing the report. "Our goal has never been to create technology for technology's sake, but to increase the happiness and productivity of every developer by keeping them in the flow longer, helping them move faster, and lowering the barrier to entry altogether. With millions of developers and over 77,000 organizations using Copilot, we feel we are making rapid progress toward that goal and realizing our vision."

Considering that two of the four Leaders are cloud giants and GitHub is owned by a cloud giant, the only non-cloud-giant Leader ended up being GitLab, which offers an AI-powered DevSecOps platform.

"AI code assistants go beyond just code generation and completion," said GitLab in its own post celebrating the report. "They're collaborative partners that boost developer efficiency by improving code quality and continuous learning. By automating routine tasks and providing intelligent suggestions, assistants like GitLab Duo -- our suite of AI-powered features -- free up developer time to focus on higher-level problem-solving."

While Duo is GitLab's offering in the space, Google was evaluated for its Gemini Code Assist tool, while AWS was evaluated for its Amazon Q Developer, formerly called Codewhisperer.

In addition to evaluating vendors, the report lists some strategic planning assumptions that shed light on Gartner's outlook for the market:

• By 2027, the number of platform engineering teams using AI to augment every phase of the software development life cycle (SDLC) will have increased from 5 percent to 40 percent.
• By 2027, 80 percent of enterprises will have integrated AI-augmented testing tools into their software engineering toolchain, which is a significant increase from approximately 15 percent in early 2023.
• By 2027, 25 percent of software defects escaping to production will result from a lack of human oversight of AI-generated code, which is a major increase from fewer than 1 percent in 2023.
• By 2028, 90 percent of enterprise software engineers will use AI code assistants, up from less than 14 percent in early 2024
• By 2028, the use of generative AI (GenAI) will reduce the cost of modernizing legacy applications by 30 percent from 2023 levels.

While Gartner typically charges for its reports, this and many Magic Quadrant reports are available for free from evaluated vendors who are granted permission to provide licensed-for-distribution editions, which can be found with a simple internet search.