Artificial Intelligence is transforming the world of software development like never before, with tools ranging from GitHub Copilot to startups leveraging large language models (LLMs) to speed up application creation. But despite the hype surrounding AI, recent insights suggest it’s not quite ready to take over all coding tasks, particularly debugging.

Researchers from Microsoft have found significant limitations in AI's ability to debug software, which is a critical part of a developer’s role that often consumes the majority of their time. They’ve developed a new tool called debug-gym, aimed at evaluating and improving AI models in this specific aspect.

Debug-gym creates an interactive environment where AI can utilize advanced debugging tools that have traditionally been outside its grasp. While this innovative approach enhances the models' performance, it reveals that they still falter compared to seasoned human developers.

Unveiling the Debug-Gym Tool

The debug-gym allows AI agents to broaden their capabilities by setting breakpoints, navigating code, and even printing variable values. According to Microsoft’s findings, when AI has access to these tools, its success rate in debugging tasks marginally improved to 48.4 percent. However, that’s still far from reliable for real-world scenarios.

AI's Debugging Limitations