CodeRabbit scales agentic software development and ships 20x more

CodeRabbit built its first product with GPT‑3.5 Turbo. Harjot says the arrival of GPT‑4 made the product effective at reviewing code. The company went on to adopt OpenAI o1 and OpenAI o3 early. Because code reviews run asynchronously in the cloud, CodeRabbit could use models that took longer to reason through a problem in exchange for more capable analysis.

The team evaluates new models against its own benchmark of known issues. At the time of the interview, Harjot estimated that GPT‑6 Astra⁠ caught 50% of the issues in that dataset, putting it at the top compared to other models in that benchmark.

The benchmark continues to evolve as agents introduce different kinds of mistakes and as fixes reveal problems earlier reviews missed. “The nature of bugs keeps evolving, so does our benchmark as the models get better,” says Harjot.

Better price-performance from OpenAI models has helped CodeRabbit move capabilities from research into its product. Harjot points to improvements in model pricing and caching that let the company use more AI, add more subagents, and pass the benefits to customers.

When CodeRabbit started, it used one or two subagents for code navigation. Lower-cost models such as GPT‑6 Luna now support more subagents throughout the platform, while models such as GPT‑6 Astra handle planning and orchestration.

“Now we are using agents everywhere on the use cases we couldn’t even afford in the past,” Harjot says. That expands the validation CodeRabbit can perform and the explanations it can give developers, helping them understand and trust changes they did not write themselves.

By using Codex, Harjot says the company has been shipping roughly 20 times more software over the last 6 months without adding engineers. He personally estimates that he pushed 2,500 to 3,000 pull requests in the 30 days before the interview.

Getting there required more than adopting Codex. Running multiple agents exposed constraints across the development process, from laptop memory and compute to testing, the tools that compile and package software, and dependencies.

The team streamlined its release process and reconsidered where human approvals belonged, while using CodeRabbit to automatically ship low-risk changes that didn’t need human intervention. “We had to actually completely rethink the entire software delivery,” Harjot says.

The company has also built a Codex-powered Slack bot that triages production incidents, Slack messages, and support tickets. It can return a pull request for the team to review and merge. Harjot sees cloud-based development as a way to reduce local-machine constraints and give teams stronger controls over access and auditing.

CodeRabbit uses Codex for work beyond engineering, including managing marketing and operational tasks through code. “We are running the company as code as much as possible,” Harjot says.

In marketing, the company manages Google Ads through pull requests. The team opens and merges a request to deploy ads. It also uses a content studio repository where Codex creates slide decks and customer-facing documents. “If you can translate your marketing problems to code problems, you will run 10x faster,” Harjot says.

CodeRabbit has run agents in sandbox environments since 2023, cloning repositories and letting agents write code to help perform reviews. More recently, CodeRabbit used OpenAI’s Cyber model through Daybreak to test and secure its own platform.

The team used the model for internal penetration testing, probing its own platform for ways an attacker could break in. Harjot says the early access helped CodeRabbit secure its own stack. “We found multiple vulnerabilities, advanced hacks that we could not even imagine could happen,” Harjot says.

It has since taken those capabilities to market, helping secure its customers’ code with its latest product, CodeRabbit Secure. Over the past two weeks, the company ran a promotional offer where it covered $1M in free deep scans for their users and found over 43,000 security vulnerabilities across 13,000 deep scans, all powered by Codex models. Thousands of these vulnerabilities have already been fixed by its customers, helping improve their code security.

As background agents begin responding to alerts and incidents without a person initiating every task, Harjot expects teams to need more help deciding which changes to take to completion.

Looking ahead, CodeRabbit is expanding toward agentic software development, helping teams validate, prioritize, explain, secure, and verify agent-generated code changes at scale.

The opportunity is to help developers stay in control as agents do more of the building, with validation, explanations, and maintenance that keep pace with the code being produced.