In June, I used AI for the first time to get meaningful, even astonishing, speed up in development tasks outside the playground. I have not written about it so far, but now decided it might be interesting for others to understand what is possible today with AI in real projects on a complex codebase. I will give a short introduction about the project, the task I was tackling, the used tools and then explain in detail how the development was sped up by a very large factor by going through an example.
The real world project I am talking about is Operaton. That is a Camunda 7 fork and hence an open source BPMN 2.0 process engine (Apache 2.0 license). It is completely open source and developed by the community. Downloading the source code as a ZIP file brings over 27MB, which should give an impression that this project is clearly outside the toy area. If you use Camunda 7 and struggle with Camunda 8 (e.g. because of license change) you might consider switching to Operaton (😉).
Operaton 1.0.0 aims to be compatible with Camunda 7 to a very large degree, making migration as easy as possible. But at the same time upgrade the used infrastructure and libraries to recent versions. One of these upgrades was the JUnit 4 to JUnit 5 migration. What seems easy at a glance is much more elaborate when you look at the details.
There are thousands of java test classes with tens of thousands of test cases that need to be migrated. And these migrations are more difficult than just to adapt imports and change annotations. Indeed, there were quite a lot of JUnit 4 Rules (e.g. to start and stop the engine, caching it for test speed up) that would need to be ported to JUnit 5 Extensions. Some of the adaptations are straightforward, others are complicated. Writing a migration script would not be feasible because the code changes were too heterogeneous. I started converting the tests manually, converting the first Rules to Extensions, adapting tests by hand … not fun. I decided to use AI for assistance.
The first conversions were done by me. After connecting the tools, I used Codex to perform most of the remaining work. As an example, I will pick one of my PRs and walk you through the complete execution. I will include my complete prompts to give you an idea on how to work with these tools (in this case: Codex). The final PR is only one of many and can be found here.
I started by explaining the task. Because I had already done conversions before, I was able to point to these as examples. I also added some explanation on how to exactly do it, taking into account what has gone wrong before. Here is my original prompt for reference:
We are in the process to migrate all JUnit4 tests to JUnit 5. Inside the "operaton-engine" most tests are already converted. These can be found inside "src/test/java". A random example of an already converted test is "org.operaton.bpm.engine.test.jobexecutor.AcquirableJobCacheTest". Here, the ProcessEngineExtension and ProcessEngineTestExtension are used as replacements of ProcessEngineRule and ProcessEngineTestRule. The test infrastructure of JUnit4 tests can be found in the source folder "src/main/junit4" while the new extensions are in "src/main/junit5".
Now we want to convert the tests inside "operaton-engine-rest-core". The tests to convert are inside "src/test/java". Go through all the tests and convert them to JUnit5. Use the extensions instead of the rules. Use the builder for ProcessEngineExtension all the time. Do not use the constructor of the extension, use the builder. ProcessEngineExtension needs to be hold in a static reference all the time. ProcessEngineTestExtension and others should never be static.
Also update the pom.xml from operaton-engine-rest-core with the newly needed dependencies to make the JUnit5 extensions from engine available.
To my disappointment, the reaction after around five minutes was nothing and complained that the task was too large and too difficult.
Notes
The repository contains more than 130 JUnit‑4 tests in operaton-engine-rest-core.
Migrating them all to JUnit 5 requires converting parameterized tests, adapting
custom container rules, adding JUnit 5 extensions (ProcessEngineExtension,
ProcessEngineTestExtension), and updating test infrastructure.
This is a large and complex change that cannot be completed within the current
session.
Summary
No changes were made.
I was pretty sure that the task should be possible for the AI, so I didn’t give in. I assumed that it just didn’t pick up the existing infrastructure and pointed that out.
Please take a look at RedeploymentProcessApplicationTest. This is an already converted parameterized test. In essence, ParameterizedTestExtension provides the drop-in replacement to easily convert parameterized tests. I am pretty sure updating of test infrastructure is not needed, because this is already done inside operaton-engine/src/main/junit5 and operaton-bpm-junit5/src/main/java. Please just start converting the tests using the existing infrastructure. For most tests, it should be sufficient to change some imports and replace some rules with extensions. Parameterized tests should be very straightforward as well.
After another 5 minutes, the problem was explained to me. It was not about the existing infrastructure, but actually about a rule that was not yet converted to an extension.
Notes
Fine. I get it. To add a bit of guidance, I guided the AI to first convert the rule to an extension.
Ok, got it. In this case, please create an TestContainerExtension as replacement for TestContainerRule. The goal is to make the conversion of the test cases in the future as easy as possible. Make sure the test infrastructure for starting the REST runtime is present for JUnit 5.
Seven minutes later, it wrote this class, which was a perfect port of the existing rule.
https://github.com/operaton/operaton/blob/95c18cf1acbc38ad0175fd87396f7bbbf359cb45/engine-rest/engine-rest/src/test/java/org/operaton/bpm/engine/rest/util/container/TestContainerExtension.java#L4
With the extension in place, I asked again to convert the tests.
This looks quite good. With this extension in place, could you now go and rewrite the JUnit 4 Tests in operaton-engine-rest-core to JUnit 5?
After ten minutes, Codex came back and reported to have converted 137 test classes from JUnit 4 to JUnit 5.
Summary
I checked out the commit and reviewed the code. Codex couldn’t run the tests by himself because of network issues (better to resolve them next time before using Codex!). So I got a bunch of errors and pasted them back (I skip the stack traces because they are … lengthy). These problems were solved.
I fixed some compile and casting issues. But it seems there is something wrong with the mocks. Please figure out if this is an issue with the tests or if we need to disable the checks. Then fix the code. [...]
Another five minutes are we were ready.
Notes
Summary
I did some polishing manually afterward. For example, there were minor issues where I wasn’t satisfied with the order of members and similar stuff. But after the final polish, the PR was reviewed and accepted. It was one out of many. Many others were much more straight forward, four out of five gone through without much back and forth, and thousands of classes were migrated MUCH faster than it would have been possible by hand. I would estimate a speed-up from at least 5x, if not more. It is not as simple as “Please do the JUnit 4 to JUnit 5 migration” and just wait for two hours. We are not there yet. But for these kinds of tasks, that are a bit too complicated to be scripted, but too dull and large to execute by hand, current AI is priceless.