Can also be used through Claude Code CLI (CLAUDE_CODE_CLI), which needs no API key.
Choosing an LLM
The Azul Payara AI Upgrade Assistant works with the LLM of your choice. You provide the API key or the self-hosted model, and you pay the token costs of your provider. You can use the tool in three ways:
-
with an API key for your provider,
-
through the Claude Code CLI you already use, without an API key, or
-
as an MCP tool from Claude Code.
The tool has been optimised for use with Claude Sonnet 5.5, Claude Opus 5.5, GPT-6 Sol and Gemini 3.8 Flash. While we support multiple LLMs, different providers and models have varying results. Extensive testing has been carried out with key providers and models. See the Supported Providers and the test results for each, so you can make an informed choice.
Supported Providers
| Provider | Tested | Notes |
|---|---|---|
Anthropic (Claude) |
Yes, see results |
|
OpenAI |
Yes, see results |
|
Google (Gemini) |
Yes, see results |
|
Mistral |
No |
Supported, not part of the published tests. |
Custom OpenAI-compatible endpoint |
No |
Supported through |
Ollama (local models) |
No |
Supported, not part of the published tests. See Self-Hosted Models. |
See Getting Started for how to configure each provider.
Only Anthropic, OpenAI and Google models have been tested so far.
Recommendations
Based on the model comparison:
| Model | Recommendation | Why |
|---|---|---|
Claude Sonnet 5.5 |
Recommended, best general choice |
Complete fix with a small, easy-to-review change, good speed, moderate token use and correct reporting. |
Gemini 3.8 Flash |
Recommended when completeness matters most |
The only model that also fixed the nested class. It used about five times more input tokens than a typical run. |
GPT-6 Sol |
Recommended (best OpenAI option) |
The same fix quality as Sonnet 5.5, with similar token use. |
Claude Opus 5.5 |
Works |
The same fix as Sonnet 5.5 without any benefit, and a whole-file change that is harder to review. |
Claude Sonnet 5, Claude Haiku 4.5, GPT-6 Luna |
Works with review |
Only the minimal fix. Review the result for what was left out. GPT-6 Luna reported that it could not verify compilation. |
Gemini 3.5 Flash Lite |
Not recommended |
The report labels findings as changed that were not changed, which is misleading in an audit. |
GPT-6 Astra |
Not recommended |
The build failed: the model did not fix anything in three rounds. |
The comparison is based on one project and only four findings that needed the AI agent, so treat it as directional. Run the tool on a representative module of your own application before you choose.
How We Tested
We ran the tool against a set of open-source and hand-written Jakarta EE applications. For each application, the advisor first reports the number of issues found, and the agent then works in rounds, with a maximum of three rounds in our tests.
A run is successful when:
-
all issues that can be fixed by deterministic rules are resolved, and
-
the issues that require the AI agent are resolved to the required threshold.
Informational findings and findings that need manual work do not decide whether a run is successful.
We also compared 9 models on one open-source project, the hmis application. In that comparison the recipes first fixed all 22,515 findings that need no AI. The models then handled the four findings that needed judgement: one removed API that needs a real code change, and three informational items where the correct answer is to change nothing.
Test Results
|
Important
|
These results were produced with the models and tool version listed. Results can differ for your application and with newer model versions. The tool is in development, and runtime and reports have improved since these tests. The AI usage should remain similar. |
Model Comparison
Nine models ran the same upgrade of the hmis project (development branch, state of 1 October 2026) with migration tool version 1.0.0-SNAPSHOT. Since then the tool has gained improvements and features, so run time and reports look different now, but a model that performed better than the others then should still do so. "AI time" is the working time of the model itself. The total run time of 33 to 68 minutes is dominated by the recipe phase and infrastructure, so it does not say anything about the model.
| Provider | Model | Build | Result of the AI fix | AI time | Token use |
|---|---|---|---|---|---|
Anthropic |
Claude Sonnet 5.5 |
Pass |
Import swap and |
1m 39s |
Typical |
Anthropic |
Claude Sonnet 5 |
Pass |
Import swap only |
1m 18s |
Typical |
Anthropic |
Claude Opus 5.5 |
Pass |
Import swap and |
3m 12s |
Not comparable |
Anthropic |
Claude Haiku 4.5 |
Pass |
Import swap only |
1m 52s |
Not comparable |
Gemini 3.8 Flash |
Pass |
Most complete: |
2m 57s |
High (about 5x) |
|
Gemini 3.5 Flash Lite |
Pass |
Import swap only, with wrong outcome labels in the report |
1m 29s |
Typical |
|
OpenAI |
GPT-6 Sol |
Pass |
Import swap and |
1m 49s |
Typical |
OpenAI |
GPT-6 Luna |
Pass |
Import swap only, could not verify compilation |
1m 21s |
Low |
OpenAI |
GPT-6 Astra |
Fail |
No fix applied in three rounds |
22.9s |
Typical |
Eight of nine runs passed the build. Every model except Gemini 3.5 Flash Lite also handled the three informational findings correctly and left them unchanged. The fixes fall in three tiers, all of which compile:
-
Most complete: Gemini 3.8 Flash.
-
Good: Claude Sonnet 5.5, Claude Opus 5.5 and GPT-6 Sol. They made the bean serializable, which CDI needs for a session-scoped bean.
-
Minimal: Claude Sonnet 5, Claude Haiku 4.5, GPT-6 Luna and Gemini 3.5 Flash Lite. They only swapped the import, so review the result for missing
Serializable.
|
Note
|
Token counts of Claude Opus 5.5 and Claude Haiku 4.5 are too low to compare, most likely because cached tokens are not counted. |
Sample Applications
"Issues found" is the total reported by the advisor before the AI agent starts. Most issues are fixed by deterministic rules. A run is successful when the issues that need the AI agent are resolved to the required threshold (see How We Tested).
| Application | Issues found | Model / mode | Rounds | Duration | Result |
|---|---|---|---|---|---|
RoyalBank |
18 |
Claude Haiku |
1 |
4m 06s |
Successful |
jakarta-ee10-bookstore |
90 |
Claude Haiku |
3 |
1h 34m |
Successful after round 1 (72 of 76 AI issues fixed); rounds 2 and 3 did not improve the result |
jakartaee-examples (branch 9.0) |
206 |
Claude Code CLI |
1 |
- |
Successful |
patched-src-javaee7-samples |
3665 |
Claude Code CLI |
1 |
- |
Successful |
patched-src-javaee8-samples |
356 |
Claude Haiku |
3 |
6m 39s |
Not successful with the AI agent. An advisor-only rescan reports success, because no actionable AI issues remain. |
sample.daytrader7 |
728 |
Claude Code CLI |
3 |
14m 13s |
Successful in round 3 |
ee8-sandbox |
1448 |
Claude Code CLI |
- |
- |
Successful, full project compiles |
Java-EE-8-Application-Development |
1089 |
Claude Code CLI |
1 |
2m 20s |
Successful |
application-modernization-javaee-quarkus |
1610 |
Claude Haiku |
2 |
- |
Successful in round 2 |
refarch-jee |
62 |
Claude Code CLI |
1 |
- |
Successful |
monolith-enterprise-application |
64 |
Claude Code CLI |
1 |
- |
Successful |
guide-monolithic-to-microservices |
406 |
Claude Code CLI |
1 |
- |
Successful |
cargotracker (jakartaee8 branch) |
380 |
Claude Code CLI |
1 |
- |
Successful |
hmis |
1534 |
Claude Code CLI |
3 |
- |
Not successful: large codebase that was already partly migrated, with |
| Model / mode | Applications tested | Successful | Not successful |
|---|---|---|---|
Claude Code CLI |
10 |
9 |
1 |
Claude Haiku |
4 |
3 |
1 |
|
Note
|
The applications are not the same for each model, so these totals are an indication and not a like-for-like comparison. For a comparison across providers, see the model comparison. |
Self-Hosted Models
You can run the tool with a self-hosted model (for example with Ollama) when your source code must not leave your environment. The quality of the result depends strongly on the model. Smaller local models can be insufficient for the complexity of the code changes, so validate the result carefully before use.
Tips for Better Results
-
Commit your project before each run, or work on a branch, and review the result with
git diff. -
Run the tool on one module at a time for large or partly migrated codebases.
-
Start with a dry run (
-Dmigration.apply=false), or limit the agent with-Dmigration.agent.maxBatchesPerRound=2for a cheap trial. -
Use interactive mode on critical code, so you approve each file change.
-
With a model that makes minimal fixes, review the result for what was left out.