Hopefully this post will prevent you from running into the same issues as we did while rolling out this feature. At the end, you can find the build scripts we use for our applications based on all the things we learned over time. Note that this post does not contain any performance measurements or comparisons, as that is a whole another story. This post is already too long for what it covers.

The concepts

If you are familiar with the JDK AOT concepts, you can skip this section. If not, you can get a short overview over all the relevant parts that go into the topic of AOT optimizations.

Timeline

The AOT cache itself has been extended with every JDK release since its introduction, with each step specified in a JEP (JDK Enhancement Proposal):

• JDK 24: JEP 483: Ahead-of-Time Class Loading & Linking introduced the AOT cache itself
• JDK 25: JEP 514: Ahead-of-Time Command-Line Ergonomics simplified the cache creation to a single step with -XX:AOTCacheOutput, and JEP 515: Ahead-of-Time Method Profiling added method profiles to the cache so that the JIT compiler can warm up faster
• JDK 26: JEP 516: Ahead-of-Time Object Caching with Any GC made cached objects usable with any garbage collector
• JDK 28: JEP 544: Ahead-of-Time Code Compilation will also add compiled code to the cache

Class-data sharing (CDS)

The AOT cache is not built from scratch. It is a natural evolution of class-data sharing (CDS), a HotSpot feature that has existed since JDK 5. CDS stores parsed class metadata in an archive file that the JVM can directly map into memory on startup. It was extended over time with application class-data sharing in JDK 10, a default CDS archive for JDK classes that is shipped with every JDK since JDK 12, and dynamic CDS archives in JDK 13. The AOT cache uses the same infrastructure, which is why you will still see CDS terms like "shared archive" or the cds log tag in many places when working with the AOT cache.

AOT class linking

AOT cache and AOT class linking are often used interchangeably, but they are not the same thing. The AOT cache is the file, which can contain different kinds of data: The loaded class data (the part inherited from CDS), method profiles, cached Java objects, some native code, and pre-linked classes. AOT class linking is just one of these optimizations. It is turned on automatically when creating an AOT cache (-XX:+AOTClassLinking) and stores classes in an already linked state, so that the JVM can skip that work on startup. You want to enable class linking, otherwise you lose out on startup performance gains.

GraalVM Native Image

The AOT cache should also not be confused with GraalVM Native Image. Native Image compiles your whole application ahead of time into a standalone native executable that runs without a JVM. To do that, it has to make a closed-world assumption: All code that can be executed at runtime has to be known at build time, so dynamic features like reflection, dynamic class loading, or resources have to be explicitly configured. In return, you get a very fast startup and low memory usage.

The AOT cache takes a different approach. Your application still runs on a normal JVM with the JIT compiler and has the full dynamic capabilities of Java, so you don't have to change or configure anything in your application. The cache only speeds up the work the JVM has to do anyway. If it can't be used, the JVM just falls back to doing all of that at runtime as usual. The startup improvements are therefore smaller than with Native Image, but you also don't have to deal with any of its restrictions. If you ever had to troubleshoot GraalVM native image issues, you are probably familiar with those.

CPU incompatibilities

In theory, the whole AOT cache idea sounds like something that should only affect the class data and not perform any native compilation. Because if it did, you would run into all sorts of issues if the individual CPU features of the training system and actual system would differ. Even if the CPU architecture is the same. Anyone who used GraalVM native image and ran into crashes on systems with missing CPU features is probably familiar with this and the GraalVM build argument -march=compatibility.

In practice, it turns out the AOT cache with class linking does generate small parts of native code. You will run into the consequences of this if your training system supports something like AVX, and the running system does not (or not to the same level, like no support for AVX-512, which corresponds to the JDK's -XX:UseAVX=3 level). Because then, your JVM will crash with:

A fatal error has been detected by the Java Runtime Environment:

SIGILL (0x4) at pc=0x00007fcc85ae2613, pid=4054, tid=4147

JRE version: OpenJDK Runtime Environment Zulu25.32+21-CA (25.0.2+10) (build 25.0.2+10-LTS)
Java VM: OpenJDK 64-Bit Server VM Zulu25.32+21-CA (25.0.2+10-LTS, mixed mode, sharing, tiered, compressed oops, compact obj headers, g1 gc, linux-amd64)

Problematic frame:
v ~AdapterBlob 0x00007fcc85ae2613

or

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ILLEGAL_INSTRUCTION (0xc000001d) at pc=0x000001d00d9f6c2b, pid=39700, tid=37844
#
# JRE version: OpenJDK Runtime Environment Zulu25.30+17-CA (25.0.1+8) (build 25.0.1+8-LTS)
# Java VM: OpenJDK 64-Bit Server VM Zulu25.30+17-CA (25.0.1+8-LTS, mixed mode, sharing, tiered, compressed oops, compact obj headers, g1 gc, windows-amd64)
# Problematic frame:
# v  ~AdapterBlob 0x000001d00d9f6c2b

This means that you for example can't run an application with an AOT cache which was trained on an 13th gen Intel Core CPU on a target system that has a 12th gen Intel Core CPU as those have different levels of CPU feature support for AVX. If you only have newer gen CPUs and ship your app to a customer who still runs older gen hardware, you will get an unhappy customer who thinks that you don't even do basic testing for their software.

Adapter caches

As it turns out, it generates adapter caches, which were added in JDK 25 with JDK-8350209: Preserve adapters in AOT cache. Adapters are small pieces of machine code that switch between interpreted and compiled Java calls (see the HotSpot source for a more detailed explanation). They're generated for the CPU features of the machine that builds the cache without taking into account that the application might be run on another system later on.

As soon as a system that lacks one of the CPU features used in the cached adapters calls one of them, the JVM crashes. This is not only a problem for desktop applications that are distributed to users. Even on backend servers, the AOT cache is often created in a CI pipeline or a container image build, and then deployed to machines whose CPUs differ from the build machine, e.g. when a cluster contains different instance types or the cloud provider moves your workload to other hardware.

How on earth is that the default behaviour for the AOT cache?

Disabling adapter caching

It turns out you can disable the AOT adapter caches by passing the arguments -XX:+UnlockDiagnosticVMOptions -XX:-AOTAdapterCaching to your AOT cache generation run. This option is not explained or documented anywhere, so you have to read the JDK source for that one. The even more confusing part was that this option is declared as false by default. It gets turned on automatically, however, by the AOT cache generation implementation if the cache contains AOT-linked classes, see here for JDK 25. You can see this with -XX:+PrintFlagsFinal, which shows the value as true {ergonomic} during the assembly run. So it is disabled by default, except in the default case.

Passing these arguments is a basically a requirement if you use the AOT cache. Otherwise, your users will run into crashes. This issue was eventually reported as JDK-8388703. JDK 27 will instead stores the CPU features of the training machine for the native code part of the AOT cache, which currently contains the adapters, and only uses the cached adapters on an exact CPU feature match.

What the AOT code cache actually is

To make things even more confusing, the term "AOT code cache" means two different things. They say there are only two hard things in computer science, cache invalidation and naming things, and this one manages to involve both. In the JDK source code and the logs, the AOT code cache (AOTCodeCache) is the part of the AOT cache that contains native code. This has existed since JDK 25 and currently contains the adapters described above. So when you see log messages like AOT Code Cache is not used: AOT Class Linking is not used or AOT Code Cache disabled: cpu features are incompatible, or read bug reports about the AOT code cache, they are mostly about the adapters, not about compiled Java code. It is also not the same as the regular code cache in which the JIT compiler stores compiled code at runtime. The CPU feature checks in JDK 27 mentioned above are also part of this. The actual compilation of Java code ahead of time, which is what most people would think of when hearing about an AOT code cache, will only come with JDK 28, and will store the compiled code in this same AOT code cache.

AOT code compilation in JDK 28

JDK 28 will release an additional feature, JEP 544: Ahead-of-Time Code Compilation, which, as the name suggests, will also do a native code compilation for your cache, with more code than just the adapters. According to the JEP, the AOT code will not be loaded if the CPU features have a mismatch, which at least will avoid crashes. This uses the same CPU feature check that already exists in JDK 27, which requires an exact match: On x86, all CPU features except hyper-threading have to be identical. So even a newer CPU that supports more features than the training machine will not load the AOT code.

However, that will also make this feature quite useless for applications that are distributed to users, as the chance of the CPU of a user exactly matching the CPU of your build machine is quite low. What would be needed is some sort of compatibility switch that disables all optional CPU features. Sure, you will get some reduced performance of the compiled code, but in exchange, a faster startup time. Technically, the compared CPU features are the ones that are enabled for the JVM, which can be reduced via options like -XX:UseAVX (see here). But you would have to pass the same options to the training run and the actual application run, and not every CPU feature has such an option, so that is not a real solution either. There needs to be a proper compatibility mode switch for this.

Scraping logs

When dealing with the AOT cache, the JVM argument -Xlog:aot is your best friend. Granted, the kind of friend who answers every question with a riddle. It will print the details when generating an AOT cache and will also show whether the AOT cache is loaded properly when you run your built application. However, there are several pitfalls as the output is quite tricky to understand and easy to misinterpret.

Two runs walk into a log

JEP 514: Ahead-of-Time Command-Line Ergonomics introduced an easier workflow for building the AOT cache. Instead of having to perform two steps, you now only need one invocation of your application to build the AOT cache. However, technically it still performs two separate JVM runs, it just hides that fact from you. As a result, you will get logs from two invocations as one, while only the second one, the assembly run that actually writes the AOT cache, tells you what ends up in the cache. Both look very similar.

If you are unaware of that, you will spend a lot of time debugging the wrong logs until you figure out that there are two separate logs printed one after another, and only the latter is relevant for the cache contents. So be careful when using CTRL+F and check whether you are looking at the correct version of the log.

Confusing log messages

Furthermore, the log messages are missing basically all the relevant information, and there are almost no docs to back them up. For example, the run that assembles the AOT cache will log something like this:

[0.024s][info][aot] full module graph: disabled
[0.024s][info][aot] initial optimized module handling: enabled
[0.024s][info][aot] initial full module graph: disabled
[0.024s][info][aot] Using AOT-linked classes: false (static archive: no aot-linked classes)
[0.032s][info][aot] JVM_StartThread() ignored: java.lang.ref.Reference$ReferenceHandler
[0.032s][info][aot] JVM_StartThread() ignored: java.lang.ref.Finalizer$FinalizerThread
[0.050s][info][aot] Not starting management agent during creation of AOT cache.

Now, your first questions are probably: What is a full module graph, why is it disabled, how can I enable it, does that influence the performance of the AOT cache? For all the answers, you will have to consult the JDK source code as that is barely explained. Why a certain feature is disabled is also not explained, so you will have to keep digging even more. What the implications of a disabled feature are, is also not documented.

The few relevant infos that are included in the java manpage are still listed in the CDS section, not in the actual Ahead-of-Time Cache section. So you will have to know where to look first. But those explanations are written with CDS in mind, so you still need to translate that to the AOT cache and know which part applies. But there is only a small amount of relevant information for this anyways.

To make it even more confusing, the full module graph: disabled lines above are printed at the start of every assembly run, even in a perfectly working setup. Whether the full module graph and AOT class linking actually ended up working is only visible at the very end of the log, in lines like Full module graph = enabled and the class summary instance classes = 9033, aot-linked = 9033, .... If something went wrong, you will instead find an error like archivedBootLayer not available, disabling full module graph somewhere in between, together with aot-linked = 0. So some messages about stuff being disabled can be ignored, while other messages that look very similar are relevant.

In summary, there is a lot of missing documentation for pretty much everything, unintuitive log statements, and an intransparent workflow for the AOT ergonomics. It takes a lot of time to even reliably understand what is going on.

Fragile class linking

AOT class linking is the most fragile part of the AOT cache, as it requires the complete module setup of your application to be stored in the cache as well (the so-called full module graph), and many conditions silently disable that. If that happens, the AOT cache is still created and used, but the classes have to be linked at runtime again, so you lose part of the startup improvement. So an AOT cache that is loaded successfully does not necessarily mean that class linking works. Going through all the work of assembling an AOT cache without class linking defeats part of the point, so the goal is to enable class linking.

Unfortunately, when assembling the AOT cache, it can feel like the AOT class linking feature is out there to get you and disable all optimizations if it even senses an opportunity, often without telling you why. For incompatible JVM options, you at least get a log message like optimized module handling: disabled due to incompatible property: ... (in JDK 24 only with -Xlog:cds, not -Xlog:aot, as someone probably forgot to rename the log tags). For some other examples below, the log does not tell you why it got disabled.

Module options

The original release of the AOT cache in JDK 24 wouldn't even allow you to specify a --add-opens in case you needed reflection for something - --add-opens, --add-exports, and --add-reads were all considered incompatible, which disabled the full module graph and with it AOT class linking. At least that was fixed in JDK 25.

GraalVM

If you are using another JDK implementation like GraalVM, you will run into more issues. GraalVM contains additional modules like jdk.internal.vm.ci that you won't find in the standard JDK implementation to provide the advanced JIT shenanigans via JVMCI (Java-Level JVM Compiler Interface). This module is present at runtime, but it won't get added automatically to the module graph of your application when GraalVM uses its native JIT compiler library (see here). As a result, you will have to figure out from the logs why the optimizations are disabled when building an AOT cache with GraalVM, just to find out that you need to do --add-modules jdk.internal.vm.ci for your AOT training runs to fix the module graph mismatch.

Final field mutation warnings

Next, you need to be careful which JVM arguments you enable. When using JDK 26, you will get warnings when a final field is modified via reflection due to JEP 500: Prepare to Make Final Mean Final. In theory a good idea, in practice quite annoying if some dependency triggers that, e.g. with some Jackson shenanigans. To remove those warnings, you will need to pass --enable-final-field-mutation=.... That option, however, disables the optimized module handling once again (the same goes for --illegal-final-field-mutation and --illegal-native-access), so the same problem as usual. There is also not a technical reason why that is strictly necessary, why not let me use it at my own risk? Right now, it is either a runtime warning for an illegal mutation of a third-party dependency or no class linking.

Incubating modules

Various module configurations disable optimizations (not documented of course, so you need to figure that out yourself). For example, if you use any incubating module of the JDK or JavaFX, no AOT class linking is possible. JavaFX's incubator modules are built with the same incubating flag as the JDK ones. If the module graph contains any incubating module, the boot layer is not archived, which disables the full module graph and class linking. Probably just for the simple reason that an archived boot layer skips the module system initialization that prints the annoying incubating module warning, not for any deeply technical reasons. It also took a fair amount of time to figure out why class linking was suddenly disabled after adding the incubating JavaFX richtext module.

Benchmark impacts

All those issues make comparisons incredibly difficult. How can a normal developer evaluate the gains of this feature if they have to be so careful that some optimizations don't get disabled? Just recently I saw someone posting AOT benchmarks made using GraalVM and I thought to myself: Unless that person actually inspected the aot log and understood all the vague messages, they probably didn't even realize that GraalVMs different module list disabled many optimizations.

The same goes for many other pitfalls. You carefully configured AOT for your application in JDK25, then upgraded to JDK26, saw these annoying illegal mutations warnings, and just disabled them, because why not. But by doing so you most likely didn't even realize that the AOT class linking is now gone.

There is also no way to fail if class linking gets disabled to detect it. If you would like to have your CI pipeline fail if the AOT class linking gets disabled for whatever reason, you can't do so. So you can't easily catch it.

Where is my AOT cache?

Previously, with CDS, it was never really a problem for the JVM to find the data. The default CDS archive is located at a fixed location inside the JDK (lib/server/classes.jsa, or bin\server\classes.jsa on Windows), or at a location that could be customized as needed. There was only one relevant JVM run environment.

With AOT, there are now multiple independent JVM runs: The training runs, the cache assembly run, and the actual target system run of your application. For a more complex application, there might even be multiple different AOT caches, e.g. if you build multiple launchers via jlink and multiple native executables via jpackage. The training and assembly runs get the cache location via -XX:AOTCacheOutput, but for the actual run of your application, you need to tell the JVM where the cache data is located with -XX:AOTCache. Otherwise it doesn't load it, there is no default value or automatic lookup.

Relative paths

This is actually not so easy to handle when it comes to relative file paths, which is basically how you have to use the option. Absolute file paths don't make sense for AOT caches as your application can be put into arbitrary directory locations on the target system. You can specify a relative path for -XX:AOTCache, but this one is relative to the working directory, not your executable. So anyone running your app in a different working directory will break the AOT cache.

jpackage installation layouts

When generating a runtime image for your application via jlink and jpackage (as is the intended way for the modern JDK), you will also have to deal with different installation layouts on each platform, so you will need platform-specific AOT cache locations. You also need to know how to specify paths relative to the installation root (also by looking through the JDK source code, as the jpackage documentation does not mention it). In summary, you need to specify the AOT cache locations in the application .cfg file of jpackage like this:

• Windows: java-options=-XX:AOTCache=$ROOTDIR/runtime/bin/server/cache.aot
• Linux: java-options=-XX:AOTCache=$APPDIR/../../lib/runtime/lib/server/cache.aot
• macOS: java-options=-XX:AOTCache=$ROOTDIR/Contents/runtime/Contents/Home/lib/server/cache.aot

The variables $ROOTDIR and $APPDIR are automatically replaced by the jpackage launcher on startup. $ROOTDIR does not work well on Linux, so $APPDIR is used as a workaround (that is a whole other story).

Generate it later

The AOT cache does not need to be generated ahead of the first launch of your application in production. It can also be generated dynamically later on, similar to CDS. If the AOTCache option handling for the actual cache file would be improved, that could also be done.

But right now, putting the newly generated AOT cache on the target system into the right spot, also considering readonly installation directories, isn't straightforward. There is no default file path. You can't use fixed absolute paths on a user's target system. Using environment variables is also not possible for the -XX:AOTCache file. You can pass a relative file path but only relative to the working directory. And if the installation directory is read-only or you lack permissions, you can't generate an AOT cache where you want it.

EDR struggles

Of course, an agent that hooks into class loading via JVMTI (a ClassFileLoadHook) also prevents the AOT cache from being loaded if it contains AOT-linked classes, that is no surprise based on the previously explained examples on how fragile the AOT cache is. However, in practice, this introduces more unexpected problems.

If such an agent is attached on startup, the AOT cache can't be used, and you will get an error message like this:

[error][aot] AOT cache has aot-linked classes. It cannot be used when JVMTI ClassFileLoadHook is in use.
[error][aot] An error has occurred while processing the AOT cache.
[error][aot] Unable to map shared spaces

This is not a terminal error, so what is the big deal? Well, it can unintentionally break the output of your application as those errors are printed to stdout, not the stderr. For example, SentinelOne will attach an agent to any started Java application, therefore they will always print this error by default.

If the user depends on your application output, e.g. by doing API_KEY=$(myapp generate-key), their workflows will break if they run an EDR or similar that tries to attach agents. Because suddenly, the usual one-line output of your command becomes multiple lines with AOT error on top.

You effectively have to run your app with -Xlog:disable for your production builds. Or you do something like -Xlog:disable -Xlog:all=warning:stderr to keep it in the stderr, where warnings+errors should normally be.

Putting it all together

To give you an idea of how all of this fits together, here is a simplified version of the Gradle task we use to create the AOT cache. It runs after jlink and jpackage have created the application image for the current platform and adds the AOT cache to that image before it is packaged into installers.

def distDir = "${project.layout.buildDirectory.get()}/dist"

tasks.register('trainAot', DefaultTask) {
    doLast {
        // You usually want your application to behave a bit differently
        // during a training run, so we pass properties to do that
        // E.g. it should not block with a dialog if it is a GUI application
        // as it should run unattended
        def properties = [
                "-D" + propertyName("dataDir") + "=$distDir/local_train",
                "-D" + propertyName("aotTrainMode") + "=true",
        ]

        def jvmName = System.getProperty("java.vendor.version")
        def graal = jvmName != null && jvmName.toLowerCase().contains("graal")

        // The generated jlink/jpackage image
        def imageDir
        // The command to launch the jlink launcher script
        def command
        // The cache output location is relative to the working directory of the jlink launcher script
        def cacheOutput
        // The jpackage app configuration file
        def cfgFile
        // The AOT cache location as seen from the jpackage launcher configuration
        def cfgCache
        if (os.isWindows()) {
            // We generated the application image here during the build
            imageDir = "$distDir/${productName}"
            command = ['cmd', '/c', "$imageDir/runtime/bin/${jpackageExecutableName}.bat"]
            cacheOutput = './server/cache.aot'
            cfgFile = "$imageDir/app/${jpackageExecutableName}.cfg"
            cfgCache = '$ROOTDIR/runtime/bin/server/cache.aot'
        } else if (os.isLinux()) {
            // We generated the application image here during the build
            imageDir = "$distDir/${productName}"
            command = ["$imageDir/lib/runtime/bin/${jpackageExecutableName}"]
            cacheOutput = './lib/runtime/lib/server/cache.aot'
            cfgFile = "$imageDir/lib/app/${jpackageExecutableName}.cfg"
            cfgCache = '$APPDIR/../../lib/runtime/lib/server/cache.aot'
        } else {
            // We generated the application image here during the build
            imageDir = "$distDir/${productName}.app"
            command = ["$imageDir/Contents/runtime/Contents/Home/bin/${jpackageExecutableName}"]
            cacheOutput = './Contents/runtime/Contents/Home/lib/server/cache.aot'
            cfgFile = "$imageDir/Contents/app/${jpackageExecutableName}.cfg"
            cfgCache = '$ROOTDIR/Contents/runtime/Contents/Home/lib/server/cache.aot'
        }

        def jvmOptions = [
                "-XX:+UnlockDiagnosticVMOptions",
                "-XX:-AOTAdapterCaching",
                "-XX:+AOTClassLinking",
                "-XX:AOTCacheOutput=$cacheOutput",
                "-Xlog:aot",
        ]
        if (graal) {
            jvmOptions += ["--add-modules", "jdk.internal.vm.ci"]
        }

        execOps.exec {
            commandLine((command + properties).collect { it.toString() })
            workingDir imageDir
            // We use the old variable CDS_JVM_OPTS of the default jlink launcher scripts
            // to pass custom AOT JVM arguments to the process
            // All jlink launcher scripts support this variable
            environment "CDS_JVM_OPTS", jvmOptions.join(" ")
        }

        def cfg = file(cfgFile)
        cfg.text = cfg.text + "java-options=-XX:AOTCache=$cfgCache\njava-options=-Xlog:disable\n"
    }
}

The training run uses a training mode property and a separate data directory, so that it does not interfere with any existing application data. In training mode, our application goes through a fixed workload, e.g. opening all the different pages of the user interface, and then exits by itself. That way, the training run covers the classes and code paths that are used in a normal session.

Conclusion

In summary, getting the AOT cache to work properly and without downsides is much harder than it should be. The feature is fragile, there is almost no documentation beyond the JEPs, and the logs often raise more questions than they answer. A lot of the knowledge in this post could only be obtained by reading the JDK source code.

That begs the question: Does a JEP even count as documentation? Is the original JEP even updated when the explained feature is updated like the AOT cache feature was multiple times? I see JEPs as a short summary for new proposed features, mainly focusing on the happy path. The goal of those pages is to convince people a feature is needed, not to extensively document it. I feel like anytime someone tells you to take a look at the JEP for documentation, there is something wrong here.

The feature also doesn't feel well-thought-out to tackle the challenges of real-world applications. Building the AOT feature on top of CDS certainly didn't help as those features have grown quite far apart in terms of their actual use case. Perhaps a separate implementation would have been better. But still, the fact that the whole CPU feature related crashes issue made it to the JDK25 release shows a lack of planning and preparation. Because this was not really a bug, it was a misplanned feature where seemingly everyone forgot that when something is generated ahead-of-time, the exact CPU features can and often will differ. This also diverged from the actual JEP, which did not list this as potential risks or considerations. The only way this could have been implemented in that form is that this was never considered or tested.

But anyways, that was a way too long article about AOT caches, and we haven't even touched performance comparisons! That might be the topic of another post.

---

Originally published at the XPipe blog.