How the code generation process in Xtext can be executed parallely

Are you developing a Xtext/Xtend-based code generator, and you want to know how to improve its performance using multiple execution threads? Then this article is just for you!

Writing a code generator that transpiles from an Xtext DSL file to code is simple. The Xtext framework provides all the necessary infrastructure, and the project wizard produces already a skeleton for a generator template out of the box. The generator automatically participates in Eclipse’s build process, so that saving a DSL file triggers the code generator and immediately produces the desired target files.

Did you know that this process happens single-threaded? And that the framework already provides infrastructure for parallelizing the work? Do you know why this isn’t enabled by default? Which limitations have to be considered?

Xtext’s default behavior for code generation

When you create a new Xtext project with the project wizard, Xtext creates an Xtend class <MyDSL>Generator in the runtime project of your DSL. This class implements the IGenerator2 interface. It is your job to transform the contents of a Resource to files through the given IFileSystemAccess2 instance.

For more complex generators, the code generator typically consists of many Xtend generator classes. The main generator class is then used as entry point and delegates to other templates to produce artifacts.

When DSL files are saved in Eclipse, the XtextBuilder computes the changed files and the relevant dependent resources and invokes its builder participants. The DSL’s implementation of the IXtextBuilderParticipant is invoked. By default, the class BuilderParticipant is bound. BuilderParticipant gets the DSL’s IGenerator2 instance injected and invokes it for each affected resource in the computed resource delta. These generator templates will write the produced strings through the file access, which is implemented by class EclipseResourceFileSystemAccess2. This all happens in the same thread.

With the amount of processed resource files, the complexity of the generator templates, amount of produced artifacts and the overall build time increases. Can you guess what the main bottleneck is? It is not the processing of Xtend templates! Xtend compiles to simple Java and executing templates is blazingly fast. Code generation means file access, and I/O is magnitudes slower than CPU processing.

How parallel building works

Starting with Xtext 2.7 another implementation of IXtextBuilderParticipant was introduced: ParallelBuilderParticipant. This builder participant basically separates the production of the generated file content (by templates) from the IO access (through the file access).

Now you could think that since IO access is the bottleneck in the generation process, it could make sense to parallelize writing of files by multiple threads. No, that’s not how it works. Instead, the generator templates are executed in multiple threads, but file access is done sequentially by a single thread.

The ParallelBuilderParticipant instance, which is invoked on the main thread, therefore holds a queue (FileSystemAccessQueue). This queue is populated with requests to write content (the generated file) to a location. To populate the queue the EclipseResourceFileSystemAccess2 is wrapped by a ParallelFileSystemAccess. Instead of writing directly content to target resources this file access just enqueues the request to write. The ParallelFileSystemAccess is then passed to the generator (IGenerator2) instance. Since the template classes only operate against the IFileSystemAccess2 interface, nothing has to be changed to the templates to support parallel building.

For the parallel invocation a thread pool is created with 2-4 threads, which are named ParallelGenerator-[0-3]. For each resource the generator is invoked on one of these threads and whenever a template has produced an artifact’s content, the writing of the files are enqueued. The ParallelBuilderParticipant processes the queue until all write requests are done. The files are written sequentially through the EclipseResourceFileSystemAccess2 instance as fast as possible. Writing of files starts when the first file content was put into the queue.

Dealing with thread-safety issues

In order to perform actions in parallel it is important that the operations are thread-safe. As you know, Xtext uses EMF, and generators heavily access the EMF-based ASTs. But EMF is not thread-safe by design, and it is up to the application to take care that parallel access on EMF objects do not modify their state. This is especially true for proxy resolution. The code generated by EMF for resolving references represented by proxies is modifying the object state. Thus, it must be assured that no proxy resolution happens while accessing the models during the execution of generator templates.

The Xtext framework resolves proxies during the linking phase. This happens before actually any generator is executed. Resources with unresolvable cross-references will be marked with linker errors. Generator templates won’t be called for any resource that has error markers, so template executions won’t trigger proxy resolution on resources with linking problems.

That is the main reason why parallel generation is not enabled by default. Enabling parallel generation should be an active decision by the framework user who are aware of the consequences and know how to build solid templates without side effects.

Another reason why Xtext stays with the default implementation is that this feature was not available from the beginning. There are also useful other alternative implementations of core framework functions.

But the Xtext project tries to be backward compatible as far as possible. This means that we can’t just exchange the default behavior which might surprise users of the framework. When Xtext users have built a language in the past, it should behave the same on newer releases. So language creators have to actively decide to use other implementations.

Parallel generators outside of Eclipse

The mentioned approach is specific to integration into Eclipse. When building “standalone” generators, e.g. for usage as command-line program or as a language server service, there is currently no framework support. The approach would be similar. Again, it must be assured that the EMF ASTs do not mutate during the generation process. Some projects use an EContentAdapter to listen on state change and abort the generation process with an error signal. But it is a project specific behavior that might not be what you want for your project.

So far the requirement was also not urgent enough to invest into a standard solution that would be suitable for most needs. If you need this, we would be more than happy if you try working on that and raise a PR. We’ll assist you when having design questions and provide reviews. Or get in contact with us directly. We are always interested in which requirements your projects could have that could be worth adding to the framework.