In five minutes to transitive imports within a DSL with Xtext

Xtext allows elements in DSLs to be referenced in several ways. One is to import elements via namespaces. This is done through the use of ImportedNamespaceAwareLocalScopeProvider, and allows the import of individual or, using wildcards (. *), all elements of a namespace.

However, there may be languages in which this behavior is not desired. In these languages, the user can explicitly import one or more resource files to access their contents.

A simple DSL with import behavior – thanks to Xtext

A DSL with such import behavior can easily be created with Xtext, by installing a parser rule in the DSL with the special attribute name importURI. The following example illustrates a simple DSL that allows you to define names in arbitrary resources and use them in greetings.

grammar org.xtext.example.mydsl.MyDsl with org.eclipse.xtext.common.Terminals
generate myDsl "http://www.xtext.org/example/mydsl/MyDsl"
Model:
	includes+=Include*
	names+=Name*
	greetings+=Greeting*;
Include:
	'import' importURI=STRING
	;
Name:
	'def' name=ID
	;
Greeting:
	'Hallo' name=[Name] '!'
	;

Suppose we want to send greetings to colleagues in our company. Since the company is large and employs many members of staff who work in different divisions, we need to create a separate file for each division that contains the names of the respective people. This increases overview and maintainability.

We want to include the name definitions in the scope by explicit import of a resource. This needs to be done in as rapid and resource-light a way as possible.

The approach here is to use the index, which eliminates the need for unnecessary and (in large models) time-consuming resource loading. First we need to write information about the relevant resources into the index. To do this, we implement a class MyDslResourceDescriptionStrategy, which derives from DefaultResourceDescriptionStrategy. The strings containing the URIs of the  resources imported into the parser rule model, are merged into a comma-separated string and stored under the key includes in the userData map of the object description in the index.

package org.xtext.example.mydsl

import com.google.inject.Inject
import java.util.HashMap
import org.eclipse.xtext.naming.QualifiedName
import org.eclipse.xtext.resource.EObjectDescription
import org.eclipse.xtext.resource.IEObjectDescription
import org.eclipse.xtext.resource.impl.DefaultResourceDescriptionStrategy
import org.eclipse.xtext.scoping.impl.ImportUriResolver
import org.eclipse.xtext.util.IAcceptor
import org.xtext.example.mydsl.myDsl.Model
import org.eclipse.emf.ecore.EObject

class MyDslResourceDescriptionStrategy extends DefaultResourceDescriptionStrategy {
	public static final String INCLUDES = "includes"
	@Inject
	ImportUriResolver uriResolver

	override createEObjectDescriptions(EObject eObject, IAcceptor<IEObjectDescription> acceptor) {
		if(eObject instanceof Model) {
			this.createEObjectDescriptionForModel(eObject, acceptor)
			return true
		}
		else {
			super.createEObjectDescriptions(eObject, acceptor)
		}
	}

	def void createEObjectDescriptionForModel(Model model, IAcceptor<IEObjectDescription> acceptor) {
		val uris = newArrayList()
		model.includes.forEach[uris.add(uriResolver.apply(it))]
		val userData = new HashMap<string,string>
		userData.put(INCLUDES, uris.join(","))
		acceptor.accept(EObjectDescription.create(QualifiedName.create(model.eResource.URI.toString), model, userData))
	}
}

To use our ResourceDescriptionStrategy, we need to bind it in the MyDslRuntimeModule.

package org.xtext.example.mydsl
import org.eclipse.xtext.resource.IDefaultResourceDescriptionStrategy
import org.eclipse.xtext.scoping.IGlobalScopeProvider
import org.xtext.example.mydsl.scoping.MyDslGlobalScopeProvider
class MyDslRuntimeModule extends AbstractMyDslRuntimeModule {
 def Class<? extends IDefaultResourceDescriptionStrategy> bindIDefaultResourceDescriptionStrategy() {
 MyDslResourceDescriptionStrategy
 }
}

So far we have only collected information and saved it in the index. To use this data, we also need our own IGlobalScopeProvider. To do this, we implement a class MyDslGlobalScopeProvider, which derives from ImportUriGlobalScopeProvider, and override the getImportedUris (Resource resource) method. This method returns a LinkedHashSet that ultimately contains all the URIs to be imported into the resource.

Reading the imported resources from the index is done by the collectImportUris method. This method queries the IResourceDescription.Manager for the resource's IResourceDescription. From this, the strings with the URIs of the imported resources of each model element, are read from the userData map, decomposed and the individual URIs are stored in a set.

package org.xtext.example.mydsl.scoping

import com.google.common.base.Splitter
import com.google.inject.Inject
import com.google.inject.Provider
import java.util.LinkedHashSet
import org.eclipse.emf.common.util.URI
import org.eclipse.emf.ecore.resource.Resource
import org.eclipse.xtext.EcoreUtil2
import org.eclipse.xtext.resource.IResourceDescription
import org.eclipse.xtext.scoping.impl.ImportUriGlobalScopeProvider
import org.eclipse.xtext.util.IResourceScopeCache
import org.xtext.example.mydsl.MyDslResourceDescriptionStrategy
import org.xtext.example.mydsl.myDsl.MyDslPackage

class MyDslGlobalScopeProvider extends ImportUriGlobalScopeProvider {
	private static final Splitter SPLITTER = Splitter.on(',');

	@Inject
	IResourceDescription.Manager descriptionManager;

	@Inject
	IResourceScopeCache cache;

	override protected getImportedUris(Resource resource) {
		return cache.get(MyDslGlobalScopeProvider.getSimpleName(), resource, new Provider<LinkedHashSet<URI>>() {
			override get() {
				val uniqueImportURIs = collectImportUris(resource, new LinkedHashSet<URI>(5))

				val uriIter = uniqueImportURIs.iterator()
				while(uriIter.hasNext()) {
					if (!EcoreUtil2.isValidUri(resource, uriIter.next()))
						uriIter.remove()
				}
				return uniqueImportURIs
			}

			def LinkedHashSet<URI> collectImportUris(Resource resource, LinkedHashSet<URI> uniqueImportURIs) {
				val resourceDescription = descriptionManager.getResourceDescription(resource)
				val models = resourceDescription.getExportedObjectsByType(MyDslPackage.Literals.MODEL)
				
				models.forEach[
					val userData = getUserData(MyDslResourceDescriptionStrategy.INCLUDES)
					if(userData !== null) {
						SPLITTER.split(userData).forEach[uri |
							var includedUri = URI.createURI(uri)
							includedUri = includedUri.resolve(resource.URI)
							if(uniqueImportURIs.add(includedUri)) {
								collectImportUris(resource.getResourceSet().getResource(includedUri, true), uniqueImportURIs)
							}
						]
					}
				]
				
				return uniqueImportURIs
			}
		});
	}
}

To use our MyDslGlobalScopeProvider, we have to bind it again in the MyDslRuntimeModule.

package org.xtext.example.mydsl

import org.eclipse.xtext.resource.IDefaultResourceDescriptionStrategy
import org.eclipse.xtext.scoping.IGlobalScopeProvider
import org.xtext.example.mydsl.scoping.MyDslGlobalScopeProvider

class MyDslRuntimeModule extends AbstractMyDslRuntimeModule {
	def Class<? extends IDefaultResourceDescriptionStrategy> bindIDefaultResourceDescriptionStrategy() {
		MyDslResourceDescriptionStrategy
	}
	override Class<? extends IGlobalScopeProvider> bindIGlobalScopeProvider() {
		MyDslGlobalScopeProvider;
	}
}

We launch the editor for our small language and start creating the model files. We decide not to import the resources of the different company divisions individually, but instead create a resource that contains all imports, then import that. For this we create the following resources:

But when creating the resource with the greetings, we notice that the names cannot be resolved. Why is that? Surely we wrote all imported resources into the index?

This is correct: all directly imported resources were written to the index. However, the imports that are themselves contained in an imported resource are ignored. The feature we need is called transitive imports: with this, importing a resource implicitly imports all the resources it itself imports.

To enable transitive imports in our language, we need to customize our MyDslGlobalScopeProvider. Instead of only storing the URI of an imported resource in the set, we also call the collectImportUris method and pass the URI as a parameter, so that its own imported resources are also processed.

package org.xtext.example.mydsl.scoping

import com.google.common.base.Splitter
import com.google.inject.Inject
import com.google.inject.Provider
import java.util.LinkedHashSet
import org.eclipse.emf.common.util.URI
import org.eclipse.emf.ecore.resource.Resource
import org.eclipse.xtext.EcoreUtil2
import org.eclipse.xtext.resource.IResourceDescription
import org.eclipse.xtext.scoping.impl.ImportUriGlobalScopeProvider
import org.eclipse.xtext.util.IResourceScopeCache
import org.xtext.example.mydsl.MyDslResourceDescriptionStrategy
import org.xtext.example.mydsl.myDsl.MyDslPackage

class MyDslGlobalScopeProvider extends ImportUriGlobalScopeProvider {
	private static final Splitter SPLITTER = Splitter.on(',');

	@Inject
	IResourceDescription.Manager descriptionManager;

	@Inject
	IResourceScopeCache cache;

	override protected getImportedUris(Resource resource) {
		return cache.get(MyDslGlobalScopeProvider.getSimpleName(), resource, new Provider<LinkedHashSet<URI>>() {
			override get() {
				val uniqueImportURIs = collectImportUris(resource, new LinkedHashSet<URI>(5))

				val uriIter = uniqueImportURIs.iterator()
				while(uriIter.hasNext()) {
					if (!EcoreUtil2.isValidUri(resource, uriIter.next()))
						uriIter.remove()
				}
				return uniqueImportURIs
			}

			def LinkedHashSet<URI> collectImportUris(Resource resource, LinkedHashSet<URI> uniqueImportURIs) {
				val resourceDescription = descriptionManager.getResourceDescription(resource)
				val models = resourceDescription.getExportedObjectsByType(MyDslPackage.Literals.MODEL)
				
				models.forEach[
					val userData = getUserData(MyDslResourceDescriptionStrategy.INCLUDES)
					if(userData !== null) {
						SPLITTER.split(userData).forEach[uri |
							var includedUri = URI.createURI(uri)
							includedUri = includedUri.resolve(resource.URI)
							if(uniqueImportURIs.add(includedUri)) {
								collectImportUris(resource.getResourceSet().getResource(includedUri, true), uniqueImportURIs)
							}
						]
					}
				]
				
				return uniqueImportURIs
			}
		});
	}
}

When we reopen our resource with the greetings after this adaptation, we see that the names are resolved by the transitive imports.

The sample project can be downloaded here.