There is currently some buzz around Microsoft’s Visual Studio Code (VS Code) editor and its ability to support rich code editing for potentially any language through its Language Server Protocol (LSP). Thanks to the work from TypeFox, the upcoming version 2.11 (release date Oct 18th) of Xtext will provide LSP support which enables integration of Xtext DSLs in code editors that support the LSP. This article explains how Xtext supports the Language Server Protocol and its implementation in detail and how a VS Code extension can be built and installed that uses an Xtext Language Server.
Xtext provides the server side of the LSP, while the client (VS Code) has to be configured to use a language server. VS Code is an extensible desktop code editor running on Windows, Linux and OS X. It has a powerful extension mechanism, thus the most convenient way to integrate language support into it is by an extension.
Architecture
The Language Server Protocol describes a standardized protocol to integrate language features such as validation and content assist in clients. TypeFox has developed a Java implementation of the protocol which is available in the ls-api repository at GitHub and explained in more detail in Miro Spoenemann's blog post. Currently they opened a project proposal to bring the LSP to Eclipse called LSP4J.
Xtext implements the LS API and enables you to bring your language to clients like VS Code or Eclipse Che.
VS Code provides an extension mechanism which allows you to hook in your own extensions. Extensions can be installed and deinstalled either from the file system or through VS Code’s marketplace.
In order to let a VS Code extension provide rich language support via a language server, such a server has to be started and the client connects to it. Basically, there are two ways to use a language server: an external server or an embedded one within the extension.
External Server
The first approach is to start a server separately and let the client talk to it.
The diagram shows how communication can take place over sockets. The server is running outside of the extension and waits for the client to connect. The client initiates the connection and asks the server about the services it provides. When the client closes the connection, the server shuts down. This scenario is very useful to debug the server side, as you can startup the server within a development environment like Eclipse. The lifecycle of the extension is bound to the lifecycle of VS Code. So the server will shut down when VS Code is closed and you have to start it up before you want to use VS Code with Xtext integration again.
Embedded Server
A more convenient scenario is to embed the server within the extension.
The diagram shows that the server can be embedded in the extension and communicate with the client. To make this happen, the extension spawns a new process where the server is running in. In this case the communication is done through process IO. A big advantage is that the lifecycles of the extension and the server are the same and you do not have to care about starting up a server separately before starting VS Code.
Xtext's LSP Implementation
As mentioned before there is a Java implementation for LS API.
Xtext implements this API in a generic way so that the implementation can be used for all languages that are built with Xtext. To embed all kind of languages Xtext uses Java’s Service Loader mechanism to load all implementations of the ISetup interface available on the classpath. The service registration is defined by the file
<mydsl>.ide/src-gen/META-INF/services/org.eclipse.xtext.ISetup.
As known from Xtext, the UI independent implementations are located in the Runtime and Generic IDE modules of an Xtext DSL implementation. So existing implementations that were used for Eclipse and IDEA or Web can be reused. No need to duplicate stuff.
Xtext Workspace Mechanism
To resolve cross-references Xtext needs to know about existing elements that are visible from other resources. In Eclipse the workspace, projects and their dependencies define what is visible and the necessary information is collected in the index. An incremental builder cares about filling that index and keeps it up to date. To enable an equal scenario for VS Code and other clients, Xtext comes with a workspace manager that is responsible to do so. The clients defines a root-location that should be handled like an Eclipse workspace root. The workspace manager picks up that location and fills the index with the existing resources. Changes to existing and new resources through the clients interface will trigger an incremental build and the index is up to date.
Showtime!
Curious enough to see it in action? Follow these simple instructions to see the current features live in VS Code. If you haven’t installed it already, download and install it now and let the fun begin.
A project that demonstrates a sample DSL with VS Code extension is available at GitHub.
The extension contains an embedded version of the simple Greeting DSL which the Xtext New Project wizard creates by default. A pre-built version is available as download on the Visual Studio Code Marketplace. Just start VS Code, open the Extensions on the left side, enter "Xtext VS Code example" and click Install.
If you want to build the project from scratch, follow the instructions on the project’s website.
- Download the VS Code extension mydsl-sc-0.0.1.vsix
- Open the extension with VS Code to install it (File -> Open)
- Restart VS Code
- Create an empty folder and open it with VS Code
- Create two files A.mydsl and B.mydsl
A.mydsl:
--------------
Hello A!
Hello B from A!
Hello name
from Other!
B.mydsl:
--------------
Hello Other!
You will see that Hello is recognized as a keyword. The identifier name in A.mydsl is marked as an error, since a validation rule in the DSL implementation checks that all identifiers have to start with capital case.
Now select the identifier Other and select ‘Go to definition’ from the context menu. This will open B.mydsl and select the definition of Other.
Structure of the VS Code Extension
package.json
Basically a VS Code extension is a Node.js module. As such its central descriptor is the package.json file. It defines project metadata, dependencies (most notably vscode-languageclient), the contributed languages, and entry points to activate the extension.
mydsl.configuration.json
This file defines a language’s symbols that recognize single and multi-line comments, brackets, auto closing pairs and selection surrounding.
src/mydsl-full.jar
To simplify the internal structure of an embedded language server extension a DSLs implementation with all its dependencies can be bundled to a Fat Jar with John Engelmann’s ShadowJar Gradle plugin. The resulting jar declares org.eclipse.xtext.ide.server.ServerLauncher as the main class to execute.
Unpacking all dependencies into one single jar has one nifty consequence: Several plugins define a plugin.properties file, and during the unpacking process one overwrites the other. However, EMF requires at least the property _UI_DiagnosticRoot_diagnostic. To work around the issue, a plugin.properties file is put into the src folder of the DSLs .ide project. It only has to contain the single property:
<mydsl>.ide/src/plugin.properties:
---------------------------------------------
_UI_DiagnosticRoot_diagnostic=foo
The resulting Fat Jar is placed by the build process in folder src.
src/extension.ts
In package.json it is declared that extension.ts is the extension’s main file. The extension.ts file contains a function called activate, which is called by VS Code on loading the extension. The activate function is the right place to spawn the language server process with a java command. Since the DSL’s language server was bundled in a single executable jar, building the java command line is trivial.
syntaxes/mydsl.tmLanguage
This file contains the definitions of text patterns that are recognized as keywords. Although the language server protocol defines a service for syntax coloring, this file can be used by the client for syntax highlighting without the need for a running language server.
Summary
Microsoft’s Language Server Protocol is implemented by Xtext to provide language servers for DSLs. Visual Studio Code is an editor that supports this protocol as a client. The required client side binding for VS Code can be provided by a VS Code extension, and even the language server can be bundled within such an extension and spawned on activation of the extension. The example given in this article bundles a simple Xtext DSL implementation in a single executable jar in an VS Code extension module and starts the language server within the extension’s lifecycle. The example DSL extension is available in VS Code's Marketplace.
Comments