📘 Ch2. Principles of SLE
This chapter presents the key principles of Software Language Engineering (SLE), a set of of software engineering methods and techniques to efficiently create and maintain new software languages.
1. Design of a software language
A software language can generally be designed with the following ingredients:
-
At the center, the abstract syntax of a software language defines the concepts of the language and the relations between said concepts. These concepts are the abstractions provided by the software language. A common way to design an abstract syntax is to rely on object-oriented modeling, for example using a simplified UML class diagram (also called a metamodel in a language engineering context) without operations.
-
Below the abstract syntax, an example of abstract syntax tree (AST) is depicted. It is composed of instances of the concepts of the abstract syntax. An AST is the "pure" representation of a program or model which can be programmatically processed by the semantics.
-
-
On the left, the concrete syntax of a software language defines the notation of concepts defined in the abstract syntax. Notations can be either textual (as most programming languages) or graphical (as most modeling languages). Notation is commonly expressed as rules telling with words or shapes correspond to each concept.
-
Below the concrete syntax, an example of concrete representation of a program/model is depicted. Here since the concrete syntax is graphical, it is composed of shapes and arrows allowed by the graphical syntax. Using the concrete syntax rules, a concrete representation can be parsed to obtain the corresponding AST.
-
-
On the right, the semantics of the software language defines the meaning of each concept defined in the abstract syntax. The meaning of a concept can be expressed in many ways, informally (with natural language and examples) or formally (with mathematical notations), and should somehow explain what happens when a concept is encountered during an execution of the AST.
-
Below the semantics, the result of the execution of AST is depicted. This result is generally a sequence of execution steps with a computational effect.
-
The official Java Language specification architectures the Java language in the following way:
-
In the center, the abstract syntax is partially shown with three concepts: ClassDeclaration represents the declaration of a class, FieldDeclaration the declaration of a typed field (also called attribute) within a class, and MethodDeclaration the declaration of a method (along with type and parameters) within a class.
-
On the left, the concrete syntax must describe how each concept should be represented textually. A ClassDeclaration is written using the
classkeyword, preceded with a modifier (public,private, etc), and followed with a name and, optionally, a superclass with theextendskeyword. -
On the right, the semantics must explain what each of the concepts mean. Here we show direct quotes from the official Java specification.
We would like a DSL to represent a specific sort of state machines.
We architecture our DSL in the following way (with only excerpts of each of the three constituents):
-
At the center, a small excerpt of the abstract syntax defines that the DSL includes concepts such as StateMachine, State and Transition.
-
At the left, a small excerpt of the concrete syntax defines that a StateMachine is represented by a drawing area, in which States are represented as rounded boxes with a name, and Transitions as solid arrows with an event name as label.
-
At the right, a small excerpt of semantics defines that executing a state machine means changing a current State from one state to another based on which Transitions are triggered by events.
| Throughout the course, we will design and implement a more detailed version of this DSL, and with an alternate textual concrete syntax instead of a graphical concrete syntax. |
2. Implementation of a software Language
Once a software language has been designed with the three ingredients we have seen in the previous section (abstract syntax, concrete syntax, semantics), the next step is to implement the software language… as software! We will distinguish:
-
the implementation of the core components of a software language, which are the minimal parts required to use the language,
-
the implementation of additional tools and of IDE integrations for a software language, to make the language convenient and productive to use for language users.
2.1. Core components of a software language implementation
Each of the three parts of a language design must be implemented as a language component, as depicted in the following figure:
-
The concrete syntax must be implemented in the form of a parser, which must be able to take as input a concrete representation of a program of the language, and produce at output an AST of the same program. This requires being able to identify (textual or graphical) patterns in the concrete representation.
While the word parser is mostly used for textual languages, here we also use this word for graphical languages. A parser for a graphical language is of course very different from a textual language parser because it must somehow recognize shapes, arrows and drawings instead of traditional text. -
The abstract syntax must be implemented in the form of a a set of data types, typically classes in an object-oriented context. The parser must be able to instantiate these data types in order to create the nodes of an AST of a program.
-
The semantics can typically be implemented in two different ways:
-
A code generator takes an AST as input and produces executable code as output. This output code is written using a separate target language that was already implemented. A target language can be either a bytecode language (such as Java bytecode), or an assembly language (such as x86 assembly), or even another programming language (such as Java, C, Rust).
-
An AST interpreter takes an AST as input and directly performs a computation based on the AST contents. This computation is typically a sequence of execution steps that changes the current state of the program.
-
A language implementation whose semantics is implemented with a code generator is commonly called a compiler, while a language implementation whose semantics is implemented with an AST interpreter is commonly called an interpreter.
|
| Just-in-time compilation is a hybrid form of semantics implementation that combines both interpretation and compilation. |
2.2. Additional tools and IDE integration
Once the core components of a software language have been implemented (which means a complete interpreter or compiler) we are now technically able to use the language by writing and executing programs. However, common tools are still missing to make it properly usable by the language user, such as:
-
An editor for the language, both to help reading a program (with syntax highlighting, an outline view, navigable symbols, etc) and writing a program (with code completion, smart renaming of elements, refactoring actions, etc.). We call editing services the set of services offered by an editor for a given software language.
-
A validator for the language (also called linter or static analyzer) for the language, able to analyze an input program and to detect structural problems as early as possible. A validator is typically implemented as a set of validation rules.
-
A debugger for the language, allowing the language user to execute a program step by step and to visualize the changes in the program state.
Moreover, a proper development environment is important to be able to use a software language efficiently, including the aforementioned tools. This can be achieved through an integration into existing Integrated Development Environments (IDEs) such as Eclipse, IntelliJ or Visual Studio Code.
With all these new concerns in mind, the scope of a software language implementation can be extended in the following way:
-
We can observe that most tools are dependent on the abstract syntax data types of the language implementation. In other words, most tools can be written without any dependency on the concrete syntax of the language.
-
At the bottom, two different ways to integrate the language with an IDE are depicted:
-
On the left, we see a plugin developed for "IDE 1" which directly uses the language implementation, whether by directly using the language implementation code, or by using the command-line interface (CLI) of the language components and tools.
-
On the right, we see plugin developed for "IDE 2" which relies on languages protocols to interact with the language implementation. A language protocol is a novel way to communicate with a language implementation in a client-server fashion, therefore decoupling as much as possible a language implementation from an IDE. This approach requires that the language implementation provides a language API, typically as a JSON-RPC web server. Two language protocols are commonly in use today: the Language Server Protocol (LSP) for editing services, and the Debug Adapter Protocol (DAP) for debugging services.
-
3. Engineering a DSL
We now know what SLE is about, and what are the ingredients of a software language. Let’s dive into a more concrete question: how to engineer a new DSL?
Since a DSL is a software system, we can proceed similarly:
-
Design the DSL, which includes designing each component of the DSL,
-
Implement the DSL, which means implementing each component of the DSL.
3.1. How to design a DSL?
Since a software language is made of software, a software language can be seen as a specific kind of product that must be delivered to a client (the language users, aka the domain experts). Hence, a great part of designing a DSL is to properly understand why such product is need, what are the needs of the client, and in which context the product will be used.
A particularity of software languages is that their purpose is always to provide the means to express something, such as a solution to a given problem, or some data relevant to a system, or the configuration of a system. Hence designing a DSL means focusing greatly on expressiveness, through the following question:
-
What do we want to express using the DSL, and why? This question aims to define the exact scope and domain covered by the DSL. It requires good understanding of the context and of the target users, and it results in a design of the abstract syntax of the DSL.
-
How do we want to express something? This question aims to define how will target users manipulate the concepts of the language: words, separators, shapes, colors, etc. This also requires a good understanding of the target users, and results in the design of the concrete syntax of the DSL.
The concrete syntax can be considered the "user interface" of the DSL. Hence it is a central matter regarding the resulting user experience. -
What is the meaning/purpose of a program written with the DSL? This question aims to define what should a program become once expressed. Is it executable, and should therefore be either interpreted or compiled? Or does it only serve purpose as documentation, or data definition? This results in a design of the semantics of the DSL.
Each following chapter will dive into a specific component of a DSL (abstract syntax, concrete syntax, semantics) and will present an approach to design said component.
3.2. How to implement a DSL?
There is a vast ecosystem of tools available to help with the implementation of a DSL.
Metalanguages are specialized software languages directly made for the engineering of other software languages. There are metalanguages to implement an abstract syntax (such as Ecore or the MPS Structure Language), metalanguages to implement a graphical concrete syntax (such as Sirius View Model), metalanguages to implement a textual concrete syntax (such as ANTLR, Xtext, Langium), and many more. Metalanguages provide abstractions directly dedicated to SLE, and can therefore facilitate the design and implementation of many aspects of a DSL.
Languages workbenches are complete environments or frameworks that assist with the end-to-end creation and packaging of DSLs. A language workbench usually provides a set of metalanguages, a set of software librairies, and tools such as generators and debuggers. A common (but not systematic) pattern is to rely on metalanguages to implement the abstract and concrete syntaxes, and to rely on general-purpose programming languages (GPLs) to implement the semantics (eg. a code generator) and to implement the tooling for the DSL (eg. CLI, validator).
There are many industry-grade language workbenches available:
-
MetaEdit+, a proprietary language workbench made by MetaCase. It allows the creation of graphical DSLs using dedicated view, forms, and metalanguages. It provides an advanced debugger to help with debugging a semantics implemented as a code generator.
Figure 4. MetaEdit+ language workbench (left) and resulting DSL (right) -
Xtext, developped by Itemis and TypeFox, and hosted by the Eclipse Foundation. It allows the creation of textual DSLs using the Ecore metalanguage for the abstract syntax, and Xtext Grammar Language for the concrete syntax. Many parts of an Xtext DSL are developped using pure Java, such as the validator, the code generator or the editor. The Eclipse IDE is recommended for implementing an Xtext DSL, since an Xtext DSL is packaged as an Eclipse plugin. An Xtext DSL can then be published as an Eclipse IDE extension, or as an LSP server that can be integrated in different IDEs.
Figure 5. An Xtext DSL running in the Eclipse IDE -
Sirius Web, developped by Obeo and hosted by the Eclipse Foundation. It allows the creation of web-based graphical DSLs using a fully web-based language workbench. It provides a Domain DSL to implement an abstract syntax, and a View DSL for the concrete syntax.
Figure 6. A Sirius Web DSL running in the browser -
Spoofax, an open-source language workbench by researchers of Delft University of Technology. It allows the creation of textual DSLs by fully using a wide range of of sophisticated metalanguages (SDF3, Statix, Stratego), and without using any GPL.
Figure 7. Spoofax language workbench -
Meta Programming System (MPS), an open-source language workbench by JetBrains. It allows the creation of hybrid textual-graphical DSLs using a using form of projectional concrete syntax. Other aspects of the DSL are fully implemented using a wide range of of provided metalanguages.
Figure 8. An MPS DSL running in MPS -
Langium, an open-source language workbench made by TypeFox and hosted by the Eclipse Foundation. It allows the creation of textual DSLs using both a metalanguage called Langium Grammar Language, and Typescript. A Langium DSL can be published as an LSP server embedded into a Visual Studio code extension.
Figure 9. Langium language workbench in Visual Studio Code
Each following chapter will dive into a specific component of a DSL (abstract syntax, concrete syntax, semantics) and will present an approach to implement said component using Langium.