Software Language Engineering (SLE): practice

1. Langium: a modern language engineering framework

langium logo w nib
Figure 1. Langium logo

Langium is an open-source language engineering framework developed by TypeFox and hosted by the Eclipse Foundation. It can be used to implement a software language in Typescript, which can then be deployed in a web environment or as an extension for the Visual Studio Code IDE.

Langium is the "spiritual successor" of an older language engineering framework called Xtext, which can be used to implement a software language in Java and to generate a powerful integration of this language in the Eclipse IDE.

2. Langium overview

The Langium framework is organized around three main components:

  • A meta-language, called the Langium grammar language, which can be used to create a Langium grammar. A Langium grammar defines both the abstract syntax and the concrete syntax of the software language being developed. The abstract syntax is written as a set of data types (with a notation similar to TypeScript), while the concrete syntax is defined in the form of a grammar rules (with a notation similar to EBNF).

  • A generator, which takes a Langium grammar as input and produces a set of Typescript language components as output (detailed below).

  • A Typescript library used by both the generator and the generated language components.

Using the Langium framework to create a new software language can be depicted as follows:

langium overview.drawio
Figure 2. Using the Langium framework to implement a software language
  1. At the top, the software language engineer uses the Langium grammar language to define both the abstract and concrete syntaxes of the developed language.

  2. The resulting Langium grammar is given to the Langium generator, which produces as output a wide array of artifacts:

    • A set of Typescript classes for all the concepts of the abstract syntax,

    • A parser, able to read a text file and to produce an AST as defined by the grammar rules,

    • A set of editing services which are essentially all the logic of a code editor for the language being developed (syntax highlighting, error highlighting, autocompletion, outline view, etc.).

    • A language API, which can be used as a single HTTP entrypoint to access all languages components. The services of this API follow the standardized Language Server Protocol (LSP).

    • A Visual Studio Code extension which can be used to easily deploy the software language implementation in any Visual Studio Code IDE. This extension is mostly an adapter that interacts with the language API to provide all services.

  3. Finally, some parts of the language implementation must be developed manually by the language engineer. This includes the implementation of the semantics, either in the form of a code generator (if implemented as a compiler) or in the form of an executor (if implemented as an interpreter).

3. Preparing a Langium project

Langium provides a Yeoman generator which can be used to rapidly quickstart a Langium project.

Yeoman can be installed on your system with npm with the following command:

npm install -g yo generator-langium
The -g option requires that you first configure your system to install npm packages in your home directory, as written in this page of the npm documentation.

Then to generate a new folder that will contain your new Langium project, run this command:

yo langium

This will ask you different questions about your Langium project, such as the name of the software language being developed, the file extension for this language, and which development environments should be supported (CLI, Visual Studio Code, web, etc.). For more information, refer to this page of the Langium documentation.

If you are unsure, you can always answer yes to all questions.

After generation of your base Langium project, you obtain a Langium project structure. This structure starts with the folder packages, which contains in the following subfolders:

  • language : Contains the main components of the software language, in particular:

    • langium-config.json : Contains metadata about your language, including the language name, the file extension, or the path to the .langium file.

    • src: Contains the .langium files defining the Langium grammar of the language, which itself defines the abstract syntax and the concrete syntax of the language (will be presented in the next section), with one subfolder:

      • generated: Contains files generated by the Langium generator from the .langium files, including the ast.ts file that contains the data types for all the language concepts.

    • test: Contains test cases for the language.

  • cli: contains a base implementation of a Command Line Interface (CLI) for the language, which by default only includes a base code generator that implements the semantics of the language (will be presented in a later section).

  • extension: Contains the code of the Visual Studio Code extension of the language.

The generated project also comes with a package.json file that defines the following important actions that can be performed using npm:

  • npm run langium:generate: Call the Langium Generator.

  • npm run build: Compiles all the Typescript code (_must be executed after any code change and after any call to the Langium Generator).

  • npm run test: Run all test cases (only works if you have generated the test folder, and if you have written test cases).

4. The Langium grammar language

A Langium grammar is written in the packages/language/src folder in the form of one or multiple .langium files written using the Langium grammar language. This language can be divided in two parts:

Quite confusingly, the Langium documentation uses the term Semantic Model to refer to the abstract syntax of the developed language. Be careful: while the word "semantic" is used, this has nothing to do with what we called earlier the semantics of the language!
The abstract and concrete syntaxes can either be defined in separate .langium files, or in the same .langium file.
This chapter does not cover all the many possibilities offered by the Langium grammar language. For more information, refer to the corresponding page in the official documentation.

4.1. Defining the abstract syntax using Declared Types

Defining a new concept of the abstract syntax is achieved using the interface keyword:

interface MyConcept {
    …
}
This is similar to a class in an object-oriented model or program, but without operations/methods.

A concept can contain properties, each with a name and a mandatory type using the : separator:

interface MyConcept {
    aPrimitiveProperty: string
    aContainmentProperty: OtherType
    aCrossReferenceProperty: @AnotherType
}

There are three main types of properties:

  • A primitive property is typed with one of the Langium primitive types: boolean, number, string, bigint or Date.

  • A containment property is typed with another existing concept of the abstract syntax. In an AST, when a node A is linked to a node B using a containment, the node B is said to be contained (or owned) by the node A. In an AST, a node can only be contained by a single container node at a time.

  • A cross-reference property is typed with another existing concept of the abstract syntax, using the character @ as prefix. In an AST, when a node A is linked to a node B using a cross-reference, the node B is said to be referenced (or known) by the node A.

A cross-reference from A to B is only possible if:

  • The concept B contains a primitive property called name and with type string. This is because the concrete syntax requires a way to encode how to identify another existing element.

  • An instance of B is always (transitively) contained in a container node of the instance A that possess this cross-reference. This means if B is contained deeper in a separate sub-tree of the AST, it won’t be accessible. This limitation can be solved by implementing a scope provider, see the Langium documentation on scoping.

Apart from the root node, each node of an AST must be contained in another node.

A property can also define a cardinality using either the optional suffix ? on the property identifier, or the array suffix [] on the type name.

interface MyConcept {
    anOptionalProperty?: string
    aArrayProperty: number[]
}
  • An optional property means that there can be zero or one value.

  • A array property means that there can be zero or multiple values.

It is of course possible to combine the different types of properties with the cardinality suffixes to define "optional containments", "arrays of primitives", "arrays of cross-references", etc.

Finally, a concept can be defined as an subtype of another existing concept using the extends keyword:

interface MyConcept extends OtherConcept {
    …
}

This gives the possibility to generalize different concepts by introducing a supertype, which is both useful to define a reference to multiple types, and to avoid redundant properties.

From UML class diagrams to Langium Declared Types

As mentioned above, Langium Declared Types are mostly similar to object-oriented classes. To make that explicit, the follow comparison can be made with UML class diagrams:

UML Langium

Primitive properties

Diagram
interface ConceptA {
    name: string
    id: number
}

Containment property

Diagram
interface ConceptA {
    b: ConceptB
}

interface ConceptB {}

Cross-reference property

Diagram
interface ConceptA {
    b: @ConceptB
}

interface ConceptB {}

Optional property

Diagram
interface ConceptA {
    b?: ConceptB
}

interface ConceptB {}

Array property

Diagram
interface ConceptA {
    b: ConceptB[]
}

interface ConceptB {}

Subtyping

Diagram
interface ConceptA {}

interface ConceptB extends ConceptA {}

Abstract types

Diagram

Unfortunately there are no abstract types in Langium:

interface ConceptA {}

Enum

Diagram

There are no enums in Langium, but we can use a union type:

type Color = 'Red' | 'Blue'
It is actually very common to design the abstract syntax of a software language using a UML class diagram, or some similar object-oriented modeling language (such as MOF or Ecore (from the EMF platform)). The class diagram that defines the abstract syntax of a software language is commonly called the metamodel of the language.
Exemple 1. Langium abstract syntax of a State Machines DSL

We consider once more the abstract syntax we designed earlier for the State Machines DSL:

Diagram

We would like to implement this DSL using Langium. We therefore start by implementing this abstract syntax using Langium Declared Types :

interface StateMachine {
    name: string
    commands: Command[]
    states: State[]
    events: Event[]
    initialState: @State
}

interface Command {
    name: string
}

interface State {
    name: string
    transitions: Transition[]
    on_entry: @Command[]
}

interface Event {
    name: string
}

interface Transition {
    trigger: @Event
    target: @State
}

4.2. Defining the concrete syntax

A Langium concrete syntax is defined in the form of a grammar and grammar rules. When a grammar is "executed", it transforms the text of an input program into a structured AST of the same program.

First, a grammar is declared with the keyword grammar:

grammar MyDSL

Another .langium file can be imported using the import keyword, without writing the .langium extension of said file:

import './otherfile' // this imports the file 'otherfile.langium'
This can for instance be used to import the abstract syntax defined in a separate file.

A grammar rule can be understood as a function that transforms a specific piece of text of the input program into a node of the output AST. This node is an instance of a concept of the abstract syntax (specified using the returns keyword).

Abstract syntax
interface MyConcept { … }
Concrete syntax
// Parser rule that produces an instance of 'MyConcept'
MyConceptRule returns MyConcept:
    …
;

// Terminal rule that recognizes a valid identifier, and outputs it as a string
terminal ID returns string:
    /[_a-zA-Z][\w_]*/
;

There are three sorts of grammar rules:

  • A parser rule defines a sequence of tokens that is possible at this location of the input program. From an abstract syntax point of view, executing a parser rule generally produces an instance of a concept.

  • A terminal rule is declared with the terminal keyword, and contains a regular expression. It defines a possible kind of lexical token that can be parsed using the grammar, such as an identifier, an arbitrary string, an integer, etc. From an abstract syntax point of view, executing a terminal rule produces the value of a primitive property.

  • A hidden terminal rule is declared with the is declared with the hidden terminal keywords, and also contains a regular expression. However, instead of defining lexical tokens, it defines which sequences of characters that must be be ignored during parsing. This is typically used to ignore whitespaces and comments.

Langium default terminal rules

By default, a Langium project comes with the following set of pre-defined terminal rules:

hidden terminal WS: /\s+/;
terminal ID: /[_a-zA-Z][\w_]*/;
terminal INT returns number: /[0-9]+/;
terminal STRING: /"(\\.|[^"\\])*"|'(\\.|[^'\\])*'/;

hidden terminal ML_COMMENT: /\/\*[\s\S]*?\*\//;
hidden terminal SL_COMMENT: /\/\/[^\n\r]*/;

Each of these terminal rules can be used to either match or ignore different sequences of characters:

  • WS is a hidden terminal rule that matches (and therefore ignore) whitespaces, including tabulations and newline characters. This means the grammar will ignore all whitespaces, which means any amounts of whitespaces are allowed between two tokens.

  • ID is a terminal rule that matches an identifier (examples: List, myVariable, _building5), and implicitly produces a _string_ primitive value. This rule is almost always used to parse the name attribute of a concept, and to define a cross-reference to an existing element.

  • INT is a terminal rule that matches an integer (examples: 15, 896, 0) and produces a number primitive value.

  • STRING is a terminal rule that matches a string enclosed either between quotations marks " or single quotations marks ' (examples: "Good morning", "hello", 'the car is red'), and implicitly produces a string primitive value. It allows whitespaces between characters.

Be careful not to mix ID and STRING! ID does not allow whitespaces and cannot contain quotation marks, while STRING can contain whitespaces and must be enclosed in quotation marks.
  • ML_COMMENT is a hidden terminal rule that matches (and therefore ignore) multi-line comments enclosed between /* and */.

  • SL_COMMENT is a hidden terminal rule that matches (and therefore ignore) single-line comments enclosed between // and a newline character.

You can customize the comments delimiters of your language by changing the ML_COMMENT and/or the SL_COMMENT rules. For example for Python-style comments (using #), you can rewrite SL_COMMENT this way:

hidden terminal SL_COMMENT: /#[^\n\r]*/;
Most of the time, we can simply reuse the default terminal rules generated in a new Langium project.

A token declared in a parser rule can be defined in two ways:

Abstract syntax
interface MyConcept {
    name: string
}
Concrete syntax
// Parser rule that produces an instance of 'MyConcept'
MyConceptRule returns MyConcept:
    'somekeyword' name=ID ';'
;
  • A keyword, written between single quotes, which means that this exact string should be found at this specific location in the program. For example, here both somekeyword and ; are keywords, and mean that MyConceptRule always starts with somekeyword and ends with ;.

  • A call to an existing grammar rule, which means that this other grammar rule should apply at this location of the program. In addition, calling another grammar rule requires assigning (=) the output of the called rule into a containment property of the output instance. For example here the rule ID is expected after somekeyword, and the output of the rule ID is stored in the property name of the output MyConcept instance.

By default Langium considers that there can be an arbitrary amount of whites spaces between two tokens.

It is possible to define repeated tokens in a parser rule:

Abstract syntax
interface MyConcept {
    name: string
    children: OtherConcept[]
}

interface OtherConcept {}
Concrete syntax
MyConceptRule returns MyConcept:
    'somekeyword' name=ID (children+=OtherConceptRule)* ';'
;

OtherConceptRule returns OtherConcept: … ;
  • A repetition of a group of tokens is declared with parentheses () followed by either a star \* or a plus `. This means that, at this location of the program, the enclosed tokens must be found _zero, one or multiple times_ for a `*`, and _one or multiple times_ for a `.

  • An assignment in a repetition must be written as +=, and must mandatorily target an containment array property of the returned concept. For instance here children is a containment array property, therefore we can have a repetition that adds multiple OtherConcept instances in the property using the OtherConceptRule.

It is possible to define optional tokens in a parser rule:

Abstract syntax
interface MyConcept {
    name: string
    isActive: boolean
}
Concrete syntax
MyConceptRule returns MyConcept:
    'somekeyword' (isActive?='active')? name=ID ';'
;
  • A optional group of tokens is declared with parentheses () followed by a question mark ?. This means that the enclosed tokens can be found either zero or one time at this location.

  • The presence of an optional token can be stored in a boolean property of the concept using the =? operator. For instance here isActive is a boolean property which is true if the keyword active is found after the keyword somekeyword, and false otherwise.

It is possible to define alternatives in a group of tokens:

Abstract syntax
interface MyConcept {
    name: string
    isActive: boolean
    others: OtherConcept[]
    anothers: AnotherConcept[]
}

interface OtherConcept {}

interface AnotherConcept {}
Concrete syntax
MyConceptRule returns MyConcept:
    'somekeyword' ( others+=OtherConceptRule | anothers+=AnotherConceptRule )* ';'
;

OtherConceptRule returns OtherConcept: … ;

AnotherConceptRule returns AnotherConcept: … ;
  • Inside any group declared with parenthese (), the pipe operator | can be used to declare mutually exclusive sequences of tokens. This can be combined with a repetition, meaning that each iteration of the repetition can have multiple possibilities. For example here the repetition will use either OtherConceptRule or AnotherConceptRule at each iteration, storing the result in others or anothers respectively.

It is possible to define a specific kind of token corresponding to a cross-reference of the abstract syntax:

Abstract syntax
interface MyConcept {
    name: string
    existingOther: @OtherConcept
}

interface OtherConcept {
    name: string
}
Concrete syntax
MyConceptRule returns MyConcept:
    'somekeyword' name=ID existingOther=[OtherConcept:ID] ';'
;

OtherConceptRule returns OtherConcept: …
    'otherkeyword' name=ID
;
  • A special cross-reference token can be declared with brackets with the following syntax <cross-reference>=[<concept>:<terminal rule>]:

    • <cross-reference> is the name of a cross-reference property

    • <concept> is the name of the concept (and not the parser rule!) referenced by the cross-reference property

    • <terminal rule> is the name of the terminal rule that is used to obtain the name attribute of the referenced concept.

For instance here the name of an OtherConcept is obtained using the ID terminal rule. Accordingly, the value of the existingOther cross-reference is an instance of OtherConcept whose name is written as an ID.

Writing parser rules can be a difficult task when writing a Langium grammar − and we only cover here a small subset of possibilities! If you want to learn more, the corresponding part in the Langium documentation is very complete.

Finally, each Langium grammar must have a unique entry parser rule, which is a grammar rule declared with the keyword entry:

entry MyConceptRule returns MyConcept: …  ;

An entry rule is the first rule executed when parsing a file with the grammar. It is comparable to the main function of a program.

Exemple 2. Langium concrete syntax of a State Machines DSL

We continue the State Machines DSL started previously, and for which we stored the abstract syntax in a file named statemachines-dsl.abstractsyntax.langium, with the following contents:

interface StateMachine {
    name: string
    commands: Command[]
    states: State[]
    events: Event[]
    initialState: @State
}

interface Command {
    name: string
}

interface State {
    name: string
    transitions: Transition[]
    on_entry: @Command[]
}

interface Event {
    name: string
}

interface Transition {
    trigger: @Event
    target: @State
}

We would like to be able to write a state machine program in the following way:

// Declares a StateMachine
statemachine threestates

// Declares the 'events' of the StateMachine
events 
    e1 e2 e3

// Declares the 'commands' of the StateMachine
commands
    c1 c2 c3

// Declares the 'initialState' of the StateMachine
initial S1

// Declares a State
state S1 

    // Declares the 'onEntry' of the State
    on_entry { c1 }

    // Declares the 'transitions' of the State
    e1 => S2

end

state S2
    on_entry { c2 }
    e2 => S3
end

state S3
    on_entry { c3 }
    e3 => S1
end

We therefore write the following grammar (which, as a reminder, relies on the concepts of the previously defined abstract syntax):

grammar StateMachinesDsl

import './statemachines-dsl.abstractsyntax'

entry StateMachineRule returns StateMachine:
    'statemachine' name=ID
    ('events' events+=EventRule+)?
    ('commands'    commands+=CommandRule+)?
    'initial' initialState=[State:ID]
    states+=StateRule*;

EventRule returns Event:
    name=ID;

CommandRule returns Command:
    name=ID;

StateRule returns State:
    'state' name=ID
        ('on_entry' '{' on_entry+=[Command:ID]+ '}')?
        transitions+=TransitionRule*
    'end';

TransitionRule returns Transition:
    trigger=[Event:ID] '=>' target=[State:ID];


hidden terminal WS: /\s+/;
terminal ID returns string: /[_a-zA-Z][\w_]*/;
terminal INT returns number: /[0-9]+/;
terminal STRING: /"(\\.|[^"\\])*"|'(\\.|[^'\\])*'/;

hidden terminal ML_COMMENT: /\/\*[\s\S]*?\*\//;
hidden terminal SL_COMMENT: /\/\/[^\n\r]*/;

Using the parser generated from this grammar, we can parse the example State Machines DSL program shown just before, which results in the following AST:

Diagram
Automatic inference of the abstract syntax

Langium offers the possibility to automatically infer the abstract syntax from the concrete syntax, which can reduce the amount of lines of code in a Langium grammar. For example, this Langium grammar file:

// Abtract syntax
interface ConceptA {
    name: string
}

// Concrete syntax
ConceptARule returns ConceptA:
    name=ID;
;

can be rewritten like this:

// Concrete syntax + automatic inference of the abstract syntax
ConceptA:
    name=ID;
;

However, because this method "hides" the underlying abstract syntax definition, we prefer in this chapter to always separate the abstract and concret syntaxes definitions.

Most Langium examples online rely on automatic inference.

5. The Langium generator

Once a complete Langium grammar has been defined in the packages/language/src folder in the form of one or multiple .langium files, it is time to call the Langium generator to automatically produce data types, an editor and a parser for our software language.

To run the Langium generator, and to compile the generated Typescript code:

npm run langium:generate && npm run build

This mainly automatically produces the following artifacts:

  • packages/language/src/generated/ast.ts: Typescript classes for all concepts of your abstract syntax, along with many utility functions and boilerplate code for an easy manipulation of your AST.

  • packages/language/src/generated/grammar.ts: a JSON representation of your grammar, which is then automatically used to configure the Langium parser.

  • packages/language/src/generated/module.ts: a Typescript module with metadata (such as the file extension) and pointers to the other language components.

You will never have to manually edit these files, as they are entirely derived from your Langium code.

6. Testing a generated editor of a Langium language

If the call to the Langium generator was successful, and if all the Typescript code compiles successfully, it is time to test the language!

Open the "Run and Debug" pane:

vscode rundebug menu

Observe that two run configurations have been prepared by Yeoman when you initialized your project:

vscode runconfs
  • "Run Extension" allows you to test your language by starting a second Visual Studio Code window in which your language in developpment will be installed as a Visual Studio Code extension.

  • "Attach to Language Server " allows you to debug your language implementation by attaching a TypeScript debugger to an already running extension of your language (requires doing Run Extension first).

When you use "Run Extension", a second Visual Studio Code window appears in an empty workspace. In this empty workspace, create a new file using the file extension of your Langium language (important). This will open the customized editor for your language:

vscode statemachines window
Figure 3. Working editor for the State Machines DSL

7. Implementing a code generator for a Langium language

Using the Langium grammar language and the Langium generator, we can efficiently obtain software components that implement both the abstract syntax and the concrete syntax of a software language. But how can we implement the semantics of a Langium-based software language?

We have seen earlier that the semantics of a software language can be implemented either as a code generator or as an AST interpreter. In this section we focus on the case of code generators. More specifically, we consider the case of a code generator whose target language is a GPL such as Java or C, or a markup language such as HTML or Markdown.

A code generator must accomplish two main types of tasks:

  • Visiting the AST of the program being compiled, which means going over all nodes of the AST. Programming this visit is achieved by the abstract syntax data types that are part of the language implementation (the ast.ts file in Langium).

  • Producing output code by constructing a string containing the code to be generated. The contents of this output string depends on the contents discovered during the visit of the AST. Constructing such string is commonly achieved using string templates.

7.1. Visiting the AST with Langium

Visiting an AST using Langium is achieved using the TypeScript types generated from the abstract syntax. These types are found in the generated ast.ts file, an can be used in an object oriented fashion.

For example, given the following abstract syntax in a Langium grammar:

interface Formular {
    name: string
    fields: MultipleChoiceField[]
}

interface MultipleChoiceField {
    name: string
    choices: Choice[]
    default: @Choice
}

interface Choice {
    name: string[]
}

Then we call the Langium generator to produce an ast.ts file containing all the abstract syntax data types.

The ast.ts file is generated in the language package of the Langium project, and can therefore be imported like so:

import type { Formular, MultipleChoiceField, Choice }
                            from 'formular-language';

where:

  • Formular, MultipleChoiceField, Choice are concepts of the abtract syntax that we need to import,

  • formular-language is the name of the package for the DSL called formular.

Going back to our visitor, this ast.ts file can therefore be used in the following way:

// Import the abstract syntax data types
import type { Formular, MultipleChoiceField, Choice }
                            from 'formular-language';

// Define a function to visit a 'Formular' node
function visitFormular(formular: Formular) {
    console.log("Visiting a Formular with name: " + formularASTRoot.name)

    for (let multipleChoiceField of formular.fields) {
        visitMultipleChoiceField(multipleChoiceField)
    }
}

// Define a function to visit a 'MultipleChoiceField' node
function visitMultipleChoiceField(multipleChoiceField: MultipleChoiceField) {
    console.log("Visiting a MultipleChoiceField with name: "
                                            + multipleChoiceField.name)
    console.log("and with default choice: "
                                + multipleChoiceField.default.ref!.name)

    for (let choice of multipleChoiceField.choices) {
        visitChoice(choice)
    }
}

// Define a function to visit a 'Choice' node
function visitChoice(choice : Choice) {
    console.log("Visiting a Choice with name: " + choice.name)
    console.log("and contained in MultipleChoiceField: " + choice.$container.name)
}

// Obtain the root of the AST produced by the parser, a 'Formular' node
const formularASTRoot : Formular = …

// Call visit function
visitFormular(formularASTRoot)
  • visitFormular is a function that visits a Formular node, and that then trigger the visit of all enclosed TextField nodes using the visitTextField function,

  • visitMultipleChoiceField is a function that visits a MultipleChoiceField node, that then trigger the visit of all enclosed Choice nodes using the visitChoice function,

    • in the expression multipleChoiceField.defaultChoice.ref!.name, the ref attribute is required to resolve a cross-reference property (here the defaultChoice cross-reference),

  • visitChoice is a function that visits a Choice node,

    • in the expression choice.$container.name the $container attribute is used to access the parent node (here the enclosing MultipleChoiceField)

While we achieve a similar goal, we are not strictly speaking following the visitor design pattern, because the code here is not object-oriented, and because it mixes the algorithm that perform the navigation between node, and the code that performs actions when visiting a node.

Then if we call this piece of code on the following AST:

Diagram

We obtain the following output:

Visiting a Formular with name: Fancy formular
Visiting a MultipleChoiceField with name: Choose color
and with default choice: red
Visiting a Choice with name: red
and contained in MultipleChoiceField: Choose color
Visiting a Choice with name: blue
and contained in MultipleChoiceField: Choose color
Visiting a MultipleChoiceField with name: Choose animal
and with default choice: cat
Visiting a Choice with name: cat
and contained in MultipleChoiceField: Choose animal
Visiting a Choice with name: dog
and contained in MultipleChoiceField: Choose animal
Exemple 3. AST visit for the State Machines DSL

We write the following sample visit code for the State Machines DSL:

import type { StateMachine, Command, Event, State, Transition }
                                from 'state-machines-language';

function visitStateMachine(stateMachine: StateMachine) {
    console.log("Visiting state machine: " + stateMachine.name)
    console.log("with initial state: " + stateMachine.initialState.ref!.name)

    for (let command of stateMachine.commands) {
        visitCommand(command)
    }

    for (let event of stateMachine.events) {
        visitEvent(event)
    }

    for (let state of stateMachine.states) {
        visitState(state)
    }

}
function visitCommand(command: Command) {
    console.log("Visiting command: " + command.name)
}

function visitEvent(event: Event) {
    console.log("Visiting event: " + event.name)
}

function visitState(state: State) {
    console.log("Visiting state: " + state.name)

    for (let transition of state.transitions) {
        visitTransition(transition)
    }
}

function visitTransition(transition: Transition) {
    console.log("Visiting transition contained in state: "
                                + transition.$container.name)
}

If we call visitStateMachine on the following AST (shown before):

Diagram

Then we obtain the following output:

Visiting state machine: threestates
with initial state: S1
Visiting command: c1
Visiting command: c2
Visiting command: c3
Visiting event: e1
Visiting event: e2
Visiting event: e3
Visiting state: s1
Visiting transition contained in state: s1
Visiting state: s2
Visiting transition contained in state: s2
Visiting state: s3
Visiting transition contained in state: s3

7.2. Producing output code using string templates

The goal when visiting an AST is to generate some output code. This requires producing a very large string using some form of string concatenation. Because string concatenation can be quite verbose and annoying to write (with lots of expressions like "first" + "second"), code generation is better achieved using string templates.

A string template is a string expression where it is possible to directly embed variables or function calls within a string. For example in TypeScript:

let name: string = "Georges"
let output: string = `Hello ${name}!`

console.log(output)

produces:

Hello Georges!

Here:

  • name is a variable,

  • output is a string we construct using a string template, which is declared using backticks. In the template we use the placeholder syntax ${…} to write a valid string expression − here we simply use the available name variable.

Another benefit of string templates is that they can be written on multiple lines:

let name: string = "Georges"
let city: string = "Nantes"

let output: string = `
    Hello ${name}
    from ${city}!
`

console.log(output)

produces:

Hello Georges
from Nantes!

We can put any sort of expression in a placeholder ${…}, including function calls, which enables code like this:

function generateHello(name: string): string {
    return `Hello ${name}`
}

function generateFrom(city: string): string {
    return `from ${city}`
}

let name: string = "Georges"
let city: string = "Nantes"

let output: string = `
    ${generateHello(name)}
    ${generateFrom(city)}!
`

console.log(output)

which produces:

Hello Georges
from Nantes!

Unfortunately, it is not possible to declare a for loop within a template in order to display a collection of values. However a good alternative is to combine on the map and join operators available in TypeScript. For example:

type Person = {
    name: string
    city: string
}

function generateHelloFromPerson(person: Person): string {
    return `
        Hello ${person.name}
        from ${person.city}!
    `
}

let manyPersons: Person[] = [
    { name: "Georges", city: "Nantes" },
    { name: "Julia", city: "Bordeaux" }
]

let output: string = `
    Let's welcome everyone:

    ${manyPersons.map(person => generateHelloFromPerson(person)).join("\n\n")}
    `

console.log(output)

which produces:

    Let's welcome everyone:

        Hello Georges
        from Nantes!

        Hello Julia
        from Bordeaux!
Exemple 4. Code generator using string templates and visit functions for the State Machines DSL

Using string templates and visit functions, we would like to implement a code generator for the State Machines DSL. The idea is to be able to generate, from a valid state machine, a Java class which roughly follows the state design pattern.

For example the following state machine:

// Declares a StateMachine
statemachine threestates

// Declares the 'events' of the StateMachine
events 
    e1 e2 e3

// Declares the 'commands' of the StateMachine
commands
    c1 c2 c3

// Declares the 'initialState' of the StateMachine
initial S1

// Declares a State
state S1 

    // Declares the 'onEntry' of the State
    on_entry { c1 }

    // Declares the 'transitions' of the State
    e1 => S2

end

state S2
    on_entry { c2 }
    e2 => S3
end

state S3
    on_entry { c3 }
    e3 => S1
end

Which can be parsed in the following AST:

Diagram

Should result in the following Java code:

package threestates;

public class threestates {

    State currentState = new S1();

    public static void c1() {
        System.out.println("Executing command c1");
        // TODO command provide implementation
    }

    public static void c2() {
        System.out.println("Executing command c2");
        // TODO command provide implementation
    }

    public static void c3() {
        System.out.println("Executing command c3");
        // TODO command provide implementation
    }

    public void processEvent(Event event) {
        System.out.println("Processing event " + event);
        switch (event) {

            case e1:
                this.currentState = this.currentState.e1();
                break;
            case e2:
                this.currentState = this.currentState.e2();
                break;
            case e3:
                this.currentState = this.currentState.e3();
                break;
        }
        System.out.println("Current state: " + 
                        this.currentState.getClass().getSimpleName());

        if (currentState != null) {
            currentState.entryCommands();
        }
    }

    public enum Event {
        e1, e2, e3
    }

    interface State {
        State e1();
        State e2();
        State e3();
        void entryCommands();
    }

    class S1 implements State {

        @Override
        public State e1() {
            return new S2();
        }

        @Override
        public State e2() {
            return this;
        }

        @Override
        public State e3() {
            return this;
        }

        @Override
        public void entryCommands() {
            threestates.c1();
        }

    }

    class S2 implements State {

        @Override
        public State e1() {
            return this;
        }

        @Override
        public State e2() {
            return new S3();
        }

        @Override
        public State e3() {
            return this;
        }

        @Override
        public void entryCommands() {
            threestates.c2();
        }

    }

    class S3 implements State {

        @Override
        public State e1() {
            return this;
        }

        @Override
        public State e2() {
            return this;
        }

        @Override
        public State e3() {
            return new S1();
        }

        @Override
        public void entryCommands() {
            threestates.c3();
        }
    }
}

We can achieve this result with the following code generator:

import type { Command, Event, State, Transition, StateMachine } from 'state-machines-language';

function generateStateMachineCode(stateMachine: StateMachine): string {
    return `
    package ${stateMachine.name};

    public class ${stateMachine.name} {
    
        State currentState = new ${stateMachine.initialState.ref?.name}();

        ${stateMachine.commands.map(command => 
                        generateCommandDeclarationCode(command)).join("\n")}   

        public void processEvent(Event event) {
            System.out.println("Processing event " + event);
            switch (event) {
                ${stateMachine.events.map(event => 
                        generateStateMachineEventHandler(event)).join("\n")}
            }
            System.out.println("Current state: " + 
                            this.currentState.getClass().getSimpleName());

            if (currentState != null) {
                currentState.entryCommands();
            }
        }

        public enum Event {
            ${stateMachine.events.map(event => event.name).join(",")}
        }

       
        interface State {
            ${stateMachine.events.map(event => 
                        generateEventDeclarationCode(event)).join("\n")}
            void entryCommands();
        }

        ${stateMachine.states.map(state => generateStateCode(state)).join("\n")}

    }
    `

}

function generateStateCode(state: State): string {
    return `

        class ${state.name} implements State {

            ${state.$container.events.map(event => 
                        generateStateEventHandlerCode(state, event)).join("\n")}

            @Override
            public void entryCommands() {
                ${state.on_entry.map(commandRef => 
                        generateCommandCall(commandRef.ref!)).join("\n")}
            }
            
        }
    `
}

function generateCommandDeclarationCode(command: Command): string {
    return `
        public static void ${command.name}() {
            System.out.println("Executing command ${command.name}");
            //TODO command provide implementation
        }
    `
}

function generateEventDeclarationCode(event: Event): string {
    return `
        State ${event.name}();
    `
}

function generateStateEventHandlerCode(state: State, event: Event): string {
    return `
        @Override
        public State ${event.name}() {
            return ${generateEventHandlerReturn(state, event)};
        }
    `
}

function generateEventHandlerReturn(state: State, event: Event) {
    let targetState : State = findTargetState(state, event);
    if (targetState === state){
        return "this"
    } else {
        return `new ${state.name}()`
    }
}

function findTargetState(state: State, event: Event): State {
    const candidateTransitions: Transition[] = 
        state.transitions.filter(transition => transition.trigger.ref! === event)
    if (candidateTransitions.length > 1) {
        throw "Generation error: cannot process a non deterministic state machine"
    } else if (candidateTransitions.length == 0) {
        return state
    } else {
        return candidateTransitions.pop()?.target.ref!
    }
}


function generateStateMachineEventHandler(event: Event): string {
    return `
        case ${event.name}:
            this.currentState = this.currentState.${event.name}();
            break;
    `
}

function generateCommandCall(command: Command): string {
    return `
        ${command.$container.name}.${command.name}();
    `
}

7.3. Integrating the code generator in the CLI

In this section we assume that, when your Langium project was generated using Yeoman, you chose to not generate an example project.

Once the code generator code has been implemented for a given Langium language, the last step is to provide a user interface to make the code generator available to language users. In Langium by default this is achieved by implementing a generate command in the Command Line Interface (CLI) of the language.

A Langium project freshly created using Yeoman should have the following files:

  • packages/cli/src/main.ts contains the entry point of the CLI, and should include a generateAction function. This function is called when the generate keyword is given to the CLI, and it accomplishes two important tasks:

    • extractAstNode is the function that will call your parser in order to produce an AST, that is then stored in the model varible,

    • generateOutput is the main function of your code generator provided in the packages/cli/src/generator.ts file.

  • packages/cli/src/generator.ts contains the code generator itself, and in particular a main function called generateOutput.

In order to integrate your code generator in this architecture, put your code generator in the packages/cli/src/generator.ts file, and call your code generation functions from the generateOutput function. Change the generateOutput as much as required to create folders as needed by your code generation logic.

When npm run build is executed, this produces the ./packages/cli/bin/cli.js containing the CLI, which can be tested by simply calling:

node ./packages/cli/bin/cli.js generate <path to program to compile>
You can also make your CLI available globally in your system by running the npm link command in your project folder. This should create a command <language name>-cli available in your console, which is a symbolic link to the ./packages/cli/bin/cli.js file of your language.

The generate command, by default, will put produced files in the generated folder.

Exemple 5. Integration of the State Machines DSL code generator in the CLI

For the State Machines DSL, we start in the packages/cli/src/generator.ts, and we look at the generateOutput function. Then we modify this function to properly create the Java package folder required for the generated Java class, and to prepare the path to the generated Java file, before calling the code generator:

export function generateOutput(model: StateMachine, source: string, destination: string): string {
    // Prepare generation paths
    const data = extractDestinationAndName(destination);
    const generatedJavaPackageName = model.name
    const generatedJavaFileName = `${model.name}.java`
    const generatedFolderPath = path.join(data.destination,
                                            generatedJavaPackageName);
    const generatedFilePath = path.join(generatedFolderPath,
                                            generatedJavaFileName)

    // Call the code generator
    const fileContents = generateStateMachineCode(model)

    // Write the generated code in a file
    fs.mkdirSync(generatedFolderPath, { recursive: true });
    fs.writeFileSync(generatedFilePath, fileContents);
    return generatedFilePath;
}

We then run npm run build and run our code generator with the CLI :

node ./packages/cli/bin/cli.js generate examples/threestates.sm

which produces the file generated/threestates/threestates.java.

8. References