Software Language Engineering (SLE): theory

The following two chapters give an introduction the field of Software Language Engineering (SLE), a set of of software engineering methods and techniques to efficiently create and maintain new software languages.

This first chapter includes an introduction explaining why this is an important advanced software engineering topic, then introduces key theoritical notions regarding the design and implementation of a software language.

The second chapter will later present an industry-grade SLE framework called Langium.

1. Introduction

1.1. Abstractions: the heart and soul of software engineering

While initially software was written in the form of binary code or assembly code that computers could directly understood, it became rapidly necessary to write software using reusable abstractions (such as loops, conditionals, functions) that generalized common redundant patterns found in code. Not only these abstractions were a way to avoid redundancies, but they also started providing higher-level concepts and ideas adapted to the mind of software engineers, and to the problems they needed to solve. This eventually led to the birth of general-purpose programming languages (GPLs) in their modern form, such as Pascal, Fortran, C, Java, and many others.

In this chapter, we understand the word abstraction as the act of identifying common characteristics of different entities, and defining these characteristics as proper self-reliant concepts. Or, as written on the Wiktionary:

The act of comparing commonality between distinct objects and organizing using those similarities; the act of generalizing characteristics; the product of said generalization.
— 6th definition of the word "abstraction"
Wiktionary (accessed on 16/11/2024)

While GPLs are key abstractions used to build software, the whole field of software engineering nowadays continuously relies on the definition of suitable and efficient new abstractions to solve always more complex problems. This abstraction-centered engineering process can be summarized as follows:

abstraction engineering.drawio
Figure 1. Engineering abstractions and using abstractions to solve problems

  • On the right, a given set of problems is depicted. The size of a set of problems can greatly vary, ranging from "writing data structures and algorithms to be executed on computers" (the very large set of problems addressed by GPLs), to "writing Java test cases" (the smaller set of problems addressed by the JUnit framework), or to something even more domain-specific such as "exchanging instant messages over the network" (problems addressed by the Matrix communication protocol).

  • On the left, the abstraction engineer is a software engineer that analyzes a given set of problems, and produces a set of reusable abstractions as a result (such as a software language, library, framework, or protocol). These abstractions are made of meticulously identified concepts fitting to solve a given set of problems. Consequently, engineering abstractions requires a very good understanding of the domain of expertise related to the set of problems.

  • At the bottom, the abstraction user is a software engineer that take advantage of existing abstractions in order to build a solution (typically a software model or a program) to an specific problem. For the abstractions to be useful, the problem at stake must be part of the set of problems targeted by the abstractions.

In 99% of cases, defining new abstractions requires pre-existing abstractions! For example, implementing a new software library is achieved using an existing software language and using other existing libraries (called dependencies). Consequently, the abstraction engineer is always also an abstraction user.

Abstractions can take many forms in the great wild of software engineering, such as:

  • Operating systems (OSs) and device drivers provide abstractions to manage hardware resources. For instance, a software application can use the file system abstraction provided by an OS to interact with a storage device.

  • Communication protocols provide abstractions allowing two or more entities to exchange information. For instance, the HTTP protocol provide abstractions (GET, POST, PUT, etc.). to interact with a web server.

  • An Application programming interface (API) provides a set of abstractions to programmatically interact with an existing piece of software (a server, a library, an application, etc.). This allows such piece of software to be used by other code without exposing any of its internals.

  • A software library is a collection of abstractions in the form of reusable data structures and functions targeting a particular computing problem, and that can serve to create new software. A software framework is a specific sort of library where parts of the library are extensible and can be redefined for a specific use.

  • Last but not least, software languages (which include both programming languages and modeling languages) are fundamental abstractions used to design and build software. We focus on this unique category of abstractions in the following section.

1.2. Software languages as powerful (but costly) abstractions

Software languages can come in many different shapes and forms:

  • General-purpose programming languages (GPLs) (Python, Java, C++, etc.) are the best known software languages. They provide all commonly expected abstractions to write algorithms and data structures. They are used to write programs for the vast majority of software today. Being general-purpose, these languages are often quite large and complex, and therefore require a lot of engineering effort for their creation and maintenance. Their lifespan is usually very long, and their user base very large.

  • A modeling language (UML, SysML, AADL, etc.) is a software language commonly used to design software and systems. They often provide high-level abstractions specialized for specific concerns or processes (for instance AADL focuses on architecture design). While some modeling languages have precise execution semantics (for example Petri nets and Statecharts) and can be executed as programming languages, other modeling languages do not have semantics precise enough to be executable (for example UML). Their notation is often based on graphical shapes, arrows, and diagrams, making them easy to use on drawing boards.

  • A domain-specific languages (DSLs) is a programming or modeling language of smaller size designed for a much smaller set of problems. The problems that are covered by a given DSL − also called the domain covered by the DSL − can be related both to business concerns (salary calculation, robotics, hospital management, etc.) and more cross-cutting technical concerns (security policies, concurrent behaviors, user interfaces). A DSL is typically created for a limited set of users of for a specific engineering needs (for example the needs of a specific branch of a specific company), with a shorter lifespan and maintenance effort than a GPL.

The line between a software language of "normal size" and a DSL is not strict at all, and it can be sometimes difficult to decide whether a given software language qualifies as a DSL or not. For example, some considers HTML to be a DSL because it focuses on a very specific and limited technical concern (which is defining the contents of a graphical user interface), while some consider it is too broadly used and has too many applications to qualify as a DSL.

Compared with other forms of abstractions (such as software librairies), software languages have a characteristic making them unique: a software language defines a fully dedicated syntax for all the abstractions it provides. This syntax can be either textual (such as Java) or graphical (such as UML), and can be extremely powerful in how it allows an engineer to directly manipulate the provided abstractions.

On the opposite, using the abstractions provided by a software library requires being skilled with the programming language the library was made for.

We presented in the previous section the always growing need to define new abstractions to tackle the always growing complexity of problems to solve. While it remains more common today to create new abstractions in the form of software libraries, there is a rising trend to rely on the engineering of new software languages − especially DSLs − to answer the need for abstractions. This is especially true when non-software experts are involved in the engineering process, and can therefore remarkably benefit from a dedicated DSL syntax both tailored for their domain of expertise, and simpler to learn than GPLs. For example, a DSL to configure the behavior of specific physical machines can be designed to be used directly by the experts of said machines, without the contant help of a software engineer.

However, it can be observed with little surprise that implementing a new software language, with its own interpreter/compiler and tools, can be a costly endeavor. The field of Software Language Engineering (SLE) emerged for this purpose: providing dedicated methods, principles, approaches and tools to efficient engineer new software languages with limited cost.

Model-Driven Engineering (MDE) and Low-code development

Different successful methods and tools rely nowadays on SLE, such as:

  • Model-driven engineering (MDE) is a method that heavily relies on models (and hence modeling languages) to design the different facets of an system, and then to process models as first-class artifacts to automatically produce simulations or generate code. While MDE can rely on well-established modeling languages, such as UML or Simulink, a great part of MDE relies on the engineering of DSLs suitable for the different facets of the system.

  • Low-code development relies on simplified development environments called Low-code development platforms (LCDP) that can be used by non-software experts to write software components. An LCDP is built around a set of well-defined DSLs easy to use and understand.

A major result in the field of SLE comes in the form of frameworks and IDEs called language workbenches, which aim to cover all aspects of engineering new software languages. We present one of such language workbenches, Langium, in the next chapter.

2. Principles of Software Language Engineering (SLE)

2.1. Anatomy of a software language

A software language can generally be designed with the following ingredients:

software language specification.drawio
Figure 2. General representation of a software language and its use

  • At the center, the abstract syntax of a software language defines the concepts of the language and the relations between said concepts. These concepts are the abstractions provided by the software language. A common way to design an abstract syntax is to rely on object-oriented modeling, for example using a simplified UML class diagram (also called a metamodel in a language engineering context) without operations.

    • Below the abstract syntax, an example of abstract syntax tree (AST) is depicted. It is composed of instances of the concepts of the abstract syntax. An AST is the "pure" representation of a program or model which can be programmatically processed by the semantics.

  • On the left, the concrete syntax of a software language defines the notation of concepts defined in the abstract syntax. Notations can be either textual (as most programming languages) or graphical (as most modeling languages). Notation is commonly expressed as rules telling with words or shapes correspond to each concept.

    • Below the concrete syntax, an example of concrete representation of a program/model is depicted. Here since the concrete syntax is graphical, it is composed of shapes and arrows allowed by the graphical syntax. Using the concrete syntax rules, a concrete representation can be parsed to obtain the corresponding AST.

  • On the right, the semantics of the software language defines the meaning of each concept defined in the abstract syntax. The meaning of a concept can be expressed in many ways, informally (with natural language and examples) or formally (with mathematical notations), and should somehow explain what happens when a concept is encountered during an execution of the AST.

    • Below the semantics, the result of the execution of AST is depicted. This result is generally a sequence of execution steps with a computational effect.

Exemple 1. Java language design (excerpt)

The official Java Language specification architectures the Java language in the following way:

software language specification java.drawio

  • In the center, the abstract syntax is partially shown with three concepts: ClassDeclaration represents the declaration of a class, FieldDeclaration the declaration of a typed field (also called attribute) within a class, and MethodDeclaration the declaration of a method (along with type and parameters) within a class.

  • On the left, the concrete syntax must describe how each concept should be represented textually. A ClassDeclaration is written using the class keyword, preceded with a modifier (public, private, etc), and followed with a name and, optionally, a superclass with the extends keyword.

  • On the right, the semantics must explain what each of the concepts mean. Here we show direct quotes from the official Java specification.

Exemple 2. Designing a state machines DSL

We would like to start creating a DSL to represent a specific sort of state machines. We would like state machines with the following characteristics:

  • A state machine has a name, and contains the following:

    • A set of commands, each with a name, representing external actions that can be invoked by the state machine.

    • A set of events, each with a name, representing external events that are received by the state machine, and that may trigger transitions.

    • A set of states, each with a name and containing the following:

      • A set of commands that are invoked when we enter the state. These commands should first be declared at the root of the state machine.

      • A set of transitions, each with a name and containing the following:

        • an event that triggers the transition when received while we are in the state

        • a target state that is reached when the transition is fired. This state should be already part of the state machine.

We design an abstract syntax using a UML class diagram (also called a metamodel):

Diagram

And we architecture our DSL in the following way (with only excerpts of each of the three constituents):

software language specification statemachines.drawio

  • At the left, a small excerpt of the concrete syntax defines that a StateMachine is represented by a drawing area, in which States are represented as rounded boxes with a name, and Transitions as solid arrows with an event name as label.

  • At the right, a small excerpt of semantics defines that executing a state machine means changing a current state from one state to another based on which transitions are triggered by events.

Later we will see a more detailed definition of the constituents of this DSL, and with an alternate textual concrete syntax instead of a graphical concrete syntax.

2.2. Implementing a software Language

Once a software language has been designed with the three ingredients we have seen in the previous section (abstract syntax, concrete syntax, semantics), the next step is to implement the software language… as software! We will distinguish:

  • the implementation of the core components of a software language, which are the minimal parts required to use the language,

  • the implementation of additional tools and of IDE integrations for a software language, to make the language convenient and productive to use for language users.

2.2.1. Core components of a software language implementation

Each of the three parts of a language design must be implemented as a language component, as depicted in the following figure:

software language implementation core.drawio
Figure 3. Core components of a software language implementation

  • The concrete syntax must be implemented in the form of a parser, which must be able to take as input a concrete representation of a program of the language, and produce at output an AST of the same program. This requires being able to identify (textual or graphical) patterns in the concrete representation.

    While the word parser is mostly used for textual languages, here we also use this word for graphical languages. A parser for a graphical language is of course very different from a textual language parser because it must somehow recognize shapes, arrows and drawings instead of traditional text.

  • The abstract syntax must be implemented in the form of a a set of data types, typically classes in an object-oriented context. The parser must be able to instantiate these data types in order to create the nodes of an AST of a program.

  • The semantics can typically be implemented in two different ways:

    • A code generator takes an AST as input and produces executable code as output. This output code is written using a separate target language that was already implemented. A target language can be either a bytecode language (such as Java bytecode), or an assembly language (such as x86 assembly), or even another programming language (such as Java, C, Rust).

    • An AST interpreter takes an AST as input and directly performs a computation based on the AST contents. This computation is typically a sequence of execution steps that changes the current state of the program.

A language implementation whose semantics is implemented with a code generator is commonly called a compiler, while a language implementation whose semantics is implemented with an AST interpreter is commonly called an interpreter.

  • GPLs are nowadays always implemented as compilers that rely either on machine languages (such as x86 or ARM assembly) or bytecode languages (such as Java Bytecode) as targets for code generation − even Python, contrary to popular knowledge!

  • Bytecode languages are always implemented as special kinds of interpreters called process virtual machines (for exemple the Java Virtual Machine (JVM) is the interpreter for Java Bytecode).

  • DSLs, on the other hand, can sometimes be implemented as interpreters when performance is not a key issue, or as compilers when targeting a specific execution environment and/or when performance is a concern.
Just-in-time compilation is a hybrid form of semantics implementation that combines both interpretation and compilation.

2.2.2. Additional tools and IDE integration

Once the core components of a software language have been implemented (which means a complete interpreter or compiler) we are now technically able to use the language by writing and executing programs. However, common tools are still missing to make it properly usable by the language user, such as:

  • An editor for the language, both to help reading a program (with syntax highlighting, an outline view, navigable symbols, etc) and writing a program (with code completion, smart renaming of elements, refactoring actions, etc.). We call editing services the set of services offered by an editor for a given software language.

  • A validator for the language (also called linter or static analyzer) for the language, able to analyze an input program and to detect structural problems as early as possible. A validator is typically implemented as a set of validation rules.

  • A debugger for the language, allowing the language user to execute a program step by step and to visualize the changes in the program state.

Moreover, a proper development environment is important to be able to use a software language efficiently, including the aforementioned tools. This can be achieved through an integration into existing Integrated Development Environments (IDEs) such as Eclipse, IntelliJ or Visual Studio Code.

With all these new concerns in mind, the score of a software language implementation can be extended in the following way:

software language implementation tools.drawio
Figure 4. Additional tools and IDE integration for a software language implementation

  • We can observe that most tools are dependent on the abstract syntax data types of the language implementation. In other words, most tools can be written without any dependency on the concrete syntax of the language.

  • At the bottom, two different ways to integrate the language with an IDE are depicted:

    • On the left, we see a plugin developed for "IDE 1" which directly uses the language implementation, whether by directly using the language implementation code, or by using the command-line interface (CLI) of the language components and tools.

    • On the right, we see plugin developed for "IDE 2" which relies on languages protocols to interact with the language implementation. A language protocol is a novel way to communicate with a language implementation in a client-server fashion, therefore decoupling as much as possible a language implementation from an IDE. This approach requires that the language implementation provides a language API, typically as a JSON-RPC web server. Two language protocols are commonly in use today: the Language Server Protocol (LSP) for editing services, and the Debug Adapter Protocol (DAP) for debugging services.

3. References