Build Automation

Gerson Sunyé gerson.sunye@univ-nantes.fr

Agenda

Introduction
Software Artifacts
Build Tools
Maven
GitLab CI
Conclusion

Definition

Build automation is the process of automating the creation of a software build and the associated processes including: compiling computer source code into binary code, packaging binary code, and running automated tests.

— Wikipedia

Build Automation Goals

Eliminate redundant tasks
Reduce dependencies on key personnel
Log the history of builds and releases in order to investigate issue
Improve product quality
Accelerate the build process
Reduce build issues.

Motivation

Build is just compilation…

Motivation (cont.)

and packaging …

Motivation (cont.)

and testing …

Motivation (cont.)

and integrating …

Motivation (cont.)

and evaluating the tests …

Motivation (cont.)

and generating some code …

Motivation (cont.)

and generating the documentation …

Motivation (cont.)

and deploying …

Motivation (cont.)

and also:

Processing resources
Keeping a history of builds
Source code analysis
Managing release versions
and several other little things!

Agenda

Introduction
Software Artifacts
Build Tools
Maven
GitLab CI
Conclusion

Understanding Software Artifacts

In software construction, the term software artifact denotes a software that has been documented and stored in a repository so it can be retrieved upon demand.

Artifact Dependencies

Figure 1. Software Artifact

Figure 2. Software Artifact Example

Dependency Scope

Dependency Scope (Cont.)

Different Scopes:

Runtime: The dependency is needed during compilation, testing and execution
Testing: The dependency is only needed during testing
Compile: The dependency is only needed during compilation
Provided: The dependency is needed during compilation and testing, but should not be embedded in the package: it will be provided by the execution environment

Dependency Versions

Version Ranges

Range	Meaning
(,1.0]	x ⇐ 1.0
1.0	It generally means 1.0 or a later version, if 1.0 is not available. Various Maven plug-ins may interpret this differently, so it is safer to use one of the other, more specific options.
[1.0]	Exactly 1.0
[1.2,1.3]	1.2 ⇐ x ⇐ 1.3

Range

Meaning

(,1.0]

x ⇐ 1.0

1.0

It generally means 1.0 or a later version, if 1.0 is not available. Various Maven plug-ins may interpret this differently, so it is safer to use one of the other, more specific options.

[1.0]

Exactly 1.0

[1.2,1.3]

1.2 ⇐ x ⇐ 1.3

Version Ranges (Cont.)

Range	Meaning
[1.0,2.0)	1.0 ⇐ x < 2.0
[1.5,)	x >= 1.5
(,1.0],[1.2,)	x ⇐ 1.0 or x >= 1.2. Multiple sets are separated by a comma.
(,1.1),(1.1,)	This excludes 1.1 if it is known not to work in combination with the library.

Range

Meaning

[1.0,2.0)

1.0 ⇐ x < 2.0

[1.5,)

x >= 1.5

(,1.0],[1.2,)

x ⇐ 1.0 or x >= 1.2. Multiple sets are separated by a comma.

(,1.1),(1.1,)

This excludes 1.1 if it is known not to work in combination with the library.

Agenda

Introduction
Software Artifacts
Build Tools
Maven
GitLab CI
Conclusion

Apache Maven

Java Build and Dependency resolution tool from Apache Foundation
Introduced several standards: project layout, artifact repository organization, build lifecycle, etc.
Most used Java build tool

Gradle

Open-source build tool, based on Maven standards.
Advantages over Maven:
- Flexibility
- Performance
- Better user experience (DSL vs. XML)

GitLab CI

Open-source, Continuous Integration and Continuous Delivery tool combined with GitLab.
Simple fast, scalable, and multi-language.
Supports Docker

SonarQube

Open-source platform used to measure and analyze the source code quality
Can analyze and manage code of more than 20 programming languages, including Java, c/c++, PL/SQL, and Cobol
Enables Continuous Code Quality Management

https://sonarcloud.io

Agenda

Introduction
Dependency Resolution
Software Artifacts
Build Tools
Maven
GitLab CI
Conclusion

Foreword

Maven was born of the practical desire to make several projects at the Apache Software Foundation(ASF) work in the same, predictable way.

Before, every project at the ASF had a different approach to compilation, distribution, and web site generation.

Ex. Build process for Tomcat was different from build process used by Struts. Turbine developers used a different approach to site generation than the developers of Jakarta, Commons, etc.

This lack of common approach to building software meant that every new project tended to copy and paste another project’s build system.

The barrier to entry for a project with a difficult build system was extremely high. Difficult to attract developer because it could take an hour to configure everything in just the right way.

Apache Maven

Maven is a:

Project management framework.
Build tool.
Set of standards.
Artifact repository.
Standard build life cycle.

Main Principles

Convention over configuration

Reuse of build logic

Conventions

Standard directory layout: sources, resources, config files, generated output, documentation, web site, etc.
A single Maven project produces a single output.
Standard naming conventions.

Features

A comprehensive software project model (POM) and tools that interact with this declarative model.
Allows developers to declare goals and dependencies and rely on default structure and plugin capabilities.

Benefits

Coherence: Maven allows organizations to standardize on a set of best practices.
Reusability (reusing the best practices).

Benefits

Agility: lowers the barrier to reuse not only of build logic but of components.
- It is easier to create a component and integrate it to a multi-project build.
- Easier for developers to jump between projects without the a steep learning curve.
Maintainability: stop building the build and focus on the application.

The Project Object Model

<project>
 <modelVersion>4.0.0</modelVersion>
  <groupId>com.mycompany.app</groupId>
  <artifactId>javaero</artifactId>
  <packaging>jar</packaging>
  <version>1.0-SNAPSHOT</version>
  <name>Javaero</name>
  <url>http://maven.apache.org</url>
  <description></description>
 <dependencies>
     <dependency>
       <groupId>junit</groupId>
       <artifactId>junit</artifactId>
       <version>4.12</version>
       <scope>test</scope>
     </dependency>
  </dependencies>
</project>

This pom will allow you to compile, test and generate basic documentation.

Build lifecycle

A series of phases where each phase can perform one or more actions, or goals, related to that phase.
For instance, the compile phase invokes a certain set of goals to compile set of classes.

Build lifecycle

If you tell Maven to compile, the validate, initialize, generate-sources, process-sources, generate-resources, and compile phases will execute.

Build lifecycle

Standard build life cycle consists of many phases and these can be thought of as extension points.
When you need to add some functionality to the build lifecycle you do so with the plugin.

Build lifecycle

Maven Plugins provide reusable build logic that can be slotted into the standard build life cycle.
Any time you need to to customize the way your projects builds you either employ the use of an existing plugin or create custom plugin for the task at hand.

Default build lifecycle

Default build lifecycle (simplified)

validate - validate the project is correct and all necessary information is available
compile - compile the source code of the project
test - test the compiled source code using a suitable unit testing framework. These tests should not require the code be packaged or deployed.
package - take the compiled code and package it in its distributable format, such as a JAR.

Build lifecycle (cont.)

integration-test - process and deploy the package if necessary into an environment where integration tests can be run
verify - run any checks to verify the package is valid and meets quality criteria
install - install the package into the local repository, for use as a dependency in other projects locally
deploy - done in an integration or release environment, copies the final package to the remote repository for sharing with other developers and projects.

Dependencies

A dependency is a reference to a specific artifact that resides in a repository.
In order for Maven to attempt to satisfy a dependency, Maven needs to know what the repository to look in as well as the dependency’s coordinates.
A dependency is uniquely identified by the following identifiers: groupId, artifactId and version.

Dependencies are declaratives

In the POM you are not telling Maven where the dependencies physically, you are simply telling Maven what a specific project expects.

Dependencies

When a dependency is declared, Maven tries to satisfy that dependency by looking in all of the remote repositories that are available, within the context of your project, for artifacts that match the dependency request.
If matching artifact is located, it transports it from remote repository to your local for general use.

Dependency Declaration

<project>
(...)
<dependencies>
     <dependency>
       <groupId>junit</groupId>
       <artifactId>junit</artifactId>
       <version>3.8.1</version>
       <scope>test</scope>
     </dependency>
  </dependencies>
(...)
</project>

Repositories

Types of repositories:

Local: ~/.m2/repository
- you must have a local repository in order for Maven to work.
Remote

General pattern for the repository layout

If the groupId is a fully qualified domain name such as z.y.x then you would end up with a directory structure like the following:

Locating Dependency Artifacts

Maven will attempt to find the artifact with groupId, artifactId and version in local repository -> ~/.m2/repository/…
If this file is not present, it will be fetched from remote repository.
By default, Maven will fetch an artifact from the central Maven repository at https://repo.maven.apache.org/
If your project’s POM contains more than one remote repository, Maven will attempt to download an artifact from each repository in the order defined in your POM.
Once dependency is satisfied, the artifact is downloaded and installed in your local repository.

Getting Started with Maven

Preparing to use Maven:

Download Maven from http://maven.apache.org and unzip it to your desired installation directory. Ex. C:\maven in Windows, or /usr/local/maven in Linux.
Add System variable M2_HOME as well as $M2_HOME/bin directory to your system path.
Type the following in a terminal or command prompt:

mvn --version

If you are behind a firewall, create a <your-home-directory>/.m2/setting.xml with the following content:

<settings>
  <proxies>
    <proxy>
      <active>true</active>
      <protocol>http</protocol>
      <host>proxy.ensinfo.sciences.univ-nantes.prive</host>
      <port>3128</port>
    </proxy>
  </proxies>
</settings>

If there is an active Internal Maven proxy running. create a <your-home-directory>/.m2/setting.xml with the following content:

<settings>
  <mirrors>
    <mirror>
      <id>maven.mycompany.com</id>
      <name>University's Maven Proxy</name>
      <url><http://maven.mycompany.com/maven3</url>
      <mirrorOf>central</mirrorOf>
    </mirror>
  </mirrors>
</settings>

Creating your First Maven Project

To create the Quick Start Maven project, execute the following:

mvn archetype:generate -DgroupId=fr.unantes.datatypes
    -DartifactId=units
    -DarchetypeArtifactId=maven-archetype-quickstart
    -DinteractiveMode=false

New directory units has been created for the new project, and this directory contains your pom.xml which looks like the following file:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>fr.unantes.datatypes</groupId>
  <artifactId>units</artifactId>
  <packaging>jar</packaging>
  <version>1.0-SNAPSHOT</version>
  <name>units</name>
  <url>http://maven.apache.org</url>
  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
    </dependency>
  </dependencies>
</project>

Figure 3. Directory structure of your project after archetype generation

Maven Basic Commands

Compile the project: mvn compile
The output were placed in target/classes
Compile rests and sources and run unit tests: mvn test
Compile your test (but not execute them): mvn test-compile
Make a JAR file or the package: mvn package
take a look at the target directory and you will see the generated JAR file.
Install the artifact(JAR file) you’ve generated into your local repository so that it can be used by other projects: mvn install

<your-home-directory>/.m2/repository is the default location for the local repository

Test Execution

Surefire plugin (which executes the test) looks for tests contained in files with a particular naming convention. By default the following tests are included:
**/*Test.java
//Test.java
**/*TestCase.java
Conversely, the following tests are excluded:
**/Abstract*Test.java
**/Abstract*TestCase.java

Website/Reports Generation

Generate a basic website for the project:
mvn site
Clean the project (remove the target directory):
mvn clean
Create descriptor for the projects
for IntelliJ IDEA:
- mvn idea:idea
for eclipse:
- mvn eclipse:eclipse

Summary

Maven is a set of standards, a repository, a framework and is a software.
Maven is more than just downloading another JAR and a set of scripts, it is the adoption of processes that allow you to take your software to the next level.

Agenda

Introduction
Dependency Resolution
Software Artifacts
Build Tools
Maven
GitLab CI
Conclusion

GitLab CI

Continuous Integration service fully integrated with GitLab.
Open source and included in any GitLab CE installation.

Runners

In GitLab CI, Runners run your .gitlab-ci.yaml file.
A runner is an isolated (virtual) machine that picks up builds through the coordinator API of GitLab CI.

Containers

GitLab CI builds run inside one or more docker containers

Simple to get started (works out of the box)
Uses containers as throw-away VMs

Starting «.gitlab-ci.yaml»

Choose a docker image for Maven: https://hub.docker.com/_/maven
Create a simple job called test
- Installs graphviz
Runs on every push or MR

.gitlab-ci.yaml file

image: maven:3.6.1-jdk-8

test:
  script:
    - apt-get update -qq
    - apt-get install -y -qq graphviz

Jobs

Jobs can be run:

using Docker containers
using Docker containers and executing job over SSH
using Docker containers with autoscaling on different clouds and virtualization hypervisors
connecting to remote SSH server

image: maven:3.6.1-jdk-8

before_script:
  - 'apt-get update -qq && apt-get install -y -qq graphviz'

job:build:compile:
  script:
    - 'mvn --settings ./.gitlab.maven.settings.xml $MAVEN_OPTS -e $MAVEN_CLI_OPTS compile'

job:build:test:
  script:
    - 'mvn --settings ./.gitlab.maven.settings.xml $MAVEN_OPTS test'

Jobs (Cont.)

Why multiple jobs?

The smaller a job the easier parallelization across multiple runners is
Smaller jobs are easier to diagnose when they break

image: maven:3.6.1-jdk-8

before_script:
  - 'apt-get update -qq && apt-get install -y -qq graphviz'

job:build:compile:
  script:
    - 'mvn --settings ./.gitlab.maven.settings.xml $MAVEN_OPTS -e $MAVEN_CLI_OPTS compile'

job:build:test:
  script:
    - 'mvn --settings ./.gitlab.maven.settings.xml $MAVEN_OPTS test'

Pipeline

Pipelines are user-defined models of a build process
Typically includes stages for building an application, testing it and then delivering it

Artifacts

Allows for transferring files from one job to the next within the same build

Why run apt-get every time?
Artifacts are passed between jobs
They may have an expiration (defaults to 1 day)
Dependencies determine a job can only run after some other job

job:build:artifacts:
  script:
- 'apt-get update -qq && apt-get install -y -qq graphviz'
  artifacts:
    expire_in: 1 day
    paths:
      - vendor/

job:build:compile:
  script:
    - 'mvn --settings ./.gitlab.maven.settings.xml $MAVEN_OPTS -e $MAVEN_CLI_OPTS compile'
  dependencies:
  - job:build:artifacts

Stages

The ordering of elements in stages defines the ordering of builds' execution:

Jobs of the same stage are run in parallel.
Jobs of the next stage are run after the jobs from the previous stage complete successfully.

image: docker-registry.univ-nantes.fr/sunye-g/docker-asciidoctor-revealjs

before_script:
  - apt-get update -qy
  - curl -sL https://deb.nodesource.com/setup_10.x | bash -

stages:
  - build
  - deploy

build:
  stage: build
  script:
   - bash ./compile.sh

pages:
  stage: deploy
  script:
    - mkdir -p public
    - rsync -r target/site/ public

Limiting builds

Sometimes we do not want certain jobs to always run:

Usually when they take lots of resources (time, compute)
- Or when they do things like deploy to staging/production
Use only
- Defines a list of git refs for which build is created

pages:
  stage: deploy
  script:
    - mkdir -p public
    - rsync -r target/site/ public
    - du -sh public
    - ls -la public
  artifacts:
    paths:
      - public
  only:
    - master

Agenda

Introduction
Dependency Resolution
Software Artifacts
Build Tools
Maven
GitLab CI
Conclusion

Conclusion

New perspective:

Software build is seen as a software artifact
It should be developed and maintained as any other artifact

Main benefits:

Eliminate redundant tasks
Reduce dependencies on key personnel
Log the history of builds and releases in order to investigate issue
Improve product quality
Accelerate the build process
Reduce build issues.

References

http://maven.apache.org