Skip to content

Lab 10

In this lab sessions you will be practicing the setup of a GitLab continuous integration pipeline. You will be working with an off-the-shelf maven template project.

Preparation

  1. Create a new maven CiTestProject, using the archetype generator: 

      mvn archetype:generate \
  -DgroupId=ca.uqam.info \
  -DartifactId=CiTestProject \
  -DarchetypeArtifactId=maven-archetype-quickstart \
  -DinteractiveMode=false

    If you are on windows, remove the \ and write the command as a single line. Protect every argument by encompassing "", e.g. "-DgroupId=ca.uqam.info".

  2. Log into GitLab and create a new project CiTestProject, using the + sign on the top-left

    • Do not initialize a README
    • Keep it public
  3. Link the projects: 
    git remote add origin git@gitlab.info.uqam.ca:yourgitlabname/citestproject.git
git add pom.xml
git add src
git commit -m "initial commit"
git push --set-upstream origin main

CI preliminaries

The main interest of continuous integration is to ensure your software never regresses. Instead of allowing arbitrary changes on the main branch, changes must be requested, and the decision to accept or reject modifications must be informed.

Protect main

Protecting main is a precondition to meaningful CI. Rather than modifying main locally and pushing new commits to the server:

  • Developers can only push to new branches.
  • Developers request a merge from their branch to main.
  • A project manager approves the merge request, and a new commit on main is created server-sided.

Your turn

In a first step we'll make sure the main branch of your repository rejects direct commits:

  • Access your project on GitLab
  • Go to Settings -> Repository -> Protected branches
  • For branch main...
    • leave Allowed to merge as it is
    • set Allowed to push and merge to No one

Once you configured the main protection, it is time to verify its effectiveness:

  1. Create a new commit on main, e.g. by creating a new empty text file or adding a javadoc comment.
  2. Attempt to push the new commit to the server.
  3. Verify the commit is rejected.
  4. Delete your local commit again (without loosing the changes you've made): git reset --soft HEAD~1
    Explanation: Reset kills the last commit, soft preserves the file content, Head~1 navigates to "1 commit before the current HEAD".

Merge requests

Next you're going to practice submitting your work the correct way. Instead of adding your commit to master, then pushing...

  1. Create and switch to a new feature branch.
  2. Commit to your feature branch.
  3. Push the feature branch to origin (the gitlab server).
  4. Issue a merge request using the GitLab webUI.
  5. Accept the merge request.

Your turn

Follow the described procedure:

  1. Use git checkout -b feature, to create a new branch.
  2. Use git add somefile and git commit -m "added a feature" to create a new commit on the feature branch.
  3. Use git push --set-upstream origin feature to send your branch to the GitLab server.
  4. Log into GitLab, navigate to your project, select your branch in the dropdown. Then select "new merge request" (blue button).
  5. On the left sidebar, select the merge request and accept it. Select delete feature branch after merge.
  6. Use git pull, to retrieve the merged version locally.

CI pipelines

So far the merge request did not provide any insight on the quality of your commit on feature branch.

  • The goal is now to configure checks and balances to investigate the state of now commits.
  • This way anyone attempting a merge, i.e. assessing a merge request, knows whether the code can be safely accepted on main.

All continuous integration configuration with GitLab takes place using a file .gitlab-ci.yml. The file only needs to exist for GitLab to run a CI pipeline.

  • Create a file .gitlab-ci.yml inside your project. It should reside at top level, i.e. right next to where you would place a pom.xml or .gitignore file.
  • At the top of the line, configure the container image to be used for CI pipeline execution. As first line of the file, enter:
    image: maven:3.9.8-amazoncorretto-21

Clutter checker

The first CI job to implement will be a clutter checker, that fails the pipeline whenever class files are stored in your repo.

  • Every runner must be associated with exactly one CI stages. The simplest option is to select one of the default stages, e.g. build.
  • Runner live in a linux environment, so you can call whatever linux script you want to test the quality of your commit.
  • If no further restricts configured, the entire CI pipeline is called for every commit on any branch sent to the server.

Your turn

  • Add a runner that returns a non-zero exit code (e.g. 1), in case class files were found in the commit, example: 
    # Search for *.class files in the repository. Store all findings in variable CLUTTER
CLUTTER=$(find . -name \*.class)
# If variable CLUTTER is not empty, return error cod 1, otherwise exist with success code 0.
if [[ ! -z $CLUTTER ]]; then exit 1; else exit 0; fi
  • After the modifications to your .gitlab-ci.yml file, create a new commit on a new branch.
  • Issue a merge request and verify the CI pipeline was activated.
  • Accept the merge request
  • Compile your project, using mvn clean compile
  • Commit again, and push. Verify the pipeline rejects your most recent commit.

Atomic maven

In this course you've already learned how to configure maven to run quality checks on your codebase. A first meaningful pipeline is to simply call the entire maven default lifecycle by a single runner.

Your turn

Set up a working atomic maven pipeline:

  1. Create a new runner that calls mvn clean package -B.
  2. Commit to a new branch atomicmaven
  3. Verify the pipeline passes.

Verify the pipeline rejects faulty code:

  1. Make a code change that breaks compilation, e.g. remove a ; at the end of a code line.
  2. Commit and push again.
  3. Verify the pipeline now fails.

Maven stages

So far we've executed all maven phases until package in a single runner.

  • Unfortunately that is not a very sophisticated pipeline. In case it fails, we do not conveniently see the reason.
  • A better approach is to match GitLab CI stages on individual phases to obtain more detailed feedback on what's going on.

Your turn

  1. Override the default CI pipeline stages, by defining a new stage for every default maven lifecycle phase.
  2. Define an individual runner for every maven stage until (including) install:
    1. The first one to call validate
    2. The second one to call ...
    3. ...
  3. Commit and push your pipeline configuration
  4. Verify the pipeline now displays individual stages for every maven lifecycle phase.

Optimizing stages

The previous CI configuration is somewhat wasteful in resources, for earlier maven phases are repeated by multiple runners, thus waisting CPU time and slowing down pipeline execution.

  • Unfortunately it is not possible to execute individual maven phases in isolation.
  • It is however possible include some optimizations, to avoid executing the same plugins over and over.

Below maven parameters can be used to skip specific maven plugins:

  • -DskipTests skips unit tests, integration tests and coverage reports at test and validate phase.
  • -Djacoco.skip=true skips coverage reports at test and validate phase.
  • -Dcheckstyle.skip=true skips checkstyle verification at verify phase.
  • -Dmaven.javadoc.skip=true skips javadoc verification and generation at package phase.

That can be taken into account for various maven runners:

  • Checkstyle is associated with the validate phase: we can skip checkstyle execution for all later phases. Verifying once is enough.

Your turn

  • Modify the maven commands of your previous runners, to skip maven plugins that have already served their purpose on earlier stages.
  • Verify your pipeline still runs correctly, i.e. still detects:
    • Checkstyle violations
    • Missing javadocs
    • Failed tests

Extracting artifacts

JUnit and Jacoco both produce artifacts beyond "Success" or "Failure" that can provide usefull details on the quality of a commit:

  • Detailed test reports: Which tests succeeded or failed.
  • Coverage reports: Which percentage of code lines is covered by tests.
  • JavaDoc: Website HTML code.

A good CI configuration provides for every commit easy to capture information, along with success or failure metrics.

Your turn: Junit reports

  • The Surefire plugin (running unit and integration tests) generates a fine-grained test report at: target/surefire-reports/
  • GitLab can use these reports to display information on individual test results on CI execution.
  • Advise GitLab to recover these artifacts from the runner, with: 
    maven-verify:
  stage: verify
  script:
    - "mvn clean verify -Dcheckstyle.skip=true -Dmaven.javadoc.skip=true -B"
  artifacts:
    when: always
    reports:
      junit:
        - target/surefire-reports/TEST-*.xml

Your turn: Jacoco reports

  • The Jacoco plugin can generate coverage reports for the combined unit and integration tests, with the following configuration (invoked at verify phase): 
    <!-- Plugin for coverage report -->
<!-- Visual test report available at:  target/site/jacoco/index.html -->
<!-- Plugin for coverage report -->
<!-- Visual test report available at:  target/site/jacoco/index.html -->
<!-- machine readable report at: target//site/jacoco/jacoco.xml -->
<plugin>
    <groupId>org.jacoco</groupId>
    <artifactId>jacoco-maven-plugin</artifactId>
    <version>0.8.12</version>
    <configuration>
        <excludes>
            <!-- ignore view package -->
            <exclude>**/view/*</exclude>
        </excludes>
    </configuration>
    <executions>
        <execution>
            <goals>
                <goal>prepare-agent</goal>
            </goals>
        </execution>
        <!-- Associate with verify phase, to grab unit AND integration test results.-->
        <execution>
            <id>report</id>
            <phase>verify</phase>
            <goals>
                <goal>report</goal>
            </goals>
        </execution>
        <execution>
            <id>jacoco-check</id>
            <goals>
                <goal>check</goal>
            </goals>
            <configuration>
                <rules>
                    <rule>
                        <element>PACKAGE</element>
                        <limits>
                            <limit>
                                <counter>LINE</counter>
                                <value>COVEREDRATIO</value>
                                <minimum>80%</minimum>
                            </limit>
                        </limits>
                    </rule>
                </rules>
            </configuration>
        </execution>
    </executions>
</plugin>
  • A test report ends up in: target/site/jacoco/index.html
  • However, GitLab needs the report not in HTML but a readable format
  • The trick is to add a linux line that extracts the total coverage from the test report: cat target/site/jacoco/index.html | grep -o 'Total[^%]*%' | sed 's/<.*>/ /; s/Total/Jacoco Coverage Total:/'
  • Gitlab knows how to consume that line printed to the runner's console:. We only need to indicate its existence with the coverage keyword.
  • Using both, the maven and the cat command, we can feed the total coverage report back to the GitLab pipeline:
maven-verify:
  stage: verify
  script:
    - "mvn clean verify -Dcheckstyle.skip=true -Dmaven.javadoc.skip=true -B"
    - "cat target/site/jacoco/index.html | grep -o 'Total[^%]*%' | sed 's/<.*>/ /; s/Total/Jacoco Coverage Total:/'"
  coverage: '/Total.*?([0-9]{1,3})%/'
  • Set up the pipeline as described, let it run through and verify the jobs tab within your finalized pipeline.
  • Verify the last job shows a coverage number.

Your turn: ApiDoc

  • The javadoc plugin creates a folder: target/apidocs
  • Extract the folder for deployment, using the artifact keyword: 
    pages:
  stage: package
  script:
   - "mvn clean package -Dcheckstyle.skip=true -DskipTests -B"
   - "mv target/apidocs public"
  artifacts:
    paths:
      - public
  • Wait for the pipeline to run through.
  • Go to: Deploy -> Pages, ensure the project pages are activated. Disable the unique domain checkbox.
  • Verify your project documentation is accessible at: https://yourgitlabname.pages.info.uqam.ca/yourproject/

Job must be named pages

For the website to be accessible, via Deploy -> Pages, it is important that the job name is pages. Otherwise the pipeline will run correctly run through, but no website will be accessible.

Markdown readme

In class, you've already seen three textual markup languages for configuration files:

  • XML, for Maven
  • JSON, for object serialization and Gradle
  • YAML, for GitLab CI pipeline configuration

All these languages are intended for machine-interpretation, that is, we write them to provide instructions for machines.

  • Markup languages are not necessarily for machines: The Markdown syntax is a simple modelling language for describing structured documents for humans.
  • MarkDown can be easily compiled to HTML, the course documentation is an example.
    • GitHub / GitLab automatically render markdown files to HTML when a .md file is selected in their webui
    • IntelliJ offers a splitview (plugin activated by default) for Markdown files, to instantly pre-visualize the HTML counterpart while drafting:

Project README.md file

For software projects, it is good practice to place a file README.md at top level.

  • If the file exists, the git server will automatically display the content as project description.
  • The file is a recommended starter point, to advice software users on:
    • Title and interest of the project
    • How to download and install the project (for non-developers, i.e. users):
      • Where / how to obtain compiled project
      • Which command to use to launch project
    • How to obtain the sources, and contribute (for developers)
      • Project requirements, i.e. what needs to be installed to compile the project.
      • How to access API documentation
    • Authors / contribution statements
    • If applicable: software licenses
  • The Markdown syntax is extremely simple and can
  • For example the XoxInternals project documentation, as by README can be inspected here.

Your turn

Familiarize yourself with the basic concepts of the markdown syntax, using: THIS syntax reference.

  • Headings, subheadings
  • Bold, italic, strikethrough text highlighting
  • Lists and enumerations:
    1. First item
    2. Second item
    3. ...
  • Hyperlinks

  • Citations:

    "Bruih ? Miow miooooow ! Croquettes !!" -- Keksli

  • Code lines and blocks:

    • Some code line
    • And also: 
      Multiple 
lines
of code.

Add a file README.md to your test project, create a very minimal documentation for project title, and run instructions.