# Mir Continuous Integration

This document outlines the journey of a contribution to Mir through its continuous integration
pipeline.

## Overview

There are a number of components to this story, and a diagram might make for a good overview:

```{mermaid} continuous-integration.mmd
```

These are run at different stages in the pipeline, balancing the time it takes to run and the
breadth of testing. We'll discuss those in more detail below.

## Mir builds

When a pull request is opened, updated or merged into `main`, we validate that the contribution
is correct by building the code and running our test suite. To facilitate this, we use
[Spread](https://github.com/canonical/spread) (or rather, our
[lightly patched version](https://snapcraft.io/spread-mir-ci)) to build Mir across a number of
environments. [`spread.yaml`](https://github.com/canonical/mir/blob/main/spread.yaml) holds
environment definitions, while the actual build tasks are maintained under
[`spread/build`](https://github.com/canonical/mir/tree/main/spread/build).

Our focus of development is the most recent Ubuntu LTS, and we maintain builds for any more recent,
supported Ubuntu releases. We also build for stable releases of other Linux distributions that
we have community interest on.

We also run builds using alternative toolchains (Clang) and development versions of Ubuntu and
those other distributions.

We also maintain Mir branches for all previous Ubuntu LTS releases under support. These will only
receive security updates and the occasional bug fix. These will always be
[`release/` branches on GitHub](https://github.com/canonical/mir/branches/all?query=release%2F),
and releases published on [the GitHub release pages](https://github.com/canonical/mir/releases).

Opening a pull request, or merging into the `main` or `release/` branches, triggers a
[build on GitHub](https://github.com/canonical/mir/actions/workflows/spread.yml).

As part of the build, the following sets of tests are run:

### Unit tests

Defined in [`tests/unit-tests`](https://github.com/canonical/mir/tree/main/tests/unit-tests),
these assert the functionality of individual components in isolation, as is usual practice.

### Acceptance tests

[`tests/acceptance-tests`](https://github.com/canonical/mir/tree/main/tests/acceptance-tests)
run more high-level tests, ensuring we maintain the contracts with external components as well as
our own loadable modules' interfaces.

One major part of this is [WLCS](https://github.com/canonical/wlcs) - asserting the behaviour
of Mir's Wayland frontend is to specification of the protocol.

### Integration tests

In [`tests/integration-tests`](https://github.com/canonical/mir/tree/main/tests/integration-tests)
lie other high level tests, ensuring we correctly integrate with some system components.

### Performance and smoke tests

Lastly, [`tests/performance-tests`](https://github.com/canonical/mir/tree/main/tests/performance-tests)
holds a handful of performance-related tests, collecting some metrics about how the system
performs end-to-end - and verifying that it works in the first place.

We also run these on a number of hardware platforms in our testing lab for every build of `main`.

## Sanitizer runs

In addition to the above, for every merge to `main` we build and run the tests using the following
sanitizers:

- **Undefined Behaviour**
  Building with UndefinedBehaviourSanitizer enabled reports no undefined behaviour, it would be a
  CI failure otherwise.
- **Address Sanitizer**
  As we have some fixing to do here, we run those builds - but don't enforce the problems reported.
- **Thread Sanitizer**
  Unfortunately due to
  [incompatibility between GLib and TSan](https://github.com/google/sanitizers/issues/490), we
  don't currently run the thread sanitizer, as that produces too many false positives.

## ABI checks

For each pull request we
[check that the exported symbols are as expected](https://github.com/canonical/mir/actions/workflows/symbols-check.yml)
- and fail if the ABI changed in any way. It doesn't necessarily mean an ABI break - but new symbols
need to be tracked.

## Coverage measurement

We [track test coverage](https://github.com/canonical/mir/actions/workflows/coverage.yml) for each pull
request and `main` builds, and the results are visible on
[Codecov.io](https://app.codecov.io/gh/canonical/mir).

## `.deb` package builds

Merges to `main`, `release/` branches as well as annotated tags are followed by `.deb` package
builds in mir-team's Launchpad's Personal Package Archives (PPAs):
- [`~mir-team/dev`](https://launchpad.net/~mir-team/+archive/ubuntu/dev) for the latest development
  builds
- [`~mir-team/rc`](https://launchpad.net/~mir-team/+archive/ubuntu/rc) for release candidate and
  release builds

At release, those get copied to the
[`~mir-team/release`](https://launchpad.net/~mir-team/+archive/ubuntu/release), general availability
PPA.

## Downstream snap builds

We [build a subset](https://github.com/canonical/mir/actions/workflows/snap.yml) of the downstream
snaps on pull requests - these are then available in `edge/mir-pr<number>`
[Snap channels](https://snapcraft.io/docs/channels) for testing, e.g.:

```shell
snap install miriway --channel edge/mir-pr1234
```

When `.deb` packages build in the PPAs above, all the affected downstream snaps get rebuilt through
the [`~mir-team` snap recipes on Launchpad](https://launchpad.net/~mir-team/+snaps) and made
available in the `edge` or `beta` channels, as appropriate.

## End-to-end testing

When the snaps get published to the store, we have Jenkins dispatch test runs on a selection of
hardware, ranging from Raspberry Pis through single-GPU systems all the way up to high performance
multi-GPU (usually hybrid) ones. We maintain a matrix of test coverage
[in this spreadsheet](https://docs.google.com/spreadsheets/d/1kUbTSt4zWVpTtgZNJvvxCdugsRUv6C5PK9Xw5dxppCc/edit#gid=893560997).

These tests ultimately verify the full story end-to-end, installing the snaps on a range of
Ubuntu versions (Core and classic alike) and verify behaviours and visuals through a number of
scenarios.

It's a somewhat complex web of things happening, so again a diagram might help

```{mermaid} end-to-end-testing.mmd
```

[checkbox-mir](https://github.com/canonical/checkbox-mir/) is the test orchestrator, collecting
the different tests (smoke, performance and functional) and packaging them in a way that can be
consumed by our lab setup. It's using our own [mir-ci](https://github.com/canonical/mir-ci) tests
along with the [mir-test-tools](https://github.com/canonical/mir-test-tools) snap.

You can find more information about Checkbox itself
[in its documentation](https://github.com/canonical/checkbox) - it's a system used by our
certification team, running thousands of tests on hundreds of systems every day.

The Jenkins job definitions go into a private repository, as they contain credentials.
[jenkins-job-builder](https://pypi.org/project/jenkins-job-builder/) is used to maintain the jobs,
and at runtime, [testflinger](https://github.com/canonical/testflinger) dispatches them to the
device under test, while [test-observer](https://github.com/canonical/test_observer) collects the
results.

## Summary

As shown above, Mir is being tested quite extensively on a large number of environments. We're
extending the coverage every day, as well. There's a lot of moving pieces involved, but it does
help us greatly in ensuring the quality level we hold ourselves to.
