# Fedora Zuul CI ## An update ![alt text](assets/zuul.svg) ![alt text](assets/Fedora_logo.svg) Note: Welcome for this quick update on the state of the Zuul-based CI infrastructure for Fedora!
## Who are we ? - 🦾 Matthieu Huin - 🤖 Fabien Boucher ### Red Hat OpenStack Production Chain Infrastructure - Zuul operators - Zuul developers Note: Good afternoon everybody I am Matthieu Huin and I am co-presenting with Fabien Boucher. Our day to day work is to maintain some Zuul deployments at Red Hat. Beside operations, we help people get started with Zuul and also we contribute code to the Zuul project. In this talk, we will give you an update about the state of Fedora Zuul CI. The platform got a couple of improvements since our last talk during devconf.cz'20.
## What is Fedora Zuul CI ? - Started around Flock'19 - **→ Provide Zuul Based CI for Fedora Packagers** - Operated by our team - Integrated into a larger Zuul Deployment - → https://softwarefactory-project.io - Triggers on events from pagure.io and src.fedoraproject.org (github.com, ...) - Jobs and workflow based on Pull Requests for distgits Note: But first, a quick reminder about Fedora Zuul CI: - this is an effort we started around FLOCK'19. The goal was to provide an alternative CI system to Fedora Packagers based on Zuul. - It is developed and operated by the Red Hat OpenStack Production Chain team. - Fedora Zuul CI is a tenant within a larger Zuul deployment hosted by our team, called Software Factory Project. It is a public deployment that hosts CI for the RDO project. - It provides CI infrastructure for projects hosted on pagure.io and src.fedoraproject.org. Projects hosted on Gerrit and Github are also supported (CI can be enabled for Fedora Projects) - Finally, it provides jobs and a validation workflow around Pull Requests for Fedora packages (distgits on src.fedoraproject.org).
## Zuul CI strengths for Fedora - Code gating system - dependencies testing → rpm build/runtime dependencies - speculative validation of the code - speculative validation of CI config: no staging infra needed Note: We started this effort because we thought a Zuul-based CI for Fedora would present many advantages: - Zuul is a code gating system that emphasizes dependencies testing and speculative validation, ensuring related changes are tested and integrated together. - The dependency approach is particularly well suited for distgits, as dependencies between packages are common. - Speculative validation of the code catches integration problems before they are merged - Since the CI configuration is also code, changes to CI can be tested and validated without the need of a staging infrastructure
## Zuul CI strengths for Fedora - Battle tested - In development for the last 9 years, next major version release in early 2021 - Used by projects and organizations where CI is critical - Scalable, can handle hundreds of jobs in parallel Note: Zuul is also a reliable service. - Zuul has been in development for more than 9 years, and improvements are still being made. Zuul has been adopted by projects where CI is critical (openstack, automobile industry, etc). It has been praised for its scalability.
## Zuul CI strengths for Fedora - "Plug and Play" - Simplified registration - Default configuration covers most use cases - Bring your own resources (if you want) - Enable PR based workflow for contributors, lowering the entry barrier for newcomers Note: And finally, - Registering a project on FZCI has become straightforward, and default CI jobs and tests environments cover most use cases. - "Bring your own resources" - should the Fedora Community want to scale usage up, dedicated resources can be added (AWS, OpenStack, bare metal, etc) - FZCI provides CI validation on Pull Requests, giving more people opportunities to contribute to packaging with a familiar workflow for submitting patches.
## Notable facts since Feb 2020 Note: Now, Fabien will tell you more about the improvements that were made to FZCI during that year.
## Fedora messaging triggers - **Before** - Using FZCI required modifications on a project's settings so that events get pushed to Zuul - **Now** - Pagure → fedora-messaging → FM-gateway → Zuul **One step less required for maintainers to attach projects to FZCI** Note: - Previously, using FZCI required changes in a project's settings. At least adding a "web-hook url" pointing at Zuul was needed. - To remove that constraint we have developed a service that listens to the fedora-messaging bus and forwards messages related to Pull Requests to Zuul as web payloads. - So thanks to this improvement there is one step less for maintainers to attach Zuul CI to a distgit.
## GitLab support - Zuul can listen to events on repositories hosted on GitLab - Fully merged in Sept 2020 and isofunctional **Fedora Zuul CI is ready for the migration to GitLab** Note: - Since recentlly, Zuul can handle events on repositories hosted on a GitLab instance. - The GitLab driver has been fully merged into Zuul in September 2020 and is isofunctional compared to other Zuul drivers. - This means that existing jobs and infrastructure provided by Fedora Zuul CI should transition seamlessly to GitLab if distgits move to GitLab.
## Finding candidates for FZCI Fedora consists of ~ 30000 distgit repositories Most contributors push changes directly into distgits **A new tool to** - Select and add suitable candidate repositories - notify the maintainers of these distgits about their new CI **Regularly attach a bunch of new repositories to Zuul** Note: - Now let's talk about our strategy to cover more distgit repositories with FZCI. - Fedora consists of around 30000 distgit repositories and we cannot, as of today, provide CI for all of them. - Also, a majority of packagers don't use PR as their way to contribute, so adding a PR-based CI workflow on these distgits is irrelevant - So we created a tool to: - search for distgit repositories that received some Pull Requests recently. We implemented it by using Fedora datagrepper to gather metrics. - Also the tool is able to notify maintainers by email when FZCI has been attached to the distgits they maintain. - So thanks to that tool we were able to regularly attach a bunch of relevant distgit repositories to Zuul.
## rpminspect-report A **rpminspect** job is run at every PR update JSON report difficult to read for humans **A rpminspect-report WEB app** - Generic and hosted on sf.io - Enabled in FZCI - Can be used by other Fedora CI systems Note: - At every PR update FZCI runs a job called rpminspect that perform like a "rpm diff" between the proposed rpms and the previous published rpms on Koji. - Unfortunatly the produced JSON report is difficult to read for a maintainer and then limits the usefulness of the job. - So to make the job more relevant we created the rpminpect-report WEB app that prettify the JSON created by rpminspect. - The webapp is generic and hosted on sf.io. It is already enabled in FZCI and it can be used easily by every Fedora CI as it takes the JSON report's url as input.
![alt text](assets/rpminspect-report.png) https://fedora.softwarefactory-project.io/rpminspect-report Note: - Here is a screenshot of the WEB app. - As you can see it takes an URL of a rpminspect report in web form making it easy to be reused by other Fedora CI. - Also you can see it allows to filter messages by severity and I think that is really useful because some packages can produce a lot of rpminspect messages.
## Functional tests job selector Some distgits do not contain any STI test definition - **check-for-tests** job - Inspect the PR content for tests/tests.yaml file - Condition the run of the right child job from: - **rpm-install-test**: Install the built rpms - **rpm-test**: Install the built rpms and run the defined STI test Note: - We recentlty figured out that we need a job selector to tell Zuul to run the right functional test job. - Indeed, some distgits do not contain any STI test definition so for them at least we need to validate the package's installation. - So we have implemented a job called "check-for-tests" that is able to detect STI test definitions and to condition the run of rpm-test or rpm-install-test jobs. - rpm-install-test job simply tries to install the built rpms - rpm-test job installs the built rpms and runs the STI tests defined in the distgit
## Arch-specific Koji scratch build - Functional tests run on x86_64 VM - Koji scratch build with x86_64 arch override **Does my package build on other architectures ?** - rpm-scratch-build- - [aarch64|armv7hl|i686|ppc64le|s390x] - Conditional job: **check-for-arches** Note: - Finally arch specific koji scratch build. - Our infrastructure only provides x86_64 VMs on which to run the functional testing jobs. Thus the default PR's jobs workflow runs a scratch build with the x86_64 arch override on Koji. - However it is important to get feedback from the CI in case of build failures on other architectures supported by Fedora. - So we have provided additional rpm-scratch-build jobs for every supported Fedora architecture: rpm-scratch-build-[aarch64|armv7hl|i686|ppc64le|s390x] - The conditional job "check-for-arches" ensures that the extra jobs aren't run on strict "noarch" packages.
## Some numbers Note: Now let's give some usage statistics of Fedora Zuul CI.
Since FZCI started: - Distgits attached: 387 - Jobs ran: 10914 - Buildsets reported: 2188 - Infrastructure failures: 15 (~0.15%) Note: - We currently provide CI for 387 distgits. - The CI ran more than 10000 jobs that we've reported back into 2000 buildsets. - The CI experiences 15 node failures, meaning that 99% of jobs executed w/o infrastructure issue.
## Top Projects by number of builds - 297: rpms/pyproject-rpm-macros - 117: rpms/python3.9 - 109: rpms/python-ogr - 86: rpms/python39 - 83: rpms/python-rpm-macros Note: - Here is the listing of the top five projects by number of builds. - You can notice that is mainly Python packages but for a good reason because we mainly worked with Python packagers to improve Fedora Zuul CI.
## FZCI HOWTO - for the packager https://fedoraproject.org/wiki/Zuul-based-ci Note: Now we will give some instructions about how, as a packager, you can interact with the CI. Also here is the wiki page of the project where you'll find the same instructions.
## Manually trigger the CI Comment the PR with **recheck** Note: - So let's pretend I'm a packager! - Hey Fabien, I hit a transient error and I would like to re-run the CI jobs, can I do that? - Yes sure, to do so just comment the PR with "recheck"
## PR dependencies Add PR dependencies with ``` Depends-on:
``` Only support the runtime depencencies rpm-tests, rpm-install-tests Note: - Hey Fabien, I'd like to use the CI to test a change related to a version bump on one of the dependencies in my package, but said bump hasn't been merged yet. Can I do that? - Yes! In the case the needed dependent package is proposed as a PR, then you can tell Zuul to include the dependent PR built artifacts (so the rpms) into the testing environment. To do so, add a "Depends-on" into the initial comment of the PR. As you can see you can also list multiple dependent PRs. - Note that this is only supported for the runtime depencencies especially for rpm-tests and rpm-install-tests jobs.
## Publish template Open a PR on pagure.io:fedora-zuul-job-config/zuul.d/projects.yaml with ```yaml - project: name: rpms/python-passlib templates: - publish ``` Basically automate **fedpkg build** once the PR is merged Note: - Hey Fabien, I'm lazy and I don't want to keep en eye constantly on my patches to run "fedpkg build" as soon as they are merged, can it be automated? - Yes, this is not part of the default workflow, but you might want to let Zuul publish on Koji on your behalf. - To do so, you can opt-in by opening a PR adding this publish template on the fedora-zuul-job-config repository.
## Debugging failing jobs - If a job fails, the execution environment can be put *on hold* for a limited amount of time instead of being destroyed - Request a autohold via IRC - #software-factory - #fedora-ci - Infos needed: project, PR id, failing job, public SSH key Note: - Hey Fabien, my CI jobs fail and I can't reproduce the problem locally, is there any way to perform some debugging on the execution environment? - Yep, with Zuul it is possible to put a test node on hold the next time a given job fails for your change. You can then SSH into that node and investigate your problem. - However it cannot be automated, it needs to be requested via IRC on the #software-factory or #fedora-ci channels on freenode.
## Attach FZCI to your package Open a PR on pagure.io:fedora-project-config/resources/fedora-distgits.yaml with ```Yaml Fedora-Distgits: source-repositories: - rpms/python-passlib: zuul/include:  ``` Note: - Well Fabien, I am convinced! How do I add my project to Fedora Zuul CI? - To attach a distgit to FZCI simply open a PR on the fedora-project-config repository. - Once merged, the default Fedora Zuul CI's jobs will be triggered by subsequent PRs creations and updates.
## Wish list - Support Depends-on at build time - TMT job support - Get more people involved Note: - Our wishlist for the near future - We would like to continue to improve FZCI as well as making it more widely used. - The next improvements we'd like to bring into it are: - Supporting the Depends-on at build time. For now we build on Fedora Koji but it is not possible to specify additionals repositories to be added in the Koji mock root. - A solution could be to use Copr where mock root can be created via an API and where additional rpm repositories can be added as attribute. - Another solution would be to use a local mock root in the test node. - Well we need you input about how to implement it efficiently. - Furthermore, similar to STI test we'd like to add a support for TMT tests. - And finally we'd love more people to be involved in maintaining or creating jobs for FZCI.
## Questions / Comments ? https://fedoraproject.org/wiki/Zuul-based-ci https://fedora.softwarefactory-project.io/devconf.cz.21-slides/ #software-factory / #fedora-ci firstname.lastname@example.org email@example.com Note: