Fedora CI and rpminspect

David Cantrell

Tim Flink


Fedora CI


How you can help

Fedora CI

  • Please go to the Future of automated build checks talk on Friday
  • A framework for automating tests and providing results
  • Primary goal is to know that component builds are tested, working, and ready for a compose
  • https://docs.fedoraproject.org/en-US/ci/


build deviation analysis tool

But first, some history...


Red Hat developed an internal QE tool called rpmdiff

(not rpmdiff from rpmlint)

Before virt, before containers, before the cloud

Tool is a service and performs tests on built packages

What kind of tests?

Package policy compliance

Legal checks

Kernel ABI verification

Security policies

Comparisons from one build to the next

About forty (40) tests overall


Developer creates an erratum

Errata Tool schedules an rpmdiff job

rpmdiff hub (Python) verifies the builds exist

If builds exist, job is handed to a worker

Worker (Python) performs the rpmdiff

rpmdiff (Python) copies builds from NFS share

Perl script invoked to unpack the RPMs

Python script invoked to run checker

checker (in C) iterates over tests with each one spawning Perl and Python scripts

Results collected in XML format and handed back to hub

Some Problems

  • Runs on physical hardware, unable to scale
  • Legacy code base with a lot of technical debt
  • Too many moving parts
  • Cannot be run at the command line
  • Only outputs XML

Call Graph

(rpmdiff-checker run on zsh builds)

Back to rpminspect


  • Ensure package reliability
  • Ensure developer is aware of all build changes
  • Allow developers to modify packages and stay in compliance
  • Provide data to make gating decisions


  1. Must be able to run locally
  2. Must be able to run in a container
  3. Must talk directly to build system
  4. Must be able to download builds over HTTP
  5. Must report results in multiple formats
  6. Must be usable as a command line tool
  7. Must support RPM builds and modules


The service side becomes part of Fedora CI

The service listens for new builds and launches rpminspect runs

The tests are performed by rpminspect

Results collected by the service and sent to ResultsDB

Modes of Operation

  • Policy Checks
  • Build differences or deviations
    • new files appeared
    • existing files removed
    • files grew or shrunk considerably
    • new packages appeared
    • existing packages removed

Policy Checks

  • License tag
  • Header metadata
  • Political concerns
  • Forbidden language
  • Use of macros or compiler flags
  • Java bytecode major version check
  • ELF object checks
  • Security concerns (setuid, setgid, ownership)

Types of Builds

Build types consists of a grouping of RPMs and an associated SRPM for all intended architectures

  • Regular Koji RPM build
  • Module build

Support Files

  • /etc/rpminspect.conf
  • Product specific data is in the rpminspect-data-fedora package
  • rpminspect-data-generic is a template package

How to Run

  • dnf install rpminspect-data-fedora
  • man rpminspect
  • rpminspect --help
  • rpminspect [options] [previous build] [new build]
  • rpminspect -v -k -T license zsh-5.7.1-3.fc31 zsh-5.7.1-4.fc31

What Can It Do Today?

  • Speaks to Koji and downloads builds
  • Can inspect module builds
  • If only one build specified, skips deviations
  • Allows user to skip tests or only perform certain tests
  • Outputs results to stdout or a file
  • Outputs results in JSON or plain text
  • Fetch-only mode to use rpminspect to download Koji builds

Tests It Currently Performs

License tag check ¤ Check for any empty RPM payloads ¤ RPM metadata check (Vendor, badwords) ¤ Validate man pages ¤ Validate XML files ¤ ELF check (execstack, relocations) ¤ Validate *.desktop files ¤ Make sure the Release tag contains %{?dist} ¤ Validate spec file name ¤ Make sure modules contain %modularity ¤ Java bytecode major JVM version check

Call Graph

(rpminspect run on zsh builds)

Call Graphs


  • Unit test suite
  • Integration test suite
  • Documenting the details of individual inspections
  • Designing new tests based on the Fedora packaging policy
  • Maintaining rpminspect-data-fedora


  • https://github.com/dcantrell/rpminspect
  • https://github.com/dcantrell/rpminspect-data-fedora
  • Packages are in rawhide, automated builds in Copr


  • #fedora-ci on FreeNode
  • dcantrell on FreeNode
  • File github Issues
  • Email me: dcantrell@redhat.com