Skip to content



Reusable workflows

These can be used by other repositories.

PR test

This workflow is meant to be used on pull requests.

First it computes the job matrix based on what was changed. Since this is time sensitive, rather than fetching the entire history we use GitHub's API to find out the precise depth to fetch in order to reach the merge base. Then it runs the test workflow for every job in the matrix.


Changes that match any of the following patterns inside a directory will trigger the testing of that target:

  • assets/configuration/**/*
  • tests/**/*
  • *.py
  • hatch.toml
  • metadata.csv
  • pyproject.toml


A matrix is limited to 256 jobs. Rather than allowing a workflow error, the matrix generator will enforce the cap and emit a warning.

Test target

This workflow runs a single job that is the foundation of how all tests are executed. Depending on the input parameters, the order of operations is as follows:

Target setup

Some targets require additional set up such as the installation of system dependencies. Therefore, all such logic is put into scripts that live under /.ddev/ci/scripts.

As targets may need different set up on different platforms, all scripts live under a directory named after the platform ID. All scripts in the directory are executed in lexicographical order. Files in the scripts directory whose names begin with an underscore are not executed.

The step that executes these scripts is the only step that has access to secrets.


Since environment variables defined in a workflow do not propagate to reusable workflows, secrets must be passed as a JSON string representing a map.

Both the PR test and Test target reusable workflows for testing accept a setup-env-vars input parameter that defines the environment variables for the setup step. For example:

    uses: DataDog/integrations-core/.github/workflows/pr-test.yml@master
      repo: "<NAME>"
      setup-env-vars: >-
        ${{ format(
            "PYTHONUNBUFFERED": "1",
            "SECRET_FOO": "{0}",
            "SECRET_BAR": "{1}"


Secrets for integrations-core itself are defined as the default value in the base workflow.

Environment variable persistence

If environment variables need to be available for testing, you can add a script that writes to the file defined by the GITHUB_ENV environment variable:

set -euo pipefail

set +x
set -x

Target configuration

Configuration for targets lives under the key inside a /.ddev/config.toml file.


Targets are referenced by the name of their directory.


Name ID Default runner
Linux linux Ubuntu 22.04
Windows windows Windows Server 2022
macOS macos macOS 12

If an integration's manifest.json indicates that the only supported platform is Windows then that will be used to run tests, otherwise they will run on Linux.

To override the platform(s) used, one can set the<TARGET>.platforms array. For example:

platforms = ["windows", "linux"]


To override the runners for each platform, one can set the<TARGET>.runners mapping of platform IDs to runner labels. For example:

runners = { windows = ["windows-2019"] }


To disable testing, one can enable the<TARGET>.exclude option. For example:

exclude = true

Target enumeration

The list of all jobs is generated as the /.github/workflows/test-all.yml file.

This reusable workflow is called by workflows that need to test everything.


During testing we use ddtrace to submit APM data to the Datadog Agent. To avoid every job pulling the Agent, these HTTP trace requests are captured and saved to a newline-delimited JSON file.

A workflow then runs after all jobs are finished and replays the requests to the Agent. At the end the artifact is deleted to avoid needless storage persistence and also so if individual jobs are rerun that only the new traces will be submitted.

We maintain a public dashboard for monitoring our CI.

Test results

After all test jobs in a workflow complete we publish the results.

On pull requests we create a single comment that remains updated:

Example comment

On merges to the master branch we generate a badge with stats about all tests:

CI testing badge


A workflow runs on merges to the master branch that, if the files defining the dependencies have not changed, saves the dependencies shared by all targets for the current Python version for each platform.

During testing the cache is restored, with a fallback to an older compatible version of the cache.

Python version

Tests by default use the Python version the Agent currently ships. This value must be changed in the following locations:


Windows performance

The first command invocation is extraordinarily slow (see actions/runner-images#6561). Bash appears to be the least affected so we set that as the default shell for all workflows that run commands.


The official checkout action is affected by a similar issue (see actions/checkout#1246) that has been narrowed down to disk I/O.

Last update: April 20, 2023