All Agent Integrations use Azure Pipelines to execute tests.
Every runner will execute test stages in the following order:
We make extensive use of Microsoft-hosted agents.
- Windows-only integrations run on Windows Server 2019 with Visual Studio 2019
- All other integrations run on Ubuntu 18.04 LTS
Some things are tested on multiple platforms, like the base package and the Disk check.
Every commit to a branch tied to an open pull request triggers a Linux and Windows job. Each runner will test any integration that was changed, with the Windows runner being further restricted to Windows-only integrations.
Every commit to the
master branch triggers one or more jobs for every integration.
Some integrations require additional set up such as the installation of system dependencies. As we only want these extra steps to occur when necessary, there is a stage ran for every job that will detect what needs to be done and execute the appropriate scripts. As integrations may need different set up on different platforms, all scripts live under a directory named after the platform. All scripts in the directory are executed in lexicographical order. Files in the scripts directory whose names begin with an underscore are not executed.
In addition to running tests on our CI, there are also some validations that are run to check for correctness of changes to various components of integrations. If any of these validations fail on your branch, then the CI will fail.
In short, each validation is a
ddev command, which fails if the component it is validating is not correct.
A list of the current validations can be found here.
ddev validate ci
This validates that all CI entries for integrations are valid. This includes checking if the integration has the correct codecov config, and has a valid CI entry if it is testable.
ddev validate ci --fix to resolve most errors.
ddev validate agent-reqs
This validates that each integration version is in sync with the
requirements-agent-release.txt file. It is uncommon for this to fail because the release process is automated.
ddev validate codeowners
Note: This validation command is only run when contributing to integrations-extras
Default configuration files¶
ddev validate config
File <INTEGRATION_SPEC> needs to be synced.To resolve this issue, you can run
ddev validate config --sync
If you see failures regarding formatting or missing parameters, see our config spec documentation for more details on how to construct configuration specs.
Dashboard definition files¶
ddev validate dashboards
This validates that dashboards are formatted correctly. This means that they need to be proper JSON and generated from Datadog's
If you see a failure regarding use of the screen endpoint, consider using our dashboard utility command to generate your dashboard payload.
ddev validate dep
- Verifies the uniqueness of dependency versions across all checks.
- Verifies all the dependencies are pinned.
- Verifies the embedded Python environment defined in the base check and requirements listed in every integration are compatible.
This validation only applies if your work introduces new external dependencies.
ddev validate manifest
This validates that the manifest files contain required fields, are formatted correctly, and don't contain common errors. See the Datadog docs for more detailed constraints.
ddev validate metadata
This checks that every
metadata.csv file is formatted correctly. See the Datadog docs for more detailed constraints.
ddev validate readmes
This ensures that every integration's README.md file is formatted correctly. The main purpose of this validation is to ensure that any image linked in the readme exists and that all images are located in an integration's
Saved views data¶
ddev validate saved-views
This validates that saved views for an integration are formatted correctly and contain required fields, such as "type".
View example saved views for inspiration and guidance.
Service check data¶
ddev validate service-checks
This checks that every service check file is formatted correctly. See the Datadog docs for more specific constraints.
ddev validate imports
from datadog_checks.base.foo import bar
See the New Integration Instructions for more examples of how to use the base package.
We use a GitHub Action to automatically add labels to pull requests.
If the Labeler CI step fails on your PR, it's probably because your PR is from a fork. Don't worry if this happens- the team can manually add labels for you.
The labeler is configured to add the following:
|integration/<NAME>||any directory at the root that actually contains an integration|
|documentation||any Markdown, config specs, |
|dev/testing||Codecov or Azure Pipelines config|
|dev/tooling||GitLab (see CD), GitHub Actions, or Stale bot config, or the |
|dependencies||any change in shipped dependencies|
|release||any base package, dev package, or integration release|
|changelog/no-changelog||any release, or if all files don't modify code that is shipped|
The changelog/<TYPE> label must be applied manually.
We forked the official action to support the following:
- a special
all:prefix modifier indicating the pattern must match every file