Test dependencies¶

E2E tests run on real VMs and clusters. Anything a test fetches from the public internet at test time is a dependency on infrastructure we don't and can't control. This is one of the biggest source of unreliability in e2e tests: DockerHub will rate-limit us, distro package mirrors often go down down, and apt install <pkg> might quietly install a different version six months from now.

A major source of flaky / unrelated failures is reliance on external dependencies: anything outside the AWS/GCP/Azure account the test runs in and internal DD systems.

We will soon block all internet access from CI for security reasons, so prepare now.

Spotting a runtime dependency¶

Common ways to remove external deps¶

ECR pull-through cache¶

Use the ECR pull-through cache set up in the datadog-agent-qa account (669783387624, us-east-1):

• DockerHub official images keep their library/ path: busybox:1.37.0 becomes …/dockerhub/library/busybox:1.37.0.
• Only those three upstreams are supported. GHCR is not — if your image is only on GHCR, use an equivalent from a supported upstream if possible, otherwise leave unproxied.
• Pulling from public.ecr.aws/… or mirror.gcr.io directly is not an approved alternative, even though neither is rate limited today. Route it through the prefixes above. Existing code that does otherwise is debt, not precedent.
• There is no pull-through cache in GCP or Azure. For a test that only ever runs on GCP, the provider's registry or mirror.gcr.io is acceptable — do not import that habit into an AWS test. Ask #agent-devx-help before relying on any other public mirror.

reg, _ := runner.GetProfile().ParamStore().GetWithDefault(parameters.ImagePullRegistry, "")
if reg != "" {
    busyboxImage = strings.SplitN(reg, ",", 2)[0] + "/dockerhub/library/busybox:1.37.0"
}

Kubernetes and Kind¶

Two different pulls happen, with two different defaults:

• The Kind node image (the VM image the cluster itself boots from) is proxied through the ECR cache by default. components/kubernetes/kind.go builds its reference from env.InternalDockerhubMirror(), which resolves to the cache on AWS and falls back to registry-1.docker.io elsewhere. Use that helper rather than writing the registry host yourself.
• Images pulled by workloads inside the cluster (your test's pods) are not proxied by default. The containerd host config the framework installs (components/kubernetes/hosts.toml) only redirects the bare docker.io host to mirror.gcr.io — a public, unauthenticated mirror, not the ECR cache — because giving every pod containerd credentials for a private registry isn't solved cluster-wide.

A workload can pull through the authenticated ECR cache, but it takes two steps together, both opt-in: rewrite the pod's image reference to point at the cache prefix (same idiom as the parameter-store snippet above), and set the imagePullRegistry/imagePullUsername/imagePullPassword runner parameters so the framework attaches a matching ImagePullSecret to the pod (common/utils/kubernetes.go, NewImagePullSecret). Do one without the other and the pull fails: a rewritten reference with no secret can't authenticate, and a secret with no rewritten reference has nothing to authenticate against.

var dockerRegistry = func() string {
    reg, _ := runner.GetProfile().ParamStore().GetWithDefault(parameters.ImagePullRegistry, "")
    if reg != "" {
        return strings.SplitN(reg, ",", 2)[0] + "/dockerhub"
    }
    return "docker.io"
}()

var cuda12DockerImage = fmt.Sprintf("%s/nvidia/cuda:%s-base-ubuntu22.04", dockerRegistry, defaultCudaVersion)

dda inv new-e2e-tests.run --targets ./tests/gpu \
  -c ddagent:imagePullRegistry=669783387624.dkr.ecr.us-east-1.amazonaws.com \
  -c ddagent:imagePullUsername=AWS \
  -c ddagent:imagePullPassword=$(aws-vault exec sso-agent-qa-read-only -- aws ecr get-login-password)

Prebake into the machine image¶

If a test needs a specific tool or system package in a VM, avoid installing it at test time at all costs. The better alternative is to:

• Use a Docker image containing that tool that you can then pull the ECR pull-through cache.
• "Bake" the dependency into the VM machine image (AMI).

We already provide -e2e AMI variants for most OSes for this purpose, that contain some common dependencies: docker itself, jq, python3 etc. These -e2e variants are already the default for tests running on AWS.

See Custom AMIs for more details on how to do this.

S3 artifact bucket¶

For a more miscellaneous third-party artifact, vendor it into s3://agent-e2e-s3-bucket and pull it from there. Hosts authenticate with their instance profile, so no credentials are involved in the test.

err := v.Env().RemoteHost.HostArtifactClient.Get("toto.txt", "toto.txt")

test/new-e2e/examples/vmenv_artifacts_test.go is the runnable example. Objects are namespaced by owning team — windows-products/xperf-5.0.8169.zip, processes/DiskSpd.zip — and the bucket URL is prepended for you, so the path in Get is the key, not a URL.

Azure and GCP¶

None of the three mechanisms above exist on Azure or GCP:

• Container images: no pull-through cache (InternalDockerhubMirror() falls back to plain registry-1.docker.io). Use the most reliable mirror available instead — mirror.gcr.io on GCP; Azure has no blessed equivalent, ask #agent-devx-help.
• Prebaked images: no -e2e AMI equivalent. docker.InstallDocker / docker.InstallCompose (test/e2e-framework/components/docker/install.go) install Docker at provision time as the accepted exception for these clouds.
• S3 artifact bucket: HostArtifactClient returns not implemented (see above), with no fallback.