Custom AMIs¶

Many E2E tests rely on extra dependencies being installed - instead of installing them at runtime (via an apt install or similar), we prefer to "bake" them into the underlying AMI that is used for provisioning the test VMs, as installing at runtime is a big source of flakiness (see here) These images are pinned by ID in this repo (test/e2e-framework/resources/aws/platforms.json) and always associated with a specific OS Descriptior in the E2E framework (see below.)

The images themselves are built in ami-builder. This page covers the consumer side — how an image is chosen, how to add or bump one, and how to pin or inspect one.

How an AMI is chosen¶

1. ec2.WithOS(e2eos.Ubuntu2204) resolves to a Descriptor{Ubuntu, "22-04", x86_64} — a (flavor, version, architecture) triple.
2. No WithOS at all? The descriptor comes from a flavor default instead (LinuxDescriptorsDefault, WindowsDescriptorsDefault, or MacOSDescriptorsDefault).
3. Either way, aws.GetAMI(descriptor) looks that triple up as platforms.json[flavor][arch][version] and returns the pinned AMI ID.

The version field is a literal platforms.json key — hence "22-04", not "22.04".

Prebaked images and the ami-builder mapping¶

Every image in platforms.json is prebaked: ubuntu/x86_64/22-04 or windows-server/x86_64/2025 already ships common test dependencies and config. See Prebake into the machine image for what that covers.

ami-builder is where the distinction lives. Its config.yaml builds a bare image ("22-04") and a prebaked one ("22-04-e2e") from the same upstream base, and maps the prebaked build to the platforms.json key it refreshes through platforms_key:

ubuntu:
  x86_64:
    "22-04-e2e":
      platforms_key: "22-04"   # refreshes platforms.json ubuntu/x86_64/22-04
      packer:
        provisioner_script: provision-e2e-apt

Entries without platforms_key refresh the key with their own name, e.g. redhat/x86_64/9-selinux-npm. Purpose-built images keep a descriptive suffix (24-04-kind, 9-selinux-npm, 22-04-gpu).

Adding or bumping an image¶

The build lives in ami-builder; this repo only holds the resulting ID.

1. Make the image change in ami-builder under ami/images/e2e/ and open a PR there.
2. Run the manual build:<os>-<arch>-<version> job in that branch's child pipeline. The build jobs are manual-only, and a feature branch produces a real usable AMI even without merging.
3. Read the ID out of the job log:

   Sharing new AMI ami-0123456789abcdef0 to account datadog-agent-sandbox
   

4. Update test/e2e-framework/resources/aws/platforms.json in this repo with that ID. Trigger a full pipeline on that PR — an image change can affect any suite, not only the ones whose files you touched.

Adding a new image¶

To add a new custom AMI, you also need to tell the e2e framework about it: this is done by creating a new OS descriptor alongside the platforms.json update 1. Update platforms.json, adding your new key in the appropriate spot. Name it after the OS version (plus a descriptive suffix for a purpose-built image).

1. Use that descriptor in your test: ec2.WithOS(e2eos.yourNewDescriptor)
2. components/os/<family>_descriptors.go — create a matching descriptor (Ubuntu2404Kind style naming, mirroring the key).

Pinning one test to a specific AMI¶

ec2.WithAMI skips resolution altogether:

ec2.WithAMI(
    "ami-04e7f0e0bde783f77", // https://gitlab.ddbuild.io/DataDog/ami-builder/-/jobs/1232214462
    compos.AmazonLinux2,
    compos.AMD64Arch,
)

The descriptor and architecture arguments must still match the image — they drive package-manager and path selection, not the lookup.

This is an escape hatch for a one-off image, as in test/new-e2e/tests/npm/ec2_1host_selinux_test.go. Always leave the build-job link in a comment, as above: an unattributed AMI ID cannot be rebuilt or audited. Do not reach for it to avoid getting a dependency prebaked properly — the ID will never be bumped by the bot and will rot.

Inspecting an AMI¶

To poke at an image by hand — check that a tool really is installed, look at a service's state:

aws-vault exec sso-agent-qa-account-admin-8h -- \
  dda inv ami.launch-instance --ami-id ami-0123456789abcdef0 --key-name <your-key>
# then ssh (or RDP) to the printed private IP

dda inv ami.create-ami / ami.delete-ami can snapshot such an instance back into an AMI.

Windows¶

Windows images need a few extra precautions when building or bumping one. Skip this section unless you are touching Windows E2E infrastructure.

• SSH bootstrap runs from EC2 user data. A fresh Windows instance has no SSH server; test/e2e-framework/components/os/scripts/setup-ssh.ps1 (wired up in scenarios/aws/ec2/os_win.go) installs OpenSSH and the test's key on boot. If a Windows host is unreachable, read this script first.
• A Packer build must end with EC2Launch.exe reset -c (or its v1 equivalent, Scripts\InitializeInstance.ps1 -Schedule). EC2Launch tracks which boot-time setup tasks it has already run in a marker file, and Packer snapshots that marker along with the disk — so without a reset, every instance launched from the image thinks setup already happened and skips it. The task that actually bites is extendRootPartition: skip it, and C: never grows to the volume size the test asked for.
• Build custom images from the EC2LaunchV2-* base AMIs, not the plain Windows_Server-* ones. The plain bases ship EC2Launch v1, which bakes its IMDS route into the image itself; an image built in one subnet and launched in another can lose IMDS entirely, which silently breaks user data and SSH bootstrap with it.
• Never set <persist>true</persist> on the build-time bootstrap script. Only the consumer-facing user data (the one setup-ssh.ps1 ships as) should persist. On EC2Launch v1, a persisted build-time script becomes a permanent scheduled task that re-runs on every instance launched from the image, fighting the real bootstrap.