Datadog Test Optimization for Ruby

Datadog's Ruby Library for instrumenting your tests. Learn more on our official website and check out our documentation for this library.

Features

• Test Visibility - collect metrics and results for your tests
• Flaky test management - track, triage, and remediate flaky tests across your organization. Quarantine or disable problematic tests to keep known flakes from breaking builds, and create cases and Jira issues to track work toward fixes.
• Auto test retries - retrying failing tests up to N times to avoid failing your build due to flaky tests
• Early flake detection - Datadog’s test flakiness solution that identifies flakes early by running newly added tests multiple times
• Test impact analysis - save time by selectively running only tests affected by code changes
• Test Health - The Test Health dashboard provides analytics to help teams manage and optimize their testing in CI. This includes sections showing the current impact of test flakiness and how Test Optimization is mitigating these problems.
• Inspect your tests' logs in Datadog
• Enhance developer workflows
• Add custom measures to your tests
• Browser tests integration with Datadog RUM

Setup

• Test visibility setup
• Test impact analysis setup (test visibility setup is required before setting up test impact analysis)

Upgrade from ddtrace v1.x

If you used test visibility for Ruby with ddtrace gem, check out our upgrade guide.

Contributing

See development guide, static typing guide and contributing guidelines.

Code of Conduct

Everyone interacting in the Datadog::CI project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the code of conduct.

Test automation exclusions

Protect selected tests from Datadog quarantine and disabling with datadog_prevent_failure_suppression. Exclude selected tests from Datadog Auto Test Retries, Early Flake Detection, and Attempt to Fix with datadog_no_auto_retry. Both declarations may be applied together. They preserve explicit skips, Test Impact Analysis selection, and framework-native retries.

RSpec uses example metadata:

it "checks a critical behavior", datadog_prevent_failure_suppression: true, datadog_no_auto_retry: true do
  expect(critical_behavior).to be_valid
end

Minitest exposes per-method declarations after the test method is defined:

class CriticalTest < Minitest::Test
  def test_critical_behavior
    assert critical_behavior.valid?
  end

  datadog_prevent_failure_suppression :test_critical_behavior
  datadog_no_auto_retry :test_critical_behavior
end

Undefined method names are ignored with a warning.

Cucumber uses scenario tags, including scenario outlines:

@datadog_prevent_failure_suppression @datadog_no_auto_retry
Scenario: A critical behavior
  Then the critical behavior succeeds

Load Datadog's instrumentation before defining these tests. Annotated executions report test.test_management.prevent_failure_suppression: true and/or test.test_management.no_auto_retry: true. Backend-assigned management tags stay visible even when annotations change the effective execution policy. Setting a span tag directly does not apply an exclusion.