Skip to content

Prometheus


Prometheus is an open source monitoring system for timeseries metric data. Many Datadog integrations collect metrics based on Prometheus exported data sets.

Prometheus-based integrations use the OpenMetrics exposition format to collect metrics.

Interface

All functionality is exposed by the OpenMetricsBaseCheck and OpenMetricsScraperMixin classes.

datadog_checks.base.checks.openmetrics.base_check.OpenMetricsBaseCheck (OpenMetricsScraperMixin, AgentCheck)

OpenMetricsBaseCheck is a class that helps scrape endpoints that emit Prometheus metrics only with YAML configurations.

Minimal example configuration:

instances:
- prometheus_url: http://example.com/endpoint
    namespace: "foobar"
    metrics:
    - bar
    - foo

Agent 6 signature:

OpenMetricsBaseCheck(name, init_config, instances, default_instances=None, default_namespace=None)
__init__(self, *args, **kwargs) special

The base class for any Prometheus-based integration.

Source code in
def __init__(self, *args, **kwargs):
    """
    The base class for any Prometheus-based integration.
    """
    args = list(args)
    default_instances = kwargs.pop('default_instances', None) or {}
    default_namespace = kwargs.pop('default_namespace', None)

    legacy_kwargs_in_args = args[4:]
    del args[4:]

    if len(legacy_kwargs_in_args) > 0:
        default_instances = legacy_kwargs_in_args[0] or {}
    if len(legacy_kwargs_in_args) > 1:
        default_namespace = legacy_kwargs_in_args[1]

    super(OpenMetricsBaseCheck, self).__init__(*args, **kwargs)
    self.config_map = {}
    self._http_handlers = {}
    self.default_instances = default_instances
    self.default_namespace = default_namespace

    # pre-generate the scraper configurations

    if 'instances' in kwargs:
        instances = kwargs['instances']
    elif len(args) == 4:
        # instances from agent 5 signature
        instances = args[3]
    elif isinstance(args[2], (tuple, list)):
        # instances from agent 6 signature
        instances = args[2]
    else:
        instances = None

    if instances is not None:
        for instance in instances:
            possible_urls = instance.get('possible_prometheus_urls')
            if possible_urls is not None:
                for url in possible_urls:
                    try:
                        new_instance = deepcopy(instance)
                        new_instance.update({'prometheus_url': url})
                        scraper_config = self.get_scraper_config(new_instance)
                        response = self.send_request(url, scraper_config)
                        response.raise_for_status()
                        instance['prometheus_url'] = url
                        self.get_scraper_config(instance)
                        break
                    except (IOError, requests.HTTPError, requests.exceptions.SSLError) as e:
                        self.log.info("Couldn't connect to %s: %s, trying next possible URL.", url, str(e))
                else:
                    self.log.error(
                        "The agent could connect to none of the following URL: %s.",
                        possible_urls,
                    )
            else:
                self.get_scraper_config(instance)
check(self, instance)
Source code in
def check(self, instance):
    # Get the configuration for this specific instance
    scraper_config = self.get_scraper_config(instance)

    # We should be specifying metrics for checks that are vanilla OpenMetricsBaseCheck-based
    if not scraper_config['metrics_mapper']:
        raise CheckException(
            "You have to collect at least one metric from the endpoint: {}".format(scraper_config['prometheus_url'])
        )

    self.process(scraper_config)
get_scraper_config(self, instance)

Validates the instance configuration and creates a scraper configuration for a new instance. If the endpoint already has a corresponding configuration, return the cached configuration.

Source code in
def get_scraper_config(self, instance):
    """
    Validates the instance configuration and creates a scraper configuration for a new instance.
    If the endpoint already has a corresponding configuration, return the cached configuration.
    """
    endpoint = instance.get('prometheus_url')

    if endpoint is None:
        raise CheckException("Unable to find prometheus URL in config file.")

    # If we've already created the corresponding scraper configuration, return it
    if endpoint in self.config_map:
        return self.config_map[endpoint]

    # Otherwise, we create the scraper configuration
    config = self.create_scraper_configuration(instance)

    # Add this configuration to the config_map
    self.config_map[endpoint] = config

    return config

datadog_checks.base.checks.openmetrics.mixins.OpenMetricsScraperMixin

create_scraper_configuration(self, instance=None)

Creates a scraper configuration.

If instance does not specify a value for a configuration option, the value will default to the init_config. Otherwise, the default_instance value will be used.

A default mixin configuration will be returned if there is no instance.

Source code in
def create_scraper_configuration(self, instance=None):
    """
    Creates a scraper configuration.

    If instance does not specify a value for a configuration option, the value will default to the `init_config`.
    Otherwise, the `default_instance` value will be used.

    A default mixin configuration will be returned if there is no instance.
    """
    if 'openmetrics_endpoint' in instance:
        raise CheckException('The setting `openmetrics_endpoint` is only available for Agent version 7 or later')

    # We can choose to create a default mixin configuration for an empty instance
    if instance is None:
        instance = {}

    # Supports new configuration options
    config = copy.deepcopy(instance)

    # Set the endpoint
    endpoint = instance.get('prometheus_url')
    if instance and endpoint is None:
        raise CheckException("You have to define a prometheus_url for each prometheus instance")

    config['prometheus_url'] = endpoint

    # `NAMESPACE` is the prefix metrics will have. Need to be hardcoded in the
    # child check class.
    namespace = instance.get('namespace')
    # Check if we have a namespace
    if instance and namespace is None:
        if self.default_namespace is None:
            raise CheckException("You have to define a namespace for each prometheus check")
        namespace = self.default_namespace

    config['namespace'] = namespace

    # Retrieve potential default instance settings for the namespace
    default_instance = self.default_instances.get(namespace, {})

    # `metrics_mapper` is a dictionary where the keys are the metrics to capture
    # and the values are the corresponding metrics names to have in datadog.
    # Note: it is empty in the parent class but will need to be
    # overloaded/hardcoded in the final check not to be counted as custom metric.

    # Metrics are preprocessed if no mapping
    metrics_mapper = {}
    # We merge list and dictionaries from optional defaults & instance settings
    metrics = default_instance.get('metrics', []) + instance.get('metrics', [])
    for metric in metrics:
        if isinstance(metric, string_types):
            metrics_mapper[metric] = metric
        else:
            metrics_mapper.update(metric)

    config['metrics_mapper'] = metrics_mapper

    # `_wildcards_re` is a Pattern object used to match metric wildcards
    config['_wildcards_re'] = None

    wildcards = set()
    for metric in config['metrics_mapper']:
        if "*" in metric:
            wildcards.add(translate(metric))

    if wildcards:
        config['_wildcards_re'] = compile('|'.join(wildcards))

    # `prometheus_metrics_prefix` allows to specify a prefix that all
    # prometheus metrics should have. This can be used when the prometheus
    # endpoint we are scrapping allows to add a custom prefix to it's
    # metrics.
    config['prometheus_metrics_prefix'] = instance.get(
        'prometheus_metrics_prefix', default_instance.get('prometheus_metrics_prefix', '')
    )

    # `label_joins` holds the configuration for extracting 1:1 labels from
    # a target metric to all metric matching the label, example:
    # self.label_joins = {
    #     'kube_pod_info': {
    #         'labels_to_match': ['pod'],
    #         'labels_to_get': ['node', 'host_ip']
    #     }
    # }
    config['label_joins'] = default_instance.get('label_joins', {})
    config['label_joins'].update(instance.get('label_joins', {}))

    # `_label_mapping` holds the additionals label info to add for a specific
    # label value, example:
    # self._label_mapping = {
    #     'pod': {
    #         'dd-agent-9s1l1': {
    #             "node": "yolo",
    #             "host_ip": "yey"
    #         }
    #     }
    # }
    config['_label_mapping'] = {}

    # `_active_label_mapping` holds a dictionary of label values found during the run
    # to cleanup the label_mapping of unused values, example:
    # self._active_label_mapping = {
    #     'pod': {
    #         'dd-agent-9s1l1': True
    #     }
    # }
    config['_active_label_mapping'] = {}

    # `_watched_labels` holds the sets of labels to watch for enrichment
    config['_watched_labels'] = {}

    config['_dry_run'] = True

    # Some metrics are ignored because they are duplicates or introduce a
    # very high cardinality. Metrics included in this list will be silently
    # skipped without a 'Unable to handle metric' debug line in the logs
    config['ignore_metrics'] = instance.get('ignore_metrics', default_instance.get('ignore_metrics', []))
    config['_ignored_metrics'] = set()

    # `_ignored_re` is a Pattern object used to match ignored metric patterns
    config['_ignored_re'] = None
    ignored_patterns = set()

    # Separate ignored metric names and ignored patterns in different sets for faster lookup later
    for metric in config['ignore_metrics']:
        if '*' in metric:
            ignored_patterns.add(translate(metric))
        else:
            config['_ignored_metrics'].add(metric)

    if ignored_patterns:
        config['_ignored_re'] = compile('|'.join(ignored_patterns))

    # Ignore metrics based on label keys or specific label values
    config['ignore_metrics_by_labels'] = instance.get(
        'ignore_metrics_by_labels', default_instance.get('ignore_metrics_by_labels', {})
    )

    # If you want to send the buckets as tagged values when dealing with histograms,
    # set send_histograms_buckets to True, set to False otherwise.
    config['send_histograms_buckets'] = is_affirmative(
        instance.get('send_histograms_buckets', default_instance.get('send_histograms_buckets', True))
    )

    # If you want the bucket to be non cumulative and to come with upper/lower bound tags
    # set non_cumulative_buckets to True, enabled when distribution metrics are enabled.
    config['non_cumulative_buckets'] = is_affirmative(
        instance.get('non_cumulative_buckets', default_instance.get('non_cumulative_buckets', False))
    )

    # Send histograms as datadog distribution metrics
    config['send_distribution_buckets'] = is_affirmative(
        instance.get('send_distribution_buckets', default_instance.get('send_distribution_buckets', False))
    )

    # Non cumulative buckets are mandatory for distribution metrics
    if config['send_distribution_buckets'] is True:
        config['non_cumulative_buckets'] = True

    # If you want to send `counter` metrics as monotonic counts, set this value to True.
    # Set to False if you want to instead send those metrics as `gauge`.
    config['send_monotonic_counter'] = is_affirmative(
        instance.get('send_monotonic_counter', default_instance.get('send_monotonic_counter', True))
    )

    # If you want `counter` metrics to be submitted as both gauges and monotonic counts. Set this value to True.
    config['send_monotonic_with_gauge'] = is_affirmative(
        instance.get('send_monotonic_with_gauge', default_instance.get('send_monotonic_with_gauge', False))
    )

    config['send_distribution_counts_as_monotonic'] = is_affirmative(
        instance.get(
            'send_distribution_counts_as_monotonic',
            default_instance.get('send_distribution_counts_as_monotonic', False),
        )
    )

    config['send_distribution_sums_as_monotonic'] = is_affirmative(
        instance.get(
            'send_distribution_sums_as_monotonic',
            default_instance.get('send_distribution_sums_as_monotonic', False),
        )
    )

    # If the `labels_mapper` dictionary is provided, the metrics labels names
    # in the `labels_mapper` will use the corresponding value as tag name
    # when sending the gauges.
    config['labels_mapper'] = default_instance.get('labels_mapper', {})
    config['labels_mapper'].update(instance.get('labels_mapper', {}))
    # Rename bucket "le" label to "upper_bound"
    config['labels_mapper']['le'] = 'upper_bound'

    # `exclude_labels` is an array of labels names to exclude. Those labels
    # will just not be added as tags when submitting the metric.
    config['exclude_labels'] = default_instance.get('exclude_labels', []) + instance.get('exclude_labels', [])

    # `type_overrides` is a dictionary where the keys are prometheus metric names
    # and the values are a metric type (name as string) to use instead of the one
    # listed in the payload. It can be used to force a type on untyped metrics.
    # Note: it is empty in the parent class but will need to be
    # overloaded/hardcoded in the final check not to be counted as custom metric.
    config['type_overrides'] = default_instance.get('type_overrides', {})
    config['type_overrides'].update(instance.get('type_overrides', {}))

    # `_type_override_patterns` is a dictionary where we store Pattern objects
    # that match metric names as keys, and their corresponding metric type overrides as values.
    config['_type_override_patterns'] = {}

    with_wildcards = set()
    for metric, type in iteritems(config['type_overrides']):
        if '*' in metric:
            config['_type_override_patterns'][compile(translate(metric))] = type
            with_wildcards.add(metric)

    # cleanup metric names with wildcards from the 'type_overrides' dict
    for metric in with_wildcards:
        del config['type_overrides'][metric]

    # Some metrics are retrieved from different hosts and often
    # a label can hold this information, this transfers it to the hostname
    config['label_to_hostname'] = instance.get('label_to_hostname', default_instance.get('label_to_hostname', None))

    # In combination to label_as_hostname, allows to add a common suffix to the hostnames
    # submitted. This can be used for instance to discriminate hosts between clusters.
    config['label_to_hostname_suffix'] = instance.get(
        'label_to_hostname_suffix', default_instance.get('label_to_hostname_suffix', None)
    )

    # Add a 'health' service check for the prometheus endpoint
    config['health_service_check'] = is_affirmative(
        instance.get('health_service_check', default_instance.get('health_service_check', True))
    )

    # Can either be only the path to the certificate and thus you should specify the private key
    # or it can be the path to a file containing both the certificate & the private key
    config['ssl_cert'] = instance.get('ssl_cert', default_instance.get('ssl_cert', None))

    # Needed if the certificate does not include the private key
    #
    # /!\ The private key to your local certificate must be unencrypted.
    # Currently, Requests does not support using encrypted keys.
    config['ssl_private_key'] = instance.get('ssl_private_key', default_instance.get('ssl_private_key', None))

    # The path to the trusted CA used for generating custom certificates
    config['ssl_ca_cert'] = instance.get('ssl_ca_cert', default_instance.get('ssl_ca_cert', None))

    # Whether or not to validate SSL certificates
    config['ssl_verify'] = is_affirmative(instance.get('ssl_verify', default_instance.get('ssl_verify', True)))

    # Extra http headers to be sent when polling endpoint
    config['extra_headers'] = default_instance.get('extra_headers', {})
    config['extra_headers'].update(instance.get('extra_headers', {}))

    # Timeout used during the network request
    config['prometheus_timeout'] = instance.get(
        'prometheus_timeout', default_instance.get('prometheus_timeout', 10)
    )

    # Authentication used when polling endpoint
    config['username'] = instance.get('username', default_instance.get('username', None))
    config['password'] = instance.get('password', default_instance.get('password', None))

    # Custom tags that will be sent with each metric
    config['custom_tags'] = instance.get('tags', [])

    # Some tags can be ignored to reduce the cardinality.
    # This can be useful for cost optimization in containerized environments
    # when the openmetrics check is configured to collect custom metrics.
    # Even when the Agent's Tagger is configured to add low-cardinality tags only,
    # some tags can still generate unwanted metric contexts (e.g pod annotations as tags).
    ignore_tags = instance.get('ignore_tags', default_instance.get('ignore_tags', []))
    if ignore_tags:
        ignored_tags_re = compile('|'.join(set(ignore_tags)))
        config['custom_tags'] = [tag for tag in config['custom_tags'] if not ignored_tags_re.search(tag)]

    # Additional tags to be sent with each metric
    config['_metric_tags'] = []

    # List of strings to filter the input text payload on. If any line contains
    # one of these strings, it will be filtered out before being parsed.
    # INTERNAL FEATURE, might be removed in future versions
    config['_text_filter_blacklist'] = []

    # Whether or not to use the service account bearer token for authentication
    # if 'bearer_token_path' is not set, we use /var/run/secrets/kubernetes.io/serviceaccount/token
    # as a default path to get the token.
    config['bearer_token_auth'] = is_affirmative(
        instance.get('bearer_token_auth', default_instance.get('bearer_token_auth', False))
    )

    # Can be used to get a service account bearer token from files
    # other than /var/run/secrets/kubernetes.io/serviceaccount/token
    # 'bearer_token_auth' should be enabled.
    config['bearer_token_path'] = instance.get('bearer_token_path', default_instance.get('bearer_token_path', None))

    # The service account bearer token to be used for authentication
    config['_bearer_token'] = self._get_bearer_token(config['bearer_token_auth'], config['bearer_token_path'])

    config['telemetry'] = is_affirmative(instance.get('telemetry', default_instance.get('telemetry', False)))

    # The metric name services use to indicate build information
    config['metadata_metric_name'] = instance.get(
        'metadata_metric_name', default_instance.get('metadata_metric_name')
    )

    # Map of metadata key names to label names
    config['metadata_label_map'] = instance.get(
        'metadata_label_map', default_instance.get('metadata_label_map', {})
    )

    config['_default_metric_transformers'] = {}
    if config['metadata_metric_name'] and config['metadata_label_map']:
        config['_default_metric_transformers'][config['metadata_metric_name']] = self.transform_metadata

    # Whether or not to enable flushing of the first value of monotonic counts
    config['_successfully_executed'] = False

    return config
parse_metric_family(self, response, scraper_config)

Parse the MetricFamily from a valid requests.Response object to provide a MetricFamily object. The text format uses iter_lines() generator.

Source code in
def parse_metric_family(self, response, scraper_config):
    """
    Parse the MetricFamily from a valid `requests.Response` object to provide a MetricFamily object.
    The text format uses iter_lines() generator.
    """
    if response.encoding is None:
        response.encoding = 'utf-8'
    input_gen = response.iter_lines(decode_unicode=True)
    if scraper_config['_text_filter_blacklist']:
        input_gen = self._text_filter_input(input_gen, scraper_config)

    for metric in text_fd_to_metric_families(input_gen):
        self._send_telemetry_counter(
            self.TELEMETRY_COUNTER_METRICS_INPUT_COUNT, len(metric.samples), scraper_config
        )
        type_override = scraper_config['type_overrides'].get(metric.name)
        if type_override:
            metric.type = type_override
        elif scraper_config['_type_override_patterns']:
            for pattern, new_type in iteritems(scraper_config['_type_override_patterns']):
                if pattern.search(metric.name):
                    metric.type = new_type
                    break
        if metric.type not in self.METRIC_TYPES:
            continue
        metric.name = self._remove_metric_prefix(metric.name, scraper_config)
        yield metric
poll(self, scraper_config, headers=None)

Returns a valid requests.Response, otherwise raise requests.HTTPError if the status code of the response isn't valid - see response.raise_for_status()

The caller needs to close the requests.Response.

Custom headers can be added to the default headers.

Source code in
def poll(self, scraper_config, headers=None):
    """
    Returns a valid `requests.Response`, otherwise raise requests.HTTPError if the status code of the
    response isn't valid - see `response.raise_for_status()`

    The caller needs to close the requests.Response.

    Custom headers can be added to the default headers.
    """
    endpoint = scraper_config.get('prometheus_url')

    # Should we send a service check for when we make a request
    health_service_check = scraper_config['health_service_check']
    service_check_name = self._metric_name_with_namespace('prometheus.health', scraper_config)
    service_check_tags = ['endpoint:{}'.format(endpoint)]
    service_check_tags.extend(scraper_config['custom_tags'])

    try:
        response = self.send_request(endpoint, scraper_config, headers)
    except requests.exceptions.SSLError:
        self.log.error("Invalid SSL settings for requesting %s endpoint", endpoint)
        raise
    except IOError:
        if health_service_check:
            self.service_check(service_check_name, AgentCheck.CRITICAL, tags=service_check_tags)
        raise
    try:
        response.raise_for_status()
        if health_service_check:
            self.service_check(service_check_name, AgentCheck.OK, tags=service_check_tags)
        return response
    except requests.HTTPError:
        response.close()
        if health_service_check:
            self.service_check(service_check_name, AgentCheck.CRITICAL, tags=service_check_tags)
        raise
process(self, scraper_config, metric_transformers=None)

Polls the data from Prometheus and submits them as Datadog metrics. endpoint is the metrics endpoint to use to poll metrics from Prometheus

Note that if the instance has a tags attribute, it will be pushed automatically as additional custom tags and added to the metrics

Source code in
def process(self, scraper_config, metric_transformers=None):
    """
    Polls the data from Prometheus and submits them as Datadog metrics.
    `endpoint` is the metrics endpoint to use to poll metrics from Prometheus

    Note that if the instance has a `tags` attribute, it will be pushed
    automatically as additional custom tags and added to the metrics
    """
    transformers = scraper_config['_default_metric_transformers'].copy()
    if metric_transformers:
        transformers.update(metric_transformers)

    for metric in self.scrape_metrics(scraper_config):
        self.process_metric(metric, scraper_config, metric_transformers=transformers)

    scraper_config['_successfully_executed'] = True
process_metric(self, metric, scraper_config, metric_transformers=None)

Handle a Prometheus metric according to the following flow: - search scraper_config['metrics_mapper'] for a prometheus.metric to datadog.metric mapping - call check method with the same name as the metric - log info if none of the above worked

metric_transformers is a dict of <metric name>:<function to run when the metric name is encountered>

Source code in
def process_metric(self, metric, scraper_config, metric_transformers=None):
    """
    Handle a Prometheus metric according to the following flow:
    - search `scraper_config['metrics_mapper']` for a prometheus.metric to datadog.metric mapping
    - call check method with the same name as the metric
    - log info if none of the above worked

    `metric_transformers` is a dict of `<metric name>:<function to run when the metric name is encountered>`
    """
    # If targeted metric, store labels
    self._store_labels(metric, scraper_config)

    if scraper_config['ignore_metrics']:
        if metric.name in scraper_config['_ignored_metrics']:
            self._send_telemetry_counter(
                self.TELEMETRY_COUNTER_METRICS_IGNORE_COUNT, len(metric.samples), scraper_config
            )
            return  # Ignore the metric

        if scraper_config['_ignored_re'] and scraper_config['_ignored_re'].search(metric.name):
            # Metric must be ignored
            scraper_config['_ignored_metrics'].add(metric.name)
            self._send_telemetry_counter(
                self.TELEMETRY_COUNTER_METRICS_IGNORE_COUNT, len(metric.samples), scraper_config
            )
            return  # Ignore the metric

    self._send_telemetry_counter(self.TELEMETRY_COUNTER_METRICS_PROCESS_COUNT, len(metric.samples), scraper_config)

    if self._filter_metric(metric, scraper_config):
        return  # Ignore the metric

    # Filter metric to see if we can enrich with joined labels
    self._join_labels(metric, scraper_config)

    if scraper_config['_dry_run']:
        return

    try:
        self.submit_openmetric(scraper_config['metrics_mapper'][metric.name], metric, scraper_config)
    except KeyError:
        if metric_transformers is not None and metric.name in metric_transformers:
            try:
                # Get the transformer function for this specific metric
                transformer = metric_transformers[metric.name]
                transformer(metric, scraper_config)
            except Exception as err:
                self.log.warning('Error handling metric: %s - error: %s', metric.name, err)

            return
        # check for wildcards in transformers
        for transformer_name, transformer in iteritems(metric_transformers):
            if transformer_name.endswith('*') and metric.name.startswith(transformer_name[:-1]):
                transformer(metric, scraper_config, transformer_name)

        # try matching wildcards
        if scraper_config['_wildcards_re'] and scraper_config['_wildcards_re'].search(metric.name):
            self.submit_openmetric(metric.name, metric, scraper_config)
            return

        self.log.debug(
            'Skipping metric `%s` as it is not defined in the metrics mapper, '
            'has no transformer function, nor does it match any wildcards.',
            metric.name,
        )
scrape_metrics(self, scraper_config)

Poll the data from Prometheus and return the metrics as a generator.

Source code in
def scrape_metrics(self, scraper_config):
    """
    Poll the data from Prometheus and return the metrics as a generator.
    """
    response = self.poll(scraper_config)
    if scraper_config['telemetry']:
        if 'content-length' in response.headers:
            content_len = int(response.headers['content-length'])
        else:
            content_len = len(response.content)
        self._send_telemetry_gauge(self.TELEMETRY_GAUGE_MESSAGE_SIZE, content_len, scraper_config)
    try:
        # no dry run if no label joins
        if not scraper_config['label_joins']:
            scraper_config['_dry_run'] = False
        elif not scraper_config['_watched_labels']:
            watched = scraper_config['_watched_labels']
            watched['sets'] = {}
            watched['keys'] = {}
            watched['singles'] = set()
            for key, val in iteritems(scraper_config['label_joins']):
                labels = []
                if 'labels_to_match' in val:
                    labels = val['labels_to_match']
                elif 'label_to_match' in val:
                    self.log.warning("`label_to_match` is being deprecated, please use `labels_to_match`")
                    if isinstance(val['label_to_match'], list):
                        labels = val['label_to_match']
                    else:
                        labels = [val['label_to_match']]

                if labels:
                    s = frozenset(labels)
                    watched['sets'][key] = s
                    watched['keys'][key] = ','.join(s)
                    if len(labels) == 1:
                        watched['singles'].add(labels[0])

        for metric in self.parse_metric_family(response, scraper_config):
            yield metric

        # Set dry run off
        scraper_config['_dry_run'] = False
        # Garbage collect unused mapping and reset active labels
        for metric, mapping in list(iteritems(scraper_config['_label_mapping'])):
            for key in list(mapping):
                if (
                    metric in scraper_config['_active_label_mapping']
                    and key not in scraper_config['_active_label_mapping'][metric]
                ):
                    del scraper_config['_label_mapping'][metric][key]
        scraper_config['_active_label_mapping'] = {}
    finally:
        response.close()
submit_openmetric(self, metric_name, metric, scraper_config, hostname=None)

For each sample in the metric, report it as a gauge with all labels as tags except if a labels dict is passed, in which case keys are label names we'll extract and corresponding values are tag names we'll use (eg: {'node': 'node'}).

Histograms generate a set of values instead of a unique metric. send_histograms_buckets is used to specify if you want to send the buckets as tagged values when dealing with histograms.

custom_tags is an array of tag:value that will be added to the metric when sending the gauge to Datadog.

Source code in
def submit_openmetric(self, metric_name, metric, scraper_config, hostname=None):
    """
    For each sample in the metric, report it as a gauge with all labels as tags
    except if a labels `dict` is passed, in which case keys are label names we'll extract
    and corresponding values are tag names we'll use (eg: {'node': 'node'}).

    Histograms generate a set of values instead of a unique metric.
    `send_histograms_buckets` is used to specify if you want to
    send the buckets as tagged values when dealing with histograms.

    `custom_tags` is an array of `tag:value` that will be added to the
    metric when sending the gauge to Datadog.
    """
    if metric.type in ["gauge", "counter", "rate"]:
        metric_name_with_namespace = self._metric_name_with_namespace(metric_name, scraper_config)
        for sample in metric.samples:
            if self._ignore_metrics_by_label(scraper_config, metric_name, sample):
                continue

            val = sample[self.SAMPLE_VALUE]
            if not self._is_value_valid(val):
                self.log.debug("Metric value is not supported for metric %s", sample[self.SAMPLE_NAME])
                continue
            custom_hostname = self._get_hostname(hostname, sample, scraper_config)
            # Determine the tags to send
            tags = self._metric_tags(metric_name, val, sample, scraper_config, hostname=custom_hostname)
            if metric.type == "counter" and scraper_config['send_monotonic_counter']:
                self.monotonic_count(
                    metric_name_with_namespace,
                    val,
                    tags=tags,
                    hostname=custom_hostname,
                    flush_first_value=scraper_config['_successfully_executed'],
                )
            elif metric.type == "rate":
                self.rate(metric_name_with_namespace, val, tags=tags, hostname=custom_hostname)
            else:
                self.gauge(metric_name_with_namespace, val, tags=tags, hostname=custom_hostname)

                # Metric is a "counter" but legacy behavior has "send_as_monotonic" defaulted to False
                # Submit metric as monotonic_count with appended name
                if metric.type == "counter" and scraper_config['send_monotonic_with_gauge']:
                    self.monotonic_count(
                        metric_name_with_namespace + '.total',
                        val,
                        tags=tags,
                        hostname=custom_hostname,
                        flush_first_value=scraper_config['_successfully_executed'],
                    )
    elif metric.type == "histogram":
        self._submit_gauges_from_histogram(metric_name, metric, scraper_config)
    elif metric.type == "summary":
        self._submit_gauges_from_summary(metric_name, metric, scraper_config)
    else:
        self.log.error("Metric type %s unsupported for metric %s.", metric.type, metric_name)

Options

Some options can be set globally in init_config (with instances taking precedence). For complete documentation of every option, see the associated configuration templates for the instances and init_config sections.

All HTTP options are also supported.

  • prometheus_url
  • namespace
  • metrics
  • prometheus_metrics_prefix
  • health_service_check
  • label_to_hostname
  • label_joins
  • labels_mapper
  • type_overrides
  • send_histograms_buckets
  • send_distribution_buckets
  • send_monotonic_counter
  • send_monotonic_with_gauge
  • send_distribution_counts_as_monotonic
  • send_distribution_sums_as_monotonic
  • exclude_labels
  • bearer_token_auth
  • bearer_token_path
  • ignore_metrics

Prometheus to Datadog metric types

The Openmetrics Base Check supports various configurations for submitting Prometheus metrics to Datadog. We currently support Prometheus gauge, counter, histogram, and summary metric types.

Gauge

A gauge metric represents a single numerical value that can arbitrarily go up or down.

Prometheus gauge metrics are submitted as Datadog gauge metrics.

Counter

A Prometheus counter is a cumulative metric that represents a single monotonically increasing counter whose value can only increase or be reset to zero on restart.

Config Option Value Datadog Metric Submitted
send_monotonic_counter true (default) monotonic_count
  false gauge

Histogram

A Prometheus histogram samples observations and counts them in configurable buckets along with a sum of all observed values.

Histogram metrics ending in:

  • _sum represent the total sum of all observed values. Generally sums are like counters but it's also possible for a negative observation which would not behave like a typical always increasing counter.
  • _count represent the total number of events that have been observed.
  • _bucket represent the cumulative counters for the observation buckets. Note that buckets are only submitted if send_histogram_buckets is enabled.
Subtype Config Option Value Datadog Metric Submitted
  send_distribution_buckets true The entire histogram can be submitted as a single distribution metric. If the option is enabled, none of the subtype metrics will be submitted.
_sum send_distribution_sums_as_monotonic false (default) gauge
    true monotonic_count
_count send_distribution_counts_as_monotonic false (default) gauge
    true monotonic_count
_bucket non_cumulative_buckets false (default) gauge
    true monotonic_count under .count metric name if send_distribution_counts_as_monotonic is enabled. Otherwise, gauge.

Summary

Prometheus summary metrics are similar to histograms but allow configurable quantiles.

Summary metrics ending in:

  • _sum represent the total sum of all observed values. Generally sums are like counters but it's also possible for a negative observation which would not behave like a typical always increasing counter.
  • _count represent the total number of events that have been observed.
  • metrics with labels like {quantile="<φ>"} represent the streaming quantiles of observed events.
Subtype Config Option Value Datadog Metric Submitted
_sum send_distribution_sums_as_monotonic false (default) gauge
    true monotonic_count
_count send_distribution_counts_as_monotonic false (default) gauge
    true monotonic_count
_quantile     gauge

Last update: July 13, 2020