HTTP¶
Whenever you need to make HTTP requests, the base class provides a convenience member that ensures consistent behavior across all integrations. Its request methods and response objects follow the shape of the popular requests library, but the member is defined by its own interface rather than by that library. Anything this page does not document is an implementation detail and may change.
The wrapper automatically parses and uses configuration from the instance, init_config, and Agent config. Also, this is only done once during initialization and cached to reduce the overhead of every call.
For example, to make a GET request you would use:
response = self.http.get(url)
and the wrapper will pass the right things to the underlying HTTP library. All methods accept the optional keyword arguments listed in HTTPRequestOptions under Reference, such as stream.
Any method-level option will override configuration. So for example if tls_verify was set to false and you do self.http.get(url, verify=True), then SSL certificates will be verified on that particular request. You can use the keyword argument persist to override persist_connections.
There is also support for non-standard or legacy configurations with the HTTP_CONFIG_REMAPPER class attribute. For example:
class MyCheck(AgentCheck):
HTTP_CONFIG_REMAPPER = {
'disable_ssl_validation': {
'name': 'tls_verify',
'default': False,
'invert': True,
},
...
}
...
Support for Unix socket is provided via requests-unixsocket and allows making UDS requests on the unix:// scheme (not supported on Windows until Python adds support for AF_UNIX, see ticket):
url = 'unix:///var/run/docker.sock'
response = self.http.get(url)
Backend-neutral mode¶
Set the AGNOSTIC_HTTP_CLIENT class attribute to opt a check into the interface this page describes:
class MyCheck(AgentCheck):
AGNOSTIC_HTTP_CLIENT = True
With it set, every client the check creates through self.http or create_http_client(), including the clients its OpenMetrics and Prometheus scrapers build, returns responses that expose only the documented members and raises the errors listed under Errors. The fake_http test fixture requires it. Without it, clients return requests responses and raise requests exceptions unchanged.
Testing configuration¶
The fake_http fixture records the configuration each client is created with, but does not apply it: every client it creates starts with defaults. Test configuration in one of these ways:
- To test how instance settings,
init_config, andHTTP_CONFIG_REMAPPERaliases resolve, create the check withoutfake_httpand assert oncheck.http.options,check.http.tls_config, orcheck.http.get_header(). Creating the client sends no request, but running the check would send real ones. - To test configuration the check builds and passes to
create_http_client(), such as a scraper configuration, assert onfake_http.created_clients. It holds aRecordedClientwith theinstance,init_config, andremapperof each client, in creation order.
If the check reads options or tls_config while it runs, set the values on check.http before running it.
Testing TLS configuration¶
Each FakeHTTPClient starts with the real client's canonical TLS defaults. Pass tls_config=... to override specific fields; overrides, including False and None, take precedence. Each client gets its own copy of the overrides, so changing one client's TLS settings doesn't affect other clients or the caller's mapping.
Calling fake_http.create_client() creates a fresh client with defaults, not a copy of the parent's TLS settings. Changing the parent fixture's tls_config does not configure the clients the check creates.
Options¶
Some options can be set globally in init_config (with instances taking precedence). For complete documentation of every option, see the associated configuration templates for the instances and init_config sections.
Disabling authentication¶
Call self.http.disable_auth() to suppress the client's configured auth (such as Basic credentials) and automatic .netrc credentials on subsequent requests, including redirects:
self.http.disable_auth()
response = self.http.get(url)
This applies to existing and newly created sessions, including requests with auth=None, without disabling other environment settings, such as proxies. It does not suppress auth_token handling or an Authorization entry in headers. Explicit per-request authentication still overrides client configuration.
Errors¶
With AGNOSTIC_HTTP_CLIENT set, a failed request raises one of the backend-neutral types from the module datadog_checks.base.utils.http_exceptions. Catch these instead:
| Exception | Parent | Raised when |
|---|---|---|
HTTPClientError | Exception | Root of the tree. Catch this to handle any HTTP failure. |
HTTPClientRequestError | HTTPClientError | The request never produced a usable response. |
HTTPClientConnectionError | HTTPClientRequestError | The connection could not be established or was lost. |
HTTPClientSSLError | HTTPClientConnectionError | The TLS handshake or certificate verification failed. |
HTTPClientInvalidURLError | HTTPClientRequestError | The URL was rejected before a connection was attempted. |
HTTPClientLineTooLongError | HTTPClientRequestError | A streamed body contained a line longer than the client's max_line_size. Also a ValueError. |
HTTPClientTimeoutError | HTTPClientRequestError | The request exceeded its configured timeout. |
HTTPClientConnectTimeoutError | HTTPClientTimeoutError | The timeout elapsed while establishing the connection. |
HTTPClientReadTimeoutError | HTTPClientTimeoutError | The timeout elapsed while waiting for response data. |
HTTPClientStatusError | HTTPClientError | An error status was raised through raise_for_status(). |
Unlike in requests, a timeout is never a connection error. Catch HTTPClientRequestError to handle both.
Errors translated from a requests exception also subclass the requests exception they replace, and therefore OSError, so that handlers in checks that have not opted in keep matching. HTTPClientLineTooLongError has no requests counterpart and subclasses ValueError instead. Opted-in checks must not rely on those bases.
For example, a check that skips optional data when its endpoint is unavailable looks like this:
from datadog_checks.base.utils.http_exceptions import HTTPClientError
try:
response = self.http.get(url)
response.raise_for_status()
except HTTPClientError as e:
self.log.debug('Could not retrieve version metadata: %s', e)
return
A few failures fall outside this tree, so handle them separately from the types listed here:
- Calling
response.json()on a body that is not valid JSON raises the standard library'sjson.JSONDecodeError, a subclass ofValueError. - Errors that an
auth_tokenreader or writer raises, other than HTTP failures, propagate unchanged.
Reference¶
Opted-in checks may use the members documented below. To annotate clients and responses, import HTTPClient and HTTPResponse from datadog_checks.base.utils.http_protocol.
datadog_checks.base.utils.http_protocol.HTTPClient ¶
Source code in datadog_checks_base/datadog_checks/base/utils/http_protocol.py
class HTTPClient(Protocol):
options: dict[str, Any]
"""Default request options. Per-request options override them.
Values use the format of `requests`, the current backend: `timeout` is `(connect, read)` and basic `auth` is
`(username, password)`. Removing a value here does not remove it from an open session, except for headers
deleted from `options['headers']` in place. Use `disable_auth()` to stop sending credentials.
"""
tls_config: dict[str, Any]
"""TLS settings for callers that build their own SSL context.
Keys are the instance's `tls_*` options, such as `tls_verify`, with defaults and remapper aliases applied.
"""
trust_env: bool
ignore_tls_warning: bool
persist_connections: bool
max_line_size: int | None
"""Longest line, in bytes, allowed when reading a streamed response body, or `None` for no limit.
Reading a longer line raises `HTTPClientLineTooLongError`.
"""
def get(self, url: str, **options: Unpack[HTTPRequestOptions]) -> HTTPResponse:
"""Perform a GET request using `HTTPRequestOptions`."""
...
def post(self, url: str, **options: Unpack[HTTPRequestOptions]) -> HTTPResponse:
"""Perform a POST request using `HTTPRequestOptions`."""
...
def head(self, url: str, **options: Unpack[HTTPRequestOptions]) -> HTTPResponse:
"""Perform a HEAD request using `HTTPRequestOptions`."""
...
def put(self, url: str, **options: Unpack[HTTPRequestOptions]) -> HTTPResponse:
"""Perform a PUT request using `HTTPRequestOptions`."""
...
def patch(self, url: str, **options: Unpack[HTTPRequestOptions]) -> HTTPResponse:
"""Perform a PATCH request using `HTTPRequestOptions`."""
...
def delete(self, url: str, **options: Unpack[HTTPRequestOptions]) -> HTTPResponse:
"""Perform a DELETE request using `HTTPRequestOptions`."""
...
def options_method(self, url: str, **options: Unpack[HTTPRequestOptions]) -> HTTPResponse:
"""Perform an OPTIONS request using `HTTPRequestOptions`; suffixed to avoid the defaults attribute."""
...
def get_header(self, name: str, default: str | None = None) -> str | None:
"""Return the wire value using case-insensitive lookup."""
...
def set_header(self, name: str, value: str) -> None:
"""Replace every case-insensitive match with one header under the first matching spelling."""
...
def disable_auth(self) -> None:
"""Suppress configured `auth` and .netrc credentials, preserving other environment settings.
Does not suppress `auth_token` handling or an `Authorization` entry in `headers`.
"""
...
def close(self) -> None: ...
def get_cookie(self, name: str, default: str | None = None) -> str | None:
"""Return default for missing or ambiguous persisted cookies."""
...
def should_bypass_proxy(self, url: str) -> bool: ...
ignore_tls_warning instance-attribute ¶
max_line_size instance-attribute ¶
Longest line, in bytes, allowed when reading a streamed response body, or None for no limit.
Reading a longer line raises HTTPClientLineTooLongError.
options instance-attribute ¶
Default request options. Per-request options override them.
Values use the format of requests, the current backend: timeout is (connect, read) and basic auth is (username, password). Removing a value here does not remove it from an open session, except for headers deleted from options['headers'] in place. Use disable_auth() to stop sending credentials.
persist_connections instance-attribute ¶
tls_config instance-attribute ¶
TLS settings for callers that build their own SSL context.
Keys are the instance's tls_* options, such as tls_verify, with defaults and remapper aliases applied.
trust_env instance-attribute ¶
close() ¶
Source code in datadog_checks_base/datadog_checks/base/utils/http_protocol.py
def close(self) -> None: ...
delete(url, **options) ¶
Perform a DELETE request using HTTPRequestOptions.
Source code in datadog_checks_base/datadog_checks/base/utils/http_protocol.py
def delete(self, url: str, **options: Unpack[HTTPRequestOptions]) -> HTTPResponse:
"""Perform a DELETE request using `HTTPRequestOptions`."""
...
disable_auth() ¶
Suppress configured auth and .netrc credentials, preserving other environment settings.
Does not suppress auth_token handling or an Authorization entry in headers.
Source code in datadog_checks_base/datadog_checks/base/utils/http_protocol.py
def disable_auth(self) -> None:
"""Suppress configured `auth` and .netrc credentials, preserving other environment settings.
Does not suppress `auth_token` handling or an `Authorization` entry in `headers`.
"""
...
get(url, **options) ¶
Perform a GET request using HTTPRequestOptions.
Source code in datadog_checks_base/datadog_checks/base/utils/http_protocol.py
def get(self, url: str, **options: Unpack[HTTPRequestOptions]) -> HTTPResponse:
"""Perform a GET request using `HTTPRequestOptions`."""
...
get_cookie(name, default=None) ¶
Return default for missing or ambiguous persisted cookies.
Source code in datadog_checks_base/datadog_checks/base/utils/http_protocol.py
def get_cookie(self, name: str, default: str | None = None) -> str | None:
"""Return default for missing or ambiguous persisted cookies."""
...
get_header(name, default=None) ¶
Return the wire value using case-insensitive lookup.
Source code in datadog_checks_base/datadog_checks/base/utils/http_protocol.py
def get_header(self, name: str, default: str | None = None) -> str | None:
"""Return the wire value using case-insensitive lookup."""
...
head(url, **options) ¶
Perform a HEAD request using HTTPRequestOptions.
Source code in datadog_checks_base/datadog_checks/base/utils/http_protocol.py
def head(self, url: str, **options: Unpack[HTTPRequestOptions]) -> HTTPResponse:
"""Perform a HEAD request using `HTTPRequestOptions`."""
...
options_method(url, **options) ¶
Perform an OPTIONS request using HTTPRequestOptions; suffixed to avoid the defaults attribute.
Source code in datadog_checks_base/datadog_checks/base/utils/http_protocol.py
def options_method(self, url: str, **options: Unpack[HTTPRequestOptions]) -> HTTPResponse:
"""Perform an OPTIONS request using `HTTPRequestOptions`; suffixed to avoid the defaults attribute."""
...
patch(url, **options) ¶
Perform a PATCH request using HTTPRequestOptions.
Source code in datadog_checks_base/datadog_checks/base/utils/http_protocol.py
def patch(self, url: str, **options: Unpack[HTTPRequestOptions]) -> HTTPResponse:
"""Perform a PATCH request using `HTTPRequestOptions`."""
...
post(url, **options) ¶
Perform a POST request using HTTPRequestOptions.
Source code in datadog_checks_base/datadog_checks/base/utils/http_protocol.py
def post(self, url: str, **options: Unpack[HTTPRequestOptions]) -> HTTPResponse:
"""Perform a POST request using `HTTPRequestOptions`."""
...
put(url, **options) ¶
Perform a PUT request using HTTPRequestOptions.
Source code in datadog_checks_base/datadog_checks/base/utils/http_protocol.py
def put(self, url: str, **options: Unpack[HTTPRequestOptions]) -> HTTPResponse:
"""Perform a PUT request using `HTTPRequestOptions`."""
...
set_header(name, value) ¶
Replace every case-insensitive match with one header under the first matching spelling.
Source code in datadog_checks_base/datadog_checks/base/utils/http_protocol.py
def set_header(self, name: str, value: str) -> None:
"""Replace every case-insensitive match with one header under the first matching spelling."""
...
should_bypass_proxy(url) ¶
Source code in datadog_checks_base/datadog_checks/base/utils/http_protocol.py
def should_bypass_proxy(self, url: str) -> bool: ...
datadog_checks.base.utils.http_protocol.HTTPRequestOptions ¶
Source code in datadog_checks_base/datadog_checks/base/utils/http_protocol.py
class HTTPRequestOptions(TypedDict, total=False):
params: Mapping[str, Any] | None
"""Query parameters appended to the request URL."""
headers: Mapping[str, str] | None
"""Headers for this request. Values override configured client headers with matching names."""
data: Any
"""Request body encoded according to the value's type."""
json: Any
"""JSON-serializable request body."""
auth: Any
"""Authentication applied to the request."""
cookies: Mapping[str, str] | None
"""Cookies sent with the request."""
timeout: float | tuple[float, float] | None
"""Timeout in seconds, or separate connect and read timeouts."""
allow_redirects: bool
"""Whether the request should follow redirects."""
verify: bool | str | None
"""Whether to verify TLS, or the path to a CA bundle."""
cert: str | tuple[str, str] | None
"""Client certificate, optionally paired with its private key."""
proxies: Mapping[str, str] | None
"""Proxy URLs keyed by scheme for this request. Omit to use the client's proxy settings.
When set, including to `None`, the client's `no_proxy` list is ignored, and for schemes the mapping does not name,
environment proxy variables, when trusted, take precedence over the configured proxies and `skip_proxy`.
"""
extra_headers: Mapping[str, str]
"""Headers merged after configured and per-request headers."""
stream: bool
"""Whether response body consumption should be deferred."""
persist: bool
"""Whether this request should use the persistent client."""
allow_redirects instance-attribute ¶
Whether the request should follow redirects.
auth instance-attribute ¶
Authentication applied to the request.
cert instance-attribute ¶
Client certificate, optionally paired with its private key.
cookies instance-attribute ¶
Cookies sent with the request.
data instance-attribute ¶
Request body encoded according to the value's type.
extra_headers instance-attribute ¶
Headers merged after configured and per-request headers.
headers instance-attribute ¶
Headers for this request. Values override configured client headers with matching names.
json instance-attribute ¶
JSON-serializable request body.
params instance-attribute ¶
Query parameters appended to the request URL.
persist instance-attribute ¶
Whether this request should use the persistent client.
proxies instance-attribute ¶
Proxy URLs keyed by scheme for this request. Omit to use the client's proxy settings.
When set, including to None, the client's no_proxy list is ignored, and for schemes the mapping does not name, environment proxy variables, when trusted, take precedence over the configured proxies and skip_proxy.
stream instance-attribute ¶
Whether response body consumption should be deferred.
timeout instance-attribute ¶
Timeout in seconds, or separate connect and read timeouts.
verify instance-attribute ¶
Whether to verify TLS, or the path to a CA bundle.
datadog_checks.base.utils.http_protocol.HTTPResponse ¶
Source code in datadog_checks_base/datadog_checks/base/utils/http_protocol.py
class HTTPResponse(Protocol):
@property
def status_code(self) -> int: ...
@property
def content(self) -> bytes: ...
@property
def text(self) -> str: ...
@property
def headers(self) -> Mapping[str, str]:
"""Case-insensitive headers whose iteration casing is backend-defined."""
...
@property
def encoding(self) -> str | None: ...
@encoding.setter
def encoding(self, value: str | None) -> None: ...
@property
def elapsed(self) -> timedelta: ...
@property
def cookies(self) -> Mapping[str, str]: ...
@property
def links(self) -> Mapping[str, Mapping[str, str]]: ...
@property
def url(self) -> str: ...
@property
def history(self) -> Sequence[HTTPResponse]: ...
@property
def ok(self) -> bool: ...
@property
def reason(self) -> str: ...
def json(self, **kwargs: Any) -> Any: ...
def raise_for_status(self) -> None: ...
def close(self) -> None: ...
def get_peer_cert(self, binary_form: bool = False) -> bytes | dict | None:
"""Return None for plain HTTP or a released connection."""
...
def iter_content(self, chunk_size: int | None = None, decode_unicode: bool = False) -> Iterator[bytes | str]: ...
def iter_lines(self, chunk_size: int | None = None, decode_unicode: bool = False) -> Iterator[bytes | str]:
"""Iterate the body line by line, with requests-compatible line boundaries."""
...
def __enter__(self) -> HTTPResponse: ...
def __exit__(self, exc_type: Any, exc_val: Any, exc_tb: Any) -> bool | None: ...
def __iter__(self) -> Iterator[bytes | str]: ...
content property ¶
cookies property ¶
elapsed property ¶
encoding property writable ¶
headers property ¶
Case-insensitive headers whose iteration casing is backend-defined.
history property ¶
links property ¶
ok property ¶
reason property ¶
status_code property ¶
text property ¶
url property ¶
__enter__() ¶
Source code in datadog_checks_base/datadog_checks/base/utils/http_protocol.py
def __enter__(self) -> HTTPResponse: ...
__exit__(exc_type, exc_val, exc_tb) ¶
Source code in datadog_checks_base/datadog_checks/base/utils/http_protocol.py
def __exit__(self, exc_type: Any, exc_val: Any, exc_tb: Any) -> bool | None: ...
__iter__() ¶
Source code in datadog_checks_base/datadog_checks/base/utils/http_protocol.py
def __iter__(self) -> Iterator[bytes | str]: ...
close() ¶
Source code in datadog_checks_base/datadog_checks/base/utils/http_protocol.py
def close(self) -> None: ...
get_peer_cert(binary_form=False) ¶
Return None for plain HTTP or a released connection.
Source code in datadog_checks_base/datadog_checks/base/utils/http_protocol.py
def get_peer_cert(self, binary_form: bool = False) -> bytes | dict | None:
"""Return None for plain HTTP or a released connection."""
...
iter_content(chunk_size=None, decode_unicode=False) ¶
Source code in datadog_checks_base/datadog_checks/base/utils/http_protocol.py
def iter_content(self, chunk_size: int | None = None, decode_unicode: bool = False) -> Iterator[bytes | str]: ...
iter_lines(chunk_size=None, decode_unicode=False) ¶
Iterate the body line by line, with requests-compatible line boundaries.
Source code in datadog_checks_base/datadog_checks/base/utils/http_protocol.py
def iter_lines(self, chunk_size: int | None = None, decode_unicode: bool = False) -> Iterator[bytes | str]:
"""Iterate the body line by line, with requests-compatible line boundaries."""
...
json(**kwargs) ¶
Source code in datadog_checks_base/datadog_checks/base/utils/http_protocol.py
def json(self, **kwargs: Any) -> Any: ...
raise_for_status() ¶
Source code in datadog_checks_base/datadog_checks/base/utils/http_protocol.py
def raise_for_status(self) -> None: ...
datadog_checks.base.utils.http_protocol.HTTPHeaders ¶
Immutable, case-insensitive HTTP headers.
Source code in datadog_checks_base/datadog_checks/base/utils/http_protocol.py
class HTTPHeaders(Mapping[str, str]):
"""Immutable, case-insensitive HTTP headers."""
__slots__ = ('_values',)
def __init__(self, headers: Mapping[str, str]) -> None:
values = {name.lower(): (name, value) for name, value in headers.items()}
self._values = MappingProxyType(values)
def __getitem__(self, name: str) -> str:
return self._values[name.lower()][1]
def __iter__(self) -> Iterator[str]:
return (name for name, _ in self._values.values())
def __len__(self) -> int:
return len(self._values)
def __repr__(self) -> str:
return f'{type(self).__name__}({dict(self)!r})'
Future¶
- Support for configuring cookies! Since they can be set globally, per-domain, and even per-path, the configuration may be complex if not thought out adequately. We'll discuss options for what that might look like. Only our
sparkandcisco_acichecks currently set cookies, and that is based on code logic, not configuration.