Class LLMObservabilityApi

Execute an analytics aggregation over LLM Observability experimentation data. Use this endpoint to compute metrics (for example average eval scores) grouped by fields such as span_id or experiment_id.

At least one compute definition and one index must be provided.

Insert, update, and delete records in a single dataset operation. By default, a new dataset version is created when the batch is applied.

Clone a dataset, copying its current records into a new dataset within the same project.

Create an annotation queue. The name and project_id fields are required. An optional annotation_schema can be provided to define the labels for the queue. Fields such as created_by, owned_by, created_at, modified_by, and modified_at are inferred by the backend.

Add one or more interactions to an annotation queue. At least one interaction must be provided. Each interaction has a type:

• trace, experiment_trace, session: content_id references the upstream entity; the server fetches the actual content.
• display_block: omit content_id and provide the rendered content in display_block. The server generates content_id as a deterministic hash of the block list.

Items of different types can be mixed in a single request.

Create a new LLM Observability dataset within the specified project.

Append one or more records to an LLM Observability dataset.

Create a new LLM Observability experiment.

Push spans and metrics for an LLM Observability experiment.

Run an LLM inference request through the specified integration and account, returning the model response and token usage.

Create a new LLM Observability project. Returns the existing project if a name conflict occurs.

Create a new prompt (and its first version) in the LLM Observability prompt registry.

Create a new version of an existing LLM Observability prompt.

Delete an annotation queue by its ID.

Delete one or more interactions from an annotation queue.

Delete one or more annotations from an annotation queue.

Delete a custom LLM Observability evaluator configuration by its name.

Submit a request to delete LLM Observability span data matching a trace ID filter within a specified time range.

Delete one or more records from an LLM Observability dataset.

Delete one or more LLM Observability datasets within the specified project.

Delete one or more LLM Observability experiments.

Delete a patterns configuration by its ID.

Delete one or more LLM Observability projects.

Soft-delete an LLM Observability prompt. The prompt's version rows are retained, but they are no longer accessible through the public prompt registry endpoints.

Download the contents of a dataset as a CSV file. The download is streamed and includes one row per dataset record.

Retrieve all interactions (traces and sessions) and their annotations for a given annotation queue.

Returns annotated interactions across all annotation queues for the given content IDs. Results include queue metadata (ID and name) for each interaction.

Retrieve the label schema for a given annotation queue.

Retrieve a custom LLM Observability evaluator configuration by its name.

Retrieve the draft state of a dataset, including whether it is currently locked for editing and which user holds the lock.

Retrieve the patterns configuration for the organization.

Retrieve the status and step-by-step progress of the current or most recent patterns run for a configuration.

Get the latest version of an LLM Observability prompt by prompt ID.

Get the full template of a single, specific version of an LLM Observability prompt.

List annotation queues. Optionally filter by project ID or queue IDs. These parameters are mutually exclusive. If neither is provided, all queues in the organization are returned.

List all custom LLM Observability evaluator configurations for the organization.

List all records in an LLM Observability dataset, sorted by creation date, newest first.

List the active versions of a dataset. A version is created each time a dataset is referenced by an experiment run.

List all LLM Observability datasets for a project, sorted by creation date, newest first.

Retrieve spans and experiment-level summary metrics for a given experiment with cursor-based pagination.

Retrieve spans with their evaluation metrics for a given experiment. Returns spans only, with no summary metrics and no pagination. Deprecated in favor of ListLLMObsExperimentEventsV3.

Retrieve spans and experiment-level summary metrics for a given experiment. Returns the full events payload without pagination. Deprecated: use ListLLMObsExperimentEventsV3 instead.

List all LLM Observability experiments sorted by creation date, newest first.

Retrieve the list of configured accounts for the specified LLM provider integration.

Retrieve the list of models available for the specified LLM provider integration and account.

List the data points grouped into a topic. For a parent topic, points from all of its leaf topics are returned.

List all patterns configurations for the organization.

List the completed patterns runs for a configuration.

List the topics discovered by a patterns run. When no run is specified, the most recent completed run is used.

List the topics discovered by a patterns run, with the clustered points attached inline to each leaf topic. When no run is specified, the most recent completed run is used.

List all LLM Observability projects sorted by creation date, newest first.

List all versions of an LLM Observability prompt, ordered newest to oldest. If the prompt does not exist, is not registered, or is archived, the response contains an empty list.

List all LLM Observability prompts in the prompt registry for the organization.

List LLM Observability spans matching the specified filters.

Acquire the draft lock on a dataset for the calling user. The lock prevents other users from concurrently editing the dataset draft.

Restore a dataset to a previous version. The dataset's current version is bumped, and its records are replaced with the records from the specified prior version.

Search across LLM Observability experimentation entities — projects, datasets, dataset records, experiments, and experiment runs — using cursor-based pagination.

The filter.scope field controls which entity types are returned. At least one valid scope must be provided.

Returns 200 OK when all results fit in a single page. Returns 206 Partial Content with a cursor in meta.after when additional pages are available.

Search LLM Observability spans using structured filters in the request body.

Search across LLM Observability experimentation entities using offset-based (page-number) pagination. Use this endpoint when you need total page count or want to navigate to a specific page number.

The filter.scope field controls which entity types are returned. At least one valid scope must be provided.

Start a patterns run for a given configuration. The run executes asynchronously.

Release the draft lock on a dataset held by the calling user, allowing other users to edit the dataset draft.

Partially update an annotation queue. The name, description, and annotation_schema fields can be updated.

Create or replace the label schema for a given annotation queue. The label schema defines the labels annotators can apply to interactions in the queue. Label names must be unique within the queue and match the pattern ^[a-zA-Z0-9_-]+$. Each label must have a valid type: score, categorical, boolean, or text.

Create or update a custom LLM Observability evaluator configuration by its name.

Partially update an existing LLM Observability dataset within the specified project.

Update one or more existing records in an LLM Observability dataset.

Partially update an existing LLM Observability experiment.

Partially update an existing LLM Observability project.

Update the title, the description, or both, for an LLM Observability prompt.

Update the description, the feature-flag environments, or both, for a specific version of an LLM Observability prompt.

Upload records to a dataset from a file. The request is a multipart/form-data upload containing a single file part. Currently only CSV is supported. The CSV must include an input column. Optional columns are id, expected_output, metadata, and tags.

The response is a Server-Sent Events stream (text/event-stream) emitting progress updates while records are processed. The stream emits the following named events:

• progress: incremental record counts written so far.
• completed: terminal event with a JSON body containing records_created.
• error: terminal event with a JSON body containing an error message.

Create or update annotations on interactions in a queue. Each annotation is matched by interaction_id and the requesting user's identity. Results and errors in the response are linked to request items by interaction_id. Errors for individual items are returned in the errors field without blocking the rest of the batch.

Create a new patterns configuration, or update an existing one when a configuration ID is provided.