Class LLMObservabilityApi

Execute an analytics aggregation over LLM Observability experimentation data. Use this endpoint to compute metrics (for example average eval scores) grouped by fields such as span_id or experiment_id.

At least one compute definition and one index must be provided.

Insert, update, and delete records in a single dataset operation. By default, a new dataset version is created when the batch is applied.

Clone a dataset, copying its current records into a new dataset within the same project.

Create an annotation queue. The name and project_id fields are required. An optional annotation_schema can be provided to define the labels for the queue. Fields such as created_by, owned_by, created_at, modified_by, and modified_at are inferred by the backend.

Add one or more interactions to an annotation queue. At least one interaction must be provided. Each interaction has a type:

• trace, experiment_trace, session: content_id references the upstream entity; the server fetches the actual content.
• display_block: omit content_id and provide the rendered content in display_block. The server generates content_id as a deterministic hash of the block list.

Items of different types can be mixed in a single request.

Create a new LLM Observability dataset within the specified project.

Append one or more records to an LLM Observability dataset.

Create a new LLM Observability experiment.

Push spans and metrics for an LLM Observability experiment.

Run an LLM inference request through the specified integration and account, returning the model response and token usage.

Create a new LLM Observability project. Returns the existing project if a name conflict occurs.

Delete an annotation queue by its ID.

Delete one or more interactions from an annotation queue.

Delete a custom LLM Observability evaluator configuration by its name.

Submit a request to delete LLM Observability span data matching a trace ID filter within a specified time range.

Delete one or more records from an LLM Observability dataset.

Delete one or more LLM Observability datasets within the specified project.

Delete one or more LLM Observability experiments.

Delete one or more LLM Observability projects.

Download the contents of a dataset as a CSV file. The download is streamed and includes one row per dataset record.

Retrieve all interactions (traces and sessions) and their annotations for a given annotation queue.

Returns annotated interactions across all annotation queues for the given content IDs. Results include queue metadata (ID and name) for each interaction.

Retrieve the label schema for a given annotation queue.

Retrieve a custom LLM Observability evaluator configuration by its name.

Retrieve the draft state of a dataset, including whether it is currently locked for editing and which user holds the lock.

List annotation queues. Optionally filter by project ID or queue IDs. These parameters are mutually exclusive. If neither is provided, all queues in the organization are returned.

List all records in an LLM Observability dataset, sorted by creation date, newest first.

List the active versions of a dataset. A version is created each time a dataset is referenced by an experiment run.

List all LLM Observability datasets for a project, sorted by creation date, newest first.

Retrieve spans and experiment-level summary metrics for a given experiment with cursor-based pagination.

List all LLM Observability experiments sorted by creation date, newest first.

Retrieve the list of configured accounts for the specified LLM provider integration.

Retrieve the list of models available for the specified LLM provider integration and account.

List all LLM Observability projects sorted by creation date, newest first.

List LLM Observability spans matching the specified filters.

Acquire the draft lock on a dataset for the calling user. The lock prevents other users from concurrently editing the dataset draft.

Restore a dataset to a previous version. The dataset's current version is bumped, and its records are replaced with the records from the specified prior version.

Search across LLM Observability experimentation entities — projects, datasets, dataset records, experiments, and experiment runs — using cursor-based pagination.

The filter.scope field controls which entity types are returned. At least one valid scope must be provided.

Returns 200 OK when all results fit in a single page. Returns 206 Partial Content with a cursor in meta.after when additional pages are available.

Search LLM Observability spans using structured filters in the request body.

Search across LLM Observability experimentation entities using offset-based (page-number) pagination. Use this endpoint when you need total page count or want to navigate to a specific page number.

The filter.scope field controls which entity types are returned. At least one valid scope must be provided.

Release the draft lock on a dataset held by the calling user, allowing other users to edit the dataset draft.

Partially update an annotation queue. The name, description, and annotation_schema fields can be updated.

Create or replace the label schema for a given annotation queue. The label schema defines the labels annotators can apply to interactions in the queue. Label names must be unique within the queue and match the pattern ^[a-zA-Z0-9_-]+$. Each label must have a valid type: score, categorical, boolean, or text.

Create or update a custom LLM Observability evaluator configuration by its name.

Partially update an existing LLM Observability dataset within the specified project.

Update one or more existing records in an LLM Observability dataset.

Partially update an existing LLM Observability experiment.

Partially update an existing LLM Observability project.

Upload records to a dataset from a file. The request is a multipart/form-data upload containing a single file part. Currently only CSV is supported. The CSV must include an input column. Optional columns are id, expected_output, metadata, and tags.

The response is a Server-Sent Events stream (text/event-stream) emitting progress updates while records are processed. The stream emits the following named events:

• progress: incremental record counts written so far.
• completed: terminal event with a JSON body containing records_created.
• error: terminal event with a JSON body containing an error message.