API ReferenceDatasets

Api Datasets Logs List List

List dataset logs with filtering, pagination, and full-object retrieval. Endpoints: GET /api/datasets/{dataset_id}/logs/list/ - List logs with pagination POST /api/datasets/{dataset_id}/logs/list/ - List logs with complex filtering Query Parameters (GET): - page (integer, optional): Page number for pagination (default: 1) - page_size (integer, optional): Number of results per page - sort_by (string, optional): Field to order results by (default: unique_id) - retrieval_mode (string, optional): "async" for background full-object loading Request Body (POST): - filters (object, optional): Complex filter criteria - page (integer, optional): Page number - page_size (integer, optional): Results per page Response (200 OK): - count (integer): Total number of logs matching filters - next (string|null): URL for next page - previous (string|null): URL for previous page - results (array): Array of log objects - filter_options (object): Available filter options ## Under-the-hood optimizations: ### 1. Async Full Object Preloading (retrieval_mode="async") - When listing logs, the API returns immediately with shallow ClickHouse fields - A background Celery task (`preload_full_objects_task`) is triggered to: a. Fetch full log objects from S3 storage (including input/output) b. Write them to Redis cache with key: `request_log_full_object_{unique_id}` c. Cache TTL: 300 seconds (5 minutes) - Subsequent detail view requests get instant cache hits - Callback: `evaluation.utils.store_dataset_log_full_objects_to_cache` ### 2. Overlay Precedence for Updated Logs - When a log is updated, an overlay file is written to S3 with the new data - ClickHouse row is updated with `updated_storage_object_key` pointing to overlay - The preload task retrieves BOTH base index AND overlay keys - Deduplication: If a `unique_id` is found in BOTH, overlay takes precedence (tracked via `ids_to_retrieve` set in `batch_retrieve_full_objects`) ### 3. ArgMax Deduplication - Uses ClickHouse's argMax to get the latest version of each log - Deduplicates by `unique_id`, sorted by `updated_at` field - Ensures only the most recent version of each log is shown ### 4. Storage Structure - Index files (.idx): `{"unique_id1": {object1}, "unique_id2": {object2}}` - Overlay files: `{object}` directly (single log, not wrapped) - Both are handled transparently in `batch_retrieve_full_objects` ## Performance characteristics: - List response: ~100-200ms (ClickHouse query only, no S3) - Cache population: 1-5s background (depends on log count) - Detail view after list: ~10ms (Redis cache hit)

Authentication

AuthorizationBearer
JWT access token or Respan API key

Path parameters

dataset_idstringRequired

Query parameters

pageintegerOptional
A page number within the paginated result set.
page_sizeintegerOptional
Number of results to return per page.

Response

countinteger
resultslist of objects
nextstring or nullformat: "uri"
previousstring or nullformat: "uri"
total_countinteger
current_filtersobject

Pydantic model for FilterParamDict. A dictionary that maps metric names to their filter parameters.

Each key is a metric name (str), and each value can be:

  • A single MetricFilterParamPydantic (one condition)
  • A List[MetricFilterParamPydantic] (multiple conditions for same metric)
  • A FilterBundlePydantic (nested filter bundle with connector)

Note: Uses extra=“allow” for dynamic metric name fields. The pydantic_extra annotation tells Pydantic what types to expect for extra fields, and generates typed additionalProperties in JSON Schema.

filters_dataobject