SageMaker (tracing) | Respan Docs

Amazon SageMaker is AWS’s fully managed machine learning platform. It provides tools for building, training, and deploying ML models at scale, including real-time inference endpoints for LLMs and other models.

Set up Respan

Create an account at platform.respan.ai and grab an API key.

Run npx @respan/cli setup to set up with your coding agent.

Use Respan Gateway

See Amazon SageMaker gateway setup to route SageMaker calls through the Respan gateway.

Example projects

Python examples

Setup

Install packages

pip install respan-ai respan-instrumentation-sagemaker boto3

Set environment variables

export AWS_ACCESS_KEY_ID="YOUR_AWS_ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="YOUR_AWS_SECRET_ACCESS_KEY"
export AWS_REGION="us-east-1"
export RESPAN_API_KEY="YOUR_RESPAN_API_KEY"

AWS credentials are used to invoke SageMaker endpoints. RESPAN_API_KEY is used to export traces to Respan.

Initialize and run

import json
import os
import boto3
from respan import Respan
from respan_instrumentation_sagemaker import SageMakerInstrumentor
respan = Respan(
    api_key=os.environ["RESPAN_API_KEY"],
    instrumentations=[SageMakerInstrumentor()],
)
client = boto3.client("sagemaker-runtime", region_name=os.environ["AWS_REGION"])
payload = json.dumps({"inputs": "Reply with one concise sentence about tracing."}).encode("utf-8")
response = client.invoke_endpoint(
    EndpointName="my-llm-endpoint",
    ContentType="application/json",
    Accept="application/json",
    Body=payload,
)
result = json.loads(response["Body"].read())
print(result)
respan.shutdown()

View your trace

Open the Traces page to see your auto-instrumented endpoint invocation spans with payload and latency.

Configuration

Parameter	Type	Default	Description
`api_key`	`str \| None`	`None`	Falls back to `RESPAN_API_KEY` env var.
`base_url`	`str \| None`	`None`	Falls back to `RESPAN_BASE_URL` env var.
`instrumentations`	`list`	`[]`	Plugin instrumentations to activate (e.g. `SageMakerInstrumentor()`).
`customer_identifier`	`str \| None`	`None`	Default customer identifier for all spans.
`metadata`	`dict \| None`	`None`	Default metadata attached to all spans.
`environment`	`str \| None`	`None`	Environment tag (e.g. `"production"`).

Attributes

In Respan()

from respan import Respan
from respan_instrumentation_sagemaker import SageMakerInstrumentor
respan = Respan(
    instrumentations=[SageMakerInstrumentor()],
    customer_identifier="user_123",
    metadata={"service": "sagemaker-api", "version": "1.0.0"},
)

With propagate_attributes

import json
import boto3
from respan import Respan, propagate_attributes
from respan_instrumentation_sagemaker import SageMakerInstrumentor
respan = Respan(instrumentations=[SageMakerInstrumentor()])
def handle_request(user_id: str, prompt: str):
    with propagate_attributes(
        customer_identifier=user_id,
        thread_identifier="conv_abc_123",
        metadata={"plan": "pro"},
    ):
        payload = json.dumps({"inputs": prompt}).encode("utf-8")
        response = boto3.client("sagemaker-runtime", region_name="us-east-1").invoke_endpoint(
            EndpointName="my-llm-endpoint",
            ContentType="application/json",
            Accept="application/json",
            Body=payload,
        )
        print(json.loads(response["Body"].read()))

Attribute	Type	Description
`customer_identifier`	`str`	Identifies the end user in Respan analytics.
`thread_identifier`	`str`	Groups related messages into a conversation.
`metadata`	`dict`	Custom key-value pairs. Merged with default metadata.