SageMaker (tracing)

Amazon SageMaker is AWS’s fully managed machine learning platform. It provides tools for building, training, and deploying ML models at scale, including real-time inference endpoints for LLMs and other models.

Set up Respan

Create an account at platform.respan.ai and grab an API key .

Paste this into your coding agent:


Help me set up Respan tracing with the SageMaker integration.
Follow https://www.respan.ai/docs/integrations/sagemaker.md

Setup

Install packages

Terminal


pip install respan-ai respan-instrumentation-sagemaker boto3

Set environment variables

Terminal


export AWS_ACCESS_KEY_ID="YOUR_AWS_ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="YOUR_AWS_SECRET_ACCESS_KEY"
export AWS_REGION="us-east-1"
export RESPAN_API_KEY="YOUR_RESPAN_API_KEY"

AWS credentials are used to invoke SageMaker endpoints. RESPAN_API_KEY is used to export traces to Respan.

Initialize and run

Python


import json
import os

import boto3
from respan import Respan
from respan_instrumentation_sagemaker import SageMakerInstrumentor

respan = Respan(
    api_key=os.environ["RESPAN_API_KEY"],
    instrumentations=[SageMakerInstrumentor()],
)

client = boto3.client("sagemaker-runtime", region_name=os.environ["AWS_REGION"])

payload = json.dumps({"inputs": "Reply with one concise sentence about tracing."}).encode("utf-8")

response = client.invoke_endpoint(
    EndpointName="my-llm-endpoint",
    ContentType="application/json",
    Accept="application/json",
    Body=payload,
)

result = json.loads(response["Body"].read())
print(result)

respan.shutdown()

View your trace

Open the Traces page to see your auto-instrumented endpoint invocation spans with payload and latency.

Configuration

Parameter	Type	Default	Description
`api_key`	`str \| None`	`None`	Falls back to `RESPAN_API_KEY` env var.
`base_url`	`str \| None`	`None`	Falls back to `RESPAN_BASE_URL` env var.
`instrumentations`	`list`	`[]`	Plugin instrumentations to activate (e.g. `SageMakerInstrumentor()`).
`customer_identifier`	`str \| None`	`None`	Default customer identifier for all spans.
`metadata`	`dict \| None`	`None`	Default metadata attached to all spans.
`environment`	`str \| None`	`None`	Environment tag (e.g. `"production"`).

Attributes

In Respan()

Python


from respan import Respan
from respan_instrumentation_sagemaker import SageMakerInstrumentor

respan = Respan(
    instrumentations=[SageMakerInstrumentor()],
    customer_identifier="user_123",
    metadata={"service": "sagemaker-api", "version": "1.0.0"},
)

With propagate_attributes

Python


import json
import boto3

from respan import Respan, propagate_attributes
from respan_instrumentation_sagemaker import SageMakerInstrumentor

respan = Respan(instrumentations=[SageMakerInstrumentor()])


def handle_request(user_id: str, prompt: str):
    with propagate_attributes(
        customer_identifier=user_id,
        thread_identifier="conv_abc_123",
        metadata={"plan": "pro"},
    ):
        payload = json.dumps({"inputs": prompt}).encode("utf-8")
        response = boto3.client("sagemaker-runtime", region_name="us-east-1").invoke_endpoint(
            EndpointName="my-llm-endpoint",
            ContentType="application/json",
            Accept="application/json",
            Body=payload,
        )
        print(json.loads(response["Body"].read()))

Attribute	Type	Description
`customer_identifier`	`str`	Identifies the end user in Respan analytics.
`thread_identifier`	`str`	Groups related messages into a conversation.
`metadata`	`dict`	Custom key-value pairs. Merged with default metadata.