Replicate (tracing) | Respan Docs

Replicate is a platform for running machine learning models in the cloud. It hosts thousands of open-source models and provides a simple API for running predictions without managing infrastructure.

Set up Respan

Create an account at platform.respan.ai and grab an API key.

Run npx @respan/cli setup to set up with your coding agent.

Use Respan Gateway

See Replicate gateway setup to route Replicate calls through the Respan gateway.

Example projects

Python examples

Setup

Install packages

pip install respan-ai respan-instrumentation-replicate replicate

Set environment variables

export REPLICATE_API_TOKEN="YOUR_REPLICATE_API_TOKEN"
export RESPAN_API_KEY="YOUR_RESPAN_API_KEY"

REPLICATE_API_TOKEN is used for Replicate predictions. RESPAN_API_KEY is used to export traces to Respan.

Initialize and run

import os
import replicate
from respan import Respan, workflow
from respan_instrumentation_replicate import ReplicateInstrumentor
respan = Respan(
    api_key=os.environ["RESPAN_API_KEY"],
    instrumentations=[ReplicateInstrumentor()],
)
@workflow(name="replicate_quickstart.workflow")
def run_prediction() -> str:
    output = replicate.run(
        "meta/meta-llama-3-8b-instruct",
        input={"prompt": "Reply with one concise sentence about tracing."},
    )
    return "".join(str(chunk) for chunk in output) if not isinstance(output, str) else output
print(run_prediction())
respan.shutdown()

View your trace

Open the Traces page and search for workflow name replicate_quickstart.workflow.

Configuration

Parameter	Type	Default	Description
`api_key`	`str \| None`	`None`	Falls back to `RESPAN_API_KEY` env var.
`base_url`	`str \| None`	`None`	Falls back to `RESPAN_BASE_URL` env var.
`instrumentations`	`list`	`[]`	Plugin instrumentations to activate (e.g. `ReplicateInstrumentor()`).
`customer_identifier`	`str \| None`	`None`	Default customer identifier for all spans.
`metadata`	`dict \| None`	`None`	Default metadata attached to all spans.
`environment`	`str \| None`	`None`	Environment tag (e.g. `"production"`).

Attributes

In Respan()

from respan import Respan
from respan_instrumentation_replicate import ReplicateInstrumentor
respan = Respan(
    instrumentations=[ReplicateInstrumentor()],
    customer_identifier="user_123",
    metadata={"service": "replicate-api", "version": "1.0.0"},
)

With propagate_attributes

import replicate
from respan import Respan, propagate_attributes
from respan_instrumentation_replicate import ReplicateInstrumentor
respan = Respan(instrumentations=[ReplicateInstrumentor()])
def handle_request(user_id: str, prompt: str):
    with propagate_attributes(
        customer_identifier=user_id,
        thread_identifier="conv_abc_123",
        metadata={"plan": "pro"},
    ):
        output = replicate.run("meta/meta-llama-3-8b-instruct", input={"prompt": prompt})
        print("".join(str(chunk) for chunk in output) if not isinstance(output, str) else output)

Attribute	Type	Description
`customer_identifier`	`str`	Identifies the end user in Respan analytics.
`thread_identifier`	`str`	Groups related messages into a conversation.
`metadata`	`dict`	Custom key-value pairs. Merged with default metadata.