Replicate (tracing) | Respan Docs

Replicate is a platform for running machine learning models in the cloud. It hosts thousands of open-source models and provides a simple API for running predictions without managing infrastructure.

Set up Respan

Create an account at platform.respan.ai and grab an API key.

Run npx @respan/cli setup to set up with your coding agent.

Use Respan Gateway

See Replicate gateway setup to route Replicate calls through the Respan gateway.

Example projects

Python examples

Setup

Install packages

$ pip install respan-ai respan-instrumentation-replicate replicate

Set environment variables

$ export REPLICATE_API_TOKEN="YOUR_REPLICATE_API_TOKEN"
$ export RESPAN_API_KEY="YOUR_RESPAN_API_KEY"

REPLICATE_API_TOKEN is used for Replicate predictions. RESPAN_API_KEY is used to export traces to Respan.

Initialize and run

1 import os
2 
3 import replicate
4 from respan import Respan, workflow
5 from respan_instrumentation_replicate import ReplicateInstrumentor
6 
7 respan = Respan(
8     api_key=os.environ["RESPAN_API_KEY"],
9     instrumentations=[ReplicateInstrumentor()],
10 )
11 
12 
13 @workflow(name="replicate_quickstart.workflow")
14 def run_prediction() -> str:
15     output = replicate.run(
16         "meta/meta-llama-3-8b-instruct",
17         input={"prompt": "Reply with one concise sentence about tracing."},
18     )
19     return "".join(str(chunk) for chunk in output) if not isinstance(output, str) else output
20 
21 
22 print(run_prediction())
23 respan.flush()
24 respan.shutdown()

View your trace

Open the Traces page and search for workflow name replicate_quickstart.workflow.

Configuration

Parameter	Type	Default	Description
`api_key`	`str \| None`	`None`	Falls back to `RESPAN_API_KEY` env var.
`base_url`	`str \| None`	`None`	Falls back to `RESPAN_BASE_URL` env var.
`instrumentations`	`list`	`[]`	Plugin instrumentations to activate (e.g. `ReplicateInstrumentor()`).
`customer_identifier`	`str \| None`	`None`	Default customer identifier for all spans.
`metadata`	`dict \| None`	`None`	Default metadata attached to all spans.
`environment`	`str \| None`	`None`	Environment tag (e.g. `"production"`).

Attributes

In Respan()

1 from respan import Respan
2 from respan_instrumentation_replicate import ReplicateInstrumentor
3 
4 respan = Respan(
5     instrumentations=[ReplicateInstrumentor()],
6     customer_identifier="user_123",
7     metadata={"service": "replicate-api", "version": "1.0.0"},
8 )

With propagate_attributes

1 import replicate
2 
3 from respan import Respan, propagate_attributes
4 from respan_instrumentation_replicate import ReplicateInstrumentor
5 
6 respan = Respan(instrumentations=[ReplicateInstrumentor()])
7 
8 
9 def handle_request(user_id: str, prompt: str):
10     with propagate_attributes(
11         customer_identifier=user_id,
12         thread_identifier="conv_abc_123",
13         metadata={"plan": "pro"},
14     ):
15         output = replicate.run("meta/meta-llama-3-8b-instruct", input={"prompt": prompt})
16         print("".join(str(chunk) for chunk in output) if not isinstance(output, str) else output)

Attribute	Type	Description
`customer_identifier`	`str`	Identifies the end user in Respan analytics.
`thread_identifier`	`str`	Groups related messages into a conversation.
`metadata`	`dict`	Custom key-value pairs. Merged with default metadata.