Pathway: RAG Frameworks

Free open-source + enterprise (contact sales)

What is Pathway?

Pathway is a high-performance Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG. The Rust-powered engine treats data as a continuous stream of changes rather than static snapshots — making it a natural fit for AI applications that need to stay in sync with live data sources.

Pathway connects to PostgreSQL, Kafka, S3, and live APIs, monitoring them for changes and automatically processing updates while incrementally maintaining vector databases. A unique capability: mixing batch and streaming logic in the same workflow, so systems can be continuously trained with new streaming data and revised without requiring full batch reuploads. The framework supports stateless and stateful transformations (joins, windowing, sorting), with many transformations implemented in Rust.

Pathway provides dedicated LLM tooling for live LLM/RAG pipelines, with wrappers for common LLM services. Used in production at NATO and Intel for real-time streaming AI workloads. Recently crossed 50K GitHub stars on the strength of its 'fresh data for AI' positioning — a deployment-first architecture that solves the real-time data challenge other RAG frameworks struggle with.

Key features

Core capabilities this platform advertises.

Rust-powered streaming engine — millions of data points/sec
Unified batch + streaming logic in the same workflow
Connectors for PostgreSQL, Kafka, S3, and live APIs
Live LLM/RAG pipelines with incremental vector DB updates
Stateful transformations (joins, windowing, sorting) in Rust

Strengths and tradeoffs

What this tool does well, and the limitations to keep in mind.

Pros

Solves real-time data challenge most RAG frameworks ignore
Rust engine genuinely fast — millions of points/sec
Unified batch + streaming is rare and powerful
Production references at NATO and Intel
Pythonic API on top of Rust performance

Cons

Streaming-first paradigm has a learning curve for batch-only teams
Smaller community than Spark Streaming or Flink
Best fit for real-time use cases — overkill for static RAG
Enterprise pricing requires sales contact

Plans & pricing

What's included in each plan, and how the tiers compare.

Open Source

Forever

Full Pathway framework — Rust engine + Python API
All connectors and transformations
Self-hostable
BSL license (free for non-competing use)

Enterprise

Contact sales

Contact sales for a quote

Production support and SLAs
Custom connectors and integrations
Multi-tenant deployment
Used at NATO, Intel, and other regulated environments

View official pricing page

Common use cases

Data engineering teams building real-time AI/RAG pipelines that need to stay in sync with live data sources

Real-time RAG pipelines kept in sync with live data sources
Streaming feature pipelines for AI/ML serving
Continuous-learning AI systems that ingest streaming events
Real-time analytics combining batch + streaming transforms
Mission-critical AI workloads at NATO/Intel-scale

Run Pathway in production with Respan

10KFree traces/mo

1,000+Models

5 minSetup

Respan lets you trace LLM and agent calls across any model or framework, A/B test prompts on production traffic, and route requests across 1,000+ models through one gateway.

Try Respan free