Build concurrent and multi-stage data ingestion and data processing pipelines with Elixir.Check our repo
Broadway is an open source library with concurrent data ingestion and processing. Outgoing data is also concurrent and can be batched.
Broadway relies on the Erlang VM and supervision trees to provide a robust failure model and graceful shutdowns.
Broadway includes integration for RabbitMQ, Amazon SQS, Google PubSub, and Apache Kafka. You can also hook your own or use any of the community packages.
Broadway builds on GenStage to provide back pressure and avoid overloads, ensuring you only ingest data the system can handle. See how Discord and Change.org use GenStage at scale.
A built-in rate limiter controls how to ingest data. This is useful when external factors dictate how fast your pipeline should run.
The Broadway Dashboard allows you to inspect your pipelines, see system parameters, and find bottlenecks.
Read our getting started guides or dive into Broadway's extensive API reference and architecture.See docs
Clone and run our example repository. It features a bike sharing pipeline that reads from RabbitMQ and writes to PostgreSQL.See example repo
Learn different ways of writing concurrent code in Elixir, covering tools like GenStage and Broadway.Buy now