Broadway is an open source library with concurrent data ingestion and processing. Outgoing data is also concurrent and can be batched.
Broadway relies on the Erlang VM and supervision trees to provide a robust failure model and graceful shutdowns.
Broadway includes integration for RabbitMQ, Amazon SQS, Google PubSub, and Apache Kafka. You can also hook your own or use any of the community packages.
A built-in rate limiter controls how to ingest data. This is useful when external factors dictate how fast your pipeline should run.
The Broadway Dashboard allows you to inspect your pipelines, see system parameters, and find bottlenecks.
Read our getting started guides or dive into Broadway's extensive API reference and architecture.See docs
Clone and run our example repository. It features a bike sharing pipeline that reads from RabbitMQ and writes to PostgreSQL.See example repo
Learn different ways of writing concurrent code in Elixir, covering tools like GenStage and Broadway.Buy now