Build concurrent and multi-stage data ingestion and data processing pipelines with Elixir.
Check our repoFeatures you will
Concurrency and Batching
Broadway is an open source library with concurrent data ingestion and processing. Outgoing data is also concurrent and can be batched.
Robust
Broadway relies on the Erlang VM and supervision trees to provide a robust failure model and graceful shutdowns.
Integrations
Broadway includes integration for RabbitMQ, Amazon SQS, Google PubSub, and Apache Kafka. You can also hook your own or use any of the community packages.
Back pressure
Broadway builds on GenStage to provide back pressure and avoid overloads, ensuring you only ingest data the system can handle. See how Discord and Change.org use GenStage at scale.
Rate limiting
A built-in rate limiter controls how to ingest data. This is useful when external factors dictate how fast your pipeline should run.
Dashboard
The Broadway Dashboard allows you to inspect your pipelines, see system parameters, and find bottlenecks.
