Batch offers foundational components that are necessary for building and maintaining complex distributed systems that utilize messaging systems.
Messaging system introspection (we support nearly all, popular messaging systems)
Automatic schema discovery (if using JSON)
In-depth protobuf support
Automatic event indexing, enabling granular search
Automatic archiving of events into a "data-science optimal" parquet format
Granular replay functionality
Support for nearly all messaging system tech
Take a look at the Use Cases section to get a better idea as to what's possible with the Batch platform.
Batch is an excellent addition to your infra if you make use of the actor model.
The following components make up the Batch platform:
Service/utility that collects data from your message bus and relays it to our event collectors (via gRPC)
Schema generators inspect your event data and update our internal parquet schemas and Athena schemas to facilitate long-term event storage
We store your events forever in our search cache and in S3 in parquet format.
The data in search cache is used for search; data in S3 is used for replays.
You can use the parsed S3 data however you wish.
All event data is indexed and can be searched using Lucene syntax
Once you find the data you are looking for, you can replay that data to a destination of your choice - Kafka, Rabbit, SQS or an HTTP endpoint.
For additional insight into how Batch works, check out the Architecture doc.