The dashboard is available at
The dashboard is one of the primary ways to interact with the Batch platform.
The dashboard enables you to:
Manage sources (hosted
Manage account, team and API tokens
Manage billing settings
When Batch receives events, it stores them using the following structure:
your original payload
Our collectors enrich your events metadata which you can use in your queries.
An ISO 8601 format timestamp
A nanosecond UNIX timestamp
A unique UUID assigned to this event
Identifier for the (Batch) system that received the event
The detailed collection view enables you to search through your event data using a
syntax (such as used by ElasticSearch) and uses full-text search.
Our search supports the following operations:
Value contains or does NOT contain
Value is greater than, less than
Value is between X and Y
Timestamps are "special". To fetch events that fit a specific time range, use the
field which uses
Timestamps in Batch metadata are using the
Client metadata is stored in parquet as a
and if queried via Athena (instead of the dashboard), you must use the following syntax:
The above applies
to when querying parquet files via Athena or another parquet-capable platform.
Fetch all results (for the picked time interval).
Any part of a string
Find any events that contain the string "foo" in any key or field.
Find all events that do not contain the string
batch.info.source: (NOT foo)
Events ingested between timestamps
batch.info.date_human: [2021-03-08T22:29:05Z TO 2021-03-08T22:30:26Z]
Chaining multiple conditions
Find all events where
is greater than
is set to "engineer".
client.payload.age: >32 AND client.payload.title: engineer.
Due to how indexing works, searching for an exact values might provide false positives.
when there are events with
will return both
You have several options to get around this:
Search for values that are unique and not part of any existing values
Add additional constraints to the search
Surround your field in double quotes so the value is treated as a single element
Sources are managed relays for various data sources
Ideal for users with long running relays that do not want to host there own
Destinations are required to use the replay functionality. A destination is an endpoint that collection data will be replayed to.
When using JSON or plain Batch will infer the schema. Protobuf schemas can not be inferred and must be uploaded.
Since replays are based off of Parquet data stored in S3, you must specify a key in order to replay data.
This will work for a replay:
Batch will translate this to the following SQL query for Athena:
SELECT ... WHERE client.payload.field = 'foo'
This won't work:
query is acceptable for searching for data, it cannot be used to facilitate a replay as there is nothing to indicate which field it represents.
Under the Hood
Replays are facilitated by searching for data via AWS Athena (Presto). To pull that off, Batch takes your Lucene query and translates it to SQL that can be executed in Athena.
One area that might be confusing is "wildcard" searches - as in, how do you write a Lucene query that will be translated to a
SELECT ... WHERE foo LIKE '%bar%`
To do so, surround your field search with
- the asterisk will cause the Batch query translator to use
in the query and replace the asterisks with a
Will be translated by Batch to:
The account section allows user to manage passwords, teams, billing, and api keys
Access your account by clicking your avatar at the top right
Update basic info under profile menu
Invite or remove team members to manage collections
Under the Team menu invite new members
Adjust plan, change billing info, and review invoices
Under billing menu manage plans and manage payment
API Keys are used to
manage your Batch account and data.
Security menu allows access to API Keys
Getting started - Previous
Next - Platform
Under the Hood