Docs
Search…
Dashboard
The dashboard is available at console.batch.sh.
The dashboard is one of the primary ways to interact with the Batch platform.
The dashboard enables you to:
    Manage collections
    Manage sources (hosted plumber)
    Manage destinations
    Manage schemas
    Manage replays
    Manage account, team and API tokens
    Manage billing settings

Manage Collections

Event Envelope

When Batch receives events, it stores them using the following structure:
1
{
2
"batch": {
3
"info": {}
4
},
5
"client": {
6
"metadata": {headers, topic, offsets, etc},
7
"payload": {your original payload}
8
}
9
}
Copied!

Batch metadata

Our collectors enrich your events metadata which you can use in your queries.
Batch metadata:
    batch.info.date_human
      An ISO 8601 format timestamp
    batch.info.date_string
      A nanosecond UNIX timestamp
    batch.info.request_id
      A unique UUID assigned to this event
    batch.info.source
      Identifier for the (Batch) system that received the event

Search Syntax

The detailed collection view enables you to search through your event data using a Lucene-like syntax (such as used by ElasticSearch) and uses full-text search.
Our search supports the following operations:
    Value contains or does NOT contain
    Value is greater than, less than
    Value is between X and Y
    Date operations
    Expression chaining

Timestamps

Timestamps are "special". To fetch events that fit a specific time range, use the batch.info.date_human field which uses ISO 8601 format.
Timestamps in Batch metadata are using the UTC timezone.

Client Metadata

Client metadata is stored in parquet as a map[string]string and if queried via Athena (instead of the dashboard), you must use the following syntax:
1
SELECT * FROM $db.$table WHERE client.metadata['request_id'] = 'foo'
Copied!
The above applies only to when querying parquet files via Athena or another parquet-capable platform.

Search Examples

All results

Fetch all results (for the picked time interval).
1
*
Copied!

Any part of a string

Find any events that contain the string "foo" in any key or field.
1
foo
Copied!

Logical NOT

Find all events that do not contain the string foo in batch.info.source.
1
batch.info.source: (NOT foo)
Copied!

Events ingested between timestamps

1
batch.info.date_human: [2021-03-08T22:29:05Z TO 2021-03-08T22:30:26Z]
Copied!

Chaining multiple conditions

Find all events where client.payload.age is greater than 32 AND client.payload.title is set to "engineer".
1
client.payload.age: >32 AND client.payload.title: engineer.
Copied!
Exact matches
Due to how indexing works, searching for an exact values might provide false positives. Example Searching for foo when there are events with foobar will return both foo and foobar.
You have several options to get around this:
    1.
    Search for values that are unique and not part of any existing values
    2.
    Add additional constraints to the search
    3.
    Surround your field in double quotes so the value is treated as a single element

Manage Sources

Sources are managed relays for various data sources
Ideal for users with long running relays that do not want to host there own plumber instances

Manage Destinations

Destinations are required to use the replay functionality. A destination is an endpoint that collection data will be replayed to.

Manage Schemas

When using JSON or plain Batch will infer the schema. Protobuf schemas can not be inferred and must be uploaded.

Manage Replays

Since replays are based off of Parquet data stored in S3, you must specify a key in order to replay data.
This will work for a replay:
1
client.payload.field: foo
Copied!
Batch will translate this to the following SQL query for Athena: SELECT ... WHERE client.payload.field = 'foo'
This won't work:
1
foo
Copied!
While the foo query is acceptable for searching for data, it cannot be used to facilitate a replay as there is nothing to indicate which field it represents.

Under the Hood

Replays are facilitated by searching for data via AWS Athena (Presto). To pull that off, Batch takes your Lucene query and translates it to SQL that can be executed in Athena.
One area that might be confusing is "wildcard" searches - as in, how do you write a Lucene query that will be translated to a SELECT ... WHERE foo LIKE '%bar%`?
To do so, surround your field search with * - the asterisk will cause the Batch query translator to use LIKE in the query and replace the asterisks with a %.
Example
1
client.payload.foo: *bar*
Copied!
Will be translated by Batch to:
1
SELECT ... FROM ... WHERE client.payload.foo LIKE '%bar%'
Copied!

Manage Account

The account section allows user to manage passwords, teams, billing, and api keys

Access your account by clicking your avatar at the top right

Update basic info under profile menu

Team Members

Invite or remove team members to manage collections
Under the Team menu invite new members

Billing

Adjust plan, change billing info, and review invoices
Under billing menu manage plans and manage payment

API Keys

API Keys are used to programmatically manage your Batch account and data.
Security menu allows access to API Keys
Last modified 1mo ago