The dashboard is one of the primary ways to interact with the Batch platform.

The dashboard enables you to:

• Manage collections

• Manage sources (hosted plumber)

• Manage destinations

• Manage schemas

• Manage replays

• Manage account, team and API tokens

• Manage billing settings

Manage Collections

Event Envelope

When Batch receives events, it stores them using the following structure:

{
    "batch": {
        "info": {}
    },
    "client": {
        "metadata": {headers, topic, offsets, etc},
        "payload": {your original payload}
    }
}

Batch metadata

Our collectors enrich your events metadata which you can use in your queries.

Batch metadata:

• batch.info.date_human

  • An ISO 8601 format timestamp

• batch.info.date_string

  • A nanosecond UNIX timestamp

• batch.info.request_id

  • A unique UUID assigned to this event

• batch.info.source

  • Identifier for the (Batch) system that received the event

Search Syntax

The detailed collection view enables you to search through your event data using a Lucene-like syntax (such as used by ElasticSearch) and uses full-text search.

Our search supports the following operations:

• Value contains or does NOT contain

• Value is greater than, less than

• Value is between X and Y

• Date operations

• Expression chaining

Timestamps

Timestamps are "special". To fetch events that fit a specific time range, use the batch.info.date_human field which uses ISO 8601 format.

Client Metadata

Client metadata is stored in parquet as a map[string]string and if queried via Athena (instead of the dashboard), you must use the following syntax:

SELECT * FROM $db.$table WHERE client.metadata['request_id'] = 'foo'

Search Examples

All results

Fetch all results (for the picked time interval).

*

Any part of a string

Find any events that contain the string "foo" in any key or field.

foo

Logical NOT

Find all events that do not contain the string foo in batch.info.source.

batch.info.source: (NOT foo)

Events ingested between timestamps

batch.info.date_human: [2021-03-08T22:29:05Z TO 2021-03-08T22:30:26Z]

Chaining multiple conditions

Find all events where client.payload.age is greater than 32 AND client.payload.title is set to "engineer".

client.payload.age: >32 AND client.payload.title: engineer.

Manage Sources

Manage Destinations

Manage Schemas

Manage Replays

Since replays are based off of Parquet data stored in S3, you must specify a key in order to replay data.

This will work for a replay:

client.payload.field: foo

This won't work:

foo

While the foo query is acceptable for searching for data, it cannot be used to facilitate a replay as there is nothing to indicate which field it represents.

Under the Hood

Replays are facilitated by searching for data via AWS Athena (Presto). To pull that off, Batch takes your Lucene query and translates it to SQL that can be executed in Athena.

One area that might be confusing is "wildcard" searches - as in, how do you write a Lucene query that will be translated to a SELECT ... WHERE foo LIKE '%bar%`?

To do so, surround your field search with * - the asterisk will cause the Batch query translator to use LIKE in the query and replace the asterisks with a %.

Example

client.payload.foo: *bar*

Will be translated by Batch to:

SELECT ... FROM ... WHERE client.payload.foo LIKE '%bar%'

Manage Account

<TODO>

Team Members

<TODO>

API Keys

<TODO>