Dashboard
The dashboard is one of the primary ways to interact with the Streamdal platform.
The dashboard enables you to:
- Manage collections
- Manage sources (hosted
plumber)
- Manage destinations
- Manage schemas
- Manage replays
- Manage account, team and API tokens
- Manage billing settings
When Batch receives events, it stores them using the following structure:
{
"batch": {
"info": {}
},
"client": {
"metadata": {headers, topic, offsets, etc},
"payload": {your original payload}
}
}
Our collectors enrich your events metadata which you can use in your queries.
Batch metadata:
batch.info.date_human
- An ISO 8601 format timestamp
batch.info.date_string
- A nanosecond UNIX timestamp
batch.info.request_id
- A unique UUID assigned to this event
batch.info.source
- Identifier for the (Batch) system that received the event
The detailed collection view enables you to search through your event data using a Lucene-like syntax (such as used by ElasticSearch) and uses full-text search.
Our search supports the following operations:
- Value contains or does NOT contain
- Value is greater than, less than
- Value is between X and Y
- Date operations
- Expression chaining
- Search modifiers
Timestamps are "special". To fetch events that fit a specific time range, use the
batch.info.date_human
field which uses ISO 8601 format.Timestamps in Batch metadata are using the UTC timezone.
Client metadata is stored in parquet as a
map[string]string
and if queried via Athena (instead of the dashboard), you must use the following syntax:SELECT * FROM $db.$table WHERE client.metadata['request_id'] = 'foo'
The above applies only to when querying parquet files via Athena or another parquet-capable platform.
It is possible to alter the results of a search by adding modifiers to the search query.
The modifier syntax is as follows:
${MODIFIER input}
Or to use multiple modifiers: ${MODIFIER input AND MODIFIER input AND ...}
Available modifiers:
- 1.
ORDER_BY <field_name>
- 1.Sort the results by a given field
- 2.Equivalent to
ORDER BY
in SQL
- 2.
SORT ASC|DESC
- 1.Sorts the returned data in ascending or descending order
- 2.Equivalent to SQL
ORDER BY ... ASC|DESC
- 3.
`LIMIT <number>`
- 1.Limits the number of results returned by the search
- 2.Equivalent to SQL
LIMIT <number>
- 4.
`UNIQUE <field_name>`
- 1.Limit results to unique values from a column
- 2.Equivalent to SQL
`GROUP BY ...`
You can chain multiple modifiers together by adding an
AND
between the modifiers.Case matters for both modifier actions and the chaining keyword (
AND
).Fetch all results (for the picked time interval).
*
Find any events that contain the string "foo" in any key or field.
foo
Find all events that do not contain the string
foo
in batch.info.source
.batch.info.source: (NOT foo)
batch.info.date_human: [2021-03-08T22:29:05Z TO 2021-03-08T22:30:26Z]
Find all events where
client.payload.age
is greater than 32
AND client.payload.title
is set to "engineer".client.payload.age: >32 AND client.payload.title: engineer.
You can query the length of an array using the length() function. The following query will match all records where my_array has 2 items
client.payload.my_array[].length(): 2
Greater than and less than operators are also supported: > , < , >=, and <=
The following query will match aall records where my_array has 3 or more items
client.payload.my_array[].length(): >=3
Specifying search modifiers
Find the event that has the oldest age:
client.payload.age: >32 ${ORDER client.payload.age AND SORT DESC AND LIMIT 1}
Exact matches
Due to how indexing works, searching for an exact values might provide false positives.
Example
Searching for
foo
when there are events with foobar
will return both foo
and foobar
. You have several options to get around this:
- 1.Search for values that are unique and not part of any existing values
- 2.Add additional constraints to the search
- 3.Surround your field in double quotes so the value is treated as a single element
Sources are managed relays for various data sources

Destinations are required to use the replay functionality. A destination is an endpoint that collection data will be replayed to.

When using JSON or plain Batch will infer the schema. Protobuf schemas can not be inferred and must be uploaded.

Since replays are based off of Parquet data stored in S3, you must specify a key in order to replay data.
This will work for a replay:
client.payload.field: foo
Batch will translate this to the following SQL query for Athena:
SELECT ... WHERE client.payload.field = 'foo'
This won't work:
foo
While the
foo
query is acceptable for searching for data, it cannot be used to facilitate a replay as there is nothing to indicate which field it represents.Replays are facilitated by searching for data via AWS Athena (Presto). To pull that off, Batch takes your Lucene query and translates it to SQL that can be executed in Athena.
One area that might be confusing is "wildcard" searches - as in, how do you write a Lucene query that will be translated to a
SELECT ... WHERE foo LIKE '%bar%`
?To do so, surround your field search with
*
- the asterisk will cause the Batch query translator to use LIKE
in the query and replace the asterisks with a %
.Example
client.payload.foo: *bar*
Will be translated by Batch to:
SELECT ... FROM ... WHERE client.payload.foo LIKE '%bar%'
The account section allows user to manage passwords, teams, billing, and api keys

Update basic info under profile menu
Invite or remove team members to manage collections

Under the Team menu invite new members
Adjust plan, change billing info, and review invoices

Under billing menu manage plans and manage payment

Security menu allows access to API Keys
Last modified 7mo ago