Collections

POST /v1/collection

Description

Create a new collection.

Header Parameters

Param	Type	Required	Description
Authorization	string	yes	Bearer token containing API key

Body Parameters

Param	Type	Required	Description
TODO	string	yes	Bearer token containing API key

Responses

200 - Collection created successfully

{
  "id": "566762f6-d06f-448f-911e-dea9f9315edb",
  "name": "cli test",
  "notes": "",
  "token": "62d11890-f3c3-42a4-9e73-0877dfe16bfc",
  "paused": false,
  "inserted_at": "2021-02-16T15:46:30.897801Z",
  "updated_at": "2021-02-16T15:46:30.897801Z",
  "disable_archiving": false,
  "team": {
    "id": "7b66ed87-5ffb-494b-9f8e-aed519054045",
    "name": "marktest-1",
    "inserted_at": "2021-02-16T15:38:51.800964Z",
    "updated_at": "2021-02-16T15:38:51.800964Z"
  },
  "schema": {
    "id": "4459dae2-6b40-408c-8379-abb36f16108f",
    "name": "Generic JSON",
    "root_type": "",
    "type": "json",
    "shared": "false",
    "archived": false,
    "inserted_at": "2021-02-16T15:38:51.800964Z",
    "updated_at": "2021-02-16T15:38:51.800964Z"
  },
  "datalake": {
    "id": "a4d9397a-bdfb-4e22-b348-2f437e7643ca",
    "type": "aws",
    "name": "Default DataLake",
    "status": "active",
    "status_full": "",
    "inserted_at": "2021-02-16T15:38:51.800964Z",
    "updated_at": "2021-02-16T15:38:51.889373Z"
  },
  "author": {
    "id": "e3ea07be-7565-4c25-9fb7-515d2b6eb582",
    "name": "mark test",
    "email": "[e-mail]"
  },
  "plan": {
    "id": "ddc4915c-b0e0-4a6b-8911-bf93529d6019",
    "stripe_plan_id": "prod_IlTxuV7gXfJic8",
    "plan_attributes": {
      "num_seats": 1,
      "replay_gb": 1,
      "storage_gb": 1,
      "num_collections": 1,
      "trial_available": true,
      "trial_length_days": 14
    },
    "plan_usage": {
      "seats": 1,
      "collections": 0
    },
    "status": "active",
    "status_reason": "account created",
    "inserted_at": "2021-02-16T15:38:51.800964Z",
    "updated_at": "2021-02-16T15:38:51.800964Z"
  }
}

412 - Out of plan resources

{"error": "max collections reached"}

422 - Invalid data provided

{
  "errors": [
    {
      "code": 422,
      "domain": "collection",
      "field": "schema_id",
      "status": "invalid",
      "raw_error": "",
      "message": "must be a valid UUID v4"
    }
  ]
}

GET /v1/collection

Description

Get all collections.

Header Parameters

Param	Type	Required	Description
Authorization	string	yes	Bearer token containing API key

Example Response

{
   "0dce7d85-8d61-41db-86bd-af005df755f4":{
      "latency_ms":0,
      "status":"error",
      "message":"Failed to connect",
      "last_checked_at":1624548189
   }
}

GET /v1/collection/{id}

Description

Get one collection.

Header Parameters

Param	Type	Required	Description
Authorization	string	yes	Bearer token containing API key

Path Parameters

Param	Type	Required	Description
TODO	string	yes	Bearer token containing API key

Example Response

{
   "0dce7d85-8d61-41db-86bd-af005df755f4":{
      "latency_ms":0,
      "status":"error",
      "message":"Failed to connect",
      "last_checked_at":1624548189
   }
}

PUT /v1/collection

Description

You can archive a collection by passing "archived": true in the payload.

Archived collections CANNOT be deleted - archive is functionally equivalent to delete.

Header Parameters

Param	Type	Required	Description
Authorization	string	yes	Bearer token containing API key

Body Parameters

Param	Type	Required	Description
TODO	string	yes	Bearer token containing API key

Example Response

{
   "0dce7d85-8d61-41db-86bd-af005df755f4":{
      "latency_ms":0,
      "status":"error",
      "message":"Failed to connect",
      "last_checked_at":1624548189
   }
}

PUT /v1/collection/{id}/toggle-pause

Description

Pause or resume a collection.

Header Parameters

Param	Type	Required	Description
Authorization	string	yes	Bearer token containing API key

Path Parameters

Param	Type	Required	Description
TODO	string	yes	Bearer token containing API key

Body Parameters

Param	Type	Required	Description
TODO	string	yes	Bearer token containing API key

Example Response

{
   "0dce7d85-8d61-41db-86bd-af005df755f4":{
      "latency_ms":0,
      "status":"error",
      "message":"Failed to connect",
      "last_checked_at":1624548189
   }
}

POST /v1/collection/{id}/search

Description

Search a collection.

Header Parameters

Param	Type	Required	Description
Authorization	string	yes	Bearer token containing API key

Path Parameters

Param	Type	Required	Description
id	string	yes	Collection ID to use for search

Body Parameters

Param	Type	Required	Description
collection_id	string	yes	Collection ID to search
query	string	yes	Lucene query
from	int	yes	Which record number to start reading from
size	int	no	How many records to fetch per page

Example Lucene Query

client.payload.hello: "world" AND batch.info.date_human: [2020-06-25T22:40:44.638Z TO 2021-06-25T22:40:44.639Z]

Example Response

{
  "total": 4,
  "is_replayable": true,
  "data": [
    {
      "@timestamp": "2021-06-17T23:46:29Z",
      "batch": {
        "info": {
          "date_human": "2021-06-17T23:46:29Z",
          "date_unix": "1623973589784716776",
          "request_id": "4df0f08e-0cc4-4e73-bda9-d82bf83858e2",
          "source": "grpc-collector_grpc-collector-deployment-65676d4f86-rk6s9"
        }
      },
      "client": {
        "metadata": {
          "header.test": "123",
          "key": "",
          "offset": "17",
          "partition": "0",
          "timestamp": "1623973589608730000",
          "topic": "foo",
          "transport": "kafka"
        },
        "payload": {
          "hello": "world"
        }
      }
    },
    {
      "@timestamp": "2021-06-17T23:42:29Z",
      "batch": {
        "info": {
          "date_human": "2021-06-17T23:42:29Z",
          "date_unix": "1623973349778615096",
          "request_id": "2d94b699-7ecd-490a-92ae-c3537c828de4",
          "source": "grpc-collector_grpc-collector-deployment-65676d4f86-8xdnh"
        }
      },
      "client": {
        "metadata": {
          "header.test": "123",
          "key": "",
          "offset": "16",
          "partition": "0",
          "timestamp": "1623973349591253000",
          "topic": "foo",
          "transport": "kafka"
        },
        "payload": {
          "hello": "world"
        }
      }
    },
    {
      "@timestamp": "2021-06-17T23:41:19Z",
      "batch": {
        "info": {
          "date_human": "2021-06-17T23:41:19Z",
          "date_unix": "1623973279784094110",
          "request_id": "c50cb190-a7b2-4da9-8729-e4962a2d56d1",
          "source": "grpc-collector_grpc-collector-deployment-65676d4f86-zpwwj"
        }
      },
      "client": {
        "metadata": {
          "header.test": "123",
          "key": "",
          "offset": "15",
          "partition": "0",
          "timestamp": "1623973279590935000",
          "topic": "foo",
          "transport": "kafka"
        },
        "payload": {
          "hello": "world"
        }
      }
    },
    {
      "@timestamp": "2021-06-17T23:39:49Z",
      "batch": {
        "info": {
          "date_human": "2021-06-17T23:39:49Z",
          "date_unix": "1623973189782415716",
          "request_id": "0115f7d5-20fa-400a-bc14-e59dd6ac3c77",
          "source": "grpc-collector_grpc-collector-deployment-65676d4f86-rk6s9"
        }
      },
      "client": {
        "metadata": {
          "header.test": "123",
          "key": "",
          "offset": "14",
          "partition": "0",
          "timestamp": "1623973189591469000",
          "topic": "foo",
          "transport": "kafka"
        },
        "payload": {
          "hello": "world"
        }
      }
    }
  ]
}

POST /v1/collection/{id}/extended-search

Description

Initiate an extended search.

Header Parameters

Param	Type	Required	Description
Authorization	string	yes	Bearer token containing API key

Path Parameters

Param	Type	Required	Description
TODO	string	yes	Bearer token containing API key

Body Parameters

Param	Type	Required	Description
TODO	string	yes	Bearer token containing API key

Example response

{
  "task_id": "0c2afdf6-3e08-477e-9770-f1aa976a898e",
  "status": "processing"
}

GET /v1/collection/{id}/extended-search-results

Description

Get extended search results.

Header Parameters

Param	Type	Required	Description
Authorization	string	yes	Bearer token containing API key

Path Parameters

Param	Type	Required	Description
TODO	string	yes	Bearer token containing API key

Example response

{
  "next_page": "fdsgfg079sdgfz9g87fd90gfd===",
  "query_id": "84d97fed-9610-4b4b-ab6c-ef9279758758",
  "total_results": 300,
  "data": [
    {"batch": {...}, "client": {...}},
    {"batch": {...}, "client": {...}},
    {"batch": {...}, "client": {...}},
    {"batch": {...}, "client": {...}},
    {"batch": {...}, "client": {...}},
  ] 
}

• Page is optional.

• Omitting it will always get the first page of results.

• Specifying it with the value of "next_page" will get the next page of 20 results.

• If it is empty, there are no more pages of results

POST /v1/collection/{id}/initial

Description

Create initial event task

Header Parameters

Param	Type	Required	Description
Authorization	string	yes	Bearer token containing API key

Path Parameters

Param	Type	Required	Description
TODO	string	yes	Bearer token containing API key

Body Parameters

Param	Type	Required	Description
TODO	string	yes	Bearer token containing API key

Example Response

{
   "0dce7d85-8d61-41db-86bd-af005df755f4":{
      "latency_ms":0,
      "status":"error",
      "message":"Failed to connect",
      "last_checked_at":1624548189
   }
}

POST /v1/collection/{id}/zip

Description

Request a ZIP archive of search results.

Header Parameters

Param	Type	Required	Description
Authorization	string	yes	Bearer token containing API key

Path Parameters

Param	Type	Required	Description
TODO	string	yes	Bearer token containing API key

Body Parameters

Param	Type	Required	Description
TODO	string	yes	Bearer token containing API key

Example Response

{
   "0dce7d85-8d61-41db-86bd-af005df755f4":{
      "latency_ms":0,
      "status":"error",
      "message":"Failed to connect",
      "last_checked_at":1624548189
   }
}