Server Mode

Overview

The plumber tool allows you to read, write, relay and tunnel data right from your CLI. While this is OK for quick one-offs, it has some drawbacks - if you are planning to make use of relay or tunnel functionality in production or in a more serious capacity:

  1. You will need to launch a new plumber instance for every relay and tunnel

  2. Difficult to pull off high-availability

  3. Potential for missed data while plumber is not running (such as a redeploy)

This might be acceptable depending on your integration needs. If plumber exists in dev to facilitate testing or enable developers to tunnel data into their local workstations - it's probably fine. But if you are planning on having many relays and/or many tunnels, it wouldn't be well-suited.

To address this, we have built plumber server mode.

Plumber's server mode enables you to run any number of plumber instances as a distributed cluster. Server mode exposes a gRPC API which can be used to run and manage multiple relays and tunnels in parallel. Plumber server can run in either of two operational modes - standalone or cluster.

Operational Modes

Standalone Mode

In standalone mode, plumber will write its config to a local JSON file (default ~/.batchsh/config.json that can be overridden via PLUMBER_SERVER_FOO).

Suitable for:

  • Local and dev environments

  • Low throughput (<100 events per sec)

  • High-reliability is not required

Cluster Mode

In cluster mode, plumber uses a message bus (NATS) to facilitate communication between the plumber instances and for storing its configs. You can run any number of plumber instances - we recommend to start with 3.

Suitable for:

  • Production environments

  • High throughput (1,000+ events per sec)

  • High-reliability is required

How to Run

Standalone mode

When running in Docker, you will want to mount this file as a writable volume to the container.

# Install plumber
$ brew tap batchcorp/public
...
$ brew install plumber
...

# Launch plumber in standalone mode
$ plumber server
INFO[0000]
█▀█ █   █ █ █▀▄▀█ █▄▄ █▀▀ █▀█
█▀▀ █▄▄ █▄█ █ ▀ █ █▄█ ██▄ █▀▄
INFO[0000] starting plumber server in 'standalone' mode...  pkg=plumber
INFO[0005] plumber server started                        pkg=plumber

Cluster mode

The NATS container SHOULD have a persistent storage volume.

# Install plumber
$ brew tap batchcorp/public
...
$ brew install plumber
...

# Git clone plumber repo (to get access to its docker-compose + assets)
$ git clone git@github.com:batchcorp/plumber.git
$ cd plumber

# Launch a NATS container
$ docker-compose up -d natsjs

# Launch plumber in cluster mode 
$ plumber server --enable-cluster
INFO[0000]
█▀█ █   █ █ █▀▄▀█ █▄▄ █▀▀ █▀█
█▀▀ █▄▄ █▄█ █ ▀ █ █▄█ ██▄ █▀▄
INFO[0000] starting plumber server in 'cluster' mode...  pkg=plumber
INFO[0015] plumber server started                        pkg=plumber

Environment Variables

Cluster Mode

All of the following environment variables are only used if plumber is launched in cluster mode.

Metrics

Plumber exposes various prometheus metrics that are exposed via its internal HTTP server:

Last updated