Tunnels

AKA "Dynamic Destination"

Our open source messaging tool Plumber can act as a dynamic replay destination. We call this "tunnels".

Tunnel mode allows Plumber to receive messages from a replay and write them to your messaging system. This means you can run Plumber from within your infrastructure, negating the need to adjust firewalls or network security settings to allow Batch to replay messages from our infrastructure to yours.

In other words, this is what tunnels do:

Prior to 02.15.2022, "tunnels" were known as "dynamic destinations". While most things reflect this change, some parts of our website and documentation may still refer to this functionality by the old name. We are expecting to have everything converted to the new "tunnel" name very soon.

Getting Started

Create API token in your Batch console

Navigate to the Account Security page and click the lock icon to create a new API token

Give your token a name to remember what it is used for

Copy your newly created API token, you will need to include it with the plumber command below

Run Plumber

You can run plumber in tunnel mode or in server mode. To keep things simple, we will use tunnel mode for now.

plumber tunnel kafka --api-token ... \
  --address localhost:9092 \
  --topics new-orders
  
INFO[0000]
█▀█ █   █ █ █▀▄▀█ █▄▄ █▀▀ █▀█
█▀▀ █▄▄ █▄█ █ ▀ █ █▄█ ██▄ █▀▄
INFO[0000] Connection authorized. Waiting for replay messages...  pkg=dproxy

Create Replay

A destination will be automatically created for you titled Dynamic Plumber [IPADDRESS] - [MESSAGE BUS]. Select it as the destination for your replay and begin the replay

Plumber will indicate when the replay has started. You may pause/resume the replay just like any other. Plumber will alert you to any replay related events such as Finish/Pause/Resume/Abort

INFO[0000]
█▀█ █   █ █ █▀▄▀█ █▄▄ █▀▀ █▀█
█▀▀ █▄▄ █▄█ █ ▀ █ █▄█ ██▄ █▀▄
INFO[0000] Connection authorized. Waiting for replay messages...  pkg=dproxy
INFO[0016] Replay starting                               pkg=dproxy replay_id=30ddb850-1aca-4ee5-870c-1bb7b339ee5d
INFO[0025] Replay paused                                 pkg=dproxy replay_id=30ddb850-1aca-4ee5-870c-1bb7b339ee5d
INFO[0038] Replay resuming                               pkg=dproxy replay_id=30ddb850-1aca-4ee5-870c-1bb7b339ee5d
INFO[0077] Replay finished!                              pkg=dproxy replay_id=30ddb850-1aca-4ee5-870c-1bb7b339ee5d

As soon as the replay has finished, you can control+c to stop running Plumber. The destination will remain in your Batch account for up to 3 hours, in case you need to reconnect and perform another replay to the same destination.

After 3 hours, the destination will be deleted, and a new one will be created if you run plumber in tunnel mode from the same machine again. When this happens, you will not be able to resume/redo any existing replays for that destination. But no worries! You can simply create a fresh replay.

When plumber is ran in server mode, tunnels are kept around until they are removed from the console.