# 7. The Runner

A **Runner** runs a folder's **scheduled flows around the clock**, on a machine the folder owner controls (a small server, a NAS, a Raspberry Pi), even when nobody has Blackslate open. It can also **send the tools' e-mails** with the owner's own mail account (7.9).

## 7.1 How it works

- The Runner is a small program, delivered as a **Docker image**: `ghcr.io/alexandrecrbpro-stack/leavenit-runner:1`.
- It belongs to **one data area** (a shared folder, My Workshop, or a folder with separate data) and only acts there.
- To run flows, it needs that data area to use **your own storage** (chapter 6). It holds **no storage key**: Blackslate signs each read and write, like for a browser. To only send e-mails, it doesn't.
- It only makes **outbound** connections: every 5 seconds it asks Blackslate whether a scheduled flow is due. **No port to open**, no domain name, no certificate.
- It runs **the same SQL as the browser**: SQLite with the same cleaning functions and the same rules (one transaction; no ATTACH, PRAGMA or VACUUM), checked by the same 40 engine checks. Since Runner 1.10 it computes **on its own disk**, not in memory: tables of tens of millions of rows (7.10). Each block runs in its own process, stopped at its time limit.
- It acts on behalf of the folder owner, within that folder, with the same rules as the website.
- Limits per block on a Runner: 2 hours; a table as large as its disk allows (about a third of its free disk, 7.10).

## 7.2 What you need

- A machine that stays on, with **Docker**, 64-bit (x86-64 or ARM64), 1 GB of memory or more.
- To run flows: the data area already on **your own storage** (chapter 6). To only send the tools' e-mails, not needed.
- To be the **owner** of the folder.
- To send e-mails: a mail account that allows sending with a password (7.9).

## 7.3 Adding a Runner in Blackslate

1. **Workshop** → the folder → **⋯** next to its name → **Runners**. (For a folder with separate data: its **⋯** → **Runners**.)
2. **Add a Runner** → name it after the machine ("Hetzner", "Home NAS") → **Get the command**.
3. Blackslate shows a **pairing code** (`LVN-XXXXX-XXXXX`, single use, valid 30 minutes) and the full command, on one line:

```
docker run -d --name leavenit-runner --restart unless-stopped --read-only --tmpfs /tmp -v leavenit-runner:/data -e LEAVENIT_URL=<Blackslate address> -e LEAVENIT_ANON_KEY=<Blackslate public key> -e LEAVENIT_PAIRING_CODE=LVN-XXXXX-XXXXX ghcr.io/alexandrecrbpro-stack/leavenit-runner:1
```

   The address and the public key are Blackslate's public values, already filled in. The pairing code is personal: don't share it.
   To send the tools' e-mails too, open **To send your tools' e-mails too** and add those lines before the image name (7.9).
4. **Copy the command**, run it on the machine (7.4). The panel shows **"✓ <name> is online."** within seconds.
5. Open a flow of this folder → **Schedule** → under **On**, choose **"<name> · Runner, always on"**, choose when, **Save**.

If the code expired, the panel offers **New code**.

**The step-by-step guide**: **Runners** → **Step-by-step guide for e-mails** (or **Set it up** above a tool that sends e-mails) does all this one step at a time: where the Runner runs, the machine, the mail account, the commands, a test e-mail. It checks what it can (the Runner online and sending as your address, then the test e-mail leaving) and remembers, on this device, where you stopped. Each step has **Ask my AI for help**: a new conversation with your AI, with the details of that step, and never a password or the pairing code (placeholders instead).

## 7.4 Machine guides

### A small cloud server: Hetzner Cloud (tested end to end)

1. **An SSH key** on your computer (PowerShell on Windows, Terminal on Mac/Linux):
   ```
   ssh-keygen -t ed25519 -C "leavenit-runner"
   ```
   Press Enter at each question. Show the public key: `Get-Content $env:USERPROFILE\.ssh\id_ed25519.pub` (Windows) or `cat ~/.ssh/id_ed25519.pub` (Mac/Linux). Never share the file without `.pub`.
2. **console.hetzner.cloud** → **New project** → **Add Server**:
   - **Location**: Falkenstein or Nuremberg.
   - **Image**: **Ubuntu 24.04**.
   - **Type**: Shared vCPU, the smallest (**CAX11** ARM64, or **CX22** x86).
   - **SSH keys**: **Add SSH key**, paste the `ssh-ed25519 …` line.
   - **Firewalls**: **Create Firewall** keeping only **inbound SSH (port 22)**.
   - **Create & Buy now**. Note the **IPv4 address**.
3. **Connect**: `ssh root@<IPv4 address>` (answer `yes` the first time).
4. **Install Docker**: `curl -fsSL https://get.docker.com | sh`
5. **Paste the command** from Blackslate (7.3).
6. **Check**: `docker logs leavenit-runner`

Scaleway (Instances) and OVHcloud (VPS or Public Cloud) work the same way: Ubuntu 24.04, your SSH key, SSH only in the firewall, then steps 3 to 6.

### A Synology NAS

1. **Package Center** → install **Container Manager** (only on models that support it).
2. **Control Panel** → **Terminal & SNMP** → enable **SSH**.
3. From a computer: `ssh <your user>@<NAS address>`.
4. Paste the Blackslate command with `sudo ` in front, then enter your password.
5. Check: `sudo docker logs leavenit-runner`.

### A Raspberry Pi

A Raspberry Pi 4 or 5 with a **64-bit** system (Raspberry Pi OS 64-bit). Install Docker with `curl -fsSL https://get.docker.com | sh`, then paste the command with `sudo ` in front.

### A computer with Docker Desktop (for a test)

Install **Docker Desktop**, start it, paste the command in a terminal (PowerShell on Windows). The Runner stops when the computer sleeps or shuts down: fine for a test, not for real use.

## 7.5 What the logs say

`docker logs leavenit-runner` (or `docker logs --tail 50 leavenit-runner`):

| Line | Meaning |
|---|---|
| `Blackslate Runner 1.0.0: engine self-test OK.` | the engine starts, computes and refuses to leave its sandbox |
| `Paired as "<name>".` | the pairing code was accepted (first start only) |
| `Runner "<name>" ready. Checking every 5 s.` | waiting for work |
| `<flow>: run for <time>` then `  <block>: v<n>, <rows> rows` | a scheduled flow ran; each block published a new version of its table |
| `  <block>: failed (<code>) <message>` | a block failed; the flow run is recorded as failed in Blackslate with this message |
| `  sent <n> rows to <warehouse>: leavenit.<table> (<replace / append / upsert>) in <s> s` | a block's table was also sent to a warehouse (4.8) |
| `  not sent (<code>): <message>` then `  <block>: v<n>, <rows> rows, not sent (<code>)` | the table was published, but the warehouse refused it; the flow run is recorded as failed, the next blocks still run |
| `  <block>: <n> rows in leavenit.<table> (<warehouse>, <replace / append>) in <s> s` | a block ran in the warehouse (4.10): the warehouse made (or added to) its table; no row came to the Runner |
| `Live query <name> (<warehouse>): <n> rows in <ms> ms` | a tool called a live query (4.9) |
| `<flow>: built now (asked in Blackslate)` | someone clicked **Build on my Runner** (7.10): the whole flow runs now |
| `Try <block>: <n> rows in <s> s.` | someone clicked **Try it on my Runner** on a block that reads a large table: nothing was published |
| `Warning: no memory limit on this system: …` | the system gives no memory limit to the engine (Windows without Docker): a block could use all the machine's memory |
| `<flow>: paused (<reason>)` | the schedule is paused (for example the flow changed since it was scheduled): review it in Blackslate |
| `Problem: <message>` | a temporary problem (network…); it tries again at the next check |
| `This Runner was removed in Blackslate. Stopping. Pair it again with a new code to restart it.` | the Runner was removed in Blackslate: it stopped and forgot its token |
| `This Runner is not paired yet: start it once with LEAVENIT_PAIRING_CODE.` | started without a code and without a saved token: get a new command in Blackslate |
| `E-mail: ready, sending from <address> (<server>).` | the mail account works: tools can send e-mails |
| `E-mail: the mail account doesn't answer (<message>). Tools can't send e-mails until it works.` | wrong server, port, login or password: see 7.9; it checks again every 5 minutes |
| `E-mail: <n> sent.` | e-mails left |
| `E-mail to <address> not sent: <message>` | the mail server refused this e-mail (address that doesn't exist, daily limit of the account…); the tool sees it as failed |
| `Blackslate expects Runner protocol <n>; this Runner speaks <m>. Update the image.` | the image is too old: update it (7.6) |
| `Engine self-test failed: …` | the machine can't run the engine: check it's 64-bit with enough memory |
| `  <block>: failed (sql_too_large) database or disk is full` | the block's table didn't fit in the disk the Runner may use (7.10): give the volume more disk, or raise `LEAVENIT_ENGINE_DISK_GB` |

## 7.6 Updating, stopping, removing

- **Update by hand** (no third-party tool): `docker rm -f leavenit-runner`, then run the start command again with `--pull always` after `docker run -d`, and **without** `-e LEAVENIT_PAIRING_CODE=…` (the token, and a mail account saved with `mail-setup`, are kept in the `leavenit-runner` volume). The step-by-step guide for e-mails gives this command, filled in.
- **Automatic updates**, with Watchtower: it updates the Runner every night at 4:00. Use its maintained version, `nickfedor/watchtower`: the original, `containrrr/watchtower`, is archived and refused by recent Docker ("client version 1.25 is too old"). Watchtower is a third-party project and controls Docker on the machine:
  ```
  docker run -d --name watchtower --restart unless-stopped -v /var/run/docker.sock:/var/run/docker.sock nickfedor/watchtower --schedule "0 0 4 * * *" leavenit-runner
  ```
  Once, right now: `docker run --rm -v /var/run/docker.sock:/var/run/docker.sock nickfedor/watchtower --run-once leavenit-runner`. Already running the old one: `docker rm -f watchtower` first.
- **Pin a version**: replace `:1` by an exact version (`:1.0.0`). Go back to the previous one by changing this tag.
- **Stop for a while**: `docker stop leavenit-runner` (`docker start leavenit-runner` to resume). After one hour offline with active schedules, the owner gets a notification: "Your Runner "<name>" has been offline for over an hour. Its scheduled flows are waiting."
- **Remove for good**: in Blackslate, **Runners** → **Remove** → confirm. The Runner stops by itself at its next check and its schedules are paused. Then on the machine: `docker rm -f leavenit-runner` and `docker volume rm leavenit-runner`. Delete the cloud server if you rented one, so billing stops.

## 7.7 Security

- The pairing code works once, for 30 minutes. The token the Runner receives is stored only on its machine (in the `leavenit-runner` volume); Blackslate keeps only a fingerprint of it.
- The token only gives access to **this folder**, only to run its scheduled flows, and is revoked by **Remove**.
- The Runner holds no storage key and no Blackslate password. The mail account's password is only in the command on your machine: Blackslate never receives it, it only learns the sending address.
- **Fetch from a web address** blocks (Runner 1.4, chapter 4.5): the Runner reads https addresses with a public name only. It resolves the name itself and refuses it if any of its addresses is private or local (your home network, a cloud server's internal address 169.254.169.254); it follows no redirection; 20 MB and 30 seconds at most (several pages: each page checked the same way, a next address only on the same name, 50 pages, 20 MB and 2 minutes together; Runner 1.5). It receives a key only for the run that needs it, only for the address the key is bound to, and never writes it in its logs.
- **Data warehouses** (Runner 1.7, chapter 4.7): the Runner connects to the warehouses the folder's **owner** declared, and only to them (the address may be in a private network: the owner chose it; the members who write blocks only choose a warehouse by its name). It receives a warehouse's sign-in details only for the run or the request that needs them, keeps them in memory only, and never writes them in its logs. A query that isn't one select is refused; files of a data lake are written in its data folder (the `leavenit-runner` volume) while a read lasts, then deleted, and the query can read no other file of the machine.
- **Writing in a warehouse** (Runner 1.8, chapter 4.8): only when the folder's owner turned **Writing** on for that warehouse, and only in the schema (or folder) the owner chose; the Runner never writes in a table Blackslate didn't create there (it checks before, and Blackslate keeps the list). A replaced table is filled on the side and swapped in at the end. The table goes from the Runner to the warehouse: nothing goes through Blackslate's servers.
- **Large tables** (Runner 1.10, 7.10): each block computes in a folder of its own in the `leavenit-runner` volume, removed after it; the SQL can read or write no other file of the machine (ATTACH, VACUUM and PRAGMA are refused, extensions can't be loaded), and the engine process gets a memory limit from the system.
- **Live queries** (Runner 1.9, chapter 4.9): the Runner runs only the selects the folder's **owner** declared, each value given to the warehouse as a parameter (never inside the SQL), in a read-only transaction where there is one; 30 seconds, 10,000 rows and 2 MB at most.
- **Web services** (Runner 1.6, chapter 5): the folder's tools call the addresses its owner or admins declared, through the Runner, which adds the key. Same checks as Fetch blocks (https, a public name, no private address, no redirection), 2 MB and 15 seconds per call. After a call, the Runner checks every second for 2 minutes (only for these calls), so a tool in use gets its answers quickly.
- Keep the machine updated (`apt update && apt upgrade -y` on Ubuntu), keep only SSH open, and protect your SSH key.

## 7.8 Troubleshooting

| What you see | Cause | Fix |
|---|---|---|
| "Flows need your own storage…" in the Runners panel | the folder's tables are in Blackslate's storage | the Runner can still send e-mails; to run flows, connect your own storage (chapter 6) with **Open Storage** |
| "A Runner runs flows only when the folder's tables are in your own storage…" when scheduling | same | connect your own storage, or schedule the flow on "This computer" |
| Watchtower stops with "client version 1.25 is too old" | the original Watchtower (`containrrr/watchtower`) is archived; recent Docker refuses it | `docker rm -f watchtower`, then the `nickfedor/watchtower` command of 7.6, or update by hand |
| The Runners panel says "No e-mail account set up" | the Runner was started without the `LEAVENIT_SMTP_…` lines, or its account doesn't answer | add them (7.9), or read `docker logs leavenit-runner` |
| The panel keeps "Waiting for the Runner…" | the command didn't run, or the code expired | check `docker logs leavenit-runner`; if needed **New code** and run the new command |
| "This pairing code is not valid. Create a new one in Blackslate." | code already used, expired, or mistyped | **Add a Runner** again |
| Windows PowerShell rejects the command | the command was cut over several lines | copy it again from Blackslate: it is on one line |
| `docker: command not found` | Docker isn't installed | install it (7.4) |
| `permission denied … docker.sock` | your user can't use Docker | put `sudo ` in front of the command |
| "This Runner belongs to another folder." when scheduling | a Runner only serves flows of its own folder | add a Runner to this folder |
| A flow fails with "The tables of this folder are no longer in your own storage." | the folder moved back to Blackslate's storage | move it to your storage again, or schedule the flow on "This computer" |
| A warehouse test says "The folder's Runner didn't answer in time" | the Runner is older than 1.7, or off | update it (7.6), check `docker logs leavenit-runner` |
| "Send now from my Runner" says "The folder's Runner didn't answer in time", or the test says "The Runner didn't check writing" | the Runner is older than 1.8 | update it (7.6) |
| A live query's "Try it on my Runner" (or a tool's call) doesn't answer | the Runner is older than 1.9, or off | update it (7.6) |
| The Runner is offline after the server rebooted | Docker didn't start | `systemctl enable --now docker`; the Runner restarts by itself (`--restart unless-stopped`) |

## 7.9 Sending the tools' e-mails

A tool can send an e-mail (an order confirmation, a reminder) with `window.leavenit.mail.send` (chapter 5.4). The Runner of the folder sends it with **your own mail account**, from your address: Blackslate doesn't send e-mails for tools. Without a Runner set up for e-mail, the tool is told so and can offer a prepared e-mail instead (chapter 9.2).

**Set it up, the simplest way** (Runner 1.3 or later, what the step-by-step guide does): the Runner asks for the password itself. A first command saves the mail account in the Runner's volume:

```
docker run -it --rm --pull always -v leavenit-runner:/data -e LEAVENIT_SMTP_HOST=smtp.gmail.com -e LEAVENIT_SMTP_USER=you@gmail.com -e "LEAVENIT_MAIL_FROM=Your name <you@gmail.com>" ghcr.io/alexandrecrbpro-stack/leavenit-runner:1 mail-setup
```

It asks for the password without showing it (a Gmail app password works with or without its spaces), checks it with your provider, and saves it only if it works: `✓ smtp.gmail.com accepted it.` Then start the Runner as in 7.3, with `--pull always` after `docker run -d`. For a Runner that already runs: the first command, then `docker rm -f leavenit-runner`, then the start command without the pairing code (its pairing stays in the volume). The `LEAVENIT_SMTP_…` variables below still work, and take precedence over a saved account.

**Or in the start command**: add these lines before the image name (the panel offers them under **To send your tools' e-mails too**), keeping the quotes around the password:

```
-e LEAVENIT_SMTP_HOST=smtp.gmail.com -e LEAVENIT_SMTP_USER=you@gmail.com -e "LEAVENIT_SMTP_PASSWORD=your-app-password" -e "LEAVENIT_MAIL_FROM=Your name <you@gmail.com>"
```

| Setting | What to put |
|---|---|
| `LEAVENIT_SMTP_HOST` | your provider's sending server (table below) |
| `LEAVENIT_SMTP_PORT` | `465` (the default) or `587`; leave it out for 465 |
| `LEAVENIT_SMTP_USER`, `LEAVENIT_SMTP_PASSWORD` | the account's login and password |
| `LEAVENIT_MAIL_FROM` | the name and address people see: `"Pizzeria Alex <contact@pizzeria.fr>"` (in quotes); by default, the login |

| Provider | Server | Notes |
|---|---|---|
| Gmail | `smtp.gmail.com` | turn on 2-Step Verification, then create an **app password** (Google account → Security → App passwords) and use it, not your usual password. About 500 e-mails a day |
| OVHcloud (e-mail with your domain) | `ssl0.ovh.net` | the address's full name as login, its password |
| Brevo | `smtp-relay.brevo.com`, port `587` | free plan: 300 e-mails a day; login and **SMTP key** under **SMTP & API** |

**An existing Runner**: stop it and start it again with the extra lines, keeping the same volume and without the pairing code: `docker rm -f leavenit-runner`, then the command without `-e LEAVENIT_PAIRING_CODE=…` and with the e-mail lines. The Runners panel then shows **"✉ Sends e-mails as <address>"**.

**Good to know**
- The password stays on your machine (in the command, or in the Runner's volume with `mail-setup`); Blackslate only learns the sending address, to tell the tools.
- Plain text only. Up to 200 e-mails a day per folder (on top of your provider's own limit); each one is listed in the folder's activity, and the content is erased after 30 days.
- E-mails can be **scheduled** (a reminder the day before): the Runner sends each one when its time comes, so it must be on then. If it's offline, an e-mail waits up to 24 hours after its time, then fails: it is never sent later than that, and the person who asked for it is told (notification in **Flows and Runners**).
- Only members who can edit the folder can send; visitors of a public link can't.
- What you send engages you: send only to people who expect it (a customer who ordered, a volunteer who signed up).

## 7.10 Large tables (Runner 1.10)

A table of more than **2,000,000 rows** (or 300 MB) is **large**. The folder's Runner computes the blocks that read it, with SQLite **on its own disk**: the same SQL and the same functions as in the browser, for tables of tens of millions of rows. The browser shows such a table's first 200 rows instead of loading it whole.

- **In a flow**: the large table says **Large** under its name, and the blocks that read it say **Runner**. **Build on my Runner** (instead of **Build all**, when a flow has blocks that run on the Runner) builds the whole flow now, without waiting for its schedule; its progress shows above the graph ("Building on Office server: block 2 of 3, …"), and a failure goes to **Last runs** as usual.
- **In a block** that reads a large table: **Try it on my Runner** runs the SQL as it is in the editor on the Runner, publishes nothing, and shows the number of rows, the columns and the first rows. (**Dry run** and **Run** are for the browser.)
- **The table** (**Open table**): its first 200 rows, **Summarize in a new block** (a new block that reads it, with a first summary to change) and **Download (SQLite, …)**: the table file itself (a gzipped SQLite file), saved straight from your storage.
- **A tool** can't read a large table whole (`err.code` `table_too_large`): it reads the smaller table a block makes from it, or a live query (4.9).
- **The Runner's disk**: a block needs about **three times** the size of its largest table, while it computes. **Runners** shows what the Runner's machine allows ("Disk: 31 GB free · tables up to about 10 GB (≈ 150 M rows) · 4 CPUs, 8 GB memory"), and names a table it couldn't compute with that disk. Give the volume more disk, or do the heavy part in the warehouse (7.11). Optional settings in the start command: `-e LEAVENIT_ENGINE_DISK_GB=40` (the most a block's working database may take; half the free disk by default) and `-e LEAVENIT_ENGINE_MEMORY_MB=2000` (half the machine's memory, 4 GB at most, by default).
- **A block reading a warehouse** can bring up to **50,000,000 rows** (At most, 4.7).
- **In a folder without a Runner**, a large table can't be read by a block: add a Runner (7.3).

## 7.11 Blocks that run in a warehouse (Runner 1.11)

A block **Run in a warehouse** (4.10) sends its query to the warehouse, which makes a table of its result in the schema Blackslate may write in. **No row comes to the Runner**: a query over billions of rows takes what the warehouse takes, and the Runner's disk and memory don't matter. The Runner wraps the select (`create table … as select`, then swaps the new table in, or adds its rows), and never touches a table Blackslate didn't create.

- **Order**: the blocks run in the order their queries name each other's tables (`leavenit.stores_clean`), then the **Read from a warehouse** blocks that read them.
- **Schedules "Only what changed"**: the blocks that read from outside (a web address, a warehouse) or run in a warehouse always run, since what they read may have changed (Runner 1.11; before, they ran only when their script changed).
- **An older Runner** refuses such a block ("Running a block in a warehouse needs Runner 1.11 or later"): update it (7.6).

## 7.12 Dev, Test and Prod (Runner 1.12)

A shared folder has three versions (4.11): Prod, Dev and Test, each with its own tables, flows and warehouses. **The folder's Runner serves the three**, on the same machine: one Runner, paired once.

- **Its log** starts a flow's lines with its version: `[dev] Budget du mois: run for 2026-11-17T05:00:00Z`, `[test] Budget du mois: built now (asked in Blackslate)`; nothing for Prod.
- **Runner 1.12** tells Blackslate which version each table it builds belongs to. An older Runner keeps serving Prod only: Dev and Test flows can't be scheduled on it nor built there ("Dev and Test need Runner 1.12 or later"), until it is updated (7.6).
- A heavy flow in Dev uses the same machine as Prod: schedule it outside Prod's hours if the machine is small.

## 7.13 Sending to a web service (Runner 1.13)

A block **Send to a web service** (4.12) sends each row of its result to one of the folder's web services: **the Runner makes the calls**, with the service's key (from Blackslate's vault, for that service only), one call per row, 5 per second, 500 per run at most, 15 seconds per call, each with the header `Idempotency-Key`. It reads back the block's log (its table) at each run, so that a row is sent once; a 5xx or a timeout is sent again at the next runs (5 tries).

- **Its log**: `sent to api.shop.com: 4 sent, 1 to retry, 1 failed, 12 already sent`; in Dev or Test without the service: `simulated api.shop.com: 6 calls (not declared in this version)`. Never the key.
- **An older Runner** builds the flow's other blocks; the sending block stops ("Sending to a web service needs Runner 1.13 or later"), until it is updated (7.6).

## 7.14 Descriptions of warehouse tables (Runner 1.14)

When someone opens the columns of a warehouse table (**Browse**, 4.13), the Runner also reads the comments the warehouse already has on the table and its columns; it never writes them. **Profile these columns** reads the table's first 10,000 rows and gives, for each column, its typical values, min – max, % empty and distinct values; nothing is kept. An older Runner shows the columns without the warehouse's comments, and can't profile ("Unknown request": update it, 7.6).
