# Monitoring and polling

> Knowing when files are done, and what to do when they are not.

[Webhooks](/vod-express/guides/webhooks) tell your server when each file is done or failed. Without a server
to receive them, or to catch up after one was down, poll. One call tells you everything about a
project's files, and `meta` counts them by state:

```bash
curl -s -H "$AUTH" $API/projects/$PROJECT/files | jq .meta
```

```json
{ "total": 3, "pending": 1, "processing": 0, "done": 1, "failed": 1 }
```

`?status=failed` lists only the failed files (`meta` still counts all of them), and
`GET /projects/{id}/files/{name}` answers one file, the call to poll when you wait for one.

## A good polling loop

- Poll `GET /projects/{id}/files` **every 30 to 60 seconds**. Files take minutes, and faster
  polling gains nothing.
- Stop when `meta.pending` and `meta.processing` are both 0. Files wait `pending` while the
  project is stopped, so a loop that runs long should check `GET /projects/{id}/status` too
  ([When a file stays pending](#when-a-file-stays-pending)). An API upload or a retry starts a
  project that ran before; a file copied in with the AWS CLI does not.
- **Renew the token** when a call answers `401`, once: a season takes longer than a token lasts
  (30 minutes). If the new token is refused too, stop: the key pair is wrong.
- For many projects, poll each one in turn: 1000 requests a minute is plenty, and a `429` says in
  `Retry-After` how long to wait.

This loop does all four, and prints each file's state as it goes:

```bash cURL
# SW_PUBLIC_KEY, SW_SECRET_KEY and PROJECT set
API=${SW_VOD_API:-https://api-vod-express.syncwords.com/v1}

new_token() {
TOKEN=$(curl -s "$API/token/create" -H 'Content-Type: application/json' -H 'Accept: application/json' \
  -d "$(jq -n --arg u "$SW_PUBLIC_KEY" --arg p "$SW_SECRET_KEY" '{username: $u, password: $p}')" \
  | jq -r .access_token)
}

new_token; RENEWED=1
while :; do
STATUS=$(curl -s -o files.json -D headers.txt -w '%{http_code}' \
  -H "Authorization: Bearer $TOKEN" -H 'Accept: application/json' "$API/projects/$PROJECT/files")
case $STATUS in
  401) [ -n "$RENEWED" ] && { echo "a new token is refused too: check the key pair"; exit 1; }
       new_token; RENEWED=1; continue ;;
  429) sleep "$(awk 'tolower($1)=="retry-after:" {print $2+0}' headers.txt)"; continue ;;
  200) RENEWED= ;;
  *) echo "HTTP $STATUS: $(cat files.json)"; exit 1 ;;
esac
echo "$(date +%T) $(jq -c .meta files.json)"
jq -e '.meta.pending + .meta.processing == 0' files.json > /dev/null && break
sleep 60
done
jq -r '.data[] | "\(.status)\t\(.name)"' files.json
```

```python Python
import os
import time

import requests

API = os.environ.get("SW_VOD_API", "https://api-vod-express.syncwords.com/v1")
project = os.environ["PROJECT"]
token = None

def new_token():
  global token
  response = requests.post(
      API + "/token/create",
      json={"username": os.environ["SW_PUBLIC_KEY"], "password": os.environ["SW_SECRET_KEY"]},
      headers={"Accept": "application/json"},
  )
  response.raise_for_status()
  token = response.json()["access_token"]

def get(path):
  renewed = token is None
  if renewed:
      new_token()
  while True:
      response = requests.get(
          API + path, headers={"Authorization": f"Bearer {token}", "Accept": "application/json"}
      )
      if response.status_code == 401:
          if renewed:
              raise RuntimeError("a new token is refused too: check the key pair")
          new_token()
          renewed = True
      elif response.status_code == 429:
          time.sleep(int(response.headers.get("Retry-After", "5")))
      else:
          response.raise_for_status()
          return response.json()

while True:
  counts = get(f"/projects/{project}/files")["meta"]
  print(time.strftime("%X"), counts)
  if counts["pending"] + counts["processing"] == 0:
      break
  time.sleep(60)
```

```javascript Node.js
const API = process.env.SW_VOD_API ?? 'https://api-vod-express.syncwords.com/v1';
const project = process.env.PROJECT;
const sleep = (seconds) => new Promise((resolve) => setTimeout(resolve, seconds * 1000));
let token;

async function newToken() {
const response = await fetch(`${API}/token/create`, {
  method: 'POST',
  headers: { 'Content-Type': 'application/json', Accept: 'application/json' },
  body: JSON.stringify({ username: process.env.SW_PUBLIC_KEY, password: process.env.SW_SECRET_KEY }),
});
if (!response.ok) throw new Error(`token: ${response.status} ${await response.text()}`);
token = (await response.json()).access_token;
}

async function get(path) {
let renewed = !token;
if (renewed) await newToken();
for (;;) {
  const response = await fetch(API + path, {
    headers: { Authorization: `Bearer ${token}`, Accept: 'application/json' },
  });
  if (response.status === 401) {
    if (renewed) throw new Error('a new token is refused too: check the key pair');
    await newToken();
    renewed = true;
  } else if (response.status === 429) await sleep(Number(response.headers.get('retry-after') ?? 5));
  else if (!response.ok) throw new Error(`GET ${path}: ${response.status} ${await response.text()}`);
  else return response.json();
}
}

for (;;) {
const { meta: counts } = await get(`/projects/${project}/files`);
console.log(new Date().toLocaleTimeString(), counts);
if (counts.pending + counts.processing === 0) break;
await sleep(60);
}
```

## When a file stays pending

### 1. Is the project running?

```bash
curl -s -H "$AUTH" $API/projects/$PROJECT/status | jq .data
```

`NOT_RUNNING` (`projectStatus: "stopped"`): it stopped itself after being idle, or was stopped.
Start it: `POST /projects/{id}/start`. A project that stopped itself after its last file starts
on its own within a minute of a new file; one stopped with `POST …/stop` only when an API upload
in parts completes or a file is retried; one with `startOnNewFiles` off only on a retry. See
[Starting again on a new file](/vod-express/concepts/projects#starting-again-on-a-new-file).

### 2. Did the service fail?

`status: "ERROR"` means the service did not launch (`lastError` says so) or launched and then
stopped working (`pod.status` is `failed`, since `pod.at`). `POST /projects/{id}/start` replaces
it. A service that has read `failed` for 2 hours is stopped, and reads `NOT_RUNNING`.

### 3. Are other files still running?

The service processes a few files at a time; the rest wait `pending` until one finishes.
Nothing to do but wait.

## When a file failed

`failureCode` says why, when the cause is one you can fix yourself:

| `failureCode` | Cause | Fix |
|---|---|---|
| `no_audio` | The file has no audio track | Nothing to dub; delete it. |
| `not_media` | The file is not media, or is truncated | Upload a good copy **under a new name**, and delete the failed one. |
| `voice_not_found` | The project config names a voice the engine does not have | Correct the voice ([Choosing a voice](/vod-express/configuration/voices)), `PATCH` the project, retry the file. |
| `interrupted` | Processing was stopped before it finished: the worker was replaced, or the project was stopped | Retry the file. |
| `internal` | Something on our side | Retry the file once. |

```bash
curl -s -X POST -H "$AUTH" $API/projects/$PROJECT/files/$(jq -rn --arg n "$NAME" '$n|@uri')/retry
```

> **Warning:**
> A failed file's replacement needs a **new name** for now. The processing service keeps its own
> copy of a failed input, found by the file's name, and processes that copy again when the name
> comes back, even with new content: `overwrite: true`, copying over it with the AWS CLI, or
> deleting and uploading it again all reprocess the old bytes.

If a retry fails again, contact [support](/vod-express/support) with the project id, the file name and its
`hash`.

## A file the engines failed on

A file whose voice or translation engine reported an error, such as a line the voice could not
say, is `failed`, not `done` with silent gaps. Retry it: a retry processes it from the start, so a
voice or engine changed in the config since applies. If it fails again, contact
[support](/vod-express/support) with the project id, the file name and its `hash`, which finds the
file's log.

## Project status at a glance

```bash
curl -s -H "$AUTH" "$API/projects?perPage=100" | jq -r '.data[] | "\(.status)\t\(.id)\t\(.name)"'
```