# Upload with the AWS CLI

> Get an IAM role for your account, then use aws s3 cp and sync for inputs and outputs.

The API creates an IAM role for your account that can read and write your account's folder in
SyncWords' storage, and only that. Your own AWS identity assumes it, with an external ID only you
and SyncWords know. After a one-time setup you use plain `aws s3` commands or any S3 SDK.

The role reaches every project's folders, but only a project that reads its input folder (the
default `S3FolderSource`) processes what you copy into it. A project that reads a
[feed](/vod-express/concepts/inputs#feeds) fills its own input folder from the feed; a file
copied in beside them is never processed, and stays `pending`. Reading a feed project's outputs
with the role works as for any project.

## One-time setup

### 1. Find your AWS principal

The identity that will upload: a role (recommended, for example the one your pipeline runs
as), a user, or a whole AWS account by its 12-digit id.

```bash
aws sts get-caller-identity --query Arn --output text
# arn:aws:iam::444455556666:role/uploader
```

### 2. Ask for the role

```bash
curl -s -X PUT $API/account/aws-access -H "$AUTH" -H 'Content-Type: application/json' \
  -d '{"iamPrincipalArn": "arn:aws:iam::444455556666:role/uploader"}'
```

```json
{
  "data": {
    "iamPrincipalArn": "arn:aws:iam::444455556666:role/uploader",
    "roleArn": "arn:aws:iam::710713970991:role/sw-vod-express-vended-upload-141",
    "externalId": "5f0c1d2e-8a4b-4c6d-9e7f-0a1b2c3d4e5f",
    "bucket": "sw-vod-express",
    "region": "us-east-1",
    "s3Prefix": "s3://sw-vod-express/projects/141/"
  }
}
```

Calling it again with the same principal returns the same role. With another principal, the role
is re-pointed to trust the new one instead; the external ID stays. `GET /account/aws-access`
returns the current settings (`404` before the first `PUT`).

The role reaches every project of your account, so only your organization's API key or one
of its administrators may create, re-point or revoke it. Any other user gets `403` with
`code: "forbidden"`.

### 3. Allow your principal to assume it

Unless your principal is an administrator, give it permission to assume the role, in **your**
AWS account:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam::710713970991:role/sw-vod-express-vended-upload-141"
    }
  ]
}
```

### 4. Add an AWS CLI profile

In `~/.aws/config`:

```ini
[profile syncwords-vod]
role_arn = arn:aws:iam::710713970991:role/sw-vod-express-vended-upload-141
external_id = 5f0c1d2e-8a4b-4c6d-9e7f-0a1b2c3d4e5f
source_profile = default
region = us-east-1
```

`source_profile` is the profile of the principal from step 1. On EC2, ECS or Lambda use
`credential_source = Ec2InstanceMetadata` (or `EcsContainer`, `Environment`) instead. The CLI
assumes the role and renews its one-hour session by itself.

```bash
aws s3 ls s3://sw-vod-express/projects/141/ --profile syncwords-vod
```

## Upload

Every project's `storage.inputPath` is inside your `s3Prefix`:

```bash
INPUT=$(curl -s -H "$AUTH" $API/projects/$PROJECT | jq -r .data.storage.inputPath)

# One file
aws s3 cp "episode 01.mp4" "$INPUT" --profile syncwords-vod

# A folder, subfolders included
aws s3 sync ./season-1 "${INPUT}season 1/" --profile syncwords-vod --exclude '*' --include '*.mp4'

# Then make sure the project is running
curl -s -X POST -H "$AUTH" $API/projects/$PROJECT/start
```

A project that [stopped itself](/vod-express/concepts/projects#auto-stop) after its last file
starts again within a minute of a new file without that call, unless its
[`startOnNewFiles`](/vod-express/concepts/projects#starting-again-on-a-new-file) is off. One stopped with `POST …/stop`, or created
with `autoStart: false` and never started, waits for it.

S3 shows an object only once its upload is complete, so the project never picks up a partial file.

> **Note:**
> Uploads through the role are not checked by the API: name inputs with media extensions and give
> them distinct base names (`trailer.mov` and `trailer.mkv` would write the same `trailer.es.mp4`).

## Download

```bash
OUTPUT=$(curl -s -H "$AUTH" $API/projects/$PROJECT | jq -r .data.storage.outputPath)
aws s3 sync "$OUTPUT" ./results --profile syncwords-vod
```

## With an SDK

```python Python (boto3)
import boto3

sts = boto3.client("sts")
creds = sts.assume_role(
  RoleArn=access["roleArn"],
  ExternalId=access["externalId"],
  RoleSessionName="vod-upload",
)["Credentials"]

s3 = boto3.client(
  "s3",
  region_name=access["region"],
  aws_access_key_id=creds["AccessKeyId"],
  aws_secret_access_key=creds["SecretAccessKey"],
  aws_session_token=creds["SessionToken"],
)

# storage.inputPath = s3://<bucket>/<key prefix>
bucket, prefix = project["storage"]["inputPath"].removeprefix("s3://").split("/", 1)
s3.upload_file("episode 01.mp4", bucket, prefix + "episode 01.mp4")
```

```javascript Node.js (AWS SDK v3)
import { S3Client } from '@aws-sdk/client-s3';
import { Upload } from '@aws-sdk/lib-storage';
import { fromTemporaryCredentials } from '@aws-sdk/credential-providers';
import { createReadStream } from 'node:fs';

const s3 = new S3Client({
region: access.region,
credentials: fromTemporaryCredentials({
  params: { RoleArn: access.roleArn, ExternalId: access.externalId, RoleSessionName: 'vod-upload' },
}),
});

// storage.inputPath = s3://<bucket>/<key prefix>
const [bucket, ...rest] = project.storage.inputPath.replace('s3://', '').split('/');
await new Upload({
client: s3,
params: { Bucket: bucket, Key: rest.join('/') + 'episode 01.mp4', Body: createReadStream('episode 01.mp4') },
}).done();
```

## What the role can do

| On | Allowed |
|---|---|
| `s3Prefix` and everything under it | list, read, write, delete, multipart uploads |
| Anything else | nothing |

The prefix holds every project of your account, one folder each.

## Revoke the role

If the principal should lose access, for example when the person or pipeline behind it leaves,
revoke the role:

```bash
curl -s -X DELETE $API/account/aws-access -H "$AUTH"   # 204
```

Nobody can assume the role from then on, and credentials already issued through it stop working.
`GET /account/aws-access` answers `404` again. A later `PUT` restores the role for the principal
you name, with a **new external ID**, so update your profile's `external_id`.

> **Warning:**
> A file's processing records are kept by name. **Replacing an input with the AWS CLI (same name,
> new content) does not reprocess it**, and neither does deleting it and copying it in again: it
> reads `pending` with `stale: true` until you call [retry](/vod-express/concepts/files#retry) on the name. An
> API upload with `overwrite: true` does this for you. To replace a `failed` file, use a new name
> ([Replace](/vod-express/concepts/files#replace)).