Upload with the AWS CLI

The API creates an IAM role for your account that can read and write your account’s folder in SyncWords’ storage, and only that. Your own AWS identity assumes it, with an external ID only you and SyncWords know. After a one-time setup you use plain aws s3 commands or any S3 SDK. The role reaches every project’s folders, but only a project that reads its input folder (the default S3FolderSource) processes what you copy into it. A project that reads a feed fills its own input folder from the feed; a file copied in beside them is never processed, and stays pending. Reading a feed project’s outputs with the role works as for any project.

One-time setup

Find your AWS principal

The identity that will upload: a role (recommended, for example the one your pipeline runs as), a user, or a whole AWS account by its 12-digit id.

aws sts get-caller-identity --query Arn --output text
# arn:aws:iam::444455556666:role/uploader

Ask for the role

curl -s -X PUT $API/account/aws-access -H "$AUTH" -H 'Content-Type: application/json' \
  -d '{"iamPrincipalArn": "arn:aws:iam::444455556666:role/uploader"}'

{
  "data": {
    "iamPrincipalArn": "arn:aws:iam::444455556666:role/uploader",
    "roleArn": "arn:aws:iam::710713970991:role/sw-vod-express-vended-upload-141",
    "externalId": "5f0c1d2e-8a4b-4c6d-9e7f-0a1b2c3d4e5f",
    "bucket": "sw-vod-express",
    "region": "us-east-1",
    "s3Prefix": "s3://sw-vod-express/projects/141/"
  }
}

Calling it again with the same principal returns the same role. With another principal, the role is re-pointed to trust the new one instead; the external ID stays. GET /account/aws-access returns the current settings (404 before the first PUT).The role reaches every project of your account, so only your organization’s API key or one of its administrators may create, re-point or revoke it. Any other user gets 403 with code: "forbidden".

Allow your principal to assume it

Unless your principal is an administrator, give it permission to assume the role, in your AWS account:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam::710713970991:role/sw-vod-express-vended-upload-141"
    }
  ]
}

Add an AWS CLI profile

In ~/.aws/config:

[profile syncwords-vod]
role_arn = arn:aws:iam::710713970991:role/sw-vod-express-vended-upload-141
external_id = 5f0c1d2e-8a4b-4c6d-9e7f-0a1b2c3d4e5f
source_profile = default
region = us-east-1

source_profile is the profile of the principal from step 1. On EC2, ECS or Lambda use credential_source = Ec2InstanceMetadata (or EcsContainer, Environment) instead. The CLI assumes the role and renews its one-hour session by itself.

aws s3 ls s3://sw-vod-express/projects/141/ --profile syncwords-vod

Upload

Every project’s storage.inputPath is inside your s3Prefix:

INPUT=$(curl -s -H "$AUTH" $API/projects/$PROJECT | jq -r .data.storage.inputPath)

# One file
aws s3 cp "episode 01.mp4" "$INPUT" --profile syncwords-vod

# A folder, subfolders included
aws s3 sync ./season-1 "${INPUT}season 1/" --profile syncwords-vod --exclude '*' --include '*.mp4'

# Then make sure the project is running
curl -s -X POST -H "$AUTH" $API/projects/$PROJECT/start

A project that stopped itself after its last file starts again within a minute of a new file without that call, unless its startOnNewFiles is off. One stopped with POST …/stop, or created with autoStart: false and never started, waits for it. S3 shows an object only once its upload is complete, so the project never picks up a partial file.

Uploads through the role are not checked by the API: name inputs with media extensions and give them distinct base names (trailer.mov and trailer.mkv would write the same trailer.es.mp4).

Download

OUTPUT=$(curl -s -H "$AUTH" $API/projects/$PROJECT | jq -r .data.storage.outputPath)
aws s3 sync "$OUTPUT" ./results --profile syncwords-vod

With an SDK

import boto3

sts = boto3.client("sts")
creds = sts.assume_role(
    RoleArn=access["roleArn"],
    ExternalId=access["externalId"],
    RoleSessionName="vod-upload",
)["Credentials"]

s3 = boto3.client(
    "s3",
    region_name=access["region"],
    aws_access_key_id=creds["AccessKeyId"],
    aws_secret_access_key=creds["SecretAccessKey"],
    aws_session_token=creds["SessionToken"],
)

# storage.inputPath = s3://<bucket>/<key prefix>
bucket, prefix = project["storage"]["inputPath"].removeprefix("s3://").split("/", 1)
s3.upload_file("episode 01.mp4", bucket, prefix + "episode 01.mp4")

import { S3Client } from '@aws-sdk/client-s3';
import { Upload } from '@aws-sdk/lib-storage';
import { fromTemporaryCredentials } from '@aws-sdk/credential-providers';
import { createReadStream } from 'node:fs';

const s3 = new S3Client({
  region: access.region,
  credentials: fromTemporaryCredentials({
    params: { RoleArn: access.roleArn, ExternalId: access.externalId, RoleSessionName: 'vod-upload' },
  }),
});

// storage.inputPath = s3://<bucket>/<key prefix>
const [bucket, ...rest] = project.storage.inputPath.replace('s3://', '').split('/');
await new Upload({
  client: s3,
  params: { Bucket: bucket, Key: rest.join('/') + 'episode 01.mp4', Body: createReadStream('episode 01.mp4') },
}).done();

What the role can do

On	Allowed
`s3Prefix` and everything under it	list, read, write, delete, multipart uploads
Anything else	nothing

The prefix holds every project of your account, one folder each.

Revoke the role

If the principal should lose access, for example when the person or pipeline behind it leaves, revoke the role:

curl -s -X DELETE $API/account/aws-access -H "$AUTH"   # 204

Nobody can assume the role from then on, and credentials already issued through it stop working. GET /account/aws-access answers 404 again. A later PUT restores the role for the principal you name, with a new external ID, so update your profile’s external_id.

A file’s processing records are kept by name. Replacing an input with the AWS CLI (same name, new content) does not reprocess it, and neither does deleting it and copying it in again: it reads pending with stale: true until you call retry on the name. An API upload with overwrite: true does this for you. To replace a failed file, use a new name (Replace).

Get started

Concepts

Configure a project

Upload media

Examples

Going to production

Help

One-time setup

Upload

Download

With an SDK

What the role can do

Revoke the role

​One-time setup

​Upload

​Download

​With an SDK

​What the role can do

​Revoke the role

One-time setup

Upload

Download

With an SDK

What the role can do

Revoke the role