input/ folder and your
client sends the bytes straight to S3, one of two ways:
A project that reads a feed takes no uploads.
One PUT
Start the upload, thenPUT the whole file to the url it answers:
200 with an empty body, and the file is in input/, listed by GET /projects/{id}/files
and processed by a running project. Nothing to complete. Until the PUT finishes, S3 shows
nothing, so the project never sees half a file. The URL works until urlExpiresAt, an hour at
most and sometimes less, and needs no header: a Content-Type you do send is stored with the file.
url is absent from the answer in two cases, and then the file goes in parts:
- The file is over 5 GB, the most S3 takes in one request.
- A file of that name is already in
input/. Replacing it in onePUTis not offered: it is the complete call of a parts upload that lets the project process the new content. Delete the file first (DELETE /projects/{id}/files/{name}), then upload it.
PUT. Any
other stopped project is not started by it (completing a parts upload does that):
POST /projects/{id}/start afterwards, which is harmless on a running project. A project whose
startOnNewFiles is off is started by neither.
Parts
The API starts an S3 multipart upload, signs a URL for each part, and completes it; any size, and resumable.The calls
1
Start
201
key is the name to send back in every later call. Cut the file into partCount parts of
partSize bytes; the last one is shorter. url and urlExpiresAt are the
one-PUT alternative, present when it is open to the file; a parts upload ignores
them.2
Send each part
Sign a part, then S3 answers
PUT its bytes to the URL with no other headers:200 with an ETag header: keep it. Sign each part just before you send it; a URL
works until expiresAt, an hour at most and sometimes less. Part signing has a budget of its own, 10,000 parts a
minute per account, so it never uses up the API’s; parts may be sent in parallel, in any order.3
Complete
200
input/ and listed by GET /projects/{id}/files. Until this call, S3 shows
nothing, so the project never sees half a file. A project that ran before and is stopped now
(auto-stopped after its last file, say) is started again, unless its startOnNewFiles is off. projectStatus is the project’s
state afterwards: stopped for a project created with autoStart: false that has never run
(start it once every file is in), error if its last start failed. Repeating the call with the
same body is safe.uploadId and key go into URLs, so URL-encode them (encodeURIComponent, jq @uri,
urllib.parse.quote(…, safe="")).
Scripts
Complete uploaders in parts, with no dependency beyond the language’s standard library (andrequests for Python). Each uploads one file of any size into a project and prints its entry from
the files listing. For a one-PUT upload, the quick start has the
three lines.
Resume an interrupted upload
An upload stays open for three days. To resume it, ask S3 which parts it already holds, and send only the others:From a browser
The calls map one to one onto the callbacks of Uppy’s@uppy/aws-s3 plugin
(version 5), so a web app gets chunking, parallel parts, retries and resume without a server of
its own: