Upload a Huge File to AWS S3 with Node

The Standard Way

Using AWS Node.js SDK we want to upload a huge file to
AWS S3. Using putObject will probably timeout for a huge file. So we need streams.

Stream the Upload

Create the read stream:

var readStream = fs.createReadStream(fileName);
var params = {Bucket: bucket, Key: key, Body: readStream};

Define how it is divided into parts (5KB), and the degree of concurrency (10):

var options = { partSize: 5 * 1024 * 1024, queueSize: 10 };

var s3 = new AWS.S3({ httpOptions: { timeout: 10 * 60 * 1000 }});
s3.upload(params, options)

Track progress:

.on('httpUploadProgress', function(evt) { 
     console.log('Completed ' + 
     (evt.loaded * 100 / evt.total).toFixed() + 
     '% of upload');
})

On completion:

.send(function(err, data) { 
     console.log('Upload done');
});