Upload a Huge File to AWS S3 with Node

The Standard Way

Using AWS Node.js SDK we want to upload a huge file to

AWS S3. Using putObject will probably timeout for a huge file. So we need streams.

Stream the Upload

Create the read stream:

var readStream = fs.createReadStream(fileName);

var params = {Bucket: bucket, Key: key, Body: readStream};

Define how it is divided into parts (5KB), and the degree of concurrency (10):

var options = { partSize: 5 * 1024, queueSize: 10 };



var s3 = new AWS.S3({ httpOptions: { timeout: 10 * 60 * 1000 }});

s3.upload(params, options)

Track progress:

.on('httpUploadProgress', function(evt) {

     console.log('Completed ' +

     (evt.loaded * 100 / evt.total).toFixed() +

     '% of upload');

})

On completion:

.send(function(err, data) {

     console.log('Upload done');

});