Upload a Huge File to AWS S3 with Node

The Standard Way

Using AWS Node.js SDK we want to upload a huge file to AWS S3. Using putObject will probably timeout for a huge file. So we need streams.

Create the read stream:


var readStream = fs.createReadStream(fileName);
var params = {Bucket: bucket, Key: key, Body: readStream};

Define how it is divided into parts (5KB), and the degree of concurrency (10):


var options = { partSize: 5 * 1024, queueSize: 10 };

var s3 = new AWS.S3({ httpOptions: { timeout: 10 * 60 * 1000 }});
s3.upload(params, options)

Track progress:


.on('httpUploadProgress', function(evt) { 
    console.log('Completed ' + (evt.loaded * 100 / evt.total).toFixed() + '% of upload');
})

On completion:


.send(function(err, data) { 
     console.log('Upload done');
});

Yoram Kornatzky

June 28, 2017