Blending System Administration with Server-Side Development

In a typical web project, the system administrators and the server-side developers each tend to different chores. Administrators write bash scripts, set cron jobs, and do data pull/push from databases. Server-side developers use PHP frameworks, MySQL databases, and plain PHP to construct the server for the application.

With the advent of Node.js - JavaScript on the server, system administration blends into server-side development. Now scripts and programs can be pulled together into processing pipelines.

Let's start by doing some basic system administration tasks in Node.js.

Periodic Jobs

Schedule with setInteval

 setInterval(function(){

 // query mysql
 var connection = mysql.createConnection(connectionOptions);
 connection.query("USE " + databaseName);
 query = connection.query(queryString);

   query.on('error', function(err) {
     console.log(err);
   });

   query.on('result', function(row) {
      // Pausing the connection is useful if your processing involves I/O
      connection.pause();

      processRow(row, function() {
         connection.resume();   
      });

   });

   query.on('end', function() {

     // all rows have been received

     processEnd();

   });

}, 5000);

The function will be executed every 5 seconds, running a MySQL query. Doing away with a cron job.

Delayed Execution

Delay with setTimeout

setTimeout(function(){

}, 3 * 60 * 1000);

Execute the function 3 minutes from now.

Execute Bash Scripts within Code

It is simple to integrate bash scripts into server-side code.

 // include the child_process module to fork new processes
 var childProcess = require("child_process");

 // execute the script
 exec(" ../scripts/run_crawler.sh " + "yahoo" ,                      
     function(err, stdout, stderr){
      console.log(err);
      console.log(stdout);
      console.log(stderr);
    });

Use Other Programs

And similarly we can execute another Node.js program, or Java, or C++, or whatever, from within a module,

 exec("node ../loader/data_fetch.js " + configuration,
     function(err, stdout, stderr){

});

Processing Pipelines

Using the Node.js async module we can construct serial and parallel processing pipelines that mix Node.js code, bash scripts, and other code modules,, like Java, or C++.

 var childProcess = require('child_process');
 var async = require('async");

 // serial execution of functions, each running a script or another module:

 async.series([    
      function(nextStep){
           exec(" ../scripts/run_crawler.sh " +
             "marketwatch" + " " + "one_week",

                    function(err, stdout, stderr){

                                  if (!err)
                                     nextStep(null, "crawler succeeded");         
                                  else     
                                     nextStep(err, "crawler failed");

           });

      },

    function(nextStep){

            exec("node ../loader/data_fetch.js " + configuration,

                    function(err, stdout, stderr){

                       if (!err)
                                 nextStep(null, "data fetch succeeded");         
                                else
                                  nextStep(err, "data fetch failed");
                });

    },

      function(nextStep){  
           exec("../scripts//export_mongodb_data.sh " + "/tmp/marketwatch_week.txt",

                function(err, stdout, stderr){
                        if (!err)
                           nextStep(null, "export succeeded");         
              else        
                 nextStep(err, "export failed");
           });

      }

 ], done);

 // on completion, results are collected by the done function

 function done(err, results)
 {
   console.log(err);
   console.log(results);
 }

For parallel execution use async.parallel.

More general pipelines are possible. See the documentation for the async module.

What All This Means

Server-side developers can do much more of the foundation without system administrators. System administrators are blended into the server-side pipeline and become developers (in bash).