This tutorial will explain how to store data with Node.js writable streams and how to create your own custom writable stream.

Version Information

  • Author: Jeff Barczewski
  • Published: August 19th, 2013
  • Tags: nodejs, streams
  • Level: Intermediate
  • Prerequisites: buffers, events, install npm modules
  • Node.js v0.10+ (latest stable is v0.10.16 as of this writing), but streams have generally been a part of Node.js from its early days
  • Streams2 Writable abstract class can be used with older versions (prior to v0.10) of node by using npm module readable-stream (tested with v1.0.15)

Storing data with writable streams

Writing text file

The simplest example is just in writing utf8 text, since the default encoding if you write strings is utf8.

var fs = require('fs');
var wstream = fs.createWriteStream('myOutput.txt');
wstream.write('Hello world!\n');
wstream.write('Another line\n');
wstream.end();

If you want to write strings with a different encoding then you can simply change the createWriteStream line or add encoding to each write.

var options = { encoding: 'utf16le' };
var wstream = fs.createWriteStream('foo', options);
// OR add the encoding to each write
wstream.write(str, 'utf16le');

Writing binary file

Writing a binary file is just a matter of using Buffers rather than strings for the write.

var crypto = require('crypto');
var fs = require('fs');
var wstream = fs.createWriteStream('myBinaryFile');
// creates random Buffer of 100 bytes
var buffer = crypto.randomBytes(100);
wstream.write(buffer);
// create another Buffer of 100 bytes and write
wstream.write(crypto.randomBytes(100));
wstream.end();

In the example above, I use crypto.createRandomBytes() to create a Buffer of data, but you can use binary data that you create or read from another source just as easily.

Knowing when file has been written

Input and output are asynchronous operations in Node.js so what if we want to know when a file has been fully written? The answer is to setup listeners to events that the stream emits.

The finish event (added in Node.js v0.10) indicates that all data has been flushed to the underlying system.

var fs = require('fs');
var wstream = fs.createWriteStream('myOutput.txt');
// Node.js 0.10+ emits finish when complete
wstream.on('finish', function () {
  console.log('file has been written');
});
wstream.write('Hello world!\n');
wstream.write('Another line');
wstream.end();

On Node.js prior to v0.10, you could add a cb to the .end() to get an indication of when that had been flushed, however finish takes into account any other transformations downstream and such.

var fs = require('fs');
var wstream = fs.createWriteStream('myOutput.txt');
wstream.write('Hello world!\n');
wstream.write('Another line');
// on Node.js older than 0.10, add cb to end()
wstream.end(function () { console.log('done'); });

Creating custom Node.js writable streams

When you need to create your own custom writable stream to store data in a database or other storage facility, it is easy if you use the new Streams2 Writable abstract class available natively in Node.js 0.10 or via the npm polyfill module readable-stream.

To create a writable stream which implements all of the normal Node.js stream behavior you only have to subclass Writable and implement _write(chunk, encoding, cb).

// Using Streams2 Writable from Node.js 0.10+
// or falling back to using a polyfill from
// npm module readable-stream
var util = require('util');
var stream = require('stream');
var Writable = stream.Writable ||
  require('readable-stream').Writable;

function MyStream(options) {
  Writable.call(this, options);
}
util.inherits(MyStream, Writable);

MyStream.prototype._write = function (chunk, enc, cb) {
 // store chunk, then call cb when done
 cb();
};

Creating writable memory stream

Here is an example which is writing to a simple in-memory datastore. We are using the name given at the creation of the stream as the key, and we append data to the value until finished.

var stream = require('stream');
var util = require('util');
// use Node.js Writable, otherwise load polyfill
var Writable = stream.Writable ||
  require('readable-stream').Writable;

var memStore = { };

/* Writable memory stream */
function WMStrm(key, options) {
  // allow use without new operator
  if (!(this instanceof WMStrm)) {
    return new WMStrm(key, options);
  }
  Writable.call(this, options); // init super
  this.key = key; // save key
  memStore[key] = new Buffer(''); // empty
}
util.inherits(WMStrm, Writable);

WMStrm.prototype._write = function (chunk, enc, cb) {
  // our memory store stores things in buffers
  var buffer = (Buffer.isBuffer(chunk)) ?
    chunk :  // already is Buffer use it
    new Buffer(chunk, enc);  // string, convert

  // concat to the buffer already there
  memStore[this.key] = Buffer.concat([memStore[this.key], buffer]);
  cb();
};


// Trying our stream out
var wstream = new WMStrm('foo');
wstream.on('finish', function () {
  console.log('finished writing');
  console.log('value is:', memStore.foo.toString());
});
wstream.write('hello ');
wstream.write('world');
wstream.end();

Writable streams are elegant and simple to use

Writing text or binary data to Node.js streams is effortless, and even creating fully functional custom writable streams is a walk in the park with the new streams2 functionality introduced in Node.js v0.10 (or using the polyfill readable-stream module).

For additional reading