jsonl Stringer - uhop/stream-json GitHub Wiki
(Since 1.6.0) jsonl/Stringer
is a Transform stream. It consumes a stream of JavaScript objects and converts it to a JSONL (AKA NDJSON) file. It is very useful when you want to edit a stream with filters and custom code, and save it back to a file.
jsonl/Stringer
has three major differences with Stringer:
- It consumes a stream of JavaScript objects, while
Stringer
consumes a stream of raw tokens. - It produces a valid JSONL file, where all values are separated by newline symbols.
- It is faster.
Introduction
const Stringer = require('stream-json/jsonl/Stringer');
const {parser} = require('stream-json/jsonl/Parser');
const {chain} = require('stream-chain');
const fs = require('fs');
const zlib = require('zlib');
chain([
fs.createReadStream('data.jsonl.gz'),
zlib.createGunzip(),
parser(),
// select only active values
data => data.value.active ? data.value : null,
new Stringer(),
zlib.createGzip(),
fs.createWriteStream('edited.jsonl.gz')
]);
API
jsonl/Stringer
has no special API. It is based on Transform with its writable part operating in object mode and its readable part operating in text mode.
constructor(options)
options
is an optional object described in details in node.js' Stream documentation. Additionally, the following optional custom flags are recognized:
replacer
can be one of two optional objects:
- It can be a function, which takes two parameters and returns a value.
- It can be an array of strings or numbers to define a whitelist for object properties.
- See JSON.stringify() for more details.
Static methods and properties
stringer(options)
and make(options)
make()
and stringer()
are two aliases of the factory function. It takes options described above, and return a new instance of jsonl/Stringer
. stringer()
helps to reduce a boilerplate when creating data processing pipelines:
const {stringer} = require('stream-json/jsonl/Stringer');
const {parser} = require('stream-json/jsonl/Parser');
const {chain} = require('stream-chain');
const fs = require('fs');
const pipeline = chain([
fs.createReadStream('sample.jsonl'),
parser(),
data => data.value,
stringer()
], {}); // empty 'options' is used to force text mode on both sides of a chain.
The code above roundtrip data from sample.jsonl
.
make.Constructor
Constructor property of make()
(and stringer()
) is set to jsonl/Stringer
. It can be used for indirect creating of stringers or metaprogramming if needed.