Skip to content
Laviyx Learn

Try

Node.js and Express course menu

Node.js Streams: Process Large Files Efficiently

Understand Node.js streams: readable, writable and transform streams, reading big files in chunks, processing files line by line with readline, pipeline for copying and compressing with gzip, async iteration, and why streams keep memory use low.

Intermediate7 min readUpdated
On this page

readFile loads a whole file into memory at once. That is fine for small files, but a 5 GB log file would exhaust the memory. Streams process data piece by piece (in chunks) as it arrives, so memory use stays small no matter how big the data is. HTTP requests and responses, files, compression and network sockets are all streams in Node.

There are four kinds:

  • Readable: a source you read from (a file, an incoming request);
  • Writable: a destination you write to (a file, a response);
  • Duplex: both (a network socket);
  • Transform: a duplex stream that changes data passing through (compression, encryption).

Create a large file to work with#

make-log.js (JavaScript)
import { createWriteStream } from "node:fs";

const out = createWriteStream("access.log");
const pages = ["/", "/products", "/cart", "/checkout"];
for (let i = 1; i <= 50000; i++) {
  const status = i % 50 === 0 ? 500 : i % 9 === 0 ? 404 : 200;
  out.write(`2026-10-25T10:00:00Z GET ${pages[i % pages.length]} ${status}\n`);
}
out.end(() => console.log("Wrote access.log"));
Terminal
node make-log.js
Output
Wrote access.log

out.write adds data to the stream's buffer, and out.end() finishes the file; the callback runs when everything has been written.

Reading in chunks#

chunks.js (JavaScript)
import { createReadStream } from "node:fs";

const stream = createReadStream("access.log", { encoding: "utf8" });
let chunks = 0;
let characters = 0;

for await (const chunk of stream) {
  chunks += 1;
  characters += chunk.length;
}
console.log(`Read ${characters} characters in ${chunks} chunks`);
Terminal
node chunks.js
Output
Read 1800000 characters in 28 chunks

A file stream reads 64 KiB at a time by default. for await...of reads each chunk as it arrives, so only one chunk needs to be in memory at a time.

Line by line with readline#

Chunks can split a line in half. readline reassembles the stream into lines, ideal for logs and CSV files:

errors.js (JavaScript)
import { createReadStream } from "node:fs";
import { createInterface } from "node:readline";

const lines = createInterface({ input: createReadStream("access.log"), crlfDelay: Infinity });
const counts = {};

for await (const line of lines) {
  const status = line.split(" ").at(-1);
  counts[status] = (counts[status] ?? 0) + 1;
}
console.log(counts);
Terminal
node errors.js
Output
{ '200': 43556, '404': 5444, '500': 1000 }

This works the same for a file of a hundred lines or a hundred million, with flat memory use.

pipeline: connecting streams#

pipeline connects a readable stream through transforms into a writable stream, handles backpressure (slowing the reader when the writer cannot keep up), and cleans up all streams if any of them fails. Compressing a file with gzip:

compress.js (JavaScript)
import { createReadStream, createWriteStream } from "node:fs";
import { stat } from "node:fs/promises";
import { pipeline } from "node:stream/promises";
import { createGzip } from "node:zlib";

await pipeline(createReadStream("access.log"), createGzip(), createWriteStream("access.log.gz"));

const before = (await stat("access.log")).size;
const after = (await stat("access.log.gz")).size;
console.log(`Compressed ${before} bytes to ${after} bytes (${((after / before) * 100).toFixed(1)}%)`);
Terminal
node compress.js
Output
Compressed 1800000 bytes to 15941 bytes (0.9%)

Log files are very repetitive, so they compress extremely well.

A custom transform stream#

upper.js (JavaScript)
import { Readable, Transform } from "node:stream";
import { pipeline } from "node:stream/promises";

const toUpper = new Transform({
  transform(chunk, encoding, callback) {
    callback(null, chunk.toString().toUpperCase());
  },
});

await pipeline(Readable.from(["namaste ", "from ", "a stream\n"]), toUpper, process.stdout);
Terminal
node upper.js
Output
NAMASTE FROM A STREAM

Readable.from turns an array (or any iterable) into a readable stream, and process.stdout is a writable stream, so the result goes to the terminal.

Streams in servers#

HTTP responses are writable streams, so a server can send a large file without loading it into memory:

JavaScript
import { createReadStream } from "node:fs";
import { createServer } from "node:http";

createServer((req, res) => {
  res.writeHead(200, { "Content-Type": "text/plain" });
  createReadStream("access.log").pipe(res);
}).listen(3000);

Express's res.sendFile and express.static stream files for you in the same way.

Common mistakes#

Practice#

Practice exercise

Using the access.log file created above, count the requests per page (/, /products, /cart, /checkout) with readline, and print the counts sorted by page name.

Show hint

The page is the third space-separated field: line.split(" ")[2]. Sort with Object.entries(counts).sort(([a], [b]) => a.localeCompare(b)).

Show solution
pages.js (JavaScript)
import { createReadStream } from "node:fs";
import { createInterface } from "node:readline";

const counts = {};
for await (const line of createInterface({ input: createReadStream("access.log"), crlfDelay: Infinity })) {
  const page = line.split(" ")[2];
  counts[page] = (counts[page] ?? 0) + 1;
}
for (const [page, n] of Object.entries(counts).sort(([a], [b]) => a.localeCompare(b))) {
  console.log(page.padEnd(10), n);
}
Terminal
node pages.js
Output
/          12500
/cart      12500
/checkout  12500
/products  12500

Key takeaway#

Next: Asynchronous Node and the Event Loop.

Finished this lesson?

Related lessons

Node.js and ExpressNode.js File System (fs): Read, Write and Manage FilesWork with files in Node.js using fs/promises: reading and writing text and JSON files, appending, checking whether files exist, handling ENOENT errors, creating and listing folders, file information with stat, and why to avoid synchronous methods in servers.Beginner8 min readNode.js and ExpressNode.js Events and EventEmitter ExplainedLearn the event-driven style at the heart of Node.js: creating an EventEmitter, emitting and listening for events with on and once, passing data, removing listeners, the special error event, extending EventEmitter in your own classes, and awaiting events with once.Beginner8 min readNode.js and ExpressBuild an HTTP Server in Node.js (Without Express)Create a web server with Node.js's built-in http module: handling requests and responses, status codes and headers, routing by method and URL, returning JSON, reading a POST body, testing with curl, and why frameworks like Express help.Intermediate9 min read