How to manage large datasets and avoid "JavaScript heap out of memory" errors in Node.js?

General Discussions
,
1

I ran my script for filesystem indexing, which crashed after 4 hours with a “heap out of memory” error.

The script processes metadata for a large number of files. Although my server has 16GB RAM and 24GB SSD swap, Node.js seems to be running into memory management issues when handling these large datasets.

Is there a way to improve JavaScript interface memory management for such large arrays or objects in Node.js?

2

Instead of loading your entire dataset into memory at once, stream it or process it in smaller chunks. This can be a game changer for filesystem indexing.

const fs = require('fs');
const readline = require('readline');

const rl = readline.createInterface({
  input: fs.createReadStream('big-file.txt'),
  crlfDelay: Infinity
});

rl.on('line', (line) => {
  // Process each line one by one
});

This drastically reduces memory usage because you’re only holding a tiny part of the data at a time.

3

Sometimes you do need more memory. You can increase the heap size like this:

node --max-old-space-size=8192 yourscript.js

This allows Node.js to use up to 8GB of memory (adjust based on your system). Use this only if optimizing logic alone doesn’t solve the problem.

4

If you’re building up a large array/object in memory, consider writing intermediate data to disk or using a lightweight DB like SQLite or LevelDB:

const fs = require('fs');
fs.appendFileSync('partial-result.json', JSON.stringify(dataChunk) + '\n');

Or use a cache-friendly database:

npm install level

This is especially helpful when you don’t need everything in memory all at once.