Skip to main content

calculating checksum is slow

We're using reed-solomon.

It is a lightweight implementation of klauspost/reedsolomon, just to be compatible with greenfield-common.

As we all know, JavaScript is not good at this kind of intensive computing. In fact, the results we tested on local Nodejs were about 10 times slower than Go.

But to be able to use it in the Javascript, we created reed-solomon, which is the core library.

Benchmarking​

If not counting big files (how big is depending on the user's device), here are the results of our tests on the Apple M2.

Note, you don't have to have webworker or worker_threads on to get faster performance. Because running worker also has performance loss.

When calculating small files, using the core is faster than using the worker.

Usage​

You can use core lib directly in the browser and Nodejs, or you can use us to run on the worker(browser use webworker, Nodejs use worker_threads).

When you are developing Greenfield and need to create objects, if you are sure that the files are small (maybe less than 5m or 10m), you can directly use the ESM solution:

core lib (ESM)
import {ReedSolomon} from '@bnb-chain/reed-solomon'

const rs = new ReedSolomon();
const res = rs.encode(new Uint8Array(fileBuffer))

It also supports UMD mode calls (simpler and more convenient):

core lib (UMD)
<!-- prefetch js -->
<link rel="prefetch" href="https://unpkg.com/@bnb-chain/reed-solomon/dist/index.aio.js" />
<script src="https://unpkg.com/@bnb-chain/reed-solomon/dist/index.aio.js"></script>

<body>
  <input type="file" id="file" />
  <button id="btn">
    get reed solomon
  </button>
  <script type="module">
    const fileInput = document.getElementById('file');

    // not use webworker
    document.getElementById('btn').onclick = async function() {
      const selectFile = fileInput.files[0];
      const arrBuffer = await selectFile.arrayBuffer()
      if (!arrBuffer) alert('no file selected');

      const sourceData = new Uint8Array(arrBuffer)
      console.time('cost')
      console.log('file size', sourceData.length / 1024 / 1024, 'm')
      const rs = new RS.ReedSolomon()
      const res = await rs.encode(sourceData)
      console.log('res', res)
      console.timeEnd('cost')
    }

  </script>
</body>

If the file is larger, this method may cause the page to freeze when calculating. We recommend using the worker mode:

webworker (only support UMD)
<!-- prefetch js -->
<link rel="prefetch" href="https://unpkg.com/@bnb-chain/reed-solomon/dist/web.adapter.aio.js" />
<link rel="prefetch" href="https://unpkg.com/@bnb-chain/reed-solomon/dist/utils.aio.js" />
<script src="https://unpkg.com/@bnb-chain/reed-solomon/dist/web.adapter.aio.js"></script>

<body>
  <input type="file" id="file" />

  <button id="worker-btn">
    get reed solomon (webworker)
  </button>

  <script>
    const rs = new WebAdapter.WebAdapterReedSolomon()
    // will create 6 webworker
    rs.initWorkers({
      workerNum: 6,
      injectWorker,
    })

    document.getElementById('worker-btn').onclick = async function() {
      const selectFile = fileInput.files[0];
      const arrBuffer = await selectFile.arrayBuffer()
      if (!arrBuffer) alert('no file selected');

      const sourceData = new Uint8Array(arrBuffer)
      const res = await rs.encodeInWorker(sourceData)
    }

    // inject worker
    function injectWorker() {
      // or download this file and put it to your CDN server
      importScripts('https://unpkg.com/@bnb-chain/reed-solomon/dist/web.adapter.aio.js');
      importScripts('https://unpkg.com/@bnb-chain/reed-solomon/dist/utils.aio.js');

      const rs = new WebAdapter.WebAdapterReedSolomon();
      onmessage = function (event) {
        const { index, chunk } = event.data;
        const encodeShard = rs.getEncodeShard(chunk, index)
        postMessage(encodeShard);
      };
    }
  </script>
</body>