@hackage / hw-json-simd

SIMD-based JSON semi-indexer

Latest0.1.1.3

About

Metadata

  • Last updated , by haskellworks
  • License BSD-3-Clause

  • Maintained by: newhoggy@gmail.com

  • Lottery factor: 1

Links

Installation

Tested Compilers

  1. 9.12.2
  2. 9.10.2
  3. 9.8.4
  4. 9.6.7

Package Flags

Use the -f option with cabal commands to enable flags

    avx2 (off by default)

    Enable avx2 instruction set

    bmi2 (off by default)

    Enable bmi2 instruction set

    sse42 (on by default)

    Enable SSE 4.2 optimisations.

Readme

hw-json-simd

This library/tool will generate semi-indexes on JSON files as per the paper: Semi-Indexing Semi-Structured Data in Tiny Space.

The command line tool

For a given JSON file, hw-json-simd will generate two semi-index files, which both together can be loaded into a single in-memory semi-index.

The semi-index files can be generated using two methods, which will be called standard and simple, which correspond to sections 4 and 5 of the Semi-Indexing Semi-Structured Data in Tiny Space paper respectively.

Navigation of the JSON text using the standard index is suppored by the hw-json project for more information. There is currently no support for navigation of the JSON text using the simple index.

Using on the command line
cat test.json | pv -t -e -b -a | time hw-json-simd create-index \
  -i /dev/stdin
  --output-ib-file test.json.ib.idx
  --output-bp-file test.json.bp.idx
  --method standard
cat test.json | pv -t -e -b -a | time hw-json-simd create-index \
  -i /dev/stdin
  --output-ib-file test.json.ib.idx
  --output-bp-file test.json.bp.idx
  --method simple

Installation

From hackage
cabal new-install hw-json-simd