should@8f711a78

link to #3054

link to 3054

link to #3054

link to #3054

With this script, we have the possibility to get structured datas from the pear preprocess to use them into the server, for quality, warning, ...
link to #3054 #3049 #3154

This will allow to more reliably add percentages that otherwise would give
meaningless results.

In the test we now have 497 reads out of 786861.

allows vidjil files to contain no clones

Fixes #2929

So that dependencies only need to be installed if VidjilParser is used.
Thanks @mikael-s :)

Being a fast parser allows us to gain a significant amount of time when
executing.

Restore old functionality and make new functionality optional.
Due to the amount of time it takes to run the vidjilparser we may need
to decide between speed and memory efficiency

See #3234

split VidjilWriter into VidjilWriter and VidjilFileWriter

use yajl2 as the ijson parser since it is much faster.

See #3234,#3235

use a list for the buffer and join strings when needed. This should be
slightly faster than concatenating strings.

due to the error in the prefix usage, we were extracting all clones from
the file instead of the top N, leading to horrendous performance issues.

changes to vidjilparser.py allow us to simply extract the root of a json
file and still apply constraints to subfields, so we no longer need to
specify all the fields we need to extract.

See #3234

fixes and issue where applying a new predicate to a subfield of a prefix
included in another field would not be taken into account.

See #3235

group all checks into a single call to any, to avoid multiple iterations
over the same data.
Also remove some comparisons that don't seem necessary.

The introduction of the buffer caused an issue with comma delimitation
when th first element of an array or map was discarded (from not meeting
predicate requirements), leading to an extra comma and therefore an
invalid JSON output.
By saving the value of the "previous" variable when buffering begins, we
can restore the variables value when discarding the buffered content.

See #3235

since we may not want to export to a file, leaving the filepath variable
at None will disable the file writing capabilities.

This is mostly for Python3 compatibility. The default mode seems to
differ between Pyton 2 and 3, causing and issue when attempting to run
fuse with Pyton 3

The object here being some optimisations to increase the performance of
fuse.py by using VidjilParser to extract only the data that is relevant
to the context we are in, before loading the data into Python objects.

See #3234

it was there for testing purposes only.

resets the parser prefixes and the writer's buffer

makes use of buffering and predicate checking to exclude or include data
to the export based on a comparison function passed when setting desired
prefixes.

See #3235,#2240

Adds a system where data can be put aside and is not written to the
output file unless requested. Also return the extracted data string at
the end of write.

See #2240

a basic version of a tool to validate and parse/extract data from large
vidjil files.
See #2240