Partially reading in array of arrays where different arrays can be of different types.

[sohaibiftikhar]

When reading in some json data which represents some values in columnar format we can have some schema struct for deciphering how to parse the data. For example:

{
  "schema": [
    {"id": "col0", "dataType": "string"},
    {"id": "col1", "dataType": "int"},
    {"id": "col2", "dataType": "float"}
  ],
  "data": [
    ["row0", "row1", "row2"],
    [100, 200, 300],
    [1.5, 2.5, 3.5]
  ]
}

This then needs to be loaded into some struct based on the schema specified. Is there a clever way to perform this style of partial reading without the #1019 implementation.

I understand I could load it into a variant but then I would have to either copy it out to different vectors or use a visitor both of which is a bit of a waste. Since the number of rows here can be fairly large only allocating what we need without the need for copy would be ideal.

Would be happy for some help.

[stephenberry]

Glaze supports partial reading in a few ways, see documentation: Partial Read

So, I would suggest creating a struct for your schema and do a partial read into that. Then based on the schema, configure your data structure and do a partial read on the input into your data.

[sohaibiftikhar]

Thank you for the reply. I have read that one. And also the one with the pointer syntax.

Initially I thought I could do a partial read using schema as you suggested and then use the pointer syntax to load data. But I was wondering about the cost for repeatedly doing "data/0", "data/1" on the buffer without any context of where the parser was last time around. I assumed that in this case the parser would go searching again right from the beginning.

In partial reads. Given this example:

std::string s = R"([1,2,3,4,5])";
std::vector<int> v(2);
expect(!glz::read<glz::opts{.partial_read = true}>(v, s));

This is pretty good except that it only allows the partial read from index 0 of the array.
If we could do something like read but start from index i and end at index j that would help read the array in parts into different containers based on data type.

[stephenberry]

It would be good to be able to target a specific range of values. This really demands another query syntax. I want to add support for JMESPATH, but that is a long term goal.

I wouldn't be afraid of glz::json_t (which uses a variant underneath) for handling generic inputs. Also, if you're afraid of copying your vectors then simply call std::move after the generic parse, this will avoid the copy and give you generic behavior.

[sohaibiftikhar]

Thank you I was able to use generic parsing without any significant red flags on performance (ran some benchmarks with uncompressed JSON's around 100MB or so).

I will call it closed with the hope that partial reads with contexts are going to make this more flexible in the future. Thanks a lot for your great work!

[stephenberry]

Glad to hear you found a solution! I do think this will become easier in the future, but it is important to get the interface right and so will take a good bit of development time.