benchmark runs with training separately than runs that use settings file

[fgregg]

@NickCrews, if i understand the benchmarking code correctly, the first time a benchmark runs it will create a settings file, and then in all subsequent runs, the benchmarking code will use that settings file.

It would be very helpful if we could separate this out, so we had benchmarks runs where each run there was not training, and other runs where each run used a settings file.

does that make sense?

[fgregg]

actually, ideal would be to track three things distinctly

• track time training
• track time matching if i trained
• track time matching if i read from settings file

[NickCrews]

Yes that makes sense and is possible. Will try to get to this in next couple days.

Also do you think adding a larger dataset would be useful? Not sure how meaningful these small datasets are, eg howuch of that 50Mb of peak memory you mentioned is just overhead

[fgregg]

would love to have some bigger data sets. those would probably need to be run on demand or nightly to and on every push.