Ability to control logging/output

[proycon]

Forwarded from proycon/python-frog#9, by @gmjonker

[kosloot]

@gmjonker wrote:

When processing large corpora, frog produces a lot of output like this:

20180711:191754:590:Wed Jul 11 19:17:54 2018 process 2 sentences
20180711:191754:592:Wed Jul 11 19:17:54 2018 done with sentence[1]
20180711:191754:607:Wed Jul 11 19:17:54 2018 done with sentence[2]

This drowns out other logging, like progress information.

It would be nice if the frog output/log level could be controlled.

[kosloot]

Yeah. you are right. his should be a bit configurable. It is always a point of discussion what and why to output.
The done with sentence[1] lines were added recently because of complaints about seeing no progress. :)

good suggestions about output levels would be nice.

[gmjonker]

For me, being able to suppress the lines mentioned above would be ideal.

Maybe a silent mode that only outputs errors?

[kosloot]

well, you could always redirect stdout to a file, and watch only stderr.

[gmjonker]

I'm using python-frog in PyCharm, which doesn't allow me to watch only stdout and not stderr, or redirect them.
Also, it seems that all the logging comes in on stderr, which I do want to see of course...

[kosloot]

yes. you are right.
Anyway. Not something I can change instantly in Frog. Let alone the Python layers...

[proycon]

Perhaps some kind of --verbosity flag with an integer value would be a good solution? Once we have that it's easy to add to python-frog.

[kosloot]

yeah, that's the plan. Discussion will start about what to include at what level...

And an implementation 'detail':
Because of the multi-threading of Frog, a single Logfile with guards is used. Which is tied to stderr. We would need 2 of those then, one to stderr, and one to stdout.

[kosloot]

one other design choice that will haunt us:
Default Frog writes it's 'tabbed' output to stdout. That's why ALL other output is send to stderr.
Introducing verbosity levels would still mean writing to stderr, until we decide to make the -o output option obligatory.

A more easy path might be using the -d option, with value 0 to be (almost) silent. value 1 some useful info, and levels above for 'real' debugging.

[gmjonker]

I think writing diagnostic information to stderr is fine.

So +1 for the --verbosity/-d option.

[kosloot]

Well, as a quick hack, I made the -d1 option much less verbose, so it is in fact a verbosity option now.
the default is -d0 and even more silent.
Maybe more output could be hidden for -d0
after my holidays then...

[kosloot]

Already many moons ago, i added a separate debug file to Frog. When -d or --debug is used a debug file frog.<random_number>.debug is created where debug info is stored.
Normal cout output is quite minimal now.