Make output UNBUFFERED for python children

[yarikoptic]

It does help

❯ duct ./test_script.py --duration 1 --memory-size=1000 2>&1 | ts
Jun 09 11:18:48 this is of test of STDERR
Jun 09 11:18:50 this is of test of STDOUT
Jun 09 11:18:50 Test completed. Consumed 1000 MB for 1 seconds with CPU load factor 10000.
Jun 09 11:18:50 duct is executing ./test_script.py --duration 1 --memory-size=1000...
Jun 09 11:18:50 Log files will be written to .duct/logs/2024.06.09T11.18.48-1439380_
Jun 09 11:18:50 
Jun 09 11:18:50 Exit Code: 0
Jun 09 11:18:50 Command: ./test_script.py --duration 1 --memory-size=1000
Jun 09 11:18:50 Log files location: .duct/logs/2024.06.09T11.18.48-1439380_
Jun 09 11:18:50 Wall Clock Time: 1.749 sec
Jun 09 11:18:50 Memory Peak Usage: 1.5%
Jun 09 11:18:50 CPU Peak Usage: 103.0%
❯ PYTHONUNBUFFERED=1 duct ./test_script.py --duration 1 --memory-size=1000 2>&1 | ts
Jun 09 11:18:54 duct is executing ./test_script.py --duration 1 --memory-size=1000...
Jun 09 11:18:54 Log files will be written to .duct/logs/2024.06.09T11.18.54-1439428_
Jun 09 11:18:54 this is of test of STDOUT
Jun 09 11:18:54 this is of test of STDERR
Jun 09 11:18:56 Test completed. Consumed 1000 MB for 1 seconds with CPU load factor 10000.
Jun 09 11:18:56 
Jun 09 11:18:56 Exit Code: 0
Jun 09 11:18:56 Command: ./test_script.py --duration 1 --memory-size=1000
Jun 09 11:18:56 Log files location: .duct/logs/2024.06.09T11.18.54-1439428_
Jun 09 11:18:56 Wall Clock Time: 1.756 sec
Jun 09 11:18:56 Memory Peak Usage: 1.5%
Jun 09 11:18:56 CPU Peak Usage: 103.0%

and with this change:

❯ duct ./test_script.py --duration 1 --memory-size=1000 2>&1 | ts
Jun 09 11:28:55 this is of test of STDOUT
Jun 09 11:28:55 this is of test of STDERR
Jun 09 11:28:57 Test completed. Consumed 1000 MB for 1 seconds with CPU load factor 10000.
Jun 09 11:28:57 duct is executing ./test_script.py --duration 1 --memory-size=1000...
Jun 09 11:28:57 Log files will be written to .duct/logs/2024.06.09T11.28.55-1441799_
Jun 09 11:28:57 
Jun 09 11:28:57 Exit Code: 0
Jun 09 11:28:57 Command: ./test_script.py --duration 1 --memory-size=1000
Jun 09 11:28:57 Log files location: .duct/logs/2024.06.09T11.28.55-1441799_
Jun 09 11:28:57 Wall Clock Time: 1.931 sec
Jun 09 11:28:57 Memory Peak Usage: 1.5%
Jun 09 11:28:57 CPU Peak Usage: 104.0%

BUT I am not yet sure we should do that (by default at least), since it changes operating conditions for the process and in some cases people would not even care to receive "online" output.

[asmacdo]

Option 1: add --python-unbuffered -u, when passed it will set PYTHONUNBUFFERED

Modifying env vars has a smell to it. And this option is only for python programs.

Option 2: Documentation

[asmacdo]

Todo(asmacdo) see how 'watch' deals with captured outputs for when comparing differences

'watch -d'

[yarikoptic]

is this still anyhow useful/desired @asmacdo ?

[asmacdo]

I think this would be useful to document to start with. Maybe we add a flag down the line, but IMO

duct --some-buffer-flag testscripy.py

isnt really much better than

PYTHONUNBUFFERED=1 duct ./test_script.py

unless that flag does a little more than set the env var

[yarikoptic]

someone need to know/discover it, while people do see --help for the ultimate RTFM.

I think we should add option (not a flag)

--flush-workarounds|-f with possible values

• all: all known, default
• python-buffer: the above
• ... later might add some others

[asmacdo]

I dont really like -f for this, since thats usually file or force. how about --buffer-flush, -b

[yarikoptic]

-b sounds good!

[yarikoptic]

replacing with

• --buffer-flush, -b option #218