New con-duct ls command #224
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #224 +/- ##
==========================================
- Coverage 95.36% 94.05% -1.32%
==========================================
Files 4 5 +1
Lines 626 740 +114
Branches 70 93 +23
==========================================
+ Hits 597 696 +99
- Misses 15 26 +11
- Partials 14 18 +4 ☔ View full report in Codecov by Sentry. |
|
Overall:
Info that seems most important:
|
|
note that command might be too long to show in full "by default", max memory resource i think would also be nice to know. |
|
@yarikoptic i've hacked together something with pyout
I'm not sure how I want to handle the "include_only" list, since ideally that would be agnostic to the output format, and I don't see a clean way to incorporate with LS_SUMMARY_FORMAT. But pyout is nice and easy! |
sorry -- could you describe what you mean by |
|
@yarikoptic https://github.com/con/duct/pull/224/files#diff-150598b33b1982d47d2bc3f78f8ba2333963da1d5ebe2b6211cba03b7c259cfcR64 Restricting the columns to add to the table |
asmacdo
force-pushed
the
con-duct-ls
branch
2 times, most recently
from
7fb180b to
39e0e0f
Compare
src/con_duct/suite/ls.py
Outdated
| VALUE_TRANSFORMATION_MAP = { | ||
| "exit_code": "{value!E}", | ||
| "wall_clock_time": "{value:.3f} sec", | ||
| "peak_rss": "{value!S}", | ||
| } |
There was a problem hiding this comment.
@yarikoptic if we use the custom formatter like this, the user can't provide their own format string, so its a bit less flexible than we did for regular duct output, but I think its a whole lot cleaner/dryer. And the user experience seems better too-- ie, provide an ordered list of fields that you want, rather than a format string that requires knowledge of how the formatter works.
If you like this approach too, I'll fill out this map with all the fields, and perhaps refactor regular duct output to function this way.
There was a problem hiding this comment.
get some way done, let us try and then lets discuss and potentially improve.
Eventually I would expect us to be able to extend list of relevant attributes to be visualized in ls based on user's inputs. Might even be some kind of a "run a regex across stdout or stderr to get information of interest to be printed in each row" to support some custom extraction but for that I do not have immediate use case hence just a wild idea (must not be yet implemented anyhow).
|
Some significantly improved UX/rendering IMO! Heres some samples of what it looks like now |
|
we should see what we should display in "LOG_PREFIX", so we display PREFIX as loaded from the files. But of interest here likely is actually "FILES_PREFIX" based on the filenames loaded. |
asmacdo
force-pushed
the
con-duct-ls
branch
2 times, most recently
from
98bad79 to
896972a
Compare
Fixes: con#185
|
🚀 PR was released in |
|
Please do not forget next time to make PR titles descriptive enough to serve as a changelog entry. |
Fixes #185
summaries
yaml
json
{"command": "sleep 0.11", "system": {"cpu_total": 20, "memory_total": 33336127488, "hostname": "fancy", "uid": 1000, "user": "austin"}, "env": {}, "gpu": null, "duct_version": "0.8.0", "schema_version": "0.2.0", "execution_summary": {"exit_code": 0, "command": "sleep 0.11", "logs_prefix": ".duct/logs/2024.11.11T22.17.33-130359_", "wall_clock_time": 0.11228775978088379, "peak_rss": 1961984, "average_rss": 1961984, "peak_vsz": 232374272, "average_vsz": 232374272, "peak_pmem": 0.0, "average_pmem": 0.0, "peak_pcpu": 0.0, "average_pcpu": 0.0, "num_samples": 1, "num_reports": 2, "start_time": 1731385053.198443, "end_time": 1731385053.3107307}, "output_paths": {"stdout": ".duct/logs/2024.11.11T22.17.33-130359_stdout", "stderr": ".duct/logs/2024.11.11T22.17.33-130359_stderr", "usage": ".duct/logs/2024.11.11T22.17.33-130359_usage.json", "info": ".duct/logs/2024.11.11T22.17.33-130359_info.json", "prefix": ".duct/logs/2024.11.11T22.17.33-130359_"}} {"command": "sleep 0.1", "system": {"cpu_total": 20, "memory_total": 33336127488, "hostname": "fancy", "uid": 1000, "user": "austin"}, "env": {}, "gpu": null, "duct_version": "0.8.0", "schema_version": "0.2.0", "execution_summary": {"exit_code": 0, "command": "sleep 0.1", "logs_prefix": ".duct/logs/2024.11.11T22.17.36-130380_", "wall_clock_time": 0.10244536399841309, "peak_rss": 1978368, "average_rss": 1978368, "peak_vsz": 232374272, "average_vsz": 232374272, "peak_pmem": 0.0, "average_pmem": 0.0, "peak_pcpu": 0.0, "average_pcpu": 0.0, "num_samples": 1, "num_reports": 2, "start_time": 1731385056.3133166, "end_time": 1731385056.415762}, "output_paths": {"stdout": ".duct/logs/2024.11.11T22.17.36-130380_stdout", "stderr": ".duct/logs/2024.11.11T22.17.36-130380_stderr", "usage": ".duct/logs/2024.11.11T22.17.36-130380_usage.json", "info": ".duct/logs/2024.11.11T22.17.36-130380_info.json", "prefix": ".duct/logs/2024.11.11T22.17.36-130380_"}}jsonpp
{'command': 'sleep 0.11', 'duct_version': '0.8.0', 'env': {}, 'execution_summary': {'average_pcpu': 0.0, 'average_pmem': 0.0, 'average_rss': 1961984, 'average_vsz': 232374272, 'command': 'sleep 0.11', 'end_time': 1731385053.3107307, 'exit_code': 0, 'logs_prefix': '.duct/logs/2024.11.11T22.17.33-130359_', 'num_reports': 2, 'num_samples': 1, 'peak_pcpu': 0.0, 'peak_pmem': 0.0, 'peak_rss': 1961984, 'peak_vsz': 232374272, 'start_time': 1731385053.198443, 'wall_clock_time': 0.11228775978088379}, 'gpu': None, 'output_paths': {'info': '.duct/logs/2024.11.11T22.17.33-130359_info.json', 'prefix': '.duct/logs/2024.11.11T22.17.33-130359_', 'stderr': '.duct/logs/2024.11.11T22.17.33-130359_stderr', 'stdout': '.duct/logs/2024.11.11T22.17.33-130359_stdout', 'usage': '.duct/logs/2024.11.11T22.17.33-130359_usage.json'}, 'schema_version': '0.2.0', 'system': {'cpu_total': 20, 'hostname': 'fancy', 'memory_total': 33336127488, 'uid': 1000, 'user': 'austin'}} {'command': 'sleep 0.1', 'duct_version': '0.8.0', 'env': {}, 'execution_summary': {'average_pcpu': 0.0, 'average_pmem': 0.0, 'average_rss': 1978368, 'average_vsz': 232374272, 'command': 'sleep 0.1', 'end_time': 1731385056.415762, 'exit_code': 0, 'logs_prefix': '.duct/logs/2024.11.11T22.17.36-130380_', 'num_reports': 2, 'num_samples': 1, 'peak_pcpu': 0.0, 'peak_pmem': 0.0, 'peak_rss': 1978368, 'peak_vsz': 232374272, 'start_time': 1731385056.3133166, 'wall_clock_time': 0.10244536399841309}, 'gpu': None, 'output_paths': {'info': '.duct/logs/2024.11.11T22.17.36-130380_info.json', 'prefix': '.duct/logs/2024.11.11T22.17.36-130380_', 'stderr': '.duct/logs/2024.11.11T22.17.36-130380_stderr', 'stdout': '.duct/logs/2024.11.11T22.17.36-130380_stdout', 'usage': '.duct/logs/2024.11.11T22.17.36-130380_usage.json'}, 'schema_version': '0.2.0', 'system': {'cpu_total': 20, 'hostname': 'fancy', 'memory_total': 33336127488, 'uid': 1000, 'user': 'austin'}}TODO