[Request] Automation: Directly receive the exported data

[piro-pp]

At the moment it is only possible to export the captured data to a file. It would be fine to be able to directly receive the data for further processing (with python).

[Marcus10110]

Hi @piro-pp,

There are technical reasons this would be tricky for us - mainly that our gRPC server is running in the browser, so I'd be worried about trying to stream large amounts of data from our data processing system through the browser over gRPC.

That said, one thing we want to provide are helper functions (perhaps in a helper package) that allow you to export data, then load it and parse it into pandas or numpy, and delete the temporary file, all in one call. Would this help?

The main limitation there is that it would only work when the python code is running on the same system as the software. It would not work over the network.

Could you tell us a little more about your application?

[piro-pp]

Thank you for the explanation.

My application is that i want to observe and protocol the communication of some hardware components that talk over UART to do some automation and testing. So i need to capture 2 singe "serial" channels and synchronize their data for further processing (i.e. checking if a request is correctly answered). Especially i want to run it in real-time and in continuous mode (or at least repeatedly to come close to it). That is why it is a bit tricky to do it manually over a file that is the appended to and at the same time in another thread is continuously read from.

The helper you suggest would indeed be helpful. I have implemented that in my script with a named pipe.

btw. for my current application the proposal with a customized hla is more appropriate. I have also successfully tested to make use of this plugin
https://github.com/LeonardMH/SaleaeSocketTransportHLA
The backside is that i have to manually configure the logic software.

The answer here #4 is helpful too. Thanks

[ypanemagics]

@Marcus10110 Is there already an implementation of the helper functions that you mentioned? Thanks

[timreyes]

@ypanemagics I suspect we may not have any updates on this topic since this Issue Post is still open. I'll double check with our software team and can get back to you in case we did in fact have a solution ready.

[Marcus10110]

We haven't revisited automation yet to create a super simple solution for this, but I did forget to mention last time that we do provide example python code for parsing our raw binary export file format, which you can export using our python API. I've used this in the past with out python api to automate export and processing. (The binary format is also quite fast to generate and to parse, compared to CSV)

The python examples to parse analog and digital binary export can be found here:
https://support.saleae.com/faq/technical-faq/binary-export-format-logic-2

[ypanemagics]

@Marcus10110: Thanks for pointing out to these examples. This is good enough for me. I can use these as a starting point.

[mfreeborn]

Just to add, I started playing with the automation api this morning (looks like a really useful bit of software for our needs) and the very first thing that I thought was a bit odd was having to write the capture to a file, then read it back into memory before operating on the raw data. Would passing an in-memory BytesIO instance even, rather than directory path, still hit the same technical limitations mentioned above? I see that I can use the binary parsing example for now and that works for me - just means I need to tidy up the file on the filesystem afterwards.