Insert markers into event buffer, and waiting on them

[dhbaird]

I have a pretty awkward way that I am invoking sysdig at the moment:

sysdig -z -w out.pcap &
sleep 0.5 # <-- awkward (need to wait for sysdig to be ready)
myProgram
sleep 0.5 # <-- awkward (sysdig might not yet be done writing events)
kill -s SIGTERM %1 # <-- cross fingers and hope all the interesting events are written!

Basically I need a way of synchronizing with events, kind of like this:

sysdig -z -w out.pcap --on-start ready & # tell sysdig to insert a "ready" marker
sysdigwait ready # wait for the "ready" marker to be written to out.pcap
myProgram
sysdigsend notifyQuit # insert a "notifyQuit" marker into out.pcap
sysdigwait notifyQuit # wait for the "notifyQuit" to be written to out.pcap
kill -s SIGTERM %1 # <-- now it's actually safe to quit, confidently capturing all data :)

A possible operational theory:

• sysdigsend <marker>

  Put marker event into the kernel log buffer.

• sysdigwait <marker>

  Communicate with the running sysdig logger instance, and waits for specified marker to be seen and written to output by sysdig. This could work as follows: Each observed marker increments a count of that marker. Each sysdigwait decrements the counter. If counter is 0, sysdigwait blocks. If sysdig is not running, sysdigwait blocks.

• sysdigclear

  Set all the counters to 0.

Markers could also be leveraged for more informed analysis, though this example is not the greatest, but hopefully inspires an idea, e.g.,

sysdigsend "Running test #1"
./runTest1
sysdigsend "Running test #2"
./runTest2

Btw, I'm very much enjoying sysdig. Many thanks for developing it.

[gianlucaborello]

That's an interesting idea, and I'd be curious to hear your practical use case as well.
As you can imagine, this is not trivial to implement.

As a temporary solution, the last 3 commits on the master branch (not yet released) contain a fix that make sysdig read the events from the kernel much more frequently than before, so with your 0.5 seconds wait after quitting your program, you should be fairly covered.

[dhbaird]

My use case (at this moment) is: synchronizing my script and the logger to ready/quit events. Using timeout delays is necessary at the moment and, even though it seems to work, it strikes me as a hack. So, I wouldn't necessarily need a full marker system, as described above. I figured that if a marker system existed though, it would solve the synchronization problem I have.

[ldegio]

David,
the first sleep (need to wait for sysdig to be ready) is caused by a limitation in the way sysdig does its initialization, and I believe it can be fixed. As you can see in the scap_open_live source, sysdig scans proc to create the process table, and then starts the capture. Scanning prog can take time, especially on busy machines, and introduces a non deterministic delay.

The choice of scanning proc first was taken because we wanted to avoid a bunch of sysdig-generated events to saturate the capture buffers when the program starts. This could be solved more elegantly by including a pid filter in the driver. At that point, the capture would start immediately, and sysdig would just efficiently drop the events that it generates when scanning proc.

We can open an issue for this, so that one of the developers will take care of it, or if you prefer feel free to send a patch.

Regarding the sysdigsend idea. Please take a look at the user_events branch, which unfortunately doesn't compile right now. The idea is creating a fake file (/dev/sysdig-events) where you can write json snippets from an arbitrary script or program. The snippets are injected in the sysdig stream, and they are parsed by the decoding engine to do things similar to the ones you suggest, plus:

• reporting metrics
• trigger sysdig to start/stop capturing
• run sysdig chisels for arbitrary snippets of your code
• measure delays

and so on. It's going to take a bit before I can merge it into master, but it would be nice to understand if you think this could solve your problem.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.