how to handle very long sysex messages

[sadguitarius]

I'm trying to send long sysex dumps through rtpmidid, and it looks like I'm running into the hard cap of 4096 bytes imposed by rtpmidid. I just tried recompiling rtpmidid with the numbers that appear in commit 7dcfed9 set to higher values, but this doesn't work. It sounds like ALSA has some general issues with long MIDI messages. I also see there are some previous conversations about adding segmentation to long midi buffers.

The other option I was wondering about, which may be easier to implement for now, would be the possibility of diverting a message to a virtual raw MIDI port created by the amidi command. Currently this doesn't work because rtpmidid waits for the full message to come in before sending it on to the connected port.

I'd be happy to take a look at this stuff in the code if anyone can help point me in the right direction of where to start looking. Thanks!

[davidmoreno]

Hi, after thinking about it hard.. I found it.

rtpmidid should break the sysex into parts. Can you change the code to print the size of sent packets (at rtppeer.cpp:675)? If its longer than about 1500 bytes.. which totally should be by your description, then there is a problem over ethernet as thats the normal frame size limit. You can check with ip a, its the mtu (max transfer unit IIRC).

The solution would be at rtpmidid.cpp:559, and the alsamidi_to_midiprotocol method to split these long message into several frames of... around 1450 bytes? The code is not prepared, but it should not be difficult to at least do a hack to check it thats the problem, by moving the buffer and send inside the alsamidi_to_midiprotocol method. Another fast way to check thats the problem, change the size of the buffer at rtpmidid.cpp:559 to 1450 bytes, which should be the limit. f you split the SysEx message into several frames, remember to end the frames with 0xF0 and start them with 0xF7, except the first and last, as should use normal SysEx start 0xF0 and end 0xF7. so it should be:

0xF0 SYSEX DATA 0xF0 
0xF7 MORE DATA 0xF0 
0xF7 FINAL PACKET 0xF7

Receiving long messages should be ok, it's only sending that was never properly implemented.

What I would do is first a proper automatic test with a long SysEx, connected both sides to alsa sequencer to double check the problem lays where... All that sending though ethernet, as loopback could have larger frame size. But I don't know when I could do it. Another alternative is just sending a long sysex and ensure it sends several packets with proper start and endings.

The hack about sending it to raw midi port.. sounds like a hack. But it could be made into a functionality adding support to send to direct hardware midi devices as well to alsa sequencer devices. Extra points if also to Jack MIDI devices. but that sounds like way more work than fising this problem.. if it's possible.

[sadguitarius]

Got it, that mostly makes sense to me. I'm getting in a bit over my head as far as networking protocol goes, but I do see that the mtu is set to 1500 bytes. I did compile the code with the changed lines, but I'm not seeing anything printed. Where should I be seeing it? Sorry if this is a newb question, but do I need to do something differently so it shows up in journalctl?

I'll try messing around with splitting long messages now that I know where to start looking. Thanks!

[davidmoreno]

Just pushed a test branch (https://github.com/davidmoreno/rtpmidid/tree/93-bugfix-send-long-sysex), that does the packet splitting.

Also ALSA sometimes gives partial SysEx packages (not fully formed, gives them in succession, but no mark of start / middle packet). I implemented the logic which should work.. but not fully sure it does.

I did not do tests yet, only manual, fighting with virtual devices... I hope it all works alright. Tell me if you can try.

[sadguitarius]

Wow that was quick! I just checked and the message splitting is almost working. I can see the incoming messages being split and prepared for output, but they're not actually being sent. I'm sure this is a relatively simple fix.

The larger issue is that I think ALSA is actually just swallowing very long incoming SysEx messages. Once they're over a certain length (> a few thousand bytes), I don't see anything printed by the debug code, either incoming or outgoing. I just found this thread and it sounds like it may actually be necessary to implement raw MIDI in order to support this use case due to ALSA (and JACK) limitations.

Also disregard my previous question about messages not printing in the journal. I was testing out Arch Linux package support and was not pulling the source from the right branch. Silly me.

[sadguitarius]

Actually now that I'm looking at it, maybe using libremidi as a backend could be the simplest solution to this... It's actively developed and specifically tackles this problem.