Remove need for short reads?

[daewok]

I was looking into what it would take to integrate this into Drakma. Besides #1, I'm guessing that the need for the base stream to support short reads would be a stumbling block.

Just to make sure I understand the problem correctly, it appears the issue stems from a difference in schannel vs openssl. In openssl (and cl+ssl), it seems that the underlying socket is completely handed over to openssl. Openssl will then perform all the reads as necessary. Whereas in schannel, the socket isn't handed over and the user is responsible for reading from the socket and filling schannel's buffers. Correct?

I know it's kind of gross, but would it be possible to remove the need for short reads by having this library read the TLS record header? So before decrypting any message, it would read the 5 byte header, extract the size (last two bytes), read-sequence that many bytes and then pass the header + record over to schannel? It could probably also be made optional, so if you're using fsocket it can continue doing short reads.

[fjames86]

Hi, I don’t know if the situation has changed but it was previously the case that Zack could not accept windows-only projects into quicklisp, maybe that’s not the case any longer.

Your understanding of the problem is correct, the windows schannel api requires the user to buffer input repeatedly until a success code is returned. You make a good point that we could be clever and compute the buffer lengths ourselves, which removes the requirement for read-sequence to support short reads. Is this something you’d like to do? Feel free to make a pull request with this functionality. Unfortunately I have very little time for this at the moment.

I do have forks of drakma and hunchentoot which use my schannel lib on windows instead of cl+ssl. Did you try these yet? I have never communicated with the maintainers of drakma to find if they would even consider such a patch.

[daewok]

I don't think QL has an issue with Windows only projects any more. At the very least, winhttp is in it :).

It'll probably be a while, but I'll have a look at computing the buffer lengths ourselves.

I can also reach out to the Drakma folks to see if they'd accept this. But even if they aren't interested, I think it's possible to use the :stream argument to drakma:http-request to force the use of schannel. I'm planning a connection pool for Drakma that would make doing such a thing (hopefully) pretty easy.

[fjames86]

I didn't know winhttp was in it, I don't recall requesting it to be added.

Someone else has requested I get fsocket added to QL so we can add schannel too at the same time.

[lispstudent]

I have never communicated with the maintainers of drakma to find if they would even consider such a patch.

A request was opened a while back. Hearing from schannel's Maintainer may offer additional help for inclusion.

[fjames86]

@lispstudent thanks, I have just commented on the linked request. I think it is the right thing to do, my only concern is the significant change in behaviour which may surprise some existing users. If we can get the proposal from @daewok to work (parsing the Tls header to compute fragment lengths) then this avoids the hard requirement for short reads which was my stumbling block. Sounds like a fun Christmas challenge for someone.

[fjames86]

@daewok I kept thinking about this so I had a go at getting it to work. I've made changes to schannel and my drakma fork to implement your suggestion of parsing the TLS header. The drakma port now uses usocket only and seems to work ok - I tried with example.com and google.com. Please try latest versions. I suspect there will still be bugs so it needs some robust testing.

[daewok]

Awesome! I'm still pretty caught up in end of year stuff, but I'll do what testing I can when it's over. I really prefer Drakma over Dexador, so I'm hoping this will let me use a single HTTP client in my projects, regardless of OS.