Use HTTPS instead of HTTP #167
Comments
|
Here is some system info: |
|
It would be good to do, but there's no straightforward path to do it. Implementations do not all provide HTTPS support, it's not straightforward to make it from scratch or use HTTPS libraries on all supported platforms. I hope to soon roll out a different form of authentication of downloads via signatures. |
|
Alternatively, for the time being I figure it is possible to point quicklisp at a http proxy that will convert requests to https, I think? |
|
Is there any news about a new form of authentication for quicklisp? |
|
SHA256 verification of downloads or similar might be easier to implement than HTTPS?... (Of course, both would be better eventually.) |
I don't know if I'm just stupid but I think SHA256 verification doesn't help without HTTPS. I think you need some kind of signature (like pgp) as @quicklisp already suggested |
|
You communicate the SHA256 hashes in advance in a secure way authenticated by a signature (as part of the dist metadata download), and then later you download over HTTP (or HTTPS) and verify that the hashes of the files are the expected ones... |
|
@Hexstream So yeah you meant a combination. I thought you meant to only use SHA256 sums. |
|
By the way, guys, have you seen an alternative package manager https://gitlab.common-lisp.net/clpm/clpm which is able to use a quicklisp distribution and to force HTTPS? I've found it recently but didn't have time to try. |
|
I guess integrity is more important than privacy in our case, so the common method would use something like a GPG key to sign the list of package checksums upon distro build. Then, once downloaded and verified, that list can be used to verify individual packages. That would mean another external dependency and calling inferiors through OS. GPG binaries are readily available on most platforms though. Otherwise one could depend on ironclad for PK cryptography and signing/verification of the package cksum list. One of GPG's "strengths" is that the distribution signing key could be acquired/verified trough already existing key server infrastructure. |
|
I have Quicklisp client code for sha256 checking and GPG signature checking in portable CL. See the gpg branch for more info. Right now the hold up is key management and a bit of infrastructure updates. |
|
I note in passing that Keybase is a pretty great modern alternative to the fairly antiquated GPG infrastructure. |
|
One straightforward approach to handling HTTPS is graceful degradation. You already have HTTP connections with GPG signature checking as the fall-through. Adding platform-specific support is unpleasant, but it's either here or in a dedicated portable SSL library that will have to be bundled into Quicklisp. From there, make it highly visible when using HTTPS. When not, keep the same interface we're used to. Is that sufficient or am I overlooking anything? |
|
Is this still in progress? |
|
One alternative to implementing a sufficient amount of GPG in CL is to port OpenBSD's signify instead. It's a lot simpler in comparison and they rely on Ed25519 which is already implemented in CL as part of Ironclad. |
|
The gpg stuff is already complete (see the "pgp" branch). It's a matter of
working out key policies and updating dist building and dist fetching at
this point.
It's not that easy to reuse parts of ironclad.
…On Fri, Oct 7, 2022 at 1:15 PM Vasilij Schneidermann < ***@***.***> wrote:
One alternative to implementing a sufficient amount of GPG in CL is to
port OpenBSD's signify instead. It's a lot simpler in comparison and they
rely on Ed25519 which is already implemented in CL as part of Ironclad.
—
Reply to this email directly, view it on GitHub
<#167 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AACPNLPSNM5GNJDUH3CYV5LWCBLCVANCNFSM4FDA2FKA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
The pgp branch is from 4 years ago, of course it would be great having proper key policies and signature verification, but in the spirit of not letting perfect be the enemy of good, what about incorporating some changes from https://github.com/rudolfochrist/ql-https in the meantime? It would be:
But in summary just shelling out to curl for https download if it's available, and printing a warning otherwise, would be a simple and immediate and significant improvement in quicklisp security. I think it's quite worthwhile to implement in the meantime as it seems dist signing may be some more years off. |
|
Shameless plug (again), but I wrote a small, single file package manager for CL (<200 LOC) that reads the Quicklisp sources but prefers git for decentralization, and creates project-local dependencies. |
|
The solution may be to include implementation-specific parts of CFFI (cffi-sys package) for minimal FFI, and use WinHTTP on Windows, OpenSSL library on Linux, and similar operating systems(OpenSSL is virtually everywhere, and version 3.0 includes HTTP(S) client, and older versions are out of support anyway), and otherwise default to plain HTTP over sockets(on Mezzano?). Not sure about the OpenSSL library situation on macOS though, but that OS has something called Core Foundation and its C API includes http-related stuff. |
|
Any updates on this? @bo-tato would you be open to submitting a fork with your suggestion? I think it's a great idea since it takes care of most of the problem. @quicklisp would you accept a fork? @quicklisp What do you think about the following: How many systems are there that don't support https anyway? I'm sure at this stage it's probably easier to make github/gitlab repositories for those than have to deal with the special cases for the security issues. You don't have to do this though. It should be the responsibility of those systems' maintainers. It's not so unreasonable to require https in 2024 and only passing a If you want to be backwards compatible to not upset the community, you could then invert it and have it In conclusion, this change should be rather simple and straightforward, with the burden placed on the system maintainers, it can even keep backwards compatibility, and probably 99.9% of users will benefit tremendously. Any thoughts? |
|
@quicklisp Is there any issue bigger than this for quicklisp? If you are too busy, can you at least comment on what type of implementation are you willing to accept and open it up to the community to submit a pull request for this? Btw, thanks for all the work on quicklisp in general! 😄 |
https://github.com/rudolfochrist/ql-https is the fork that implements those suggestions (well just checking md5 not sha256 as we can't change the hash format of main quicklisp dist). If quicklisp wants to incorporate those changes I can work on a PR that better merges it into quicklisp proper. |
|
I recently noticed Quicklisp was downloading over HTTP. If @quicklisp would accept it I was thinking about making a PR that would merge the changes from ql-https into Quicklisp. I think this is a reasonable solution given that the user probably will have already installed curl, since the instructions at https://www.quicklisp.org/beta/ tell them to. Alternatively, if there's some way I can help with the pgp branch, I could do that. I'm not too familiar with how that stuff works, but if there's any way I can help with this issue I'd be happy to. |
|
I support @bo-tato and @hpoggie with their proposal with the added detail: We can actually check if curl is available with (uiop:run-program "curl -V")So therefore this could be a valid PR where we can check if the above signals some error/condition (wrap it in a Additionally the user could set up some setting That way this would not break quicklisp for anyone, and at the same time, if curl is available, it would enable by default HTTPS. I think such a PR would be accepted. @quicklisp what do you think? This doesn't break anything, and it enables it by default. In the meantime, this is probably the best solution until something better is developed. |
|
I've just read a couple of articles why lisp is not so popular, there no much jobs and projects. Then, I read a thread 6 years old where people are discussing to make http->https. |
haha it's fair enough that lisp is not popular, there's not much jobs, and ecosystem is quite fractured (or diverse whether you want to call it a word with good or bad connotation). See review of json and testing libraries. I'm not sure if any other language has so many, let alone a language with as few users as common lisp. Also with package management there is many solutions. Some are using ocicl, some qlot, some clpm, some ql-https, some guix, some nix, and some projects like nyxt browser just use git submodules to manage dependencies. There's probably other solutions I haven't heard of and weirder solutions I can't imagine. None of those solutions use insecure http though. I don't think almost anyone doing real development in common lisp now is using quicklisp and http. It is a shame though that most resources for learning common lisp are older so quicklisp is what new users will see recommended, and the maintainer of quicklisp is mostly inactive and seems they won't fix this anytime soon, or even just add a warning to the documentation that quicklisp uses insecure http and recommend some more secure alternatives. To be fair though, 2007 is a little exaggerated. Maven, the main package management for the most used enterprise language with by far the most dollar resources poured into it's development, was supporting http until 2020, and iirc was connecting over http on default setups as recently as 2015-16, and they say 25% was still connecting over http in 2019... |
|
@bo-tato thanks for providing information. I'm new in lisp and just have found couple of packages like ql-https. But it is anyway quite hard to start. |
|
Given the conversation on this issue I wanted to leave an example of This is a condensed and slightly modified version of my quicklisp https The general approach requires three steps
Once you have code locally other lisp implementations (e.g. those that There are ways this could be made more robust, such as removing the In the examples below you should adjust paths to match your system. mkdir /tmp/bootstrap-ql-https
pushd /tmp/bootstrap-ql-https
git clone --branch version-2021-02-13 --depth=1 https://github.com/quicklisp/quicklisp-client.git quicklisp
git clone --depth=1 https://github.com/rudolfochrist/ql-https.git /tmp/ql-https
# when using docker this step can be run with networking disabled to ensure nothing accidentally leaks
sbcl --no-userinit --non-interactive --load bootstrap-1.lisp
sbcl --core continue.core --no-userinit --non-interactive --load bootstrap-2.lisp
sbcl --no-userinit --non-interactive --load bootstrap-3.lisp
(asdf:initialize-source-registry
(list :source-registry (list :tree #p"/tmp/bootstrap-ql-https/ql-https")
(list :tree #p"/tmp/bootstrap-ql-https/quicklisp") :inherit-configuration))
(defpackage #:ql-setup)
; don't forget the trailing slash for files >_<
(defvar ql-setup::*quicklisp-home* (probe-file "/tmp/bootstrap-ql-https/quicklisp/"))
(load "/tmp/bootstrap-ql-https/ql-https/ql-setup.lisp")
(asdf:load-system :ql-https)
(save-lisp-and-die "/tmp/bootstrap-ql-https/continue.core" :purify nil)
(setf ql-https::*quietly-use-https* t)
(quicklisp:setup)
(ql:quickload :dexador)
(let ((quicklisp-init "/tmp/bootstrap-ql-https/quicklisp/setup.lisp"))
(if (probe-file quicklisp-init)
(load quicklisp-init)
(error "path to quicklisp-init is missing!? ~a" quicklisp-init)))
#-dexador
(asdf:load-system :dexador :verbose nil)
#-dexador
(push :dexador *features*)
#-quicklisp-https-always
(asdf:load-system :quicklisp-https-always :verbose nil)
#-quicklisp-https-always
(push :quicklisp-https-always *features*)
(defpackage quicklisp-https-always
(:use :cl)
(:export :setup))
(in-package :quicklisp-https-always)
#+quicklisp
(defun fetch-via-dexador (url file &key (follow-redirects t) quietly (maximum-redirects 10))
"Request URL and write the body of the response to FILE."
(declare (ignorable follow-redirects quietly maximum-redirects))
(let ((url-obj (ql-http:url url)))
(setf (ql-http::scheme url-obj) "https")
(let ((url-string (ql-http::urlstring url-obj)))
(format t "FETCHING VIA dex: ~s~%" url-string) ; comment out this line if you are confident that https is working
(dex:fetch url-string file :if-exists :supersede)))
(values (make-instance 'ql-http::header :status 200)
(probe-file file)))
(defun setup (&key (overwirte t) (methods '("https" "http")))
(declare (ignorable overwirte methods))
#+quicklisp
(dolist (x methods)
(when (or (not (find x ql-http:*fetch-scheme-functions* :test 'equal :key 'first))
overwirte)
(setf ql-http:*fetch-scheme-functions*
(acons x 'fetch-via-dexador
(remove x ql-http:*fetch-scheme-functions* :key 'first :test 'equal))))))
(setup)
(in-package :asdf-user)
(defsystem quicklisp-https-always
:version "0.0.0"
:author "Tom Gillespie" ; adapted from "SANO Masatoshi"
:license "MIT"
:depends-on (:dexador)
:components ((:file "quicklisp-https-always"))
:description "Patch to force quicklisp to always use https. Adapted from quicklisp-https.") |
Hi!
I've noticed that archives are downloaded via
http:But I found that they are also available if I change schema to https:
It would be great (and more secure) to switch to the HTTPS. What do you think, @xach?
The text was updated successfully, but these errors were encountered: