Tokenize OPAMFETCH value properly, not splitting by whitespaces

[scjung]

# opam config report
# opam-version         2.2.0~alpha~dev (7f4558d65bbfbf4524e413ab7d06d8ac5f4484d3)
# self-upgrade         no
# system               arch=x86_64 os=macos os-distribution=homebrew os-version=13.2.1
# solver               builtin-mccs+glpk
# install-criteria     -removed,-count[avoid-version,changed],-count[version-lag,request],-count[version-lag,changed],-count[missing-depexts,changed],-changed
# upgrade-criteria     -removed,-count[avoid-version,changed],-count[version-lag,solution],-count[missing-depexts,changed],-new
# jobs                 7
# repositories         1 (http), 1 (version-controlled) (default repo at 5811dd22)
# pinned               0
# current-switch       5.0
# invariant            ocaml-base-compiler >= 5.0 & ocaml-base-compiler < 5.1
# compiler-packages    ocaml-base-compiler.5.0.0, ocaml-options-vanilla.1
# ocaml:native         true
# ocaml:native-tools   true
# ocaml:native-dynlink true
# ocaml:stubsdir       /Users/mukka/.opam/5.0/lib/ocaml/stublibs:/Users/mukka/.opam/5.0/lib/ocaml
# ocaml:preinstalled   false
# ocaml:compiler       5.0.0

Because OPAM splits OPAMFETCH value by spaces, the use-case below does not work as expected.

# The package is in private Github repo, so we supply access token in Authorize header.
$ OPAMFETCH='wget --header "Authorization: token >>>redacted<<<" %{url}% -O %{out}%' opam install coreutil

... snip ...
+ /usr/bin/wget "--header" "\"Authorization:" "token" ">>>redacted<<<\"" "-O" "/home/scjung/.opam/4.12.1/.opam-switch/sources/coreutil.1.10/v1.10.tar.gz.part" "%{url}"
- --2023-03-24 10:31:27--  http://token/
- Resolving token (token)... failed: Name or service not known.
- wget: unable to resolve host address ‘token’

As you may see in the log, "Authorization: token ..." argument is parsed into seperate words.

OPAM should tokenize OPAMFETCH value in the way shell tokenizes[1, 2], so that a quoted argument is parsed as a single string rather than separate words.

[1] https://pubs.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html
[2] https://github.com/dbuenzli/bos/blob/master/src/bos_cmd.ml#L46-L112

[scjung]

Related discussion:
https://discuss.ocaml.org/t/how-can-i-install-opam-package-from-private-github-repo/11745

[kit-ty-kate]

Thanks for opening this bug report.

I don't think opam has the necessary code to parse commands so I've sent a message to @dbuenzli to see how we can leverage his code without pulling all the dependencies from bos. Copying his code with the necessary acknowledgments would be a bit sad but that's an option.

[kit-ty-kate]

@scjung #5492 should fix your issue. I can't promised it'll be merged soon but unless some other issues appear it should be merged before the final release of opam 2.2.0. In the meantime using this branch should be fine if you're ok with that.

[scjung]

It works! Thanks for the fix 👍

Processing  1/3: [coreutil.1.70: http]
+ /usr/local/bin/wget "--header" "Authorization: token >>>redacted<<" "https://github.com/>>redacted<</coreutil/archive/refs/tags/v1.70.tar.gz" "-O" "/Users/mukka/.opam/5.0/.opam-switch/sources/coreutil.1.70/v1.70.tar.gz.part"
- --2023-03-27 09:18:43--  https://github.com/>>redacted<</coreutil/archive/refs/tags/v1.70.tar.gz
- Resolving github.com (github.com)... 20.200.245.247
- Connecting to github.com (github.com)|20.200.245.247|:443... connected.
- HTTP request sent, awaiting response... 302 Found