Add CLI flag & directive for clang/LLVM target triples when using Scala Native

[bynux-gh]

This commit is meant to resolve issue #2766 by adding both a directive:

//> using nativeTargetTriple arch-mnfctr-opsys

and a command line option (--native-target-triple) that can successfully pass the target triple to Scala Native for cross-compiling projects.

As previously mentioned, this is a very experimental option with limited reach. Use with caution, and don't hesitate to provide suggestions for a better approach :)

[Gedochao]

We need integration tests, if this was to be merged. The right place would be one of:

• https://github.com/VirtusLab/scala-cli/blob/main/modules/integration/src/test/scala/scala/cli/integration/RunScalaNativeTestDefinitions.scala
• https://github.com/VirtusLab/scala-cli/blob/main/modules/integration/src/test/scala/scala/cli/integration/PackageTestDefinitions.scala

[Gedochao]

cc @WojciechMazur

[WojciechMazur]

The scala-native-lci integration itself LGTM, however as it was previously mentioned we need to have first a working integration test to prove that it can be actually usable. I left my concerns and mentioned problems with this flag in the comment #2766 (comment)
We should not merge it if we cannot guarantee it is somehow usable and using scala-cli we can actually create a working, cross-compiled binary and execute it.

[tgodzik]

@Gedochao do you think it would be good if we merged it under experimental? We don't actually provide any guarantees at that point.

[bynux-gh]

You all will be very pleased to hear that I got cross-compilation working using this setting! I was able to compile an AArch64 copy of a simple single-file Scala program using an x86_64 machine, then run it in qemu with minimal issues.

The last thing I hadn't accounted for is that Scala-CLI automatically adds system libraries, which is normally the desired behavior, but not when they're being added to a redefined sysroot. So, in the latest commit I have it overriding that behavior when a target triple is defined.

After making sure my rootfs was properly configured, I was able to point clang at it with

--native-linking "--sysroot=/usr/aarch64-linux-gnu"                                      

Everything else is handled automatically by clang/LLVM when the target triple is handed over, so in total the command I needed was:

scala-cli package Test.scala --native --native-target-triple aarch64-linux-gnu --native-linking "--sysroot=/usr/aarch64-linux-gnu" --destination $OUTDIR

The resulting output can be run with qemu-aarch64 as long as it is also told where to find its libraries. It also runs without any additional input on AArch64 Arch Linux (using proot on Termux for Android on my Galaxy Note 9).

Someone who already does cross-compilation on the regular and has sysroots set up will have an easier time of making use of this for small shell programs!

Needless to say, integration tests will still be needed, and the feature should remain experimental while we only have confirmation of one particular workflow working as planned, but it's a massive step in the right direction. Next challenge will be using mingw to see if I can cross-compile Scala code to Windows :)

TL;DR: I have it working on two of my machines, so we just need integration tests to demonstrate reproducible results. I was able to compile a simple one-file Scala program into a native binary that both the qemu emulator and a Linux installation on my Android phone can both execute without issues.

[Gedochao]

@bynux-gh do you need help with setting up the integration test?

[bynux-gh]

@bynux-gh do you need help with setting up the integration test?

Hello hello, yes! I said something in the Scala Discord server but meant to bring it over to this thread. I haven't had much use for GitHub's CI system until now, so I'm not familiar with how it functions or how it integrates with test code. I could definitely use assistance setting that up and making sure the test code "tells" GitHub's systems what we want to hear.

[Gedochao]

Hello hello, yes! I said something in the Scala Discord server but meant to bring it over to this thread. I haven't had much use for GitHub's CI system until now, so I'm not familiar with how it functions or how it integrates with test code. I could definitely use assistance setting that up and making sure the test code "tells" GitHub's systems what we want to hear.

@bynux-gh Okay, so let's not worry about the CI too much for now.
Let's start with adding an integration test which tests the feature and passes on your local machine, and then we'll try to figure out how to make that pass on the CI, if there are problems.

As mentioned before, the test would probably be best added to one of:

• https://github.com/VirtusLab/scala-cli/blob/main/modules/integration/src/test/scala/scala/cli/integration/RunScalaNativeTestDefinitions.scala
• https://github.com/VirtusLab/scala-cli/blob/main/modules/integration/src/test/scala/scala/cli/integration/PackageTestDefinitions.scala

More info on our integration tests can be found around here: https://github.com/VirtusLab/scala-cli/blob/main/DEV.md#run-integration-tests-with-the-jvm-launcher

As a side note, the CI is currently failing on reference docs and formatting, the commands for making those pass can be found here: https://github.com/VirtusLab/scala-cli/blob/main/CONTRIBUTING.md#rules-for-a-well-formed-pr

Let me know if you get stuck.