Want to contribute to the Setup tasks that run when setting up the workspace

[harlequin]

I want to start looking into the Setup tasks that run when setting up the workspace from the Roadmap.

My idea would be to implement the "tasks" into the workspace.yml file.

With this the end user would have the full flexibility, which they have also inside a project.

@milesj , is it okay to start at this point?
Or do you have an other point of view at this topic?

[milesj]

The tasks themselves wouldn't be configured in workspace.yml, it would just be a list of task targets from existing projects (most likely the root) to run during setup.

[harlequin]

So this means that workspace.yml will get an config Item as vector of strings
which indicates the Tasks to execute?

Do you have already a naming proposal for this config Item?

[milesj]

Yeah just something like this:

setupTasks:
  - 'project:task'

I'm not sure what the name should be.

We also need to figure out what scenarios do these setup tasks actually run.

[harlequin]

We have several use cases for working with this:-

1. Hashicorp Vault Authentication

The workspace shall login into Vault and will derive a token which the sub-projects can use for receiving secrets

2. Conan config install

When using conan as dependency manager for the projects a common config shall be installed and configured on workspace level.

3. Some "preparation" steps shall be executed which will validate the correct environment for the user

For example, has the git config the correct email format and so on ...

4. proto use shall be called to get the tools

Remark:

One big issue we are facing currently is the the option.envfile is evaluated at the very beginning.
Conan and Vault will generate these during runtime ...

A goal from my thinking is that the

• workspace tasks will be executed and generates the dotenv files
• the moon processor for project tasks will evaluate after the SyncWorkspace all the inputs and envFiles

Another approach would be that the proto toolchain feature will be implemented for

• proto
• vault
• conan

What is complicated at this stage is that all 3 tools can be deployed and distributed via toml.
When we want to use the toolchain feature from plugins, these tools have to be converted into wasm,
which from my point of view is a huge overhead compared to the toml approach.

Conclusion:

Guess the main message would be, how can we produce a dotenv before the project tasks will be evaluated?

[milesj]

Guess the main message would be, how can we produce a dotenv before the project tasks will be evaluated?

I don't think you can in a single process, because tasks are expanded (dotenv eval'd) when the action graph is built, not when it is running. This is necessary because env vars are inputs into the cache hash.

You would probably need your users to run moon setup before at some point.

[harlequin]

Got it, moon setup is also called indirectly via toolchains.

So bringing the toolchain proto topic further, would solve my riddle.

Is there any task which I can contribute?

[milesj]

I just landed uv support in proto: https://moonrepo.dev/blog/proto-v0.45#uv

Maybe update the python toolchain to support pip or uv, similar to how node package managers work?

[harlequin]

Hey @milesj

I will take care about this uv package manager in python, no problem.

But one wish before implementing this:

Can you agree that the toolchain plugins will be than in the release 1.33 (For example)
Because than I would likely to include the new uv logic into the python proto/toolchain plugin.

You have said in other discussions that the toolchain plugin stuff is very hard coded at the moment.
Maybe you can write a very short bullet list how I can achieve the workaround.

The reason why I want to combine uv and toolchain plugins is the fact that many of our internal projects would increase the user experience if some common tasks are covered into the toolchain plugin.
Additional contributing new languages into moon, like Java, seems not to be the future.

What do you think about this strategy?

[milesj]

The toolchain plugins will definitely not land in 1.33, there's still too much work to be done.

The uv support would work just like Node.js, where the user can configure which package manager to use. It would then just change the pip commands being ran to uv. Should be super simple.

[milesj]

Actually, I've been making a lot of bun/deno/rust improvements in 1.32. I'll handle this python change since it should be super simple. Started here: #1810

If you're looking for something interesting to work on, do you have any experience with daemon's?

[harlequin]

Wow, you are so fast in response and implementation!
I have looked through #1810 and seems really good and easy.

[milesj]

I've been thinking about setup tasks a bit more, and I don't think we can actually use moon tasks for this. Reasons:

1 - We can't run setup tasks from the SyncWorkspace action, because tasks must depend on the SyncProject, SetupToolchain, etc, actions in the graph, because they come after SyncWorkspace. So it would create in an invalid DAG.
2 - We can't (maybe) run setup tasks from moon setup because of the reason above. If these setup tasks "should run before the toolchain exists", that's simply not possible. If they can run after the toolchain is setup, then they could work here.

With that said, isntead of moon tasks, we could just run arbitrary commands? Like so:

setupCommands:
  - 'git do --something'
  - './setup-script.sh'

This would avoid the issues above, but is kind of gross, and we lose all the infrastructure that currently exists around tasks (like caching).

And lastly, if you have tasks that you want to run before all tasks, you can do that now with implicitDeps: https://moonrepo.dev/docs/config/tasks#implicitdeps

[harlequin]

If you're looking for something interesting to work on, do you have any experience with daemon's?

What is your aim with a daemon system?

Would it be the same like the bazel remote execution service?

[milesj]

It would just be a background process that moon can pass of heavy/large processing tasks to. Stuff like project graph caching, file watching, remote uploading, etc.

It would be similar to Nx or Bazel daemons.

[harlequin]

Maybe I can explain my current challenges which I have to solve (somehow)

Use Case:

We are working in the software department with conan for C development.
You have to image that with conan you can bootstrap/install compilers, gnu make and libraries like with other package manager.

There are 3 essential commands:

Command	Remark
conan config install	This installs from Git or HTTP the configuration for conan
conan install	Installs the dependencies from conanfile.txt or conanfile.py into your local environment, defaults to $HOME/.conan2
conan build	Execute the build of the C application/program

Essential on this is the fact the conan install creates an dotenv file with the location where the tools and libraries can be found in your $HOME/.conan2

So here a very naive thinking would be that the dotenv from conan can be referenced as options.envfile

But this can't work at the moment, because the conan install command is within the task graph and therefore the options.envfile is already evaluated.

What I have now in mind as solution(s):-

1.) Create a own language, like the Python one, so that conan config install and conan install can be executed within the toolchain and that the dotenv from conan will than be evaluated for the task graph.

This is also the reason why I think that the toolchain plugin topic is so important, because than we would not have to modify the moon application all the time in case of "new" languages and or dependency management system and users can implement in his responsibility.
But I also understand that a lot of work has to be done before we can start this toolchain plugin topic.

2.) Setup tasks
Would be great if the 2 commands can be executed in the proposed setupCommands, but here I would see the risk that I have to put the conan installation itself into the .prototools file. Because this shall be installed, with the correct version like in any other toolchain/language.
Also how is the setupCommands knowing which tool (here in the case it's conan) have to be installed, without the toolchain or platform reference?

Maybe you have, as always, an excellent idea how to coop this challenge.

[milesj]

This would definitely be solved by the plugin system, but for now, I think your best bet is using tag-base tasks? Maybe something like this.

# .moon/tasks/tag-conan.yml
tasks:
  conan-install-config:
    command: 'conan config install' 
    options:
      internal: true
  conan-install-deps:
    command: 'conan install' 
    deps:
      - 'conan-install-config'
    options:
      internal: true
  conan-build:
    command: 'conan build' 
    deps:
      - 'conan-install'
    options:
      internal: true

And then int he project, inherit the conan stuff.

# project/moon.yml
tag: ['conan']
tasks:
  build:
    deps: ['conan-build']

The only blocker is the env file. Could it be solved with env vars?

[harlequin]

Using the tag approach is excactly what we are using.
But how can env variables help us?

Just Image conan Install defines an env variable calles HELLO

conan build can't use this because it will open a new shell which don't know about HELLO env

[milesj]

Ah so the conan commands sets the env var? Not you? Yeah that's a bit complicated. You could bundle this all into a single script?

tasks:
  setup-conan:
    script: 'conan config install && conan install && conan build' 
    options:
      internal: true

Or wrap it in a bash script so you can have more control over all the pieces.

tasks:
  setup-conan:
    command: './setup-conan.sh' 
    options:
      internal: true