Hitting lots of "stale" build errors on 5.0.6

[mlms13]

A few of us were talking in the #bucklescript channel on Discord, and we've been noticing an increase in the number of "It's possible that your build is stale" errors. Here's an example:

>>>> Start compiling 
[1/33] Building ...s/offerData/OfferData.cmj

  We've found a bug for you!
  /home/michael/Projects/Homebay/homebay/app/_assets/re/models/offerData/OfferData.re
  
  It's possible that your build is stale.
  Try to clean the artifacts and build again?
  
  Here's the original error message
  The files ./app/_assets/re/models/offerData/OfferData_CostAllocations.cmi
  and ./app/_assets/re/models/offerData/OfferData_Other.cmi
  make inconsistent assumptions over interface Txt_Offers
  
>>>> Finish compiling(exit: 1)

It seems to happen pretty often (but not quite all the time) when I edit a file that two or more unique modules depend on.

In the above example, I had Bucklescript's watch mode enabled, and I had just tried to save a change to Txt_Offers.re, which is a module that is used by quite a few other modules (including OfferData_Other.re, which had recently been modified, and OfferData_CostAllocations.re, which had not).

[bobzhang]

To help me diagnose.
Can you reproduce this in 5.1.0?
Can you reproduce this not using editor/watch mode
e.g,

touch Txt_Offers.re
bsb -- -v -d explain

Here -v means running ninja in verbose mode, -d explain means explain why it does rebuild

[TheSpyder]

I can reproduce in 5.1.0. I don't know if I can build a small test case, but I can confirm it happens when not using editor/watch mode.

I have a fairly central module, used at the tip of a growing list of dependencies. Very few parts of my system depend on this module directly; the majority are via a module alias or [@bs.deriving abstract] generated types. It will probably take me a while to track down a sample project that triggers it.

When I change this central module, the explain output has a large list of X.cmj is dirty and X.cmi is dirty for various X, but very few of them are rebuilt.

if I change something else that depends on this central module, the next build throws the stale error with the files X.cmi and Y.cmi make inconsistent assumptions over interface <something that uses the central module>

[bobzhang]

When I change this central module, the explain output has a large list of X.cmj is dirty and X.cmi is dirty for various X, but very few of them are rebuilt.

This is possible when X.cmi is rebuilt but its content is actually not changed. A workspace that I can reproduce would be very helpful

[TheSpyder]

This is possible when X.cmi is rebuilt but its content is actually not changed

There are a lot of modules that rebuild when this core piece changes, which haven't themselves changed. But not enough of them it seems 🤔

As another example of the same issue, if I comment out one of the APIs on this core piece there is no build failure until I do a rebuild; not enough of the modules that depend on it are rebuilt when it changes.

I built a sample project, Module aliases seem to be the cause!
stale.zip

There are comments in Demo.re explaining the replication cases; both for the lack of build failure I just mentioned and also the stale issue (two examples). Mapping this sample to my previous comment, Demo__Internal.re is module X and Other.re is module Y. Failure example 3 seems like the likely cause of my personal issues with stale builds, I'm building a new project and things are changing / expanding all the time.

Looking at the explain output this is a lot simpler than my exact scenario - nothing rebuilds when the internal module is changed, in my project quite a few things do - but it seems like the same underlying cause.

[bobzhang]

stale>ocamlobjinfo lib/bs/src/core/DemoCore-Stale.cmi 
File lib/bs/src/core/DemoCore-Stale.cmi
Unit name: DemoCore-Stale
Interfaces imported:
        99a12ec23baaa373bcbdf195890e5994        DemoCore-Stale
        144a6ab4da176665193ed0a16717f331        Pervasives
        8b069fca1e4d9316b588e5554f38bb24        CamlinternalFormatBasics

It seems Demo__internal was not recorded properly

[bobzhang]

So the issue is that in a namespaced project.

module B = Project_b

is actually

module B = Project.Project_b

If Project_b is only used as alias in project_a.ml, then only Project is recorded as a dependency, since Project is a special file(namespace file), it is created upfront, it does not record the dependency on Project_b (otherwise a cycle dep would be introduced), so Project_b changes would not trigger Project_a in this special case.

I guess dune would expect the same issues, can you confirm that?

Edit: since we plan only to use -no-alias-deps in namespace file, our fix could be easier

[bobzhang]

relevant issues: ocaml/ocaml#6843

[TheSpyder]

I guess dune would expect the same issues, can you confirm that?

Me? I'm not very familiar with dune, sorry. I made some toy projects back in the jbuilder days but that's about it.

[bobzhang]

Here is a small example that showing the property that cmi containing the transitive dependencies no longer hold after bring the -no-alias-deps in single place:

ns.ml

module X = List

mty.ml

module X = X

ocaml>./ocamlc -c -no-alias-deps  ns.ml && ocamlobjinfo ns.cmi
Path: List
File ns.cmi
Unit name: Ns
Interfaces imported:
        f8cd3d72903d320741a5b370ed6d5e7f        Ns
        0d015a5a2136659b0de431be7f1545be        Pervasives
        --------------------------------        List
        8b069fca1e4d9316b588e5554f38bb24        CamlinternalFormatBasics

ocaml>./ocamlc -c -open Ns mty.ml && ocamlobjinfo mty.cmi
Mty_alias:Ns.X vs List
Path: Ns.X
File mty.cmi
Unit name: Mty
Interfaces imported:
        b1a71b460aaa32e87fa76b08c17a6bd3        Mty
        0d015a5a2136659b0de431be7f1545be        Pervasives
        f8cd3d72903d320741a5b370ed6d5e7f        Ns
        8b069fca1e4d9316b588e5554f38bb24        CamlinternalFormatBasics

mty no longer depends on Ns. Note we can not take the approach dune adopted by imposing transitive dependencies in the system level , that would render our effort of content-based build system not available any more

[bobzhang]

This is indeed a tough issue, great that we own the whole compiler that we can do a clean fix. can you confirm that everything looks good after #3805?

[TheSpyder]

Oooh thank you for the quick update! I’m happy to try, how do I install from master? Check out the codebase and build it myself? 🤔

[bobzhang]

@TheSpyder I made a change in the last commit, so that install from master should work out of box (no git submodule needed), feedback is welcome and appreciated (build errors are hard to debug)

[TheSpyder]

Sorry I took a few days to get around to this. I was able to just use the git repo as a direct dependency in my package.json, it took ages to install but that's ok for now.

It's working for me! No more stale errors! When do you think this will become a full release?

[edit] nevermind, no rush as I can't upgrade until jaredly/reason-language-server#320 is fixed

[bobzhang]

@TheSpyder I think both are fixed with 5.2.0-dev.2 would you have a try?

[TheSpyder]

Yes, I've been using it all day since your announcement on discord and it seems good 👍

[mlms13]

Sorry I didn't do more to follow up on this, but I can also confirm that with 5.2.0-dev.2, things seem to be great.