Editor tooling scenarios in expression proc macros

[cart]

If I were to build a custom jsx-like format with a proc_macro (lets call it bsx), what is the feasibility of the following scenarios in the context of rust-analyzer + VSCode. And if they are possible, do you have pointers to prior art / examples / docs / etc?

Important point of clarity: this would be a "fully custom" format. I want to query Rust type information and use it to improve the experience of composing the custom format.

Custom syntax highlighting inside of the macro:

bsx!(Node [
  Label("hello")
  Label("world")
])

Auto-completing information inside of the macro:

bsx!(Node [
  /* 
    when I place my cursor here, query RA for relevant Struct and Enum types that implement
    a given trait and presenting those names as autocomplete options
   */ 
  Label("world")
])

bsx!(Sprite {
    path: "player.png"
     /* 
        when I place my cursor here and type, query RA for missing fields in Sprite and autocomplete them  
    */ 
})

Identifying missing symbols

bsx!(Node [
  // Oh no the Label isn't in scope!  Underline it in red!
  Label("world")
  -------
])

Importing Missing Symbols

bsx!(Node [
  // Present a "import missing symbol code action" for the missing Label symbol 
  Label("world")
  -------
] )

[Veykril]

(Reminds me that I wanted to write a small blog post about this topic at some point, though that'll have to wait until after the token map rewrite)

There are a few ways to make this work well with rust-analyzer (and probably IntelliJ as well), the most important part being that you try to recover from incorrect syntax being passed to your macro (so plain syn based parsing will not suffice, you'll need to put more work into the parsing). That is, if your macro only gets partial input (let's say, a certain identifier is missing that you expect), then your macro should generate output that mostly resembles what it should be if the input was complete (instead of turning the entire output into a single compile_error!(...) invocation, as that will erase basically all the information the IDE could make use of.

The best way to tell you what you can do is probably to roughly explain how the relevant r-a features work in regards to macros.

Custom syntax highlighting inside of the macro

This will just work if you do what was said above, when we see an identifier in a macro input we map it into the expansion (via tracking spans, our span handling is currently being rewritten to work better in general), then we check what the identifier's purpose is in the expansion instead. This will work just fine if you re-use the proc-macro tokens as is. If you re-use the tokens span for more things r-a might get confused though (thats what our span rewrite will hopefully somewhat fix).

Auto-completing information inside of the macro

Auto completion is a bit special, basically when r-a calculates completions for a specific position, we insert a dummy identifier at the cursor position such that the parse tree gets fixed up (and completion always occurs on an identifier), ex. (where $0 represents the cursor) foo!($0) will be invoked as foo!() for the normal analysis, and as foo!(intellijRulezz) (the dummy identifier might change in the future) for completions, we then try to match up the expansions of these two to do analysis, so ideally their expansions should not differ by too much. We basically use the dummy identifier to track where our cursor should be in the expansion then to then calculate completions from that position on, so whereever the intellijRulezz token will end up in the expansion, that's where r-a will think the cursor actually is, and as such it will generate completions for that as the context.

(Note if there current code is foo!(bar$0)r-a will invoke the macro call asfoo!(bar)andfoo!(barintellijRulezz)` respectively, and the same tracking as before still applies)

Identifying missing symbols / Importing Missing Symbols

If Label is a plain path in the output being used in a corrsponding context (let's say as a function call), then this will just work as is. r-a will notice that the identifier is not in scope in the expansion and will offer the assist. Similarly for auto-import completions. If Label in this case is a special identifier that the macro consumes to then generate some other information (so that Label no longer appears in the expansion), that we don't currently have a solution for.

Note that r-a currently lacks diagnostics for unresolved paths, it merely has syntax highlighting for unresolved things.

[cart]

Thanks a ton that is super helpful and exactly what I was hoping to hear! I'm starting to feel optimistic about this :)

[flodiebold]

Edit: Oh, I missed that Veykril already answered in the meantime...

You can't query type information during expansion because it doesn't exist yet; not even names are fully resolved at that time. (Unless you mean having custom code that runs later for specific IDE functions; that would of course have access to type information, but there is currently no way of expanding rust-analyzer in that manner.)

All IDE support in macros currently works by looking at the expansion and deriving the meaning of the original tokens from that. Hence the main thing that proc macros can and should do for better IDE support is to pass through span information properly, and to do parsing in an error resilient way.

So, for (semantic) syntax highlighting there is currently AFAIK no way to customize it. But I think (I'm not an expert on our semantic highlighting code) if e.g. you expand to some code where Label refers to a type and its span points back to the span of the original Label token in the macro input, it will get highlighted as a type.

Autocomplete is where error-resilient parsing comes in. E.g. in this situation:

bsx!(Node [<|>])

(where <|> is the cursor), the proc macro will be passed something like

bsx!(Node [__intellijRulezz])

i.e. rust-analyzer inserts a dummy token where the cursor is and expands the macro with that. If the dummy token appears in the expansion and now is e.g. parsed as a type name, rust-analyzer will use that context to provide completions. If it doesn't appear, e.g. because the macro expands to an error message or panics, then RA will not be able to provide completions. (Also, if the expansion contains further macro calls, rust-analyzer can for technical reasons only name-resolve these currently if they also occur in the expansion without the fake token, i.e. bsx!(Node []). So we need a certain amount of "continuity" in the macro expansion.)
There is no way to restrict completions to types implementing a certain trait or anything like that. There are some ideas to allow macros to provide custom completions, but nothing implemented.

For missing symbols, you should be able to either emit an error from your proc macro (just please still emit an expansion as well), or rely on rustc's error message if the token expands to e.g. a type name. As long as the spans are correct, the error message should show up in the right place.

Same goes for autoimport; both the autoimport completion and the autoimport assist work in macros (as long as the token expands to an unresolved type or value path, of course).

[cart]

Brilliant thanks for the help!