Use winnow to parse proc_macro::TokenStream or proc_macro2::TokenStream

[evan0greenup]

When writing procedural macro, when the input token stream is not Rust code, using winnow to parse is much easier.

Is there any elegant way to use winnow to parse proc_macro::TokenStream or proc_macro2::TokenStream?

[epage]

proc_macro2 works with

• An iterator, while we require a sliceable buffer. That is mostly because we've not seen enough of a use case for working with Iterators to add the complexity
• A tree of tokens, rather than flat tokens.

It looks like syn flattens the TokenStream and puts it into an array, see https://github.com/dtolnay/syn/blob/master/src/buffer.rs

I don't know if we'd want to add direct support for proc macros but a winnow-proc-macro with something like syns TokenBuffer would work.

[dtolnay]

It looks like syn flattens the TokenStream and puts it into an array

FWIW this is a relatively recent change. For more than 5 years the parser used a tree representation. https://github.com/dtolnay/syn/blob/1.0.101/src/buffer.rs

At least in the way Syn's parsing internals are designed, flattening tokens into an array is not the relevant thing that makes efficient parsing possible. Having a cheaply copyable cursor is.

[epage]

To be clear, this was more exploring gaps between proc-macro2 (iterable, tree) and winnow's expectations (flat slice). We can explore whether to support parsing more complicated, transient structures, or we can just copy what was done with TokenBuffer.