Making workspace/symbols fast & useful

[matklad]

If I open rust-analyzer's own code base in Emacs with eglot, and use C-u M-., it would miss a lot of symbols (eg, handle_workspace_symbol), which more or less completely breaks the feature from the user's perspective.

What happens here is that eglot sends workspace/symbol with an empty query, expecting to get all the results there are. However, rust-analyzer manually limits the number of the results returned. There's a disagreement about semantics of workspace/symbol, which leads to missing symbols.

Now, it is 100% true that eglot follows the LSP spec, and rust-analyzer is in violation of it. However, from the perspective of rust-analyzer that's intenional violation, because following the spec here leads to fundamentally suboptimal behavior, which significantly impacts user experience for one of the most important features.

If, from eglot's perspective, following the spec is a top priority above all else, then it's fair to just close this as a won't fix, and put the blame on rust-analyzer.

From the perspective of delivering optimal user experience, I think a change in Emacs' behavior (and the spec) is warranted, and the former seems easier, and also in some sense orthogonal to LSP. Non-LSP based backend for C-u M-. would see the same problem, because, as far as i understand it, the limitation is in the Emacs' xref API.

Here's my explanation of design of rust-analyzer here:

Getting all symbols is a fundamentally global operation. As it is global, the size of the result is unbounded, and this can be arbitrary slow (eg, worst case is querying google's code indexer for the list of all symbols in google's codebase).

At the same time, getting all results is useless for the user. The user can see only a handful of results on the screen, anything else is a waste. Even if you stream results, that still creates latency bloat and burns the CPU needlessly. The response which allows for minimizing latency is "here are the first 128 results for your query, please refine it to see more", and that's exactly what rust-analyzer implements.

Although this violates the spec, this actually works fine with most clients, because the LSP API is setup in such a way that the client is supposed to send updated query string to the server on user's typing. Eglot is the exception here. My understanding is that this stems from the fact Emacs' API does not allow incremental refining of the result via talking to an external entity.

TL;DR: Emacs assumes that it is possible to get "the list of all symbols in the project". This assumption doesn't match reality, for performance reasons, the full list might be unknowable. I feel like Emacs' core API should be extended to support "server side" fuzzy matching.

[nemethf]

There is a draft PR partially addressing your question (which I forgot to continue) here: #832

But I did implement rust-analyzer's Workspace Symbols Filtering in eglot-x.el. Do you think that has the functionality you are looking for? To decide it is enough to search for "Workspace Symbols Filtering" and look at relevant screenshots at https://github.com/nemethf/eglot-x

[matklad]

Yeah, that works in terms of functionality, but I think it’s important that default, built-in feature works that way

[joaotavora]

TL;DR: Emacs assumes that it is possible to get "the list of all symbols in the project".

No, it doesn't. 😃 Or at least it shouldn't. In fact I agree with just about everything you wrote and I am familiar with this problem generally. It doesn't happen just for workspace symbols, it also happens for completions (which is an even older problem, AFAIK, and there it's moderately solved by the isIncomplete flag).

So, last time I looked into this (which was somewhat long ago), I tweaked Eglot's C-u M-. to allow just the behaviour you describe (i.e. work with an incomplete list, and re-request the list as user types more input). I didn't give it a whole lot of testing because I didn't have a server whose behaviour i could precisely predict. In particular, I don't think I tried with rust-analyzer either because I didn't have it at the time, or I didn't see a clear test case.

This assumption doesn't match reality, for performance reasons, the full list might be unknowable.

Right, and I agree.

I feel like Emacs' core API should be extended to support "server side" fuzzy matching.

Have a look at the starting commentary of Emacs's lisp/progmodes/external-completion.el and the docstring of the external-completion-table function and tell me what you think.

Eglot makes use of this, precisely to support your use case. But maybe that use is broken in some specific case. So, ideally, you would provide a MRE example that a mostly Rust-ignorant person like me can follow, containing with your expectations and observations for each command. Follow the steps in https://joaotavora.github.io/eglot/#Troubleshooting-Eglot to show such a recipe. You probably can skip step 3, as I have a moderately recent version of rust-analyzer installed.

[matklad]

Fascinating! So it indeed works as it should with built-in completion framework, but doesn't work with ivy

Here's a minimal init.el which shows the problem

(setq package-selected-packages '(rust-mode ivy))
(package-refresh-contents)
(package-initialize)
(package-install-selected-packages)

(ivy-mode)
(setq-default eglot-workspace-configuration
              '(:rust-analyzer
        (:workspace (:symbol (:search (:limit 3))))))

(I configured rust-analyzer to limit to at most 3 symbols, to make the behavior obvious)

Here's the recording:

https://asciinema.org/a/j9KeohR8qZrhMJ3nYtPR1RedB

Disabling (ivy-mode) fixes the thing.

And, for completeness, trivial main.rs, though this should be visible with more or less any project (you can type cargo new foo to create a trivial rust project, and more or less any rust-analyzer binary should work to show this behavior):

fn foo1() {}
fn foo2() {}
fn foo3() {}
fn foobar1() {}
fn foobar2() {}
fn foobar3() {}

fn main() {
    foobar3();
    println!("Hello, world!");
}

[joaotavora]

Thanks for the simple repro. I'm looking into this again and I have detected one bug at least, even with the built-in completion mechanism of fido-vertical-mode (which you may look into for an alternative to ivy).

I'll get back here with more results.

[joaotavora]

I did a small fix to M-x fido-vertical-mode tried your example, which worked fine. See the commit message

https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=0e8d8a72284f6b3aaa1bbce73d41c7d84bbc4d3c

particularly the example, to understand exactly what I fixed.

From my POV the problem is fixed (in fact it never existed if you used any other Emacs completion UI other than fido-mode).

[matklad]

Yup, confirmed that its working now! And thanks for the tip about fido-vertical-mode!