feat: Introduce term search to rust-analyzer #16092

kilpkonn · 2023-12-11T16:27:41Z

Introduce term search to `rust-analyzer`

I've marked this as draft as there might be some shortcomings, please point them out so I can fix them. Otherwise I think it is kind of ready as I think I'll rather introduce extra functionality in follow up PRs.

Term search (or I guess expression search for rust) is a technique to generate code by basically making the types match.
Consider the following program

fn wrap(arg: i32) -> Option<i32> {
    todo!();
}

From the types of values in scope and constructors of Option, we can produce the expected result of wrapping the argument in Option

Dependently typed languages such as Idris2 and Agda have similar tools to help with proofs, but this can be also used in everyday development as a "auto-complete".

Demo videos

2023-12-11_16-28-17.mp4

2023-12-11_15-41-30.mp4

What does it currently do

It works well with locals, free functions, type constructors and non-static impl methods that take items by value.
Works with functions/methods that take shared references, but not with unique references (very conservative).
Can handle projections to struct fields (eg. foo.bar.baz) but this might me more conservative than it has to be to avoid conflicting with borrow checker
Should create only valid programs (no type / borrow checking errors). Tested with rust-analyzer analysis-stats /path/to/ripgrep/Cargo.toml --run-term-search --validate-term-search (basically running cargo check on all of the generated programs and only error seems to be due to type inference which is more of issue of testing method.

Performace / fitness

ripgrep (latest)
Tail Expr syntactic hits: 130/1692 (7%)
Tail Exprs found: 523/1692 (30%)
Term search avg time: 9ms
Term search:         15.64s, 97ginstr, 8mb

rust-analyzer (on this branch)
Tail Expr syntactic hits: 804/13860 (5%)
Tail Exprs found: 6757/13860 (48%)
Term search avg time: 78ms
Term search:         1088.23s, 6765ginstr, 98mb

Highly generic code seems to blow up the search space so currently the amount of generics allowed is functions/methods is limited down to 0 (1 didn't give much improvement and 2 is already like 0.5+s search time)

Plans for the future (not in this PR)

``~~Add impl methods that do not take self type (should be quite straight forward)~~ Done
Be smarter (aka less restrictive) about borrow checking - this seems quite hard but since the current approach is rather naive I think some easy improvement is available.
``~~See if it works as a autocomplete while typing~~ Done

Feel free to ask questions / point of shortcoming either here or on Zulip, I'll be happy to address them. I'm doing this as part of my MSc thesis so I'll be working on it till summer anyway 😄

Young-Flash · 2023-12-12T02:07:46Z

interesting, it would be nicer if there give a bit more docs for crates/hir/src/term_search/*.rs to help understand the idea and the implementation approach under the hood.

kilpkonn · 2023-12-12T14:38:28Z

interesting, it would be nicer if there give a bit more docs for crates/hir/src/term_search/*.rs to help understand the idea and the implementation approach under the hood.

I've added some more documentation there. If you find some caps / stuff that doesn't make sense etc let me know as it definitely can be improved further 😄

Veykril · 2023-12-12T14:47:31Z

Just a headsup but it will take some time (I'm potentially speaking months) until this gets reviewed as there is a massive backlog + me having little time atm + holidays coming up.

kilpkonn · 2023-12-12T14:55:33Z

Just a headsup but it will take some time (I'm potentially speaking months) until this gets reviewed as there is a massive backlog + me having little time atm + holidays coming up.

That's unfortunate :( But thanks for the heads-up, appreciate it!

kilpkonn · 2024-01-08T19:48:33Z

I added also autocomplete part to this PR.
Demo can be seen here:

2024-01-08_21-26-53.mp4

Snippet is a little misleading `auto completion kind for it, but couldn't find any better :p

Veykril · 2024-01-17T10:07:45Z

CI fails because Generator stuff was renamed to Coroutine (i'll try giving this PR a review this or next week)

kilpkonn · 2024-01-17T10:10:05Z

CI fails because Generator stuff was renamed to Coroutine (i'll try giving this PR a review this or next week)

Yep, rebasing onto master now.
Would be awesome to get some feedback next week!

bors · 2024-01-18T10:31:41Z

☔ The latest upstream changes (presumably #16395) made this pull request unmergeable. Please resolve the merge conflicts.

bors · 2024-01-27T11:08:22Z

☔ The latest upstream changes (presumably #16434) made this pull request unmergeable. Please resolve the merge conflicts.

Veykril · 2024-01-27T11:45:54Z

Sorry I haven't gotten around to this yet, been somewhat sick last week.

kilpkonn · 2024-01-27T13:12:32Z

Sorry I haven't gotten around to this yet, been somewhat sick last week.

No worries, hope your well now 😄
I've done the rebase so everything should be ready for review again

crates/hir-def/src/attr.rs

crates/hir-ty/src/infer/unify.rs

crates/hir-ty/src/mir/borrowck.rs

crates/ide-assists/src/handlers/term_search.rs

crates/ide-completion/src/render.rs

crates/ide-diagnostics/src/tests.rs

crates/rust-analyzer/src/cli/flags.rs

crates/rust-analyzer/src/config.rs

Veykril · 2024-02-12T14:43:52Z

Then let's merge this I guess :) Excited to try this out!
@bors r+

bors · 2024-02-12T14:43:54Z

📌 Commit 1257913 has been approved by Veykril

It is now in the queue for this repository.

bors · 2024-02-12T14:47:15Z

⌛ Testing commit 1257913 with merge 818c30c...

bors · 2024-02-12T14:58:34Z

☀️ Test successful - checks-actions
Approved by: Veykril
Pushing 818c30c to master...

feat: Add "make tuple" tactic to term search Follow up to #16092 Now term search also supports tuples. ```rust let a: i32 = 1; let b: f64 = 0.0; let c: (i32, (f64, i32)) = todo!(); // Finds (a, (b, a)) ``` In addition to new tactic that handles tuples I changed how the generics are handled. Previously it tried all possible options from types we had in scope but now it only tries useful ones that help us directly towards the goal or at least towards calling some other function. This changes O(2^n) to O(n^2) where n is amount of rounds which in practice allows using types that take generics for multiple rounds (previously limited to 1). Average case that also used to be exponential is now roughly linear. This means that deeply nested generics also work. ````rust // Finds all valid combos, including `Some(Some(Some(...)))` let a: Option<Option<Option<bool>>> = todo!(); ```` _Note that although the complexity is smaller allowing more types with generics the search overall slows down considerably. I hope it's fine tho as the autocomplete is disabled by default and for code actions it's not super slow. Might have to tweak the depth hyper parameter tho_ This resulted in a huge increase of results found (benchmarks on `ripgrep` crate): Before ```` Tail Expr syntactic hits: 149/1692 (8%) Tail Exprs found: 749/1692 (44%) Term search avg time: 18ms ``` After ``` Tail Expr syntactic hits: 291/1692 (17%) Tail Exprs found: 1253/1692 (74%) Term search avg time: 139ms ```` Most changes are local to term search except some tuple related stuff on `hir::Type`.

rustbot added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Dec 11, 2023

kilpkonn force-pushed the term_search_1 branch 3 times, most recently from dc56fa6 to 0a912d0 Compare December 11, 2023 16:53

kilpkonn force-pushed the term_search_1 branch from fa93910 to 4da835f Compare December 18, 2023 14:33

kilpkonn changed the title ~~Draft: Introduce term search to rust-analyzer~~ Feat: Introduce term search to rust-analyzer Dec 23, 2023

kilpkonn marked this pull request as ready for review December 23, 2023 13:23

kilpkonn changed the title ~~Feat: Introduce term search to rust-analyzer~~ feat: Introduce term search to rust-analyzer Dec 23, 2023

kilpkonn force-pushed the term_search_1 branch 2 times, most recently from 8c0be25 to e860e1a Compare January 9, 2024 09:53

kilpkonn force-pushed the term_search_1 branch from e860e1a to 42b2a1d Compare January 17, 2024 09:58

kilpkonn force-pushed the term_search_1 branch 3 times, most recently from 232bc6d to a2bdd37 Compare January 17, 2024 14:14

kilpkonn force-pushed the term_search_1 branch from a2bdd37 to 62194da Compare January 19, 2024 20:19

kilpkonn force-pushed the term_search_1 branch from bb7dab7 to 50d9a91 Compare January 27, 2024 13:07

kilpkonn force-pushed the term_search_1 branch 2 times, most recently from 21adceb to 23f37ab Compare January 28, 2024 19:32

Veykril reviewed Jan 30, 2024

View reviewed changes

kilpkonn added 6 commits February 11, 2024 13:33

Add more documentation for term search

35eb0db

Add static method tactic

627255d

Initial version of term_search for autocomplete

bdbdd83

Add quantified trees to reduce autocomplete options

a946970

Expand target for autocompletion

0b838e3

Optionally disable term search for autocompletion

88964c0

kilpkonn force-pushed the term_search_1 branch from 587ab93 to 1257913 Compare February 11, 2024 12:35

Cleanup term search related changes

1257913

bors merged commit 818c30c into rust-lang:master Feb 12, 2024
11 checks passed

kilpkonn mentioned this pull request Feb 26, 2024

feat: Add "make tuple" tactic to term search #16687

Merged

philips-software-forest-releaser bot mentioned this pull request Jun 4, 2024

chore(deps,rust): update rust-lang.rust-analyzer philips-software/amp-devcontainer#436

Merged

This was referenced Jun 11, 2024

chore(deps,rust): update rust-lang.rust-analyzer philips-software/amp-devcontainer#445

Merged

chore(deps,rust): update rust-lang.rust-analyzer philips-software/amp-devcontainer#447

Merged

philips-software-forest-releaser bot mentioned this pull request Jun 18, 2024

chore(deps,rust): update rust-lang.rust-analyzer philips-software/amp-devcontainer#456

Merged

philips-software-forest-releaser bot mentioned this pull request Jun 25, 2024

chore(deps,rust): update rust-lang.rust-analyzer philips-software/amp-devcontainer#469

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Introduce term search to rust-analyzer #16092

feat: Introduce term search to rust-analyzer #16092

kilpkonn commented Dec 11, 2023 •

edited

Loading

Young-Flash commented Dec 12, 2023

kilpkonn commented Dec 12, 2023

Veykril commented Dec 12, 2023

kilpkonn commented Dec 12, 2023

kilpkonn commented Jan 8, 2024 •

edited

Loading

Veykril commented Jan 17, 2024

kilpkonn commented Jan 17, 2024 •

edited

Loading

bors commented Jan 18, 2024

bors commented Jan 27, 2024

Veykril commented Jan 27, 2024

kilpkonn commented Jan 27, 2024

Veykril commented Feb 12, 2024

bors commented Feb 12, 2024

bors commented Feb 12, 2024

bors commented Feb 12, 2024

feat: Introduce term search to rust-analyzer #16092

feat: Introduce term search to rust-analyzer #16092

Conversation

kilpkonn commented Dec 11, 2023 • edited Loading

Introduce term search to rust-analyzer

Demo videos

What does it currently do

Performace / fitness

Plans for the future (not in this PR)

Young-Flash commented Dec 12, 2023

kilpkonn commented Dec 12, 2023

Veykril commented Dec 12, 2023

kilpkonn commented Dec 12, 2023

kilpkonn commented Jan 8, 2024 • edited Loading

Veykril commented Jan 17, 2024

kilpkonn commented Jan 17, 2024 • edited Loading

bors commented Jan 18, 2024

bors commented Jan 27, 2024

Veykril commented Jan 27, 2024

kilpkonn commented Jan 27, 2024

Veykril commented Feb 12, 2024

bors commented Feb 12, 2024

bors commented Feb 12, 2024

bors commented Feb 12, 2024

kilpkonn commented Dec 11, 2023 •

edited

Loading

Introduce term search to `rust-analyzer`

kilpkonn commented Jan 8, 2024 •

edited

Loading

kilpkonn commented Jan 17, 2024 •

edited

Loading