[Auto Suggest] PPL & SQL Value Suggestion #8275

paulstn · 2024-09-20T22:25:17Z

Description

Only done for opensearch SQL + PPL. For both languages, value completion only occurs inside of a binary comparison predicate (for example column = value, and for an in-predicate with the keyword IN (e.g. column IN ("value1", "value2", "value3")). These are the only cases in which it makes sense to suggest values for a particular column.

SQL:

Screen.Recording.2024-10-09.at.11.51.51.AM.mov

PPL:

Screen.Recording.2024-10-09.at.11.42.04.AM.mov

`src/plugins/data/public/antlr/*`

For PPL & SQL, uses the SQL query

SELECT <column> FROM <table> GROUP BY <column> ORDER BY COUNT(<column>) DESC LIMIT <limit>

to find the most popular column values to display within autocomplete whenever a value should be suggested.

`src/plugins/data/server/ui_settings.ts`

There were two UI settings added.
query:enhancements:suggestValues is a boolean that determines if value suggestion will be used
query:enhancements:suggestValuesLimit is a number that specifies how many values should be queried, defaulting to 200

`src/plugins/data/public/autocomplete/autocomplete_service.ts`

The ability to add a value suggestion provided was included in this pr, but isn't in use so far. Creating a value suggestion provider that would make the api call (with all of the same parameters that exist right now) could be done but would require a lot of complexity for what is essentially done in 10 lines today.

This PR also contains various fixes and quality of life updates within SQL & PPL suggestions.

Issues Resolved

Screenshot

Testing the changes

Field level security testing:

Screen.Recording.2024-10-21.at.2.22.46.PM.mov

Field level masking testing:

Screen.Recording.2024-10-21.at.2.09.44.PM.mov

Changelog

feat: Autocomplete Value Suggestion

Check List

All tests pass
- yarn test:jest
- yarn test:jest_integration
New functionality includes testing.
New functionality has been documented.
Update CHANGELOG.md
Commits are signed per the DCO using --signoff

Signed-off-by: Paul Sebastian <[email protected]>

github-actions · 2024-09-20T22:25:52Z

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

codecov · 2024-09-20T22:39:27Z

Codecov Report

Attention: Patch coverage is 75.44910% with 41 lines in your changes missing coverage. Please review.

Project coverage is 61.61%. Comparing base (82689bb) to head (26782e2).
Report is 24 commits behind head on main.

Files with missing lines	Patch %	Lines
...ntlr/opensearch_sql/opensearch_sql_autocomplete.ts	68.62%	12 Missing and 4 partials ⚠️
...ntlr/opensearch_ppl/opensearch_ppl_autocomplete.ts	63.88%	7 Missing and 6 partials ⚠️
...ata/public/antlr/opensearch_sql/code_completion.ts	78.78%	3 Missing and 4 partials ⚠️
...ugins/data/public/ui/query_editor/query_editor.tsx	0.00%	3 Missing ⚠️
...ata/public/antlr/opensearch_ppl/code_completion.ts	81.81%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8275      +/-   ##
==========================================
+ Coverage   61.01%   61.61%   +0.59%     
==========================================
  Files        3812     3813       +1     
  Lines       91385    91610     +225     
  Branches    14438    14498      +60     
==========================================
+ Hits        55762    56441     +679     
+ Misses      32065    31587     -478     
- Partials     3558     3582      +24

Flag	Coverage Δ
Linux_1	`29.00% <5.48%> (-0.08%)`	⬇️
Linux_2	`?`
Linux_3	`39.08% <75.44%> (+1.05%)`	⬆️
Linux_4	`28.93% <5.48%> (-0.10%)`	⬇️
Windows_1	`29.01% <5.48%> (-0.08%)`	⬇️
Windows_2	`56.40% <ø> (+<0.01%)`	⬆️
Windows_3	`39.08% <75.44%> (+1.06%)`	⬆️
Windows_4	`28.93% <5.48%> (-0.10%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Paul Sebastian <[email protected]>

paulstn · 2024-12-24T19:07:56Z

src/plugins/data/public/antlr/opensearch_sql/code_completion.test.ts

+      });
+    });
+
+    it.skip('should suggest aggregate functions when appropriate', async () => {


NOTE: This test is skipped for now due to the working functionality requiring a new 'rerun and combine' functionality, where preferred rules are removed one at a time and then the completion candidate process is run without that rule.
This new system is already written in this PR: #9120
Decoupling it from the rest of that PR would be too much unnecessary effort.

mengweieric

Still need more time to test it in app.

mengweieric · 2024-12-24T21:00:53Z

src/plugins/data/public/antlr/opensearch_ppl/opensearch_ppl_autocomplete.test.ts

+});
+
+describe('processVisitedRules', () => {
+  //   // createTokenStream takes a list of 'tokens' defined only by its type or an actual object to be returned as the Token


should we remove these commented out tests?

I left this in as an accident, I was going to write these tests but I was able to cover these tests with end to end tests in code_completion.test.ts.
I'll put some basic processVisitedRules tests in just as a placeholder if more testing is needed for PPL's visitedRules

src/plugins/data/public/antlr/shared/utils.ts

mengweieric · 2024-12-24T21:24:31Z

src/plugins/data/public/autocomplete/providers/value_suggestion_provider.ts

-export type ValueSuggestionsGetFn = (args: ValueSuggestionsGetFnArgs) => Promise<any[]>;
+export type ValueSuggestionsGetFn = (
+  args: ValueSuggestionsGetFnArgs | ValueSuggestionsSQLGetFnArgs
+) => Promise<any[]>;


I think what @joshuali925 suggests is to make ValueSuggestionsGetFn generic by parameterizing args and return types to look something like:

export type ValueSuggestionsGetFn<ArgsType = any, ReturnType = any[]> = ( args: ArgsType ) => Promise<ReturnType>;

with this refactoring, for different types of languages you can now use it to be something like

const sqlValueSuggestions: ValueSuggestionsGetFn<ValueSuggestionsSQLGetFnArgs, string[]> = async (args) => { // Implementation specific to SQL return ['value1', 'value2']; }; const otherLanguageSuggestions: ValueSuggestionsGetFn<CustomArgsType, CustomReturnType> = async (args) => { // Implementation for another language return [{ id: 1, name: 'value1' }]; };

mengweieric · 2024-12-24T21:32:39Z

src/plugins/data/public/autocomplete/autocomplete_service.ts

+        return provider(args);
+      }
+    }
+    return this.defaultGetValueSuggestions


just a question: what would be a valid use case where we don't have an autocomplete provider for a language? If this is the case, should we even provide the default provider as default provider may not even be the one works with this language?

the default value provider exists for the discover filter bar, i've set the default like that because i'm aiming to make it so that it will still exist for every case other than if the language is sql/ppl

mengweieric · 2024-12-24T21:40:21Z

src/plugins/data/public/ui/query_editor/query_editor.tsx

@@ -263,7 +263,7 @@ export const QueryEditorUI: React.FC<Props> = (props) => {
                  range: s.replacePosition ?? defaultRange,
                  detail: s.detail,
                  command: { id: 'editor.action.triggerSuggest', title: 'Trigger Next Suggestion' },
-                  sortText: s.sortText, // when undefined, the falsy value will default to the label
+                  sortText: s.sortText ?? s.text, // when undefined, the falsy value will default to the label


will s.text also be undefined or null in corner cases? do we want something like

sortText: s.sortText ?? s.text ?? ""

s.text can't be undefined or null because its type (MonacoCompatibleQuerySuggestion) has text as a required field

mengweieric · 2024-12-24T21:42:37Z

src/plugins/data/public/antlr/shared/utils.ts

+  column: string,
+  services: IDataPluginServices,
+  fieldInOsd: IndexPatternField | undefined
+): Promise<any[]> => {


seems like we can use string[] as the return type?

I originally wrote this to return a string array, but I changed it to any because if the value suggestions change in the future and return more types than just string, this method would be able to handle those cases too.
Should I change it back to string[]?

I think we should be able to follow the similar above: #8275 (comment), or if string is the only type supported as of right now we can stick with string and extend it later when we want dynamic typing.

Changed it to this in an earlier commit, agree with extending it later since currently only string is supported

src/plugins/data/public/antlr/shared/utils.ts

src/plugins/data/public/autocomplete/providers/value_suggestion_provider.ts

mengweieric · 2024-12-24T21:56:31Z

src/plugins/data/public/antlr/shared/utils.ts

-  );
+      })
+    )
+  ).body.fields[0].values;


this line of code will fail if any field in this chain is missing is it? Also seems like fetchFromAPI is rejecting the promise when there's issue, is this case handled correctly?

In the case when fetchFromAPI rejects the promise with the error, the rejected promise will travel up to the caller of fetchColumnValues. I wrote a test for this mocking an error from the http fetch function that fetchFromAPI calls:
https://github.com/opensearch-project/OpenSearch-Dashboards/pull/8275/files/2a167e91cc27b7be0091cd010495ed87ef9cd0e7#diff-b547688af7b19722a725a97405371301f010a16e1d6dd8dfbfdae74b0e3c784eR374-R382

In the case where the fetched object doesn't match this caller chain, it will throw a rejected promise about the format. I wrote it this way because currently the returned object has this exact format, and there is no other standard to flatted this object to that I'm aware of.

src/plugins/data/public/antlr/opensearch_sql/code_completion.ts

Signed-off-by: Paul Sebastian <[email protected]>

…indexes Signed-off-by: Paul Sebastian <[email protected]>

Signed-off-by: Paul Sebastian <[email protected]>

opensearch-trigger-bot · 2025-01-17T23:32:56Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch-Dashboards/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch-Dashboards/backport-2.x
# Create a new branch
git switch --create backport/backport-8275-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 a7aeb76b70b79880ff807895df3bb5cc19eb1178
# Push it to GitHub
git push --set-upstream origin backport/backport-8275-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch-Dashboards/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-8275-to-2.x.

Signed-off-by: Paul Sebastian <[email protected]> Signed-off-by: Paul Sebastian <[email protected]> Co-authored-by: Paul Sebastian <[email protected]> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com> (cherry picked from commit a7aeb76) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

paulstn added 3 commits September 18, 2024 11:44

basic value suggestion + other stuff

bf6a69b

Signed-off-by: Paul Sebastian <[email protected]>

allow for more than one value suggestion provider

ab915ad

Signed-off-by: Paul Sebastian <[email protected]>

allow for parenthesis suggestions

462d067

Signed-off-by: Paul Sebastian <[email protected]>

github-actions bot added the valued-contributor label Sep 20, 2024

github-actions bot added the failed changeset label Sep 20, 2024

paulstn added 4 commits September 20, 2024 16:26

ui settings for suggestion and limit

da5c691

Signed-off-by: Paul Sebastian <[email protected]>

pass in dataset for mds support

4d39d57

Signed-off-by: Paul Sebastian <[email protected]>

update sql with table name and spaces after insertion

5335744

Signed-off-by: Paul Sebastian <[email protected]>

fix values for sql

11618a4

Signed-off-by: Paul Sebastian <[email protected]>

paulstn marked this pull request as ready for review September 23, 2024 16:44

paulstn requested review from ananzh, kavilla, AMoo-Miki, ashwin-pc, joshuarrrr, abbyhu2000, zengyan-amazon, zhongnansu, manasvinibs, ZilongX, Flyingliuhub, curq, bandinib-amzn, SuZhou-Joe, ruanyl, BionIT, xinruiba and zhyuanqi as code owners September 23, 2024 16:44

paulstn force-pushed the autosuggest-value-suggestion branch from 4bb08f5 to 603a1d9 Compare December 19, 2024 22:22

paulstn added 3 commits December 19, 2024 22:52

basic opensearch ppl autocomplete object tests w/o preferred rules

d854f59

Signed-off-by: Paul Sebastian <[email protected]>

Merge branch 'main' into autosuggest-value-suggestion

77ba770

skip aggregate test, requires new rerun system

470a184

Signed-off-by: Paul Sebastian <[email protected]>

paulstn commented Dec 24, 2024

View reviewed changes

mengweieric reviewed Dec 24, 2024

View reviewed changes

paulstn added 3 commits December 27, 2024 23:57

pr comments incl. basic visited rules tests

92d7061

Signed-off-by: Paul Sebastian <[email protected]>

update utils types to be generic instead of 'any'

e97c32e

Signed-off-by: Paul Sebastian <[email protected]>

move suggestion importance to constants

0a0ae50

Signed-off-by: Paul Sebastian <[email protected]>

paulstn force-pushed the autosuggest-value-suggestion branch from 2a167e9 to 0a0ae50 Compare January 6, 2025 23:19

paulstn added 3 commits January 6, 2025 15:20

Merge branch 'main' into autosuggest-value-suggestion

9d4cd1f

remove value suggestion providor changes

65de34e

Signed-off-by: Paul Sebastian <[email protected]>

take in dataset type to only send agg queries for index patterns and …

00b02e9

…indexes Signed-off-by: Paul Sebastian <[email protected]>

paulstn mentioned this pull request Jan 8, 2025

[Auto Suggest] Fix Grammar Changes #9120

Merged

7 tasks

paulstn added 6 commits January 9, 2025 12:09

Merge branch 'main' into autosuggest-value-suggestion

7dd9475

Signed-off-by: Paul Sebastian <[email protected]>

fix name issue with merge

911c52b

Signed-off-by: Paul Sebastian <[email protected]>

fix merge again

161038a

Signed-off-by: Paul Sebastian <[email protected]>

remove jest reference

ad431b4

Signed-off-by: Paul Sebastian <[email protected]>

sql predicate pref rule field name expansion

25c06cd

Signed-off-by: Paul Sebastian <[email protected]>

ppl pref rule field name expansion

26782e2

Signed-off-by: Paul Sebastian <[email protected]>

mengweieric approved these changes Jan 16, 2025

View reviewed changes

joshuali925 approved these changes Jan 17, 2025

View reviewed changes

joshuali925 merged commit a7aeb76 into opensearch-project:main Jan 17, 2025
69 checks passed

opensearch-trigger-bot bot added the failed backport label Jan 17, 2025

abbyhu2000 added backport 2.x and removed backport 2.x failed backport labels Jan 21, 2025

opensearch-trigger-bot bot mentioned this pull request Jan 21, 2025

[Backport 2.x] [Auto Suggest] PPL & SQL Value Suggestion #9244

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Auto Suggest] PPL & SQL Value Suggestion #8275

[Auto Suggest] PPL & SQL Value Suggestion #8275

paulstn commented Sep 20, 2024 •

edited

Loading

github-actions bot commented Sep 20, 2024

codecov bot commented Sep 20, 2024 •

edited

Loading

paulstn Dec 24, 2024

mengweieric left a comment

mengweieric Dec 24, 2024

paulstn Dec 27, 2024

mengweieric Dec 24, 2024

mengweieric Dec 24, 2024

paulstn Jan 6, 2025

mengweieric Dec 24, 2024

paulstn Dec 27, 2024

mengweieric Dec 24, 2024

paulstn Dec 27, 2024

mengweieric Jan 16, 2025

paulstn Jan 17, 2025

mengweieric Dec 24, 2024

paulstn Dec 27, 2024

opensearch-trigger-bot bot commented Jan 17, 2025

[Auto Suggest] PPL & SQL Value Suggestion #8275

[Auto Suggest] PPL & SQL Value Suggestion #8275

Conversation

paulstn commented Sep 20, 2024 • edited Loading

Description

src/plugins/data/public/antlr/*

src/plugins/data/server/ui_settings.ts

src/plugins/data/public/autocomplete/autocomplete_service.ts

Issues Resolved

Screenshot

Testing the changes

Changelog

Check List

github-actions bot commented Sep 20, 2024

❌ Empty Changelog Section

codecov bot commented Sep 20, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

mengweieric left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

opensearch-trigger-bot bot commented Jan 17, 2025

paulstn commented Sep 20, 2024 •

edited

Loading

`src/plugins/data/public/antlr/*`

`src/plugins/data/server/ui_settings.ts`

`src/plugins/data/public/autocomplete/autocomplete_service.ts`

codecov bot commented Sep 20, 2024 •

edited

Loading