Feature/data importer #9408

ruchidh · 2025-02-18T02:11:27Z

Description

Description:
This PR introduces the P0 features for the Data Importer, enabling users to import structured data into OpenSearch in a lightweight manner. The initial release is functional for testing and includes core import functionalities.

Key Features Implemented:
✅ Cluster/Domain/Collection Selection & Permission Verification - Users can select a target OpenSearch cluster and verify if they have the necessary permissions to add data or create an index.
✅ Single File Import - Supports importing a single file up to the specified size limit.
✅ Partial Success Handling & Error Notifications - If some data fails to import, users receive notifications with details about the failed records.
✅ Common Data Format Support - Allows importing structured data in widely used formats such as CSV, JSON, and NDJSON.
✅ Preview Top Rows of Structured Data - Before import, users can preview the top X rows of the dataset.

Not in Scope for This Release:
🚫 Reviewing the entire file before import.
🚫 Importing multiple files simultaneously.
🚫 Handling very large files beyond the defined size limit.

Next Steps:
If adoption meets the defined threshold, we will proceed with P1 features, including an editable table for resolving data inconsistencies and post-import verification enhancements.

Testing & Validation:

Verified import functionality for supported file types.
Tested permission validation for cluster selection.
Checked error handling for partial failures.
Confirmed correct previewing of structured data.

Screenshot

Changelog

feat: Data importer P0 release

Check List

All tests pass
- yarn test:jest
- yarn test:jest_integration
New functionality includes testing.
New functionality has been documented.
Update CHANGELOG.md
Commits are signed per the DCO using --signoff

Signed-off-by: Ruchi Sharma <[email protected]>

github-actions · 2025-02-18T02:11:58Z

ℹ️ Manual Changeset Creation Reminder

Please ensure manual commit for changeset file 9408.yml under folder changelogs/fragments to complete this PR.

If you want to use the available OpenSearch Changeset Bot App to avoid manual creation of changeset file you can install it in your forked repository following this link.

For more information about formatting of changeset files, please visit OpenSearch Auto Changeset and Release Notes Tool.

kavilla · 2025-02-18T05:24:44Z

config/opensearch_dashboards.yml

@@ -371,7 +371,7 @@
 # dynamic_config_service.enabled: false

 # Set the value to true to enable direct data import from a file
-# data_importer.enabled: false
+data_importer.enabled: true


i'd recommend not source controlling this config file. historically, people have created PRs with passwords or IPs that they had to clean up later.

You can add a new package.json script or you can start OSD like

yarn start --data_importer.enabled=true --opensearch_security.enabled=false

+1, I usually use git skip to locally ignore config file changes. But TIL you can add configs on the fly

kavilla · 2025-02-18T05:26:41Z

src/plugins/data_importer/public/components/preview_table.tsx

+          className="customSearchBar"
+        />
+      </div>
+      <div style={{ height: 'calc(100% - 40px)', overflowY: 'auto' }}>


avoid inline styling

huyaboo · 2025-02-18T17:56:20Z

yarn.lock

@@ -1527,6 +1527,29 @@
  resolved "https://registry.yarnpkg.com/@faker-js/faker/-/faker-7.6.0.tgz#9ea331766084288634a9247fcd8b84f16ff4ba07"
  integrity sha512-XK6BTq1NDMo9Xqw/YkYyGjSsg44fbNwYRx7QK2CuoQgyy+f1rrTDHoExVM5PsyXCtfl2vs2vVJ0MN0yN6LppRw==

+"@fast-csv/[email protected]":


Do we have dependency changes that require a lockfile change?

huyaboo · 2025-02-18T18:00:13Z

src/plugins/data_importer/public/components/data_importer_app.tsx

+    if (file) {
+      setIsLoadingPreview(true);
+      const reader = new FileReader();
+      reader.onload = (event) => {


Can we avoid reading the entire file into memory? This can be problematic for larger files. I added in the /_preview route a way to parse only the first 10 entries of the document so the file contents (and index mappings) can be found by calling /_preview.

The /_preview can handle file parsing since not all file formats are the same (e.g. entries may not be newline delimited)

huyaboo · 2025-02-18T18:04:28Z

src/plugins/data_importer/public/components/data_importer_app.tsx

+                onSelectedDataSources: onDataSourceSelect,
+                onManageDataSource: () => {}, // Add a proper handler if needed
+              }}
+              onManageDataSource={function (): void {


Do we need this? The DataSourceMenuComponent is needed to only read/grab dataSource information

huyaboo · 2025-02-18T18:11:38Z

src/plugins/data_importer/public/components/data_importer_app.tsx

@@ -132,7 +162,7 @@ export const DataImporterPluginApp = ({
          http,
          inputFile,
          // TODO This should be determined from the index name textbox/selectable
-          false,
+          true,


is there a reason this is set to true? This boolean controls whether the index name is a new or pre-existing OpenSearch index.

huyaboo · 2025-02-18T18:21:25Z

src/plugins/data_importer/public/components/preview_table.tsx

+      </div>
+      {loadedRows < totalRows && (
+        <EuiButton onClick={loadMoreRows} style={{ marginTop: '20px' }}>
+          Click to See More


If we have some sort of lazy loading, then we probably need changes server side to return more docs. Not a blocking comment.

huyaboo · 2025-02-18T18:23:03Z

src/plugins/data_importer/public/components/data_importer_app.tsx

  const [showDelimiterChoice, setShowDelimiterChoice] = useState<boolean>(shouldShowDelimiter());
  const [delimiter, setDelimiter] = useState<string | undefined>(
    dataType === CSV_FILE_TYPE ? CSV_SUPPORTED_DELIMITERS[0] : undefined
  );
+  const [visibleRows, setVisibleRows] = useState<number>(10);


Since we are using 10 a lot as the increment, can we export these instances as a constant in constant.ts?

huyaboo · 2025-02-18T18:26:18Z

DCO check failed. Can you edit your commits to --signoff

zhongnansu

@ruchidh Could you add more details of this PR and linked the issue to the description?

codecov · 2025-02-18T20:13:03Z

Codecov Report

Attention: Patch coverage is 22.00000% with 39 lines in your changes missing coverage. Please review.

Please upload report for BASE (feature/data-importer@7d2c8a7). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
...a_importer/public/components/data_importer_app.tsx	31.25%	19 Missing and 3 partials ⚠️
.../data_importer/public/components/preview_table.tsx	6.25%	15 Missing ⚠️
...mporter/public/components/import_type_selector.tsx	0.00%	2 Missing ⚠️

Additional details and impacted files

@@                   Coverage Diff                    @@
##             feature/data-importer    #9408   +/-   ##
========================================================
  Coverage                         ?   61.72%           
========================================================
  Files                            ?     3833           
  Lines                            ?    92185           
  Branches                         ?    14603           
========================================================
  Hits                             ?    56897           
  Misses                           ?    31618           
  Partials                         ?     3670

Flag	Coverage Δ
Linux_1	`28.99% <ø> (?)`
Linux_2	`56.46% <ø> (?)`
Linux_4	`28.91% <ø> (?)`
Windows_1	`29.00% <ø> (?)`
Windows_2	`56.41% <ø> (?)`
Windows_3	`39.34% <22.00%> (?)`
Windows_4	`28.91% <ø> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Ruchi Sharma <[email protected]>

huyaboo · 2025-02-25T00:33:05Z

src/plugins/data_importer/server/routes/import_file.ts

@@ -87,10 +87,11 @@ export function importFileRoute(
        }
      } else {
        try {
+          const mapping = request.body.mapping ? JSON.parse(request.body.mapping) : {};


thanks for catching that

huyaboo · 2025-02-25T00:34:11Z

src/plugins/data_importer/server/routes/cat_indices.ts

-          dataSource: schema.maybe(schema.string()),
-        }),
-      },
+      validate: false,


why aren't we validating here? shouldn't we enforce dataSourceId if it's provided as a query param?

Signed-off-by: Ruchi Sharma <[email protected]>

github-actions · 2025-02-25T19:21:59Z

❌ Invalid Prefix For Manual Changeset Creation

Invalid description prefix. Found "feat". Only "skip" entry option is permitted for manual commit of changeset files.

If you were trying to skip the changelog entry, please use the "skip" entry option in the ##Changelog section of your PR description.

Signed-off-by: Ruchi Sharma <[email protected]>

ruchidh added 3 commits February 17, 2025 21:12

move header to top and define sectiobna nd add preview table

501ce52

add preview load

b62e78d

Signed-off-by: Ruchi Sharma <[email protected]>

fix ui and add preview table

22a3644

Signed-off-by: Ruchi Sharma <[email protected]>

github-actions bot added the first-time-contributor label Feb 18, 2025

github-actions bot added the failed changeset label Feb 18, 2025

kavilla reviewed Feb 18, 2025

View reviewed changes

huyaboo reviewed Feb 18, 2025

View reviewed changes

zhongnansu reviewed Feb 18, 2025

View reviewed changes

ruchidh added 2 commits February 24, 2025 22:05

add data source, indices and mismtach in table

2fd4543

Signed-off-by: Ruchi Sharma <[email protected]>

revert changes from yml

86d92c6

Signed-off-by: Ruchi Sharma <[email protected]>

huyaboo reviewed Feb 25, 2025

View reviewed changes

ruchidh added 2 commits February 25, 2025 00:43

comment data import flag

445156e

Signed-off-by: Ruchi Sharma <[email protected]>

fix line

44db71a

Signed-off-by: Ruchi Sharma <[email protected]>

revert validate check and give rows count as constant

77bd9b0

Signed-off-by: Ruchi Sharma <[email protected]>

huyaboo approved these changes Feb 25, 2025

View reviewed changes

huyaboo merged commit 1a51a92 into opensearch-project:feature/data-importer Feb 25, 2025
1 of 2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/data importer #9408

Feature/data importer #9408

ruchidh commented Feb 18, 2025 •

edited

Loading

github-actions bot commented Feb 18, 2025

kavilla Feb 18, 2025

huyaboo Feb 18, 2025 •

edited

Loading

kavilla Feb 18, 2025

huyaboo Feb 18, 2025

huyaboo Feb 18, 2025

huyaboo Feb 18, 2025

huyaboo Feb 18, 2025

huyaboo Feb 18, 2025

huyaboo Feb 18, 2025

huyaboo Feb 18, 2025

huyaboo commented Feb 18, 2025

zhongnansu left a comment

codecov bot commented Feb 18, 2025 •

edited

Loading

huyaboo Feb 25, 2025

huyaboo Feb 25, 2025

github-actions bot commented Feb 25, 2025

Feature/data importer #9408

Feature/data importer #9408

Conversation

ruchidh commented Feb 18, 2025 • edited Loading

Description

Screenshot

Changelog

Check List

github-actions bot commented Feb 18, 2025

ℹ️ Manual Changeset Creation Reminder

Choose a reason for hiding this comment

huyaboo Feb 18, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

huyaboo commented Feb 18, 2025

zhongnansu left a comment

Choose a reason for hiding this comment

codecov bot commented Feb 18, 2025 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Feb 25, 2025

❌ Invalid Prefix For Manual Changeset Creation

ruchidh commented Feb 18, 2025 •

edited

Loading

huyaboo Feb 18, 2025 •

edited

Loading

codecov bot commented Feb 18, 2025 •

edited

Loading