Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/data importer #9408

Merged
merged 8 commits into from
Feb 25, 2025
+467 −159
Merged

Feature/data importer #9408

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
501ce52
move header to top and define sectiobna nd add preview table
ruchidh Feb 6, 2025
b62e78d
add preview load
ruchidh Feb 6, 2025
22a3644
fix ui and add preview table
ruchidh Feb 18, 2025
2fd4543
add data source, indices and mismtach in table
ruchidh Feb 24, 2025
86d92c6
revert changes from yml
ruchidh Feb 25, 2025
445156e
comment data import flag
ruchidh Feb 25, 2025
44db71a
fix line
ruchidh Feb 25, 2025
77bd9b0
revert validate check and give rows count as constant
ruchidh Feb 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion config/opensearch_dashboards.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -371,7 +371,7 @@
# dynamic_config_service.enabled: false

# Set the value to true to enable direct data import from a file
# data_importer.enabled: false
data_importer.enabled: true
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'd recommend not source controlling this config file. historically, people have created PRs with passwords or IPs that they had to clean up later.

You can add a new package.json script or you can start OSD like

yarn start --data_importer.enabled=true --opensearch_security.enabled=false

Copy link
Member

@huyaboo huyaboo Feb 18, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, I usually use git skip to locally ignore config file changes. But TIL you can add configs on the fly


# Set the backend roles in groups or users, whoever has the backend roles or exactly match the user ids defined in this config will be regard as dashboard admin.
# Dashboard admin will have the access to all the workspaces(workspace.enabled: true) and objects inside OpenSearch Dashboards.
Expand All @@ -382,3 +382,4 @@

# Set the value to true to enable the new UI for savedQueries in Discover
# data.savedQueriesNewUI.enabled: true
opensearch_security.enabled: false
211 changes: 123 additions & 88 deletions src/plugins/data_importer/public/components/data_importer_app.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,11 @@
EuiPageContent,
EuiPageContentHeader,
EuiPageHeader,
EuiFlexGroup,
EuiFlexItem,
EuiTitle,
EuiPageSideBar,
EuiBasicTable,
EuiLoadingSpinner,
EuiFieldText,
EuiSpacer,
} from '@elastic/eui';
Expand All @@ -40,13 +43,10 @@
import { PublicConfigSchema } from '../../config';
import { ImportTextContentBody } from './import_text_content';
import { ImportFileContentBody } from './import_file_content';
import {
CSV_FILE_TYPE,
CSV_SUPPORTED_DELIMITERS,
PLUGIN_NAME_AS_TITLE,
} from '../../common/constants';
import { CSV_FILE_TYPE, CSV_SUPPORTED_DELIMITERS } from '../../common/constants';
import { DelimiterSelect } from './delimiter_select';
import { previewFile } from '../lib/preview';
import { PreviewComponent } from './preview_table';

interface DataImporterPluginAppProps {
basename: string;
Expand All @@ -69,21 +69,27 @@
dataSourceEnabled,
dataSourceManagement,
}: DataImporterPluginAppProps) => {
const DataSourceMenu = dataSourceManagement?.ui.getDataSourceMenu<DataSourceSelectableConfig>();
const DataSourceMenuComponent = dataSourceManagement?.ui.getDataSourceMenu<
DataSourceSelectableConfig
>();
const [indexName, setIndexName] = useState<string>();
const [importType, setImportType] = useState<ImportChoices>(IMPORT_CHOICE_FILE);
const [disableImport, setDisableImport] = useState<boolean>();
const [dataType, setDataType] = useState<string | undefined>(
config.enabledFileTypes.length > 0 ? config.enabledFileTypes[0] : undefined
);
const [filePreviewData, setFilePreviewData] = useState<any[]>([]);
const [inputText, setText] = useState<string | undefined>();
const [inputFile, setInputFile] = useState<File | undefined>();
const [dataSourceId, setDataSourceId] = useState<string | undefined>();
const [selectedDataSource, setSelectedDataSource] = useState<DataSourceOption | undefined>();
const [filePreviewColumns, setFilePreviewColumns] = useState<any[]>([]);
const [isLoadingPreview, setIsLoadingPreview] = useState<boolean>(false);
const [showDelimiterChoice, setShowDelimiterChoice] = useState<boolean>(shouldShowDelimiter());
const [delimiter, setDelimiter] = useState<string | undefined>(
dataType === CSV_FILE_TYPE ? CSV_SUPPORTED_DELIMITERS[0] : undefined
);
const [visibleRows, setVisibleRows] = useState<number>(10);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we are using 10 a lot as the increment, can we export these instances as a constant in constant.ts?


const onImportTypeChange = (type: ImportChoices) => {
if (type === IMPORT_CHOICE_FILE) {
Expand All @@ -107,8 +113,32 @@

const onFileInput = (file?: File) => {
setInputFile(file);
if (file) {
setIsLoadingPreview(true);
const reader = new FileReader();
reader.onload = (event) => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we avoid reading the entire file into memory? This can be problematic for larger files. I added in the /_preview route a way to parse only the first 10 entries of the document so the file contents (and index mappings) can be found by calling /_preview.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The /_preview can handle file parsing since not all file formats are the same (e.g. entries may not be newline delimited)

const text = event.target?.result as string;
const rows = text.split('\n').map((row) => row.split(','));
const columns = rows[0].map((header, index) => ({

Check warning on line 122 in src/plugins/data_importer/public/components/data_importer_app.tsx

View check run for this annotation

Codecov / codecov/patch

src/plugins/data_importer/public/components/data_importer_app.tsx#L117-L122 

Added lines #L117 - L122 were not covered by tests
field: `column_${index}`,
name: header,
}));
const data = rows.slice(1, 11).map((row) =>
row.reduce((acc: { [key: string]: string }, value, index) => {
acc[`column_${index}`] = value;
return acc;

Check warning on line 129 in src/plugins/data_importer/public/components/data_importer_app.tsx

View check run for this annotation

Codecov / codecov/patch

src/plugins/data_importer/public/components/data_importer_app.tsx#L126-L129 

Added lines #L126 - L129 were not covered by tests
}, {})
);
setFilePreviewColumns(columns);
setFilePreviewData(data);
setIsLoadingPreview(false);

Check warning on line 134 in src/plugins/data_importer/public/components/data_importer_app.tsx

View check run for this annotation

Codecov / codecov/patch

src/plugins/data_importer/public/components/data_importer_app.tsx#L132-L134 

Added lines #L132 - L134 were not covered by tests
};
reader.readAsText(file);

Check warning on line 136 in src/plugins/data_importer/public/components/data_importer_app.tsx

View check run for this annotation

Codecov / codecov/patch

src/plugins/data_importer/public/components/data_importer_app.tsx#L136 

Added line #L136 was not covered by tests
} else {
setFilePreviewColumns([]);
setFilePreviewData([]);

Check warning on line 139 in src/plugins/data_importer/public/components/data_importer_app.tsx

View check run for this annotation

Codecov / codecov/patch

src/plugins/data_importer/public/components/data_importer_app.tsx#L138-L139 

Added lines #L138 - L139 were not covered by tests
}
};

const onTextInput = (text: string) => {
setText(text);
};
Expand All @@ -132,7 +162,7 @@
http,
inputFile,
// TODO This should be determined from the index name textbox/selectable
false,
true,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a reason this is set to true? This boolean controls whether the index name is a new or pre-existing OpenSearch index.

// TODO This should be determined from the file type selectable
fileExtension,
indexName!,
Expand Down Expand Up @@ -241,18 +271,24 @@
const renderDataSourceComponent = useMemo(() => {
return (
<div>
<DataSourceMenu
dataSourceManagement={dataSourceManagement}
componentType={'DataSourceSelectable'}
componentConfig={{
fullWidth: true,
savedObjects: savedObjects.client,
notifications,
onSelectedDataSources: onDataSourceSelect,
selectedOption: selectedDataSource,
}}
/>
<EuiSpacer size="m" />
{DataSourceMenuComponent && (
<>
<DataSourceMenuComponent
componentType={'DataSourceSelectable'}
componentConfig={{
fullWidth: true,
savedObjects: savedObjects.client,
notifications,
onSelectedDataSources: onDataSourceSelect,
onManageDataSource: () => {}, // Add a proper handler if needed
}}
onManageDataSource={function (): void {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this? The DataSourceMenuComponent is needed to only read/grab dataSource information

throw new Error('Function not implemented.');

Check warning on line 286 in src/plugins/data_importer/public/components/data_importer_app.tsx

View check run for this annotation

Codecov / codecov/patch

src/plugins/data_importer/public/components/data_importer_app.tsx#L286 

Added line #L286 was not covered by tests
}}
/>
<EuiSpacer size="m" />
</>
)}
</div>
);
// eslint-disable-next-line react-hooks/exhaustive-deps
Expand Down Expand Up @@ -281,87 +317,86 @@
);
}

const loadMoreRows = () => {
setVisibleRows((prevVisibleRows) => prevVisibleRows + 10);

Check warning on line 321 in src/plugins/data_importer/public/components/data_importer_app.tsx

View check run for this annotation

Codecov / codecov/patch

src/plugins/data_importer/public/components/data_importer_app.tsx#L321 

Added line #L321 was not covered by tests
};

return (
<Router basename={basename}>
<I18nProvider>
<>
<navigation.ui.TopNavMenu appName={PLUGIN_ID} useDefaultBehaviors={true} />
<EuiPage>
<EuiPageSideBar>
<ImportTypeSelector
updateSelection={onImportTypeChange}
initialSelection={importType}
/>
{showDelimiterChoice && (
<DelimiterSelect
onDelimiterChange={onDelimiterChange}
initialDelimiter={delimiter}
/>
)}
<EuiTitle size="xs">
<span>
{i18n.translate('dataImporter.dataSource', {
defaultMessage: 'Data Source Options',
})}
</span>
</EuiTitle>
<EuiFieldText placeholder="Index name" onChange={onIndexNameChange} />
<EuiSpacer size="m" />
{dataSourceEnabled && renderDataSourceComponent}
<EuiButton fullWidth={true} isDisabled={disableImport} onClick={importData}>
Import
</EuiButton>
<EuiSpacer size="m" />
<EuiButton fullWidth={true} isDisabled={disableImport} onClick={previewData}>
Preview
</EuiButton>
</EuiPageSideBar>
<EuiPageBody component="main">
<EuiPageHeader>
<EuiTitle size="l">
<h1>
<FormattedMessage
id="dataImporter.mainTitle"
defaultMessage="{title}"
values={{ title: PLUGIN_NAME_AS_TITLE }}
/>
<FormattedMessage id="dataImporter.mainTitle" defaultMessage="Data Importer" />
</h1>
</EuiTitle>
</EuiPageHeader>
<EuiPageContent>
<EuiPageContentHeader>
<EuiTitle>
<h2>
{importType === IMPORT_CHOICE_TEXT && (
<FormattedMessage
id="dataImporter.textTitle"
defaultMessage="Import Data"
/>
)}
{importType === IMPORT_CHOICE_FILE && (
<FormattedMessage
id="dataImporter.fileTitle"
defaultMessage="Import Data from File"
/>
)}
</h2>
</EuiTitle>
</EuiPageContentHeader>
{importType === IMPORT_CHOICE_TEXT && (
<ImportTextContentBody
onTextChange={onTextInput}
enabledFileTypes={config.enabledFileTypes}
initialFileType={dataType!}
characterLimit={config.maxTextCount}
onFileTypeChange={onDataTypeChange}
/>
)}
{importType === IMPORT_CHOICE_FILE && (
<ImportFileContentBody
enabledFileTypes={config.enabledFileTypes}
onFileUpdate={onFileInput}
/>
)}
<EuiFlexGroup>
<EuiFlexItem grow={1}>
<ImportTypeSelector
updateSelection={onImportTypeChange}
initialSelection={importType}
/>
{showDelimiterChoice && (
<DelimiterSelect
onDelimiterChange={onDelimiterChange}
initialDelimiter={delimiter}
/>
)}
<EuiTitle size="xs">
<span>
{i18n.translate('dataImporter.dataSource', {
defaultMessage: 'Data Source Options',
})}
</span>
</EuiTitle>
<EuiFieldText placeholder="Index name" onChange={onIndexNameChange} />
<EuiSpacer size="m" />
{dataSourceEnabled && renderDataSourceComponent}
{importType === IMPORT_CHOICE_FILE && (
<ImportFileContentBody
enabledFileTypes={config.enabledFileTypes}
onFileUpdate={onFileInput}
/>
)}
<EuiButton fullWidth={true} isDisabled={disableImport} onClick={previewData}>
Preview
</EuiButton>
<EuiSpacer size="m" />
<EuiButton fullWidth={true} isDisabled={disableImport} onClick={importData}>
Import
</EuiButton>
</EuiFlexItem>
<EuiFlexItem grow={2}>
{importType === IMPORT_CHOICE_TEXT && (
<ImportTextContentBody
onTextChange={onTextInput}
enabledFileTypes={config.enabledFileTypes}
initialFileType={dataType!}
characterLimit={config.maxTextCount}
onFileTypeChange={onDataTypeChange}
/>
)}
{importType === IMPORT_CHOICE_FILE && (
<div>
{isLoadingPreview ? (
<EuiLoadingSpinner size="xl" />
) : (
<PreviewComponent
previewData={filePreviewData}
visibleRows={visibleRows}
loadMoreRows={loadMoreRows}
/>
)}
</div>
)}
</EuiFlexItem>
</EuiFlexGroup>
</EuiPageContent>
</EuiPageBody>
</EuiPage>
Expand Down
60 changes: 32 additions & 28 deletions src/plugins/data_importer/public/components/import_type_selector.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,35 +61,39 @@
),
}}
>
<EuiCheckableCard
id={'file-selection'}
label={createLabel({
text: i18n.translate('dataImporter.file', {
defaultMessage: 'Upload from file',
}),
tooltip: i18n.translate('dataImporter.fileTooltip.file', {
defaultMessage: 'Upload data from a file',
}),
})}
checked={importType === IMPORT_CHOICE_FILE}
onChange={() => onChange(IMPORT_CHOICE_FILE)}
/>
<EuiFlexGroup>
<EuiFlexItem>
<EuiCheckableCard
id={'file-selection'}
label={createLabel({
text: i18n.translate('dataImporter.file', {
defaultMessage: 'Upload',
}),
tooltip: i18n.translate('dataImporter.fileTooltip.file', {
defaultMessage: 'Upload data from a file',
}),
})}
checked={importType === IMPORT_CHOICE_FILE}
onChange={() => onChange(IMPORT_CHOICE_FILE)}

Check warning on line 77 in src/plugins/data_importer/public/components/import_type_selector.tsx

View check run for this annotation

Codecov / codecov/patch

src/plugins/data_importer/public/components/import_type_selector.tsx#L77 

Added line #L77 was not covered by tests
/>
</EuiFlexItem>

<EuiSpacer size="s" />

<EuiCheckableCard
id={'text-selection'}
label={createLabel({
text: i18n.translate('dataImporter.text', {
defaultMessage: 'Text box',
}),
tooltip: i18n.translate('dataImporter.fileTooltip.text', {
defaultMessage: 'Type/paste data',
}),
})}
checked={importType === IMPORT_CHOICE_TEXT}
onChange={() => onChange(IMPORT_CHOICE_TEXT)}
/>
<EuiFlexItem>
<EuiCheckableCard
id={'text-selection'}
label={createLabel({
text: i18n.translate('dataImporter.text', {
defaultMessage: 'Text',
}),
tooltip: i18n.translate('dataImporter.fileTooltip.text', {
defaultMessage: 'Type/paste data',
}),
})}
checked={importType === IMPORT_CHOICE_TEXT}
onChange={() => onChange(IMPORT_CHOICE_TEXT)}

Check warning on line 93 in src/plugins/data_importer/public/components/import_type_selector.tsx

View check run for this annotation

Codecov / codecov/patch

src/plugins/data_importer/public/components/import_type_selector.tsx#L93 

Added line #L93 was not covered by tests
/>
</EuiFlexItem>
</EuiFlexGroup>

<EuiSpacer size="m" />
</EuiFormFieldset>
Expand Down
6 changes: 6 additions & 0 deletions src/plugins/data_importer/public/components/preview_table.scss
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
// stylelint-disable-next-line @osd/stylelint/no_modifying_global_selectors
.customSearchBar .euiFormControlLayoutIcons {
height: 20px;
display: flex;
align-items: center;
}
Loading
Loading