Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(openspg): version 0.6 #451

Merged
merged 5 commits into from
Jan 8, 2025
Merged

feat(openspg): version 0.6 #451

merged 5 commits into from
Jan 8, 2025

Conversation

andylau-55
Copy link
Collaborator

New Features

  1. Support for Abstractive Summarization Tasks

    • Native support for abstractive summarization tasks without sacrificing multi-hop factual reasoning accuracy. On the CSQA dataset, while comprehensiveness, diversity, and empowerment metrics are slightly lower than LightRAG (-1.2/10), the factual accuracy metric is better than LightRAG (+0.1/10). On multi-hop question answering datasets such as HotpotQA, TwoWiki, and MuSiQue, since LightRAG and GraphRAG do not provide a factual QA evaluation entry, the EM metric using the default entry is close to 0. For quantitative evaluation results, please refer to the KAG code repository under examples/csqa/README.md and follow the steps to reproduce.
  2. Domain Schema Management

    • The product provides SPG schema management capabilities, allowing users to optimize knowledge base construction and inference Q&A performance by customizing schemas.
  3. Knowledge Exploration

    • Added a knowledge exploration feature to enable visual query and analysis of knowledge base data, and provided an HTTP API for integration with other systems.
  4. Support for Mounting Domain Knowledge in Knowledge Base Construction (Developer Mode)

    • In developer mode, the system supports injecting domain knowledge (domain vocabulary, relationships between terms) into the knowledge base, which can significantly improve knowledge base construction and inference Q&A performance (with a 10%+ improvement in the medical domain).
  5. Added Knowledge Alignment Component to the Build Pipeline

    • Kag-Builder provides a default knowledge alignment component that includes features such as invalid data filtering and similar entity linking, to optimize the structure and data quality of the knowledge graph.

User Experience Optimization

  1. Resumable Tasks

    • Provide resumable capabilities for knowledge base construction tasks at the file level and chunk level in both product mode and developer mode, to reduce the time and token consumption caused by full re-runs after task failures.
  2. User Login & Permission System

    • Implement a user login and permission system to prevent unauthorized access and operations on the knowledge base data.
  3. Optimized Knowledge Base Construction Task Scheduling

    • Provide database-based knowledge base construction task scheduling to avoid task anomalies or interruptions after container restarts.
  4. Support for Different Models at Different Stages (Developer Mode)

    • Provide a registry-based component management mechanism that allows users to instantiate component objects via configuration files. This supports the development and integration of custom components into the KAG-Builder and KAG-Solver workflows. Additionally, it enables the configuration of different large models at different stages of the workflow to enhance overall inference and Q&A performance.
  5. Enhanced Layout Analysis for Markdown, PDF, and Word Files

    • Prioritize chunk division based on file sections for Markdown, PDF, and Word files to ensure that the content within the same chunk is more cohesive.
  6. Global Configuration and Knowledge Base Configuration

    • Provide global configuration for the knowledge base, allowing unified settings for storage engines, generation models, and representation model access information.
  7. Support for Schema-Constrained Extraction and Linking (Developer Mode)

    • Offer schema-constrained mode, where during the knowledge base construction phase, operations are strictly performed according to the definitions in the schema, enabling more granular and complex knowledge extraction.

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

license-eye has checked 1755 files.

Valid Invalid Ignored Fixed
1681 45 29 0
Click to see the invalid file list
  • common/util/src/main/java/com/antgroup/openspg/common/util/DozerBeanMapperUtil.java
  • common/util/src/main/java/com/antgroup/openspg/common/util/converter/EnumToStringConverter.java
  • common/util/src/main/java/com/antgroup/openspg/common/util/converter/LocalDateTimeToDateDozerConverter.java
  • common/util/src/main/java/com/antgroup/openspg/common/util/converter/LocalDateTimeToLocalDateTimeDozerConverter.java
  • common/util/src/main/java/com/antgroup/openspg/common/util/converter/LocalDateToLocalDateDozerConverter.java
  • common/util/src/main/java/com/antgroup/openspg/common/util/converter/LocalTimeToLocalTimeDozerConverter.java
  • common/util/src/main/java/com/antgroup/openspg/common/util/pemja/PemjaUtils.java
  • common/util/src/main/java/com/antgroup/openspg/common/util/pemja/PythonInvokeMethod.java
  • common/util/src/main/java/com/antgroup/openspg/common/util/pemja/model/PemjaConfig.java
  • common/util/src/main/resources/dozer-custom-convert.xml
  • server/api/facade/src/main/java/com/antgroup/openspg/server/api/facade/dto/common/request/DataQueryRequest.java
  • server/api/http-client/src/main/java/com/antgroup/openspg/server/api/http/client/account/AccountService.java
  • server/biz/common/src/main/java/com/antgroup/openspg/server/biz/common/AccountManager.java
  • server/biz/common/src/main/java/com/antgroup/openspg/server/biz/common/ConfigManager.java
  • server/biz/common/src/main/java/com/antgroup/openspg/server/biz/common/PermissionManager.java
  • server/biz/common/src/main/java/com/antgroup/openspg/server/biz/common/impl/AccountManagerImpl.java
  • server/biz/common/src/main/java/com/antgroup/openspg/server/biz/common/impl/ConfigManagerImpl.java
  • server/biz/common/src/main/java/com/antgroup/openspg/server/biz/common/impl/PermissionManagerImpl.java
  • server/common/model/src/main/java/com/antgroup/openspg/server/common/model/account/Account.java
  • server/common/model/src/main/java/com/antgroup/openspg/server/common/model/config/Config.java
  • server/common/model/src/main/java/com/antgroup/openspg/server/common/model/data/DataRecord.java
  • server/common/model/src/main/java/com/antgroup/openspg/server/common/model/data/EntitySampleData.java
  • server/common/model/src/main/java/com/antgroup/openspg/server/common/model/permission/Permission.java
  • server/common/model/src/main/java/com/antgroup/openspg/server/common/model/project/AccountRoleInfo.java
  • server/common/service/src/main/java/com/antgroup/openspg/server/common/service/account/AccountRepository.java
  • server/common/service/src/main/java/com/antgroup/openspg/server/common/service/config/ConfigRepository.java
  • server/common/service/src/main/java/com/antgroup/openspg/server/common/service/config/DefaultValue.java
  • server/common/service/src/main/java/com/antgroup/openspg/server/common/service/permission/PermissionRepository.java
  • server/infra/dao/src/main/java/com/antgroup/openspg/server/infra/dao/dataobject/AccountDO.java
  • server/infra/dao/src/main/java/com/antgroup/openspg/server/infra/dao/dataobject/BuilderJobDO.java
  • server/infra/dao/src/main/java/com/antgroup/openspg/server/infra/dao/dataobject/ConfigDO.java
  • server/infra/dao/src/main/java/com/antgroup/openspg/server/infra/dao/dataobject/DataSourceDO.java
  • server/infra/dao/src/main/java/com/antgroup/openspg/server/infra/dao/dataobject/PermissionDO.java
  • server/infra/dao/src/main/java/com/antgroup/openspg/server/infra/dao/mapper/AccountMapper.java
  • server/infra/dao/src/main/java/com/antgroup/openspg/server/infra/dao/mapper/ConfigMapper.java
  • server/infra/dao/src/main/java/com/antgroup/openspg/server/infra/dao/mapper/PermissionMapper.java
  • server/infra/dao/src/main/java/com/antgroup/openspg/server/infra/dao/repository/common/AccountRepositoryImpl.java
  • server/infra/dao/src/main/java/com/antgroup/openspg/server/infra/dao/repository/common/ConfigRepositoryImpl.java
  • server/infra/dao/src/main/java/com/antgroup/openspg/server/infra/dao/repository/common/PermissionRepositoryImpl.java
  • server/infra/dao/src/main/java/com/antgroup/openspg/server/infra/dao/repository/common/convertor/AccountConvertor.java
  • server/infra/dao/src/main/java/com/antgroup/openspg/server/infra/dao/repository/common/convertor/ConfigConvertor.java
  • server/infra/dao/src/main/java/com/antgroup/openspg/server/infra/dao/repository/common/convertor/PermissionConvertor.java
  • server/infra/dao/src/main/resources/mapper/AccountMapper.xml
  • server/infra/dao/src/main/resources/mapper/ConfigMapper.xml
  • server/infra/dao/src/main/resources/mapper/PermissionMapper.xml
Use this command to fix any missing license headers
```bash

docker run -it --rm -v $(pwd):/github/workspace apache/skywalking-eyes header fix

</details>

@@ -0,0 +1,43 @@
package com.antgroup.openspg.common.util;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
package com.antgroup.openspg.common.util;
/*
* Copyright 2023 OpenSPG Authors
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
* in compliance with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License
* is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
* or implied.
*/
package com.antgroup.openspg.common.util;

@@ -0,0 +1,52 @@
package com.antgroup.openspg.common.util.converter;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
package com.antgroup.openspg.common.util.converter;
/*
* Copyright 2023 OpenSPG Authors
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
* in compliance with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License
* is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
* or implied.
*/
package com.antgroup.openspg.common.util.converter;

@@ -0,0 +1,49 @@
package com.antgroup.openspg.common.util.converter;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
package com.antgroup.openspg.common.util.converter;
/*
* Copyright 2023 OpenSPG Authors
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
* in compliance with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License
* is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
* or implied.
*/
package com.antgroup.openspg.common.util.converter;

@@ -0,0 +1,46 @@
package com.antgroup.openspg.common.util.converter;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
package com.antgroup.openspg.common.util.converter;
/*
* Copyright 2023 OpenSPG Authors
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
* in compliance with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License
* is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
* or implied.
*/
package com.antgroup.openspg.common.util.converter;

@@ -0,0 +1,45 @@
package com.antgroup.openspg.common.util.converter;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
package com.antgroup.openspg.common.util.converter;
/*
* Copyright 2023 OpenSPG Authors
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
* in compliance with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License
* is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
* or implied.
*/
package com.antgroup.openspg.common.util.converter;

@@ -0,0 +1,46 @@
package com.antgroup.openspg.common.util.pemja;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
package com.antgroup.openspg.common.util.pemja;
/*
* Copyright 2023 OpenSPG Authors
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
* in compliance with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License
* is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
* or implied.
*/
package com.antgroup.openspg.common.util.pemja;

@@ -0,0 +1,71 @@
package com.antgroup.openspg.common.util.pemja.model;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
package com.antgroup.openspg.common.util.pemja.model;
/*
* Copyright 2023 OpenSPG Authors
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
* in compliance with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License
* is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
* or implied.
*/
package com.antgroup.openspg.common.util.pemja.model;

@@ -0,0 +1,79 @@
<?xml version="1.0" encoding="UTF-8"?>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Copyright 2023 OpenSPG Authors
~
~ Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
~ in compliance with the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software distributed under the License
~ is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
~ or implied.
-->
<?xml version="1.0" encoding="UTF-8"?>

@@ -0,0 +1,16 @@
package com.antgroup.openspg.server.api.facade.dto.common.request;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
package com.antgroup.openspg.server.api.facade.dto.common.request;
/*
* Copyright 2023 OpenSPG Authors
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
* in compliance with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License
* is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
* or implied.
*/
package com.antgroup.openspg.server.api.facade.dto.common.request;

@@ -0,0 +1,102 @@
package com.antgroup.openspg.server.api.http.client.account;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
package com.antgroup.openspg.server.api.http.client.account;
/*
* Copyright 2023 OpenSPG Authors
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
* in compliance with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License
* is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
* or implied.
*/
package com.antgroup.openspg.server.api.http.client.account;

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

license-eye has checked 1755 files.

Valid Invalid Ignored Fixed
1686 40 29 0
Click to see the invalid file list
  • common/util/src/main/java/com/antgroup/openspg/common/util/DozerBeanMapperUtil.java
  • common/util/src/main/java/com/antgroup/openspg/common/util/pemja/PemjaUtils.java
  • common/util/src/main/java/com/antgroup/openspg/common/util/pemja/PythonInvokeMethod.java
  • common/util/src/main/java/com/antgroup/openspg/common/util/pemja/model/PemjaConfig.java
  • common/util/src/main/resources/dozer-custom-convert.xml
  • server/api/facade/src/main/java/com/antgroup/openspg/server/api/facade/dto/common/request/DataQueryRequest.java
  • server/api/http-client/src/main/java/com/antgroup/openspg/server/api/http/client/account/AccountService.java
  • server/biz/common/src/main/java/com/antgroup/openspg/server/biz/common/AccountManager.java
  • server/biz/common/src/main/java/com/antgroup/openspg/server/biz/common/ConfigManager.java
  • server/biz/common/src/main/java/com/antgroup/openspg/server/biz/common/PermissionManager.java
  • server/biz/common/src/main/java/com/antgroup/openspg/server/biz/common/impl/AccountManagerImpl.java
  • server/biz/common/src/main/java/com/antgroup/openspg/server/biz/common/impl/ConfigManagerImpl.java
  • server/biz/common/src/main/java/com/antgroup/openspg/server/biz/common/impl/PermissionManagerImpl.java
  • server/common/model/src/main/java/com/antgroup/openspg/server/common/model/account/Account.java
  • server/common/model/src/main/java/com/antgroup/openspg/server/common/model/config/Config.java
  • server/common/model/src/main/java/com/antgroup/openspg/server/common/model/data/DataRecord.java
  • server/common/model/src/main/java/com/antgroup/openspg/server/common/model/data/EntitySampleData.java
  • server/common/model/src/main/java/com/antgroup/openspg/server/common/model/permission/Permission.java
  • server/common/model/src/main/java/com/antgroup/openspg/server/common/model/project/AccountRoleInfo.java
  • server/common/service/src/main/java/com/antgroup/openspg/server/common/service/account/AccountRepository.java
  • server/common/service/src/main/java/com/antgroup/openspg/server/common/service/config/ConfigRepository.java
  • server/common/service/src/main/java/com/antgroup/openspg/server/common/service/config/DefaultValue.java
  • server/common/service/src/main/java/com/antgroup/openspg/server/common/service/permission/PermissionRepository.java
  • server/infra/dao/src/main/java/com/antgroup/openspg/server/infra/dao/dataobject/AccountDO.java
  • server/infra/dao/src/main/java/com/antgroup/openspg/server/infra/dao/dataobject/BuilderJobDO.java
  • server/infra/dao/src/main/java/com/antgroup/openspg/server/infra/dao/dataobject/ConfigDO.java
  • server/infra/dao/src/main/java/com/antgroup/openspg/server/infra/dao/dataobject/DataSourceDO.java
  • server/infra/dao/src/main/java/com/antgroup/openspg/server/infra/dao/dataobject/PermissionDO.java
  • server/infra/dao/src/main/java/com/antgroup/openspg/server/infra/dao/mapper/AccountMapper.java
  • server/infra/dao/src/main/java/com/antgroup/openspg/server/infra/dao/mapper/ConfigMapper.java
  • server/infra/dao/src/main/java/com/antgroup/openspg/server/infra/dao/mapper/PermissionMapper.java
  • server/infra/dao/src/main/java/com/antgroup/openspg/server/infra/dao/repository/common/AccountRepositoryImpl.java
  • server/infra/dao/src/main/java/com/antgroup/openspg/server/infra/dao/repository/common/ConfigRepositoryImpl.java
  • server/infra/dao/src/main/java/com/antgroup/openspg/server/infra/dao/repository/common/PermissionRepositoryImpl.java
  • server/infra/dao/src/main/java/com/antgroup/openspg/server/infra/dao/repository/common/convertor/AccountConvertor.java
  • server/infra/dao/src/main/java/com/antgroup/openspg/server/infra/dao/repository/common/convertor/ConfigConvertor.java
  • server/infra/dao/src/main/java/com/antgroup/openspg/server/infra/dao/repository/common/convertor/PermissionConvertor.java
  • server/infra/dao/src/main/resources/mapper/AccountMapper.xml
  • server/infra/dao/src/main/resources/mapper/ConfigMapper.xml
  • server/infra/dao/src/main/resources/mapper/PermissionMapper.xml
Use this command to fix any missing license headers
```bash

docker run -it --rm -v $(pwd):/github/workspace apache/skywalking-eyes header fix

</details>

@@ -0,0 +1,98 @@
package com.antgroup.openspg.server.biz.common;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
package com.antgroup.openspg.server.biz.common;
/*
* Copyright 2023 OpenSPG Authors
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
* in compliance with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License
* is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
* or implied.
*/
package com.antgroup.openspg.server.biz.common;

@@ -0,0 +1,98 @@
package com.antgroup.openspg.server.biz.common;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
package com.antgroup.openspg.server.biz.common;
/*
* Copyright 2023 OpenSPG Authors
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
* in compliance with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License
* is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
* or implied.
*/
package com.antgroup.openspg.server.biz.common;

Copy link
Collaborator

@xionghuaidong xionghuaidong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Collaborator

@zhuzhongshu123 zhuzhongshu123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Collaborator

@northmachine northmachine left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Collaborator

@caszkgui caszkgui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

L

@andylau-55 andylau-55 merged commit e53b20a into master Jan 8, 2025
5 of 6 checks passed
@github-actions github-actions bot locked and limited conversation to collaborators Jan 8, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants