From e3359279ae869a511a847c880c81224ff6ec9d99 Mon Sep 17 00:00:00 2001 From: Sai Suraj Date: Sun, 12 Feb 2023 11:32:45 +0530 Subject: [PATCH] Corrected more spelling/grammatical mistakes in other some other files, removed unnecessary spaces at some places. Signed-off-by: Sai Suraj --- docs/distributed-synergy-ai-benchmarking.md | 81 ++++++++++--------- docs/index.rst | 4 +- .../industrial-defect-detection/pcb-aoi.md | 16 ++-- docs/proposals/simulation/simulation.md | 14 +++- ...ndustrial-defect-detection-with-pcb-aoi.md | 19 ++--- ...ndustrial-defect-detection-with-pcb-aoi.md | 23 +++--- 6 files changed, 88 insertions(+), 69 deletions(-) diff --git a/docs/distributed-synergy-ai-benchmarking.md b/docs/distributed-synergy-ai-benchmarking.md index 08a921f3..a1a6c4cd 100644 --- a/docs/distributed-synergy-ai-benchmarking.md +++ b/docs/distributed-synergy-ai-benchmarking.md @@ -1,55 +1,65 @@ # Distributed Synergy AI Benchmarking -Edge computing emerges as a promising technical framework to overcome the challenges in cloud computing. In this machine-learning era, the AI application becomes one of the most critical types of applications on the edge. Driven by the increasing computation power of edge devices and the increasing amount of data generated from the edge, edge-cloud synergy AI and distributed synergy AI techniques have received more and more attention for the sake of device, edge, and cloud intelligence enhancement. -Nevertheless, distributed synergy AI is at its initial stage. For the time being, the comprehensive evaluation standard is not yet available for scenarios with various AI paradigms on all three layers of edge computing systems. According to the landing challenge survey 2022, developers suffer from the lack of support on related datasets and algorithms; while end users are lost in the sea of mismatched solutions. That limits the wide application of related techniques and hinders a prosperous ecosystem of distributed synergy AI. A comprehensive end-to-end distributed synergy AI benchmark suite is thus needed to measure and optimize the systems and applications. +Edge computing emerges as a promising technical framework to overcome the challenges in cloud computing. In this machine-learning era, the AI application becomes one of the most critical types of applications on the edge. Driven by the increasing computation power of edge devices and the increasing amount of data generated from the edge, edge-cloud synergy AI and distributed synergy AI techniques have received more and more attention for the sake of device, edge, and cloud intelligence enhancement. + +Nevertheless, distributed synergy AI is at its initial stage. For the time being, the comprehensive evaluation standard is not yet available for scenarios with various AI paradigms on all three layers of edge computing systems. According to the landing challenge survey 2022, developers suffer from the lack of support on related datasets and algorithms; while end users are lost in the sea of mismatched solutions. That limits the wide application of related techniques and hinders a prosperous ecosystem of distributed synergy AI. A comprehensive end-to-end distributed synergy AI benchmark suite is thus needed to measure and optimize the systems and applications. Ianvs thus provides a basic benchmark suite for distributed synergy AI, so that AI developers and end users can benefit from efficient development support and best practice discovery. ## Goals -For developers or end users of distributed synergy AI solutions, the goals of the distributed synergy AI framework are: + +For developers or end users of distributed synergy AI solutions, the goals of the distributed synergy AI framework are: + - Facilitating efficient development for developers by preparing - test cases including dataset and corresponding tools - benchmarking tools including simulation and hyper-parameter searching - Revealing best practices for developers and end users - presentation tools including leaderboards and test reports - ## Scope -The distributed synergy AI benchmarking ianvs aims to test the performance of distributed synergy AI solutions following recognized standards, in order to facilitate more efficient and effective development. + +The distributed synergy AI benchmarking ianvs aims to test the performance of distributed synergy AI solutions following recognized standards, in order to facilitate more efficient and effective development. The scope of ianvs includes -- Providing end-to-end benchmark toolkits across devices, edge nodes and cloud nodes based on typical distributed-synergy AI paradigms and applications. - - Tools to manage test environment. For example, it would be necessary to support the CRUD (Create, Read, Update and Delete) actions in test environments. Elements of such test environments include algorithm-wise and system-wise configuration. + +- Providing end-to-end benchmark toolkits across devices, edge nodes, and cloud nodes based on typical distributed-synergy AI paradigms and applications. + - Tools to manage test environment. For example, it would be necessary to support the CRUD (Create, Read, Update, and Delete) actions in test environments. Elements of such test environments include algorithm-wise and system-wise configuration. - Tools to control test cases. Typical examples include paradigm templates, simulation tools, and hyper-parameter-based assistant tools. - - Tools to manage benchmark presentation, e.g., leaderboard and test report generation. -- Cooperation with other organizations or communities, e.g., in KubeEdge SIG AI, to establish comprehensive benchmarks and developed related applications, which can include but are not limited to + - Tools to manage benchmark presentation, e.g., leaderboard and test report generation. +- Cooperation with other organizations or communities, e.g., in KubeEdge SIG AI, to establish comprehensive benchmarks and developed related applications, which can include but are not limited to - Dataset collection, re-organization, and publication - - Formalized specifications, e.g., standards + - Formalized specifications, e.g., standards - Holding competitions or coding events, e.g., open source promotion plan - - Maintaining solution leaderboards or certifications for commercial usage + - Maintaining solution leaderboards or certifications for commercial usage Targeting users + - Developers: Build and publish edge-cloud collaborative AI solutions efficiently from scratch - End users: view and compare distributed synergy AI capabilities of solutions The scope of ianvs does NOT include to + - Re-invent existing edge platform, i.e., kubeedge, etc. -- Re-invent existing AI framework, i.e., tensorflow, pytorch, mindspore, etc. +- Re-invent existing AI frameworks, i.e., tensorflow, pytorch, mindspore, etc. - Re-invent existing distributed synergy AI framework, i.e., kubeedge-sedna, etc. - Re-invent existing UI or GUI toolkits, i.e., prometheus, grafana, matplotlib, etc. ## Design Details + ### Architecture and Modules + The architectures and related concepts are shown in the below figure. The ianvs is designed to run within a single node. Critical components include + - ``Test Environment Manager``: the CRUD of test environments serving for global usage -- ``Test Case Controller``: control the runtime behavior of test cases like instance generation and vanish - - ``Generation Assistant``: assist users to generate test cases based on certain rules or constraints, e.g., the range of parameters +- ``Test Case Controller``: control the runtime behavior of test cases like instance generation and vanish + - ``Generation Assistant``: assist users to generate test cases based on certain rules or constraints, e.g., the range of parameters - ``Simulation Controller``: control the simulation process of edge-cloud synergy AI, including the instance generation and vanishment of simulation containers - ``Story Manager``: the output management and presentation of the test case, e.g., leaderboards ![](guides/images/ianvs_arch.png) -Ianvs includes Test-Environment Management, Test-case Controller and Story Manager in the Distributed Synergy AI benchmarking toolkits, where +Ianvs includes Test-Environment Management, Test-case Controller, and Story Manager in the Distributed Synergy AI benchmarking toolkits, where + 1. Test-Environment Manager basically includes - Algorithm-wise configuration - Public datasets @@ -59,44 +69,39 @@ Ianvs includes Test-Environment Management, Test-case Controller and Story Manag - System-wise configuration - Overall architecture - System constraints or budgets - - End-to-end cross-node + - End-to-end cross-node - Per node -2. Test-case Controller includes but is not limited to the following components - - Templates of common distributed-synergy-AI paradigms, which can help the developer to prepare their test case without too much effort. Such paradigms include edge-cloud synergy joint inference, incremental learning, federated learning, and lifelong learning. +2. Test-case Controller includes but is not limited to the following components + - Templates of common distributed-synergy-AI paradigms, which can help the developer to prepare their test case without too much effort. Such paradigms include edge-cloud synergy joint inference, incremental learning, federated learning, and lifelong learning. - Simulation tools. Develop simulated test environments for test cases - Note that simulation tools are not yet available in early versions until v0.5 - - It is NOT in scope of this open-sourced Ianvs to simulate different hardware devices, e.g., simulating NPU with GPU and even CPU - - Other tools to assist test-case generation. For instance, prepare test cases based on a given range of hyper-parameters. + - It is NOT in the scope of this open-sourced Ianvs to simulate different hardware devices, e.g., simulating NPU with GPU and even CPU + - Other tools to assist test-case generation. For instance, prepare test cases based on a given range of hyper-parameters. 3. Story Manager includes but is not limited to the following components - Leaderboard generation - Test report generation - ### Definitions of Objects Quite a few terms exist in ianvs, which include the detailed modules and objects. To facilitate easier concept understanding, we show a hierarchical table of terms in the following figures, where the top item contains the items below it. ![](guides/images/ianvs_concept.png) -The concept definition of modules has been shown in the Architecture Section. In the following, we introduce the concepts of objects for easier understanding. -- ``Benchmark``: standardized evaluation process recognized by the academic or industry. -- ``Benchmarking Job``: the serving instance for an individual benchmarking with ianvs, which takes charge of the lifetime management of all possible ianvs components. - - Besides components, a benchmarking job includes instances of a test environment, one or more test cases, a leaderboard, or a test report. - - Different test environments lead to different benchmarking jobs and leaderboards. A benchmarking job can include multiple test cases. -- ``Test Object``: the targeted instance under benchmark testing. A typical example would be a particular algorithm or system. +The concept definition of modules has been shown in the Architecture Section. In the following, we introduce the concepts of objects for easier understanding. + +- ``Benchmark``: standardized evaluation process recognized by the academic or industry. +- ``Benchmarking Job``: the serving instance for an individual benchmarking with ianvs, which takes charge of the lifetime management of all possible ianvs components. + - Besides components, a benchmarking job includes instances of a test environment, one or more test cases, a leaderboard, or a test report. + - Different test environments lead to different benchmarking jobs and leaderboards. A benchmarking job can include multiple test cases +- ``Test Object``: the targeted instance under benchmark testing. A typical example would be a particular algorithm or system. - ``Test Environment``: setups or configurations for benchmarking, typically excluding the test object. - It can include algorithm-wise and system-wise configurations. - It serves as the unique descriptor of a benchmarking job. Different test environments thus lead to different benchmarking jobs. -- ``Test Case``: the executable instance to evaluate the performance of the test object under a particular test environment. Thus, the test case is usually generated with a particular test environment and outputs testing results if executed. +- ``Test Case``: the executable instance to evaluate the performance of the test object under a particular test environment. Thus, the test case is usually generated with a particular test environment and outputs testing results if executed. - It is the atomic unit of a benchmark. That is, a benchmarking job can include quite a few test cases. -- ``Attribute (Attr.) of Test Case``: Attributes or descriptors of a test case, e.g., id, name, and time stamp. +- ``Attribute (Attr.) of Test Case``: Attributes or descriptors of a test case, e.g., id, name, and time stamp. - ``Algorithm Paradigm``: acknowledged AI process which usually includes quite a few modules that can be implemented with replaceable algorithms, e.g., federated learning which includes modules of local train and global aggregation. - ``Algorithm Module``: the component of the algorithm paradigm, e.g., the global aggregation module of the federated learning paradigm. -- ``Leaderboard``: the ranking of the test object under a specific test environment. - - The local node holds the local leaderboard for private usage. - - The global leaderboard is shared (e.g., via GitHub) by acknowledge organization. -- ``Test Report``: the manuscript recording how the testing is conducted. - - - - - +- ``Leaderboard``: the ranking of the test object under a specific test environment. + - The local node holds the local leaderboard for private usage. + - The global leaderboard is shared (e.g., via GitHub) by the acknowledged organization. +- ``Test Report``: the manuscript recording how the testing is conducted. diff --git a/docs/index.rst b/docs/index.rst index ce540997..1b28a4c9 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -14,8 +14,8 @@ Ianvs also reveals best practices for developers and end users with presentation The scope of Ianvs is mainly two folds. -First, Ianvs aims to provide end-to-end benchmark toolkits across devices, edge nodes and cloud nodes based on typical distributed-synergy AI paradigms and applications. -- Tools to manage test environment. For example, it would be necessary to support the CRUD (Create, Read, Update and Delete) actions in test environments. Elements of such test environments include algorithm-wise and system-wise configuration. +First, Ianvs aims to provide end-to-end benchmark toolkits across devices, edge nodes, and cloud nodes based on typical distributed-synergy AI paradigms and applications. +- Tools to manage test environment. For example, it would be necessary to support the CRUD (Create, Read, Update, and Delete) actions in test environments. Elements of such test environments include algorithm-wise and system-wise configuration. - Tools to control test cases. Typical examples include paradigm templates, simulation tools, and hyper-parameter-based assistant tools. - Tools to manage benchmark presentation, e.g., leaderboard and test report generation. diff --git a/docs/proposals/scenarios/industrial-defect-detection/pcb-aoi.md b/docs/proposals/scenarios/industrial-defect-detection/pcb-aoi.md index 1fba247b..e74427a7 100644 --- a/docs/proposals/scenarios/industrial-defect-detection/pcb-aoi.md +++ b/docs/proposals/scenarios/industrial-defect-detection/pcb-aoi.md @@ -1,25 +1,28 @@ -# Industrial defect detection: the PCB-AoI dataset +# Industrial defect detection: the PCB-AoI dataset Download link: [Kaggle](https://www.kaggle.com/datasets/kubeedgeianvs/pcb-aoi), [Huawei OBS](https://kubeedge.obs.cn-north-1.myhuaweicloud.com:443/ianvs/pcb-aoi/dataset.zip) ## Authors + - China Telcom Research Institute: Dongdong Li, Dan Liu, Yun Shen, Yaqi Song - Raisecom Technology Co.,ltd.: Liangliang Luo ## Background -Surface-mount technology (SMT) is a technology that automates electronic circuits production in which components are mounted or placed onto the surface of printed circuit boards. Solder paste printing (SPP) is the most delicate stage in SMT. It prints solder paste on the pads of an electronic circuit panel. Thus, SPP is followed by a solder paste inspection (SPI) stage to detect defects. SPI scans the printed circuit board for missing/less paste, bridging between pads, miss alignments, and so forth. Boards with anomaly must be detected, and boards in good condition should not be disposed of. Thus SPI requires high precision and a high recall. -The PCB-AoI dataset is a part of the open-source distributed synergy AI benchmarking project KubeEdge-Ianvs. Ianvs is honored to be the ``First`` site that this dataset is released and the Ianvs working group put it on Kaggle as [The PCB-AoI public dataset](https://www.kaggle.com/datasets/kubeedgeianvs/pcb-aoi). It is released by KubeEdge SIG AI members from China Telecom and Raisecom Technology. +Surface-mount technology (SMT) is a technology that automates electronic circuits production in which components are mounted or placed onto the surface of printed circuit boards. Solder paste printing (SPP) is the most delicate stage in SMT. It prints solder paste on the pads of an electronic circuit panel. Thus, SPP is followed by a solder paste inspection (SPI) stage to detect defects. SPI scans the printed circuit board for missing/less paste, bridging between pads, miss alignments, and so forth. Boards with anomaly must be detected, and boards in good condition should not be disposed of. Thus SPI requires high precision and a high recall. -Below shows two example figures in the dataset. +The PCB-AoI dataset is a part of the open-source distributed synergy AI benchmarking project KubeEdge-Ianvs. Ianvs is honored to be the ``First`` site that this dataset is released and the Ianvs working group put it on Kaggle as [The PCB-AoI public dataset](https://www.kaggle.com/datasets/kubeedgeianvs/pcb-aoi). It is released by KubeEdge SIG AI members from China Telecom and Raisecom Technology. -![](images/PCB-AoI_example.png) +Below shows two example figures in the dataset. +![](images/PCB-AoI_example.png) ## Data Explorer + In this dataset, more than 230 boards are collected and the number of images is enhanced to more than 1200. Detailedly, the dataset include two parts, i.e., the train and the test set. The train set includes 173 boards while the test set includes 60 boards. That is, the train-test ratio is around 3:1 in terms of PCB boards. Data augmentation is conducted, boosting the train-test ratio to 1211:60 (about 20:1) in term of images. Both directories of train_data and test_data include the ``index`` file which recodes the mapping between the raw images and the label of annotation. -The directories of this dataset is as follows: +The directories of this dataset is as follows: + ``` ├── PCB-AoI Dataset │   ├── train_data @@ -37,6 +40,7 @@ The directories of this dataset is as follows: ``` The following is part of `index.txt`: + ```shell ./JPEGImages/20161019-SPI-AOI-1.jpeg ./Annotations/20161019-SPI-AOI-1.xml ./JPEGImages/20161020-SPI-AOI-5.jpeg ./Annotations/20161020-SPI-AOI-5.xml diff --git a/docs/proposals/simulation/simulation.md b/docs/proposals/simulation/simulation.md index 4c95a9d2..46107a14 100644 --- a/docs/proposals/simulation/simulation.md +++ b/docs/proposals/simulation/simulation.md @@ -1,6 +1,6 @@ # Benchmarks for Edge-cloud Collaborative Lifelong Learning -Artificial intelligence technology has served us in all aspects of life, especially in image, video, voice, recommendation system, etc. It has brought breakthrough results. AI Benchmark is designed to measure the performance and efficacy of AI models. There are already many authoritative AI Benchmark systems in the traditional AI field. As an emerging direction of edge AI, the corresponding Benchmark development is not perfect, and there are generally problems such as insufficient paradigm coverage, inactive communities, no edge/cloud side involved, and few test samples. +Artificial intelligence technology has served us in all aspects of life, especially in image, video, voice, recommendation systems, etc. It has brought breakthrough results. AI Benchmark is designed to measure the performance and efficacy of AI models. There are already many authoritative AI Benchmark systems in the traditional AI field. As an emerging direction of edge AI, the corresponding Benchmark development is not perfect, and there are generally problems such as insufficient paradigm coverage, inactive communities, no edge/cloud side involved, and few test samples. The specific research results of AI Benchmark for cloud-side-device collaboration can be found in this [document(in chinese)](https://github.com/iszhyang/AI-Benchmark-for-Cloud-Edge-Device). As a very promising community in edge AI Benchmarking, ianvs still has the following problems to be solved @@ -12,7 +12,7 @@ This proposal provides features in datasets, edge-cloud collaborative AI system ## Goals -The project is committed to building an edge-cloud collaborative AI Bechmark for edge-cloud collaborative lifelong detection in the Kubeedge open source community. +The project is committed to building an edge-cloud collaborative AI Benchmark for edge-cloud collaborative lifelong detection in the Kubeedge open source community. - Provide downloadable open source datasets and data processing scripts for the difficulty in obtaining datasets and supporting algorithms - Provides industrial-grade distributed collaborative system simulation for the problem that the repeated deployment of edge-cloud collaboration is too heavy @@ -99,7 +99,9 @@ We will provide the cityscapes dataset and corresponding methods for ianvs. The ### Industrial Distributed Collaborative System Simulation Provide ianvs with the feature of industrial distributed system simulation. + #### Framework + ![img](images/simulation_framework.jpg) In the framwork of ianvs, the `simulation controller` is the core module of system simulation. The `simulation controller` has been supplemented, which build and deploy local edge-cloud simulation environment with K8s. @@ -107,6 +109,7 @@ In the framwork of ianvs, the `simulation controller` is the core module of syst ![img](images/simulation_controller.jpg) The models in `simulation controller` are as follows: + - The `Simulation System Administrator` is used to 1. parse the system config(simulation) 2. check the host enviroment, e.g. check if the host has installed docker, kind, and whether memory > 4GB @@ -120,19 +123,24 @@ The models in `simulation controller` are as follows: 4. list-watch the results of `simulation job` in K8s #### Flowchart + ![img](images/simulation_dataflow.jpg) The `System Config` denotes the system config of the current benchmarkingjob, such as the `cloud number` and `edge number`. In the flowchart diagram above, the expected flow is as follows: + 1. start the benchmarkingjob 2. check whether to start emulation based on `system config` 3. `Simulation System Administrator` check environment and parse system config + - check environment: e.g. check if the host has installed docker, kind, and whether the memory > 4GB - parse simulation config + 4. `Simulation System Administrator` build the environment and create needed moudles. - build the environment: deploying a cluster including K8s and GM (global manager) locally through [all-in-one scripts of sedna](https://github.com/kubeedge/sedna/blob/527c574a60d0ae87b0436f9a8b38cf84fb6dab21/docs/setup/all-in-one.md) - create and deploy the `simulation job controller` + 5. `Simulation Job Administrator` generate and deploy simulation jobs. - build the docker images of algorithms to be tested - generate the YAML of simulation job base on `testenv.yaml` and `algorithm.yaml` @@ -164,4 +172,4 @@ The roadmap would be as follows 1. (0901-0904) Start implementing the preliminary framework through code. 2. (0905-0911) Improve the code and verify the reliability. 3. (0912-0918) Prepare the final PR. -2. (0919-0930) Linkage integration with other projects +4. (0919-0930) Linkage integration with other projects diff --git a/docs/proposals/test-reports/testing-incremental-learning-in-industrial-defect-detection-with-pcb-aoi.md b/docs/proposals/test-reports/testing-incremental-learning-in-industrial-defect-detection-with-pcb-aoi.md index c1444283..209346fc 100644 --- a/docs/proposals/test-reports/testing-incremental-learning-in-industrial-defect-detection-with-pcb-aoi.md +++ b/docs/proposals/test-reports/testing-incremental-learning-in-industrial-defect-detection-with-pcb-aoi.md @@ -1,26 +1,27 @@ # Testing incremental learning in industrial defect detection -## About Industrial Defect Detection +## About Industrial Defect Detection -In recent years, the manufacturing process is moving towards a higher degree of automation and improved manufacturing efficiency. During this development, smart manufacturing increasingly employs computing technologies, for example, with a higher degree of automation, there is also a higher risk in product defects; thus, a number of machine learning models have been developed to detect defectives in the manufacturing process. +In recent years, the manufacturing process is moving towards a higher degree of automation and improved manufacturing efficiency. During this development, smart manufacturing increasingly employs computing technologies, for example, with a higher degree of automation, there is also a higher risk of product defects; thus, a number of machine learning models have been developed to detect defectives in the manufacturing process. -Defects are an unwanted thing in manufacturing industry. There are many types of defect in manufacturing like blow holes, pinholes, burr, shrinkage defects, mould material defects, pouring metal defects, metallurgical defects, etc. For removing this defective product all industry have their defect detection department. But the main problem is this inspection process is carried out manually. It is a very time-consuming process and due to human accuracy, this is not 100\% accurate. This can because of the rejection of the whole order. So it creates a big loss in the company. +Defects are an unwanted thing in the manufacturing industry. There are many types of defects in manufacturing like blow holes, pinholes, burr, shrinkage defects, mould material defects, pouring metal defects, metallurgical defects, etc. For removing this defective product all industry have their defect detection department. But the main problem is this inspection process is carried out manually. It is a very time-consuming process and due to human accuracy, this is not 100\% accurate. This can be because of the rejection of the whole order. So it creates a big loss for the company. ## About Dataset -The printed circuit board (PCB) industry is not different. Surface-mount technology (SMT) is a technology that automates PCB production in which components are mounted or placed onto the surface of printed circuit boards. Solder paste printing (SPP) is the most delicate stage in SMT. It prints solder paste on the pads of an electronic circuit panel. Thus, SPP is followed by a solder paste inspection (SPI) stage to detect defects. SPI scans the printed circuit board for missing/less paste, bridging between pads, miss alignments, and so forth. Boards with anomaly must be detected, and boards in good condition should not be disposed of. Thus SPI requires high precision and a high recall. +The printed circuit board (PCB) industry is not different. Surface-mount technology (SMT) is a technology that automates PCB production in which components are mounted or placed onto the surface of printed circuit boards. Solder paste printing (SPP) is the most delicate stage in SMT. It prints solder paste on the pads of an electronic circuit panel. Thus, SPP is followed by a solder paste inspection (SPI) stage to detect defects. SPI scans the printed circuit board for missing/less paste, bridging between pads, miss alignments, and so forth. Boards with anomaly must be detected, and boards in good condition should not be disposed of. Thus SPI requires high precision and a high recall. -As an example in this document, we are using [the PCB-AoI dataset](https://www.kaggle.com/datasets/kubeedgeianvs/pcb-aoi) released by KubeEdge SIG AI members on Kaggle. See [this link](../scenarios/industrial-defect-detection/pcb-aoi.md) for more information of this dataset. Below also shows two example figures in the dataset. +As an example in this document, we are using [the PCB-AoI dataset](https://www.kaggle.com/datasets/kubeedgeianvs/pcb-aoi) released by KubeEdge SIG AI members on Kaggle. See [this link](../scenarios/industrial-defect-detection/pcb-aoi.md) for more information on this dataset. Below also shows two example figures in the dataset. ![](images/PCB-AoI_example.png) - ## About Incremental Learning -Traditionally, the data is collected manually and periodically retrained on the cloud to improve the model effect. However, data is continuously generated on the edge side. Traditional method wastes a lot of human resources, and the model update frequency is slow. + +Traditionally, the data is collected manually and periodically retrained on the cloud to improve the model effect. However, data is continuously generated on the edge side. The traditional method wastes a lot of human resources, and the model update frequency is slow. Incremental learning allows users to continuously monitor the newly generated data and by configuring some triggering rules to determine whether to start training, evaluation, and deployment automatically, and continuously improve the model performance. Its goals include: + * Automatically retrains, evaluates, and updates models based on the data generated at the edge. * Support time trigger, sample size trigger, and precision-based trigger. * Support manual triggering of training, evaluation, and model update. @@ -30,7 +31,7 @@ This report is testing the basic incremental algorithm based on FPN and interest ## Benchmark Setting -Key settings of the test environment to incremental learning are as follows: +Key settings of the test environment for incremental learning are as follows: ``` yaml # testenv.yaml @@ -95,7 +96,7 @@ algorithm: modules: # type of algorithm module; string type; # currently the options of value are as follows: - # 1> "basemodel": contains important interfaces such as train¡¢ eval¡¢ predict and more; required module; + # 1> "basemodel": contains important interfaces such as train�� eval�� predict and more; required module; - type: "basemodel" # name of python module; string type; # example: basemodel.py has BaseModel module that the alias is "FPN" for this benchmarking; diff --git a/docs/proposals/test-reports/testing-single-task-learning-in-industrial-defect-detection-with-pcb-aoi.md b/docs/proposals/test-reports/testing-single-task-learning-in-industrial-defect-detection-with-pcb-aoi.md index 6184324e..1604e814 100644 --- a/docs/proposals/test-reports/testing-single-task-learning-in-industrial-defect-detection-with-pcb-aoi.md +++ b/docs/proposals/test-reports/testing-single-task-learning-in-industrial-defect-detection-with-pcb-aoi.md @@ -1,27 +1,29 @@ # Testing single task learning in industrial defect detection -## About Industrial Defect Detection +## About Industrial Defect Detection -In recent years, the manufacturing process is moving towards a higher degree of automation and improved manufacturing efficiency. During this development, smart manufacturing increasingly employs computing technologies, for example, with a higher degree of automation, there is also a higher risk in product defects; thus, a number of machine learning models have been developed to detect defectives in the manufacturing process. - -Defects are an unwanted thing in manufacturing industry. There are many types of defect in manufacturing like blow holes, pinholes, burr, shrinkage defects, mould material defects, pouring metal defects, metallurgical defects, etc. For removing this defective product all industry have their defect detection department. But the main problem is this inspection process is carried out manually. It is a very time-consuming process and due to human accuracy, this is not 100\% accurate. This can because of the rejection of the whole order. So it creates a big loss in the company. +In recent years, the manufacturing process is moving towards a higher degree of automation and improved manufacturing efficiency. During this development, smart manufacturing increasingly employs computing technologies, for example, with a higher degree of automation, there is also a higher risk of product defects; thus, a number of machine learning models have been developed to detect defectives in the manufacturing process. +Defects are an unwanted thing in the manufacturing industry. There are many types of defects in manufacturing like blow holes, pinholes, burr, shrinkage defects, mould material defects, pouring metal defects, metallurgical defects, etc. For removing this defective product all industry have their defect detection department. But the main problem is this inspection process is carried out manually. It is a very time-consuming process and due to human accuracy, this is not 100\% accurate. This can be because of the rejection of the whole order. So it creates a big loss for the company. ## About Dataset -The printed circuit board (PCB) industry is not different. Surface-mount technology (SMT) is a technology that automates PCB production in which components are mounted or placed onto the surface of printed circuit boards. Solder paste printing (SPP) is the most delicate stage in SMT. It prints solder paste on the pads of an electronic circuit panel. Thus, SPP is followed by a solder paste inspection (SPI) stage to detect defects. SPI scans the printed circuit board for missing/less paste, bridging between pads, miss alignments, and so forth. Boards with anomaly must be detected, and boards in good condition should not be disposed of. Thus SPI requires high precision and a high recall. +The printed circuit board (PCB) industry is not different. Surface-mount technology (SMT) is a technology that automates PCB production in which components are mounted or placed onto the surface of printed circuit boards. Solder paste printing (SPP) is the most delicate stage in SMT. It prints solder paste on the pads of an electronic circuit panel. Thus, SPP is followed by a solder paste inspection (SPI) stage to detect defects. SPI scans the printed circuit board for missing/less paste, bridging between pads, miss alignments, and so forth. Boards with anomaly must be detected, and boards in good condition should not be disposed of. Thus SPI requires high precision and a high recall. -As an example in this document, we are using [the PCB-AoI dataset](https://www.kaggle.com/datasets/kubeedgeianvs/pcb-aoi) released by KubeEdge SIG AI members on Kaggle. See [this link](../scenarios/industrial-defect-detection/pcb-aoi.md) for more information of this dataset. Below also shows two example figures in the dataset. +As an example in this document, we are using [the PCB-AoI dataset](https://www.kaggle.com/datasets/kubeedgeianvs/pcb-aoi) released by KubeEdge SIG AI members on Kaggle. See [this link](../scenarios/industrial-defect-detection/pcb-aoi.md) for more information on this dataset. Below also shows two example figures in the dataset. ![](images/PCB-AoI_example.png) ## About Single Task Learning -Single task learning is a traditional learning pooling all data together to train a single model. It typically includes a specialist model laser-focused on a single task and requires large amounts of task-specific labeled data, which is not always available on the early stage of a distributed synergy AI project. -This report is testing the single task learning algorithm based on ``FPN_TensorFlow``. It is a tensorflow re-implementation of Feature Pyramid Networks for Object Detection, which is based on Faster-RCNN. More detailedly, feature pyramids are a basic component in recognition systems for detecting objects at different scales. But recent deep learning object detectors have avoided pyramid representations, in part because they are compute and memory intensive. Researchers have exploited the inherent multi-scale, pyramidal hierarchy of deep convolutional networks to construct feature pyramids with marginal extra cost. A top-down architecture with lateral connections is developed for building high-level semantic feature maps at all scales. The architecture, called a Feature Pyramid Network (FPN), shows significant improvement as a generic feature extractor in several applications. Using FPN in a basic Faster R-CNN system, the method achieves state-of-the-art single-model results on the COCO detection benchmark without bells and whistles, surpassing all existing single-task entries including those from the COCO 2016 challenge winners. In addition, FPN can run at 5 FPS on a GPU and thus is a practical and accurate solution to multi-scale object detection. The ``FPN_TensorFlow`` is also open sourced and completed by YangXue and YangJirui. For those interested in details of ``FPN_TensorFlow``, an example implementation is available [here](https://github.com/DetectionTeamUCAS/FPN_Tensorflow) and is extended with the Ianvs algorithm inferface [here](https://github.com/ECIL-EdgeAI/FPN_Tensorflow). Interested readers can refer to [the FPN](../algorithms/single-task-learning/fpn.md) for more details. +Single task learning is a traditional learning pooling all data together to train a single model. It typically includes a specialist model laser-focused on a single task and requires large amounts of task-specific labeled data, which is not always available in the early stage of a distributed synergy AI project. + +This report is testing the single task learning algorithm based on ``FPN_TensorFlow``. It is a Tensorflow re-implementation of Feature Pyramid Networks for Object Detection, which is based on Faster-RCNN. More detailedly, feature pyramids are a basic component in recognition systems for detecting objects at different scales. But recent deep learning object detectors have avoided pyramid representations, in part because they are compute and memory intensive. Researchers have exploited the inherent multi-scale, pyramidal hierarchy of deep convolutional networks to construct feature pyramids with marginal extra cost. A top-down architecture with lateral connections is developed for building high-level semantic feature maps at all scales. The architecture, called a Feature Pyramid Network (FPN), shows significant improvement as a generic feature extractor in several applications. Using FPN in a basic Faster R-CNN system, the method achieves state-of-the-art single-model results on the COCO detection benchmark without bells and whistles, surpassing all existing single-task entries including those from the COCO 2016 challenge winners. In addition, FPN can run at 5 FPS on a GPU and thus is a practical and accurate solution to multi-scale object detection. The ``FPN_TensorFlow`` is also open sourced and completed by YangXue and YangJirui. For those interested in details of ``FPN_TensorFlow``, an example implementation is available [here](https://github.com/DetectionTeamUCAS/FPN_Tensorflow) and is extended with the Ianvs algorithm interface [here](https://github.com/ECIL-EdgeAI/FPN_Tensorflow). Interested readers can refer to [the FPN](../algorithms/single-task-learning/fpn.md) for more details. ## Benchmark Setting -Key settings of the test environment to single task learning are as follows: + +Key settings of the test environment to single task learning are as follows: + ``` yaml # testenv.yaml testenv: @@ -86,5 +88,4 @@ algorithm: ## Benchmark Result We release the -leaderboard [here](../leaderboards/leaderboard-in-industrial-defect-detection-of-PCB-AoI/leaderboard-of-single-task-learning.md) -. \ No newline at end of file +leaderboard [here](../leaderboards/leaderboard-in-industrial-defect-detection-of-PCB-AoI/leaderboard-of-single-task-learning.md).