This web application project provides a friendly to use interface in which a user can upload a csv file he wishes to train, and another csv file with possible anomalies. The uploaded data is processed by an anomaly detection algorithm, and the answers that are sent back from the algorithm are presented to the user in a few informative graphs and by spans that specify in which lines the anomalies occurred. The entire project is written in JavaScript and is divided into 2 main sections:
- The client side
- A RESTful API handling the logic by interacting with a server that detects anomalies.
To get a local copy up and running follow these steps.
- npm
npm install npm@latest -g
-
Clone the repo
git clone https://github.com/or-gottman/AnomalyDetectionWebApp
-
Install NPM packages - Make sure that all libraries (mentioned in the package.json) are installed.
npm install
-
Client side website - Once cloned in order to initiate the website please enter: npm start
git clone https://github.com/giladaube/WebClientAnomaliesDetector
-
Algorithm Server - need to install npm as mentioned before.
git clone https://github.com/y-jacobs/algoServer
Firstly, the user uploads a csv file to be trained and also specifies which type of algorithm (hybrid/regression) needs to be used by the algorithm server. After the train file is uploaded the user is presented with a notice that a new model has been created, and the time of creation is also specified. At this point, the csv file's data is presented visually in a graph to the user. The user is able to select a feature, in a drop-down menu, and the relevant data will be presented on the graph - in which the Y axis contains values of the selected feature, and the X axis is time. The user can then upload yet another file containing possible anomalies. The data is processed by the API and passed to the algorithm server for anomaly detection. When the detection is done the algorithm server sends back a message that contains a list of spans for each 2 correlated features. The spans are in the format of [start,end] where start is the first line and end is the line after the last line in which the anomalies were found. This list of lines is passed to the client side to be presented to the user. Moreover, the information concerning the anomalies is presented graphically in the website. At this point, the user is able to select a feature in the drop-down menu, and the graph will get updated - the X axis will represent the selected feature's values, and the Y axis will represent values of the selected feature's most correlated feature. The points which are considered anomalies will be colored in red, and the user has the option to show or hide them and the regular points.
- POST api/model - This path expects a query parameter containing the model type and the data in the body. The data sent in the body is trained by the algorithm server. After that a new model is created that contains a unique ID for this model, the exact time the model was created, the type of algorithm that the model was trained with, and the features that were trained.
- GET api/model - This path expects a query parameter containing the unique model ID. It sends back the trained model saved in the database.
- DELETE api/model - This path expects a query parameter containing the unique model ID. It deletes the model from the database.
- GET api/models - This path sends all the models that were trained and saved in the database.
- POST api/anomaly - This path expects a query parameter containing the unique model ID and the data in the body. The data given in the body is sent to the algorithm server for anomaly detection. It sends back the spans calculated by the algorithm.
The website is built with react. Each component has its own responsibility and thus we've kept OOP principles throughout the project. The site is built in way that with each render of the site the model's status updates (if a model was trained previously, its status would be pending and after the next rendering of the site it will change to ready). In addition, each request from the user is translated into an API request for the API server. The client processes the input csv files given by the user (a train file or a file with possible anomalies) and parses them into useable data for the API and sends the data to the server. Once the results from the API server return the client presents them to the user graphically. If any anomalies were found the user can observe them in the graph (they are colored in red) and the lines in the table that are lines that were stated in the spans are colored in red as well. Moreover, any request or update is followed by a notification in order to keep the user informed of any change in the data.
This server detects anomalies with two different algorithms. The server receives CSV files both for the learning part and the detection part. The algo server contains a few parts: Server- receives and sends data from and to clients that has connected with the program. Algo- the algorithms find, based on the ‘train’ CSV, the correlation between the CSV features and calculate the anomalies points bases on the ‘test’ CSV. The data is received and the ‘learn normal’ finds the correlated features in the CSV, each correlated feature is saved. The ‘detect’ finds the points in the ‘train’ CSV that are not suited for the correlation that was found during the ‘learn’. The anomaly points are saved, and the client can ask for both list of the points or span of them. The span is calculated that points that are closed together will be featured as one. In this project there are two algorithms that find the correlation and find anomalies. The first algorithm is a line regression algorithm and the second is a Hybrid algorithm. Both algorithms can be used. The data that is received is in CSV format, the Parser extracts the relevant data from the file and saves it for further use by the algorithms.
https://github.com/or-gottman/AnomalyDetectionWebApp/blob/master/DEMO.mp4