Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Signed-off-by: Jaehyun Nam <[email protected]>
- Loading branch information
1 parent
eba964a
commit b0ab5e2
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| name: ci-test-py | ||
| on: | ||
| push: | ||
| branches: [main] | ||
| pull_request: | ||
| branches: [main] | ||
|
|
||
| jobs: | ||
| py-pip-ai-sentryflow: | ||
| runs-on: ubuntu-20.04 | ||
| steps: | ||
| - uses: actions/checkout@v3 | ||
| - uses: actions/setup-python@v4 | ||
| with: | ||
| python-version: '3.11' | ||
| cache: 'pip' | ||
|
|
||
| - name: check Python pip3 requirements | ||
| run: | | ||
| pip install -r requirements.txt | ||
| working-directory: ai-engine | ||
|
|
||
| py-lint-ai-sentryflow: | ||
| runs-on: ubuntu-20.04 | ||
| steps: | ||
| - uses: actions/checkout@v3 | ||
| - uses: actions/setup-python@v4 | ||
| with: | ||
| python-version: '3.11' | ||
| cache: 'pip' | ||
|
|
||
| - name: Install dependencies | ||
| run: | | ||
| python -m pip install --upgrade pip | ||
| pip install -r requirements.txt | ||
| working-directory: ai-engine | ||
|
|
||
| - name: Lint with Ruff | ||
| run: | | ||
| pip install ruff | ||
| ruff --output-format=github . | ||
| continue-on-error: true | ||
| working-directory: ai-engine | ||
|
|
||
| py-pep8-ai-sentryflow: | ||
| runs-on: ubuntu-20.04 | ||
| steps: | ||
| - uses: actions/checkout@v3 | ||
| - name: 'Run PEP8' | ||
| uses: quentinguidee/pep8-action@v1 | ||
| with: | ||
| arguments: '--max-line-length=120' |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,3 +1 @@ | ||
| .DS_Store | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| .idea | ||
| .git | ||
| .gitignore | ||
| protobuf | ||
| Dockerfile | ||
| __pycache__/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| .idea/ | ||
| __pycache__/ | ||
| protobuf/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| # Dockerfile | ||
| FROM ubuntu:latest | ||
|
|
||
| RUN apt-get update && apt-get -y install python3 python3-pip wget git | ||
|
|
||
| RUN git clone https://github.com/isu-kim/stringlifier.git | ||
| WORKDIR ./stringlifier | ||
| RUN pip install . | ||
|
|
||
| RUN mkdir /app | ||
| WORKDIR /app | ||
| COPY /ai-engine . | ||
|
|
||
| # Build protobuf for Python | ||
| RUN pip install grpcio grpcio-tools | ||
| RUN mkdir protobuf/ | ||
| COPY /protobuf ./protobuf | ||
|
|
||
| # Due to python import bugs, we have to compile protoc using this command | ||
| # Refer to https://github.com/protocolbuffers/protobuf/issues/1491#issuecomment-261621112 for more information on this | ||
| RUN python3 -m grpc_tools.protoc --python_out=. --pyi_out=. --grpc_python_out=. -I=. protobuf/sentryflow_metrics.proto | ||
|
|
||
| WORKDIR /app | ||
| RUN pip install -r requirements.txt | ||
|
|
||
| CMD ["python3", "ai-engine.py"] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| IMAGE_NAME = 5gsec/sentryflow-ai-engine | ||
| TAG = v0.1 | ||
|
|
||
| .PHONY: build | ||
|
|
||
| build: | ||
| docker build -t $(IMAGE_NAME):$(TAG) -f ./Dockerfile ../ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,94 @@ | ||
| import os | ||
| import grpc | ||
|
|
||
| from stringlifier.api import Stringlifier | ||
| from concurrent import futures | ||
|
|
||
| from protobuf import sentryflow_metrics_pb_grpc | ||
| from protobuf import sentryflow_metrics_pb | ||
|
|
||
|
|
||
| class HandlerServer: | ||
| """ | ||
| Class for gRPC Servers | ||
| """ | ||
| def __init__(self): | ||
| try: | ||
| self.listen_addr = os.environ["AI_ENGINE_ADDRESS"] | ||
| except KeyError: | ||
| self.listen_addr = "0.0.0.0:5000" | ||
|
|
||
| self.server = None | ||
| self.grpc_servers = list() | ||
|
|
||
| def init_grpc_servers(self): | ||
| """ | ||
| init_grpc_servers method that initializes and registers gRPC servers | ||
| :return: None | ||
| """ | ||
| self.server = grpc.server(futures.ThreadPoolExecutor(max_workers=10)) | ||
| self.grpc_servers.append(APIClassificationServer()) # @todo: make this configurable | ||
|
|
||
| grpc_server: GRPCServer | ||
| for grpc_server in self.grpc_servers: | ||
| grpc_server.register(self.server) | ||
|
|
||
| def serve(self): | ||
| """ | ||
| serve method that starts serving gRPC servers, this is blocking function. | ||
| :return: None | ||
| """ | ||
| self.server.add_insecure_port(self.listen_addr) | ||
|
|
||
| print("[INFO] Starting to serve on {}".format(self.listen_addr)) | ||
| self.server.start() | ||
| self.server.wait_for_termination() | ||
|
|
||
|
|
||
| class GRPCServer: | ||
| """ | ||
| Abstract class for an individual gRPC Server | ||
| """ | ||
| def register(self, server): | ||
| """ | ||
| register method that registers gRPC service to target server | ||
| :param server: The server | ||
| :return: None | ||
| """ | ||
| pass | ||
|
|
||
|
|
||
| class APIClassificationServer(sentryflow_metrics_pb2_grpc.SentryFlowMetricsServicer, GRPCServer): | ||
| """ | ||
| Class for API Classification Server using Stringlifier | ||
| """ | ||
|
|
||
| def __init__(self): | ||
| self.stringlifier = Stringlifier() | ||
| print("[Init] Successfully initialized APIClassificationServer") | ||
|
|
||
| def register(self, server): | ||
| sentryflow_metrics_pb2_grpc.add_SentryFlowMetricsServicer_to_server(self, server) | ||
|
|
||
| def GetAPIClassification(self, request_iterator, context): | ||
| """ | ||
| GetAPIClassification method that runs multiple API ML Classification at once | ||
| :param request_iterator: The requests | ||
| :param context: The context | ||
| :return: The results | ||
| """ | ||
|
|
||
| for req in request_iterator: | ||
| paths = req.paths | ||
| ml_results = self.stringlifier(paths) | ||
| print("{} -> {}".format(paths, ml_results)) | ||
|
|
||
| results = [sentryflow_metrics_pb2.APIClassificationSingleResponse(merged=ml_result, fields=[]) for ml_result | ||
| in ml_results] | ||
| yield sentryflow_metrics_pb2.APIClassificationResponse(response=results) | ||
|
|
||
|
|
||
| if __name__ == '__main__': | ||
| hs = HandlerServer() | ||
| hs.init_grpc_servers() | ||
| hs.serve() | ||
Binary file not shown.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,2 +1,3 @@ | ||
| .idea/ | ||
| *.pb.go | ||
| *.tar |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.