diff --git a/README.md b/README.md index 0243d63..3d25c2c 100644 --- a/README.md +++ b/README.md @@ -2,12 +2,13 @@ [![Linter and Builder 🚀](https://github.com/SkywardAI/chat-backend/actions/workflows/linter-and-builder.yaml/badge.svg)](https://github.com/SkywardAI/chat-backend/actions/workflows/linter-and-builder.yaml) [![Release Drafter 🚀](https://github.com/SkywardAI/chat-backend/actions/workflows/release-drafter.yml/badge.svg)](https://github.com/SkywardAI/chat-backend/actions/workflows/release-drafter.yml) [![Releasing Image 🚀](https://github.com/SkywardAI/kirin/actions/workflows/release-image.yaml/badge.svg)](https://github.com/SkywardAI/kirin/actions/workflows/release-image.yaml) [![CodeQL](https://github.com/SkywardAI/kirin/actions/workflows/github-code-scanning/codeql/badge.svg)](https://github.com/SkywardAI/kirin/actions/workflows/github-code-scanning/codeql) + # Architecture ![](./imgs/SkywardAI(Mind%20Map)%20-%20APIs%20aggregator.svg) -# Phi3-minimal-4k-gguf on 8 CPUs +# Doing inference on 8 CPUs https://github.com/SkywardAI/kirin/assets/8053949/143b016c-2264-4644-a7ae-304c908fbedf @@ -51,11 +52,17 @@ The above-listed technologies are just the main ones. There are other technologi See [Requirements.md](./docs/Requirements.md) + ## Quick setup See [quick start](https://skywardai.github.io/skywardai.io/docs/quick-start.html) -## Deployment + +## Build and Deployment + + + +[![Build and Deployment](https://img.youtube.com/vi/63OxSmcBkhI/1.jpg)](https://youtu.be/63OxSmcBkhI?si=G82BOtcwRvQE7dLU) See [deployment](https://skywardai.github.io/skywardai.io/docs/development/build_and_run.html). And if you are interested in docker-in-docker development, see [Development.md](./docs/Development.md) diff --git a/backend/README.md b/backend/README.md deleted file mode 100644 index a2815f7..0000000 --- a/backend/README.md +++ /dev/null @@ -1,169 +0,0 @@ -

Kirin Platform

- -This repo consists of: - -* Configure the [TOML](https://toml.io/en/) file for testing and linting. -* Create the backend server with FastAPI and Uvicorn, -* Connect the backend app with PostgreSQL server via the asynchronous SQLAlchemy with AsyncPG driver. -* Set up Alembic for the auto-generation of database migrations. -* Create `Account` (domain) class as our database entity `account`. -* Create data validation for our database entity with [Pydantic](https://pydantic-docs.helpmanual.io/). -* Create formatters for `json` convention and `datetime`. -* Create authorization with JWT Token generator. -* Create hashing function generator and password generator. -* Create custom error/exception handlers for both source code and HTTP. -* Create dependency injections for utilizing our asynchronous session and C. R. U. D. repository class. -* Create account specific routes: `signup`, `signin`, `get all`, `get by id`, `update by id`, and `delete by id`. - -**P. S. This README will walk you through the local development setup. If you prefer to run your backend app with docker, see the main README.** - ---- - -## Python, VEnv, & Requirements Installation - -**INFO**: All related to Python will be setup **IN** and **FROM** the `backend/` directory! - -* Step 1 $\rightarrow$ Open your project root directory and set up your Python via `PyEnv`: - - ```shell - pyenv install 3.11.0 - ``` - -* Step 2 $\rightarrow$ Create our virtual environment: - - ```shell - pyenv virtualenv 3.11.0 YOUR_VENV_NAME - ``` - -* Step 3 $\rightarrow$ Set the newly created virtual environment as your main Python interpreter in the root directory: - - ```shell - pyenv local YOUR_VENV_NAME - ``` - -* Step 4 $\rightarrow$ Install the initial project requirements with `pip3`: - - ```shell - pip3 install -r https://raw.githubusercontent.com/SkywardAI/containers/main/requirements.txt - ``` - ---- - -## SSHhhhttt 🤫 It's a Secret! - -All secret variables are configured in the `.env`, but in this case since it is listed in our `.gitignore` you should create one and save it in the root directory. - -The secret variables are accessed by 2 different files: - -1. `backend/src/config/settings/base.py` $\rightarrow$ The `BackendBaseSettings` class contains all the secret variables. This allows us to set the values dynamically through out the backend application ("dynamic" because we only need to change the value in `.env` file without breaking any code). -2. `docker-compose.yaml` $\rightarrow$ When we initialize the `backend_app` container, we register all the secret variables as well for our containerized backend application. - -## Database - -**INFO**: No postgres server, no connection! - -* Step 1 $\rightarrow$ Run your postgres server (if you use MacOS, I suggest the installation via [postgre14 homebrew](https://formulae.brew.sh/formula/postgresql@14)): - - ```shell - brew services start postgresql@14 - ``` - -* Step 2 $\rightarrow$ Go to your database editor choice and create a database for this application. - -* Step 3 $\rightarrow$ Register your database credentials respectively in our `.env` file in the root directory. - -* Step 4 (Optional) $\rightarrow$ To stop your postgres server, run: - ```shell - brew services stop postgres@14 - ``` - -### Alembic for Automigration - -**INFO**: Files generated by `Alembic` can be found in `backend/src/repository/migrations/**` (if you don't see the `versions/` directory, make one!) - -**INFO**: You need to import your domain classes in `backend/src/repository/base.py` for alembic's auto migration! - -* Step 1 $\rightarrow$ Create the initialization of your database migration: - ```shell - alembic revision --autogenerate -m "YOUR NOTES ABOUT THE DATABASE MIGRATION HERE" - ``` - -* Step 2 $\rightarrow$ Push the registered database objects to your database: - ```shell - alembic upgrade head - ``` - ---- - -## Pre-Commit - -**INFO**: Run **Step 1** every time you `git add` a file to identify any mistakes before `git commit`. Otherwise, you will re-write your perfect commit message again 👿🤬🤮 - -* Step 1 $\rightarrow$ Install the pre-commit hook: - ```shell - pre-commit - ``` - -* Step 2 $\rightarrow$ For good practive, let#s update the hooks: - ```shell - pre-commit autoupdate - ``` - ---- - -## TOML Configuration - -Check the `pyproject.toml` as the main configuration file for the following packages: - -* Project documentation -* Black -* Isort -* MyPy -* PyTest -* Converage - ---- - -## FastAPI Backend Server - -**INFO**: Make sure that you are in the `backend/` directory! - -* Stp 1 $\rightarrow$ Run the server: - ```shell - uvicorn src.main:backend_app --reload - ``` - -* Step 2 (Optional) $\rightarrow$ To stop the server click simultaneously `control` and `C` - ---- - -## Test with PyTest - -**INFO**: For running the test, make sure you are in the root directory and NOT in the `backend/` directory! - -**INFO**: The testing report is automatically generated and saved in `backend/coverage/**` - -* Step 1: Run PyTest: - - ```shell - pytest - ``` - ---- - -## Python Package Info Board - -So what are we actually installing? You can find the main packages in the below table. - -Server | Database | Test | Linter | Others ---|--|--|--|-- -[fastapi](https://fastapi.tiangolo.com/) | [sqlalchemy (version 2.0.0b3)](https://docs.sqlalchemy.org/en/20/orm/extensions/asyncio.html) | [pytest](https://docs.pytest.org/en/7.2.x/) | [black](https://pypi.org/project/black/) | [python-decouple](https://pypi.org/project/python-decouple/) -[uvicorn](https://www.uvicorn.org/) | [alembic](https://alembic.sqlalchemy.org/en/latest/) | [pytest-asyncio](https://pypi.org/project/pytest-asyncio/) | [isort](https://pypi.org/project/isort/) | [email-validator](https://pypi.org/project/email-validator/) -[pydantic](https://docs.pydantic.dev/usage/exporting_models/) | [asyncpg](https://magicstack.github.io/asyncpg/current/) | [pytest-cov](https://pypi.org/project/pytest-cov/) | [mypy](https://pypi.org/project/mypy/) | [python-dotenv](https://pypi.org/project/python-dotenv/) -[starlette](https://www.starlette.io/) | [greenlet](https://pypi.org/project/greenlet/) | [pytest-xdist](https://pypi.org/project/pytest-xdist/) | [pre-commit](https://pypi.org/project/pre-commit/) | [passlib](https://passlib.readthedocs.io/en/stable/) -[asgi_lifespan](https://pypi.org/project/asgi-lifespan/) | - | | [colorama](https://pypi.org/project/colorama/) | [loguru](https://github.com/Delgan/loguru) -[httpx](https://www.python-httpx.org/) | - | - | - | [python-jose](https://python-jose.readthedocs.io/en/latest/) -[trio](https://pypi.org/project/trio/) | - | - | - | [pathlib](https://pathlib.readthedocs.io/en/pep428/) - \- | - | - | - | [python-slugify](https://pypi.org/project/python-slugify/) - ---- diff --git a/backend/src/repository/rag/chat.py b/backend/src/repository/rag/chat.py index 5d296cc..1623e7a 100644 --- a/backend/src/repository/rag/chat.py +++ b/backend/src/repository/rag/chat.py @@ -271,7 +271,8 @@ async def inference( inference_helper.completion_url, headers={'Content-Type': 'application/json'}, json=data, - timeout=httpx.Timeout(timeout=5.0) + # We disable all timeout and trying to fix streaming randomly cutting off + timeout=httpx.Timeout(timeout=None) ) as response: response.raise_for_status() async for chunk in response.aiter_text():