docs: Add object detection docs (#61)

## Description This PR adds useObjectDetection to the computer vision section of the docs ### Type of change - [ ] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [x] Documentation update (improves or adds clarity to existing documentation) ### Tested on - [ ] iOS - [ ] Android ### Testing instructions  ### Screenshots  ### Related issues  ### Checklist - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have updated the documentation accordingly - [ ] My changes generate no new warnings ### Additional notes  --------- Co-authored-by: Mateusz Kopciński <[email protected]>
software-mansion · Dec 18, 2024 · bbf6aee · bbf6aee
1 parent 69802ee
commit bbf6aee
Show file tree

Hide file tree

Showing 2 changed files with 135 additions and 0 deletions.
diff --git a/docs/docs/computer_vision/_category_.json b/docs/docs/computer_vision/_category_.json
@@ -0,0 +1,8 @@
+{
+    "label": "Computer Vision",
+    "position": 3,
+    "link": {
+      "type": "generated-index"
+    }
+}
+
diff --git a/docs/docs/computer_vision/useObjectDetection.mdx b/docs/docs/computer_vision/useObjectDetection.mdx
@@ -0,0 +1,127 @@
+---
+title: useObjectDetection
+sidebar_position: 1
+---
+
+Object detection is a computer vision technique that identifies and locates objects within images or video. It’s commonly used in applications like image recognition, video surveillance or autonomous driving.
+`useObjectDetection` is a hook that allows you to seamlessly integrate object detection into your React Native applications.
+
+:::caution
+It is recommended to use models provided by us, which are available at our [Hugging Face repository](https://huggingface.co/software-mansion/react-native-executorch-ssdlite320-mobilenet-v3-large). You can also use [constants](https://github.com/software-mansion/react-native-executorch/blob/69802ee1ca161d9df00def1dabe014d36341cfa9/src/constants/modelUrls.ts#L28) shipped with our library.
+:::
+
+## Reference
+```jsx
+import { useObjectDetection, SSDLITE_320_MOBILENET_V3_LARGE } from 'react-native-executorch';
+
+function App() {
+  const ssdlite = useObjectDetection({
+    modelSource: SSDLITE_320_MOBILENET_V3_LARGE, // alternatively, you can use require(...)
+  });
+
+  ...
+  for (const detection of await ssdlite.forward("https://url-to-image.jpg")) {
+    console.log("Bounding box: ", detection.bbox);
+    console.log("Bounding label: ", detection.label);
+    console.log("Bounding score: ", detection.score);
+  }
+  ...
+}
+```
+
+<details>
+<summary>Type definitions</summary>
+
+```typescript
+interface Bbox {
+  x1: number;
+  x2: number;
+  y1: number;
+  y2: number;
+}
+
+interface Detection {
+  bbox: Bbox;
+  label: keyof typeof CocoLabel;
+  score: number;
+}
+
+interface ObjectDetectionModule {
+  error: string | null;
+  isReady: boolean;
+  isGenerating: boolean;
+  forward: (input: string) => Promise<Detection[]>;
+}
+```
+</details>
+
+### Arguments
+
+`modelSource`
+
+A string that specifies the path to the model file. You can download the model from our [HuggingFace repository](https://huggingface.co/software-mansion/react-native-executorch-ssdlite320-mobilenet-v3-large/tree/main). 
+For more information on that topic, you can check out the [Loading models](https://docs.swmansion.com/react-native-executorch/fundamentals/loading-models) page.
+
+### Returns
+
+The hook returns an object with the following properties:
+
+
+| **Field**            | **Type**                              | **Description**                                                                                                  |
+|-----------------------|---------------------------------------|------------------------------------------------------------------------------------------------------------------|
+| `forward`            | `(input: string) => Promise<Detection[]>` | A function that accepts an image (url, b64) and returns an array of `Detection` objects.                                   |
+| `error`              | <code>string &#124; null</code>       | Contains the error message if the model loading failed.    |
+| `isGenerating`  | `boolean`                            | Indicates whether the model is currently processing an inference.                                               |
+| `isReady`       | `boolean`                            | Indicates whether the model has successfully loaded and is ready for inference.                                 |
+
+
+## Running the model
+
+To run the model, you can use the `forward` method. It accepts one argument, which is the image. The image can be a remote URL, a local file URI, or a base64-encoded image. The function returns an array of `Detection` objects. Each object contains coordinates of the bounding box, the label of the detected object, and the confidence score. For more information, please refer to the reference or type definitions.
+
+## Detection object
+The detection object is specified as follows:
+```typescript
+interface Bbox {
+  x1: number;
+  y1: number;
+  x2: number;
+  y2: number;
+}
+
+interface Detection {
+  bbox: Bbox;
+  label: keyof typeof CocoLabels;
+  score: number;
+}
+```
+The `bbox` property contains information about the bounding box of detected objects. It is represented as two points: one at the bottom-left corner of the bounding box (`x1`, `y1`) and the other at the top-right corner (`x2`, `y2`).
+The `label` property contains the name of the detected object, which corresponds to one of the `CocoLabels`. The `score` represents the confidence score of the detected object.
+
+
+
+## Example
+```tsx
+import { useObjectDetection, SSDLITE_320_MOBILENET_V3_LARGE } from 'react-native-executorch';
+
+function App() {
+  const ssdlite = useObjectDetection({
+    modelSource: SSDLITE_320_MOBILENET_V3_LARGE,
+  });
+
+  const runModel = async () => {
+    const detections = await ssdlite.forward("https://url-to-image.jpg");
+    for (const detection of detections) {
+      console.log("Bounding box: ", detection.bbox);
+      console.log("Bounding label: ", detection.label);
+      console.log("Bounding score: ", detection.score);
+    }
+  }
+}
+```
+
+## Supported Models
+
+| Model                                                                                                           | Number of classes | Class list                                                                                                                                                                 |
+| --------------------------------------------------------------------------------------------------------------- | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [SSDLite320 MobileNetV3 Large](https://pytorch.org/vision/main/models/generated/torchvision.models.detection.ssdlite320_mobilenet_v3_large.html#torchvision.models.detection.SSDLite320_MobileNet_V3_Large_Weights) | 91              | [COCO](https://github.com/software-mansion/react-native-executorch/blob/69802ee1ca161d9df00def1dabe014d36341cfa9/src/types/object_detection.ts#L14) |