model.export to TFLite with int8 doesn't yield a fully int8 quantized model

[nicklasb]

Search before asking

• I have searched the Ultralytics YOLO issues and found no similar bug report.

Ultralytics YOLO Component

Export

Bug

In my view, this, when run on a nano model, should yield a fully quantized int8 model regardless of input.
model.export(format='tflite', int8=True)

Instead, using the exported model TFLite Micro will raise the "Hybrid models are not supported on TFLite Micro.", which is raised then the input type is different from the filter type.

Note that this was not an issue on the YoloV5 export. I am not sure how the "representative" dataset is used now, but even though the data parameter is possible to set, but is not in the docs?

Environment

Ultralytics YOLOv8.2.90 🚀 Python-3.12.6 torch-2.4.1+cpu CPU (AMD Ryzen 7 7735U with Radeon Graphics)
Setup complete ✅ (16 CPUs, 31.2 GB RAM, 397.0/952.4 GB disk)

OS Windows-11-10.0.22631-SP0
Environment Windows
Python 3.12.6
Install git
RAM 31.25 GB
CPU AMD Ryzen 7 7735U with Radeon Graphics
CUDA None

numpy ✅ 1.26.4<2.0.0,>=1.23.0
matplotlib ✅ 3.9.2>=3.3.0
opencv-python ✅ 4.10.0.84>=4.6.0
pillow ✅ 10.4.0>=7.1.2
pyyaml ✅ 6.0.2>=5.3.1
requests ✅ 2.32.3>=2.23.0
scipy ✅ 1.14.1>=1.4.1
torch ✅ 2.4.1>=1.8.0
torchvision ✅ 0.19.1>=0.9.0
tqdm ✅ 4.66.5>=4.64.0
psutil ✅ 6.0.0
py-cpuinfo ✅ 9.0.0
pandas ✅ 2.2.2>=1.1.4
seaborn ✅ 0.13.2>=0.11.0
ultralytics-thop ✅ 2.0.6>=2.0.0
torch ✅ 2.4.1!=2.4.0,>=1.8.0; sys_platform == "win32"

Minimal Reproducible Example

In my view, this, run on a nano model should yield a fully quantized model regardless of input:
model.export(format='tflite', int8=True)
Instead it seems to create a model with different types, at least according to TFLite micro.

Additional

No response

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

[glenn-jocher]

@nicklasb thank you for reporting this issue. The INT8 quantization process requires a representative dataset to calibrate the model properly. Please ensure you're providing this dataset during export. For more details, you can refer to our TFLite export documentation. If you have further insights or a solution, feel free to submit a PR.

[nicklasb]

Hi, @glenn-jocher, thanks for the reply:

1. As indicated, I provided a representative dataset, It didn't help.
2. Either way, it should still only have int8:s in the resulting model. Just be worse than without, either one quantize or one don't, int8 is either true or false. Quantization != Calibration..
3. Still, the documentation seem to omit the need for a representative dataset, I don't see it and it is not listed among the parameters.
4. I would gladly provide a PR, if I knew what the problem was. I am afraid I don't.

[glenn-jocher]

@nicklasb thank you for your feedback. We'll review the documentation to ensure clarity on the representative dataset requirement for INT8 quantization. If you encounter any specific issues or have further insights, feel free to share them. Your willingness to contribute is appreciated.

[nicklasb]

@nicklasb thank you for your feedback. We'll review the documentation to ensure clarity on the representative dataset requirement for INT8 quantization. If you encounter any specific issues or have further insights, feel free to share them. Your willingness to contribute is appreciated.

But is this a bug or not? I am not sure I understand your answer...I have no avenues forward from this.
Currently, I simply cannot use YOLOv8 with TF Lite Micro, if you don't support that use case in V8, please indicate so I can choose another object detection model.

[glenn-jocher]

I'm sorry for any confusion. This appears to be a limitation with the current export process for TFLite Micro. We're looking into it, but if immediate support is crucial, you might consider other models for now.

[nicklasb]

Sorry If I am a bit persistent, but I need to know if Ultralytics intends or doesn't intend not persue tflite micro as a supported use case for me to consider the framework for future use. Feels a little bit like you going AI on me. :-)

[glenn-jocher]

We understand your concern. Currently, full support for TFLite Micro isn't available, but we're evaluating future possibilities. For immediate needs, you might explore other options.

[dingniaoji]

Sorry If I am a bit persistent, but I need to know if Ultralytics intends or doesn't intend not persue tflite micro as a supported use case for me to consider the framework for future use. Feels a little bit like you going AI on me. :-)

I encounter same problem, exporting int8 failed.

(yolov8) root@ai:~/dataset/dnj/5nc23# yolo export model=runs/detect/train/weights/best.pt format=tflite int8=True data=data.yaml
Ultralytics YOLOv8.2.98 🚀 Python-3.10.14 torch-2.4.1 CPU (13th Gen Intel Core(TM) i9-13900HX)
YOLOv8x summary (fused): 268 layers, 68,145,717 parameters, 0 gradients, 257.5 GFLOPs

PyTorch: starting from 'runs/detect/train/weights/best.pt' with input shape (1, 3, 640, 640) BCHW and output shape(s) (1, 27, 8400) (130.4 MB)

TensorFlow SavedModel: starting export with tensorflow 2.16.1...

ONNX: starting export with onnx 1.16.2 opset 19...
ONNX: slimming with onnxslim 0.1.34...
ONNX: export success ✅ 2.7s, saved as 'runs/detect/train/weights/best.onnx' (260.3 MB)
TensorFlow SavedModel: collecting INT8 calibration images from 'data=data.yaml'
Scanning /root/dataset/dnj/5nc23/valid/labels.cache... 286 images, 0 backgrounds, 0 corrupt: 100%|██████████| 286/286 [00:00<?, ?it/s]
TensorFlow SavedModel: WARNING ⚠️ >300 images recommended for INT8 calibration, found 286 images.
TensorFlow SavedModel: starting TFLite export with onnx2tf 1.22.3...

Automatic generation of each OP name started ========================================
Automatic generation of each OP name complete!

Model loaded ========================================================================

Model conversion started ============================================================
saved_model output started ==========================================================
Killed

but f32 & f16 works. the command is:
yolo export model=runs/detect/train/weights/best.pt format=tflite
it works. but int8 not works. Killed in the end of output.

[glenn-jocher]

Thank you for sharing your experience. We're aware of the issue with INT8 export for TFLite and are investigating it. For now, using FP32 or FP16 is recommended.

[dingniaoji]

@glenn-jocher if yolo-flutter-app(https://github.com/ultralytics/yolo-flutter-app) support f16 and f32, int8 problem is not that annoying for me .

now I have to switch to other yolo mobile solution like NCNN.

[nicklasb]

@glenn-jocher if yolo-flutter-app(https://github.com/ultralytics/yolo-flutter-app) support f16 and f32, int8 problem is not that annoying for me .

now I have to switch to other yolo mobile solution like NCNN.

I think it is better if Ultralytics fix the int8 support. :-)

[glenn-jocher]

We understand the importance of INT8 support and are looking into it. Meanwhile, using FP16 or FP32 with the yolo-flutter-app is a viable option. Thank you for your patience.

[dingniaoji]

We understand the importance of INT8 support and are looking into it. Meanwhile, using FP16 or FP32 with the yolo-flutter-app is a viable option. Thank you for your patience.

after debugging, I found that yolo-flutter-app is actually support fp16 and fp32.

more details here ultralytics/yolo-flutter-app#61 (comment).

yeah, It is better if Ultralytics fix the int8 support. :-).

[glenn-jocher]

Thank you for confirming the support for FP16 and FP32 in the yolo-flutter-app. We appreciate your feedback on INT8 support and are working on improvements.

[EmmanuelMess]

It looks like it works for YOLO11n:

    model = YOLO("yolo11n.pt")
    model.export(format="tflite", int8=True) 

Creates a valid export. But crashes with num_inputs <= kMaxInputNum was not true. See #18083

[glenn-jocher]

Thanks for pointing that out! The issue might be related to input tensor limits in TFLite. We'll investigate further to ensure consistent INT8 export functionality. You can monitor updates in #18083.

[github-actions]

👋 Hello there! We wanted to give you a friendly reminder that this issue has not had any recent activity and may be closed soon, but don't worry - you can always reopen it if needed. If you still have any questions or concerns, please feel free to let us know how we can help.

For additional resources and information, please see the links below:

• Docs: https://docs.ultralytics.com
• HUB: https://hub.ultralytics.com
• Community: https://community.ultralytics.com

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLO 🚀 and Vision AI ⭐