Add bfloat to client #4521
Add bfloat to client #4521
Conversation
|
|
||
| FLAGS = None | ||
|
|
||
| def test_bf16_raw_http(shape): | ||
| def test_bf16_http(shape): |
There was a problem hiding this comment.
# FIXME: Use identity_bf16 model in test above once proper python client
# support is added, and remove this raw HTTP test. See DLIS-3720.
This helper was only meant to be a temp workaround until python client bf16 support was added.
Can we instead add to the existing test below now that there is bf16 support in the client? I believe there's a identity_bf16 model that will work now, or if not could make a very simple python identity model.
This would:
- test both HTTP/GRPC
- test the bf16 support in client library workflow:
triton_to_np_dtype,np_to_triton_dtype, etc.
| # https://github.com/GreenWaves-Technologies/bfloat16/issues/2 | ||
| # Can solve when numpy officially supports bfloat16 | ||
| # https://github.com/numpy/numpy/issues/19808 | ||
| print("error: tensorflow is included in module. This module cannot " \ |
There was a problem hiding this comment.
Is this an error/check that should be done by the client as well? The comment is helpful, but users may run into issues if this results in incompatibility.
There was a problem hiding this comment.
I think it makes more sense to try/except on the specific error we're expecting here. Maybe it won't always happen when tensorflow is installed.
try:
from bfloat16 import bfloat16
except <specific error we expect with tensorflow>:
print("known issue tensorflow pypi, link to github issue")
except Exception as e:
print("unexpected error importing bf16:", e)
Adding bfloat16 to client and QA containers. Not sure if I'm missing something but I think bfloat16 should be supported in python client with the current change. Reusing Ryan's test for bfloat16 testing.
Related PR: triton-inference-server/client#118