gRPC throws error on large input size

[ankita4AI]

Hello,
gRPC throws an error when the size of input request is bulky, which is not the case with REST requests.
works seamlessly for smaller batch sizes.
Where am I supposed to increase max size parameter?

PFB the error log
debug_error_string = "{"created":"@1611067667.277075000","description":"Error received from peer ipv6:[::1]:7070","file":"src/core/lib/surface/call.cc","file_line":1063,"grpc_message":"Received RST_STREAM with error code 8","grpc_status":1}"

Any help would be appreciated.
TIA

[maaquib]

@ankita4AI Can you try increasing max_request_size parameter to something larger (doc)

[ankita4AI]

hi @maaquib thanks for your response.
But this max_request_size in ConfigManager.java is not the one overriding grpc's message size.
I wonder where grpc client settings are configured. can you point me where we can set max message Inbound size for gRPC.
I can say that with certainty coz its maximum size remains 4194304 no matter the max_request_size set.

please find below ts_logs for more information
WARNING: Exception processing message
io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: gRPC message exceeds maximum size 4194304: 9198552

[ankita4AI]

gRPC uses per-message size limits to manage incoming and outgoing messages. By default, gRPC limits incoming messages to 4 MB. There is no limit on outgoing messages.
I tried updating the grpc client file where we set the channel, but I see the same error.

def get_inference_stub():
    channel = grpc.insecure_channel(
    'localhost:7070',
    options=[
        ('grpc.max_send_message_length', 15000000),
        ('grpc.max_receive_message_length', 15000000),
    ],
    )
    stub = inference_pb2_grpc.InferenceAPIsServiceStub(channel)
    return stub

[maaquib]

Looks like this has to be made configurable on the server side by adding this parameter. Adding this to 0.4 roadmap

[ankita4AI]

I was trying to update startGRPCServer method in ModelServer.java
one can update ServerBuilder's setMaxMessageSize to unlimited.
But struggled at rebuilding torch serve for this build to be used.
Please share steps to do that as well, It would help a lot, Thanks.

[maaquib]

@ankita4AI You can run the scripts here to build TS from source.

[cbasavaraj]

Hi, I tried changing client side code just like @ankita4AI . Can confirm that it's not enough. Looking forward to server side changes in v0.4. Thanks

[cbasavaraj]

Hi, will v0.4.0 be released sometime soon? Thanks

[lxning]

Yes. Thanks, Li From: Chandrachud Basavaraj ***@***.***> Reply-To: pytorch/serve ***@***.***> Date: Friday, April 16, 2021 at 9:29 AM To: pytorch/serve ***@***.***> Cc: "Ning, Li" ***@***.***>, State change ***@***.***> Subject: Re: [pytorch/serve] gRPC throws error on large input size (#942) Hi, will v0.4.0 be released sometime soon? Thanks — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub<#942 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AFTASZGWCKI544OGWEKILZLTJBQSFANCNFSM4WI4KGXQ>.