Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add realtime building blocks to the SDK #61

Merged
merged 4 commits into from
Jan 29, 2024
Merged

feat: add realtime building blocks to the SDK #61

merged 4 commits into from
Jan 29, 2024

Conversation

isidentical
Copy link
Collaborator

@isidentical isidentical commented Jan 26, 2024

Building realtime apps without these are super hard and makes the code look terrible, this basically abstracts all the required stages for the user and lets them just define the inference behavior similiar to how the regular HTTP stuff works. Real world example of this code clutter: https://github.com/fal-ai/serverless-registry/blob/9f49dad913e10404a647eff3f7fa2f2e79e7f721/registry/image/lcm/inference.py#L292-L431

example:

class RealtimeApp(fal.App, _scheduler="nomad"):
    requirements = ["websockets", "msgpack"]

    def setup(self):
        pass

    @fal.endpoint("/")
    def generate(self, input: Input) -> Output:
        return Output(text=input.prompt)

    @fal.endpoint("/ws", is_websocket=True)
    async def generate_ws(self, websocket: WebSocket) -> None:
        await websocket.accept()
        for _ in range(3):
            await websocket.send_json({"message": "Hello world!"})
        await websocket.close()

    @fal.realtime("/realtime")
    def generate_rt(self, input: Input) -> Output:
        return Output(text=input.prompt)

@isidentical isidentical marked this pull request as ready for review January 26, 2024 00:48
Comment on lines +291 to +292
await websocket.send_json(
{
"type": "x-fal-error",
"error": "INTERNAL_ERROR",
"reason": str(exc),
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how does the consumer determine whether the response is a msgpack struct or a json? or why don't we use msgpack for all the messages?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fal emits its own messages as json, so this is done to mimic that. binary protocol is used only for I/O, and errors and other stuff goes as json. the users are expected to decode the frames to see if they are a text frame or a binary frame, and use the right format for that (json and msgpack respectively). https://github.com/fal-ai/fal-js/blob/12887d4b876793ef060924e5bc10a08d39d6313a/libs/client/src/realtime.ts#L473-L495

@isidentical isidentical force-pushed the realtime branch 2 times, most recently from 9b20389 to e9ee6ae Compare January 29, 2024 22:25
turbo1912
turbo1912 previously approved these changes Jan 29, 2024
@isidentical isidentical merged commit 1411fe5 into main Jan 29, 2024
4 checks passed
@isidentical isidentical deleted the realtime branch January 29, 2024 22:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants