[FEATURE] Support multimodal input

[RyanMarten]

Example use cases with curator

• synthetic data: generating synthetic captions for images
• structured data extraction: getting itemized costs from receipts

Tasks:

• Basic Design interface for multi modal support #412
• Open AI online multi modal support (Online) #413

[marianna13]

Hey @RyanMarten can help with captioning (we have a common project with LAION exactly on that)

[vutrung96]

@marianna13 would be great if you could help! i think we were thinking of support multimodal as a input modality more generally (captioning is an example). we can brainstorm on this :D

[RyanMarten]

That would be awesome @marianna13!

[madiator]

Thanks Marianna! Sent via Superhuman ( ***@***.*** )

…

On Fri, Jan 10, 2025 at 10:28 AM, Ryan Marten < ***@***.*** > wrote: That would be awesome @ marianna13 ( https://github.com/marianna13 ) ! — Reply to this email directly, view it on GitHub ( #319 (comment) ) , or unsubscribe ( https://github.com/notifications/unsubscribe-auth/AAIJX25V74PKFJN7L7MJWBL2KAGORAVCNFSM6AAAAABUYM4I7SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKOBTGUYDIMJWG4 ). You are receiving this because you are subscribed to this thread. Message ID: <bespokelabsai/curator/issues/319/2583504167 @ github. com>

[adamoptimizer]

Tasks:

• Design an interface for multi modality
• Support images/videos as input (url/local path)

[marianna13]

I think loading images/videos from URLs will not be sustainable (for large datasets)

[adamoptimizer]

I think loading images/videos from URLs will not be sustainable (for large datasets)

For large datasets, we have batch processing!
we will have a basic support of multi modality starting with OpenAI. (Online mode)
Then progress with other providers along with edge cases!
Thanks

[marianna13]

what do you want to use for batch processing?