Experiment with extracting contact details extraction and if good make a miniactor integration

[metalwarrior665]

Compare if we could improve https://apify.com/vdrmota/contact-info-scraper

Basically, we need to figure out the best prompts and if it will do more than the contact, we can release it as a miniactor

[jancurn]

Just a note, if you guys do a miniactor for these sub-usecases, please let's use metamorph and make it open source, so that we can use it for marketing and show others how easily to do this.

[foxt451]

I guess I'll try to look into it a bit

[metalwarrior665]

Sounds good, let's not write any code yet, just play with it. We need to collect some pages for testing. The ideal case for GPT would be to match emails, phones, etc. with names. We had a student project that tried to do this based on HTML element proximity but that's super tricky.

[foxt451]

🙂 we would probably need some quite clever prompt

[metalwarrior665]

Ouch :D

[jancurn]

Wow, maybe we'll need to use another model :)

[foxt451]

So, I haven't yet started tinkering with the model settings, but I've set up basic boilerplate, which includes the miniactor code with metamorpth and tests. Tests basically list a set of urls with expected contacts to be found on them and check if they get returned when using model settings, schema and prompt exported from @packages/contact-scraper. They also check that there are no hallucinations (and, well, they all fail for now)
To tinker with it at this point, you go to @packages/contact-scraper and change either the exported model name, the model settings, the prompt or the schema and rerun npm test until it succeeds.
The model/prompt settings themselves would be not available on the miniactor. And here is the draft miniactor itself, built from the https://github.com/apify-projects/store-gpt-scraper/tree/feat/contact-details branch

[metalwarrior665]

I created a PR here https://github.com/apify-projects/store-gpt-scraper/pull/50/files.

I don't think this implementation offers any advantage over the current Contact Details Scraper since all of these things can be regexed. The next step that we are missing and GPT could manage (although I don't believe it is powerful enough) is to link the contacts together also with names. This is something we tried as student project based on HTML proximity but we didn't have good generic name regex. I would just play with it a bit and we can release it even if it is not great.

[foxt451]

I guess this actor/repo is not even needed then (and the issue belongs rather to contact details repo)? All the HTML/contacts is extracted in the contact-details scraper and GPT API requests could be sent directly from that actor. There is no need in this (gpt) actor's crawling capabilities (on the other hand, one could say that contact details is not needed, and we could use crawlee's social module here)

[foxt451]

So, I've closed that old PR as it seemed to go in the way wrong direction. And opened another one #51. What do you think @jancurn @metalwarrior665 ?