Scene recognition (I think)

[dwaterford75]

I have the AI server operating in my home environment. I have used it for simple object detection such as birds near my koi pond and stuff like that. I would like to take the operation up a level though. I want to use the AI server to recognise scenes (security related) and create alerts. Is this possible?

So by example, if someone is spotted climbing a wall or if someone is on my wall inside my electric fence.

I fed the images in question to an online AI asking if the image is suspicious and the server would reply saying yes this looks suspicious for the following reasons. Like a man is on the wall behind the security electric fence or it seems like this person is trying to open a window form the outside.

I don't want to feed 43,200 images (per camera) over the internet to an online server to do this scene recognition so would like to put a server on site to do it.

Can CodeProject do this?

Thanks in advance and WOW, what a great system!

[ChrisMaunder]

CodeProject.AI can actually do this, but you'll need to use the Multi-modal LLM module. You can pass this an image and ask "Describe this image" and it'll do exactly what you are asking for.

Caveat is that you'll need a decent GPU for this since it's very resource intensive. I have managed to get it to work on just the CPU but it takes minutes, rather than seconds.

I would probably rig up something that detects people, and when it sees a person it sends the frames (one every few seconds) to the multimodal module.

Alternatively there is an ActionNet in the YOLO object detector that can detect actions (running, sitting etc). It's super basic but fast. Maybe that could fill in some gapes or do extra pre-processing before you decide to ask the big LLM

[dwaterford75]

Aha ... Interesting. I don't mind putting a peoper GPU into the machine so that is not a hindrance.
Where do I get this LLM Modeule?

[ChrisMaunder]

On the "Modules" tab of the dashboard. Note that installing the module gives you access to the module's LLM, but it will be up to you to make the API calls to the module to achieve what you're after

[dwaterford75]

LOL – Apologies, that was a really stupid question! I didn’t realize this would fall under generative. Thanks. From: Chris Maunder ***@***.***> Sent: Wednesday, 08 January 2025 22:31 To: codeproject/CodeProject.AI-Server ***@***.***> Cc: dwaterford75 ***@***.***>; Author ***@***.***> Subject: Re: [codeproject/CodeProject.AI-Server] Scene recognition (I think) (Discussion #259) On the "Modules" tab of the dashboard. Note that installing the module gives you access to the module's LLM, but it will be up to you to make the API calls to the module to achieve what you're after — Reply to this email directly, view it on GitHub <#259 (reply in thread)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/APC43W5KBXI7DJA26DEJSEL2JWDHLAVCNFSM6AAAAABUZOHRUCVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCNZXHAZDIMI> . You are receiving this because you authored the thread. <https://github.com/notifications/beacon/APC43W34BS2DE4NIGMIII6T2JWDHLA5CNFSM6AAAAABUZOHRUCWGG33NNVSW45C7OR4XAZNRIRUXGY3VONZWS33OINXW23LFNZ2KUY3PNVWWK3TUL5UWJTQAWO4MC.gif> Message ID: ***@***.*** ***@***.***> >

[tbc725]

Or, if LLM is a bit resource intensive, you could also simply set the motion detection that is filtered by the object detection and only set certain area. That could potentially help you to catch it beforehand; rather than after it is happening. But you could also have both in place as well, which you would know someone was near the wall or fence and now is climbing it.

Or simply a set the region of detection on the wall to detect person. Nobody should be on the wall anyway, so if it detects person on your wall, that could send an alarm to you.