Failed to fetch Frigate clip

[twister36]

Bug Description

Unable to fetch any frigate event.

I am using the frigate api to get a valid frigate event id. However, any id I use results in the same "failed to fetch" error. Please note that my frigate is not hosted on the Home Assistant, it is a remote instance outside of Home Assistant. I am using Frigate Proxy to allow my Home Assistant to see my remote instance of Frigate.

Is running frigate remotely supported? It appears this is assuming that Frigate is running within Home Assistant and all video/images are local to Home Assistant.

In my setup, Frigate is external to Home assistant and is not accessible through port 5000 of Home Assistant. Image/Video files would be externally hosted on another IP/Host where Frigate is hosted.

Version:

Service Call

action: llmvision.video_analyzer
data:
remember: false
frigate_retry_attempts: 2
frigate_retry_seconds: 1
max_frames: 5
include_filename: false
target_width: 1280
max_tokens: 100
temperature: 0.2
generate_title: false
expose_images: false
expose_images_persist: false
provider: 01JGN58MMMD4Q40KFE2X85RPBV
model: gemini-1.5-flash
event_id: 1736006221.203278-ezta1o
message: what is happening in this video?

Logs

No errors in the logs found for LLM Vision or Frigate

Additional context

[andyp05]

I am experiencing the same error using standard frigate addon for HA. Using the blueprints.

Two minutes later, the automation was run again and it worked as expected. Not sure if this is a timing issue where the frigate files are not written to disk yet. However, I am running on a very fast 8 core pc with a coral device and GPU acceleration.

Failed Automation Trace:

Executed: January 4, 2025 at 12:32:59 PM
Error: Failed to fetch frigate clip 1736011975.094501-0xrh96
Result:
params:
  domain: llmvision
  service: video_analyzer
  service_data:
    event_id: 1736011975.094501-0xrh96
    provider: 01JGPAT5VKKFHVEFTTZ25QQ34X

Running HA 2024.12.5 on amd64
LLM Vision 1.3.6

[twister36]

@andyp05 - Are you running this right after the event occurs from Frigate? For me I'm testing with old events from today.

[andyp05]

The automation is running using the blueprints and is triggered by frigate person detection.

[quincy5891]

I have the same problem. It cannot fetch clip, but if I line comment out the llmvision action and go straight to the notification with the video clip it will include the clip to watch on my phone.

Run Analysis
Executed: January 4, 2025 at 2:38:00 PM
Error: Failed to fetch frigate clip 1736012466.033084-lw9qku
Result:
params:
  domain: llmvision
  service: video_analyzer
  service_data:
    event_id: 1736012466.033084-lw9qku
    provider: 01JGANNYZGRCC70X026JD1XV07
    model: gemini-1.5-flash
    

Executed: January 4, 2025 at 2:36:38 PM
Result:
params:
  domain: notify
  service: mobile_app_iphone
  service_data:
    title: Car seen
    message: ''
    data:
      video: /api/frigate/notifications/1736012466.033084-lw9qku/clip.mp4    

[twister36]

I think we're talking about different issues. I'm referencing that I can not call frigate events and get an image/video because frigate is installed remotely from Home Assistant

[quincy5891]

My frigate is separate machine(unraid) than homeassistant (proxmox) but still in the internal lan. My issue is very intermittent too. Sometimes works sometimes doesn’t. If its been to long it will never pull that clip but I believe that is due to the unauthenticated event proxy timer. I say that since I still have access to clip for alerts if I test run the same event id for notifications to phone.

[twister36]

@quincy5891 ok, so you're using Frigate Proxy in Home Assistant?

[quincy5891]

Sorry, no. I just use the integration.

[valentinfrlch]

This error usually happens when the clip is not yet available (not yet written to disk by Frigate) and can therefore not be accessed.

LLM Vision uses the API endpoint provided by the Frigate Integration which means that it should still work even if Frigate is running on another machine. Just make sure that you have the Frigate Integration installed, as otherwise it wont work.

[andyp05]

I am using the frigate addon and integration and frigate notifications are working fine. When I replace using the blueprints in this repository, I get no image or a 404 error 80% of the time. Can't figure out why it works sometimes.

[valentinfrlch]

In that case the clip isn't always ready (not written to disk). I'll add an option to configure retries in the blueprint so when the request fails, the blueprint will continue to try to fetch the clip.

[twister36]

This error usually happens when the clip is not yet available (not yet written to disk by Frigate) and can therefore not be accessed.

LLM Vision uses the API endpoint provided by the Frigate Integration which means that it should still work even if Frigate is running on another machine. Just make sure that you have the Frigate Integration installed, as otherwise it wont work.

Ok, what IP is LLM-Vision calling when trying to reach the frigate API? The Home Assistant IP on port 5000? If so, I don't think frigate proxy expose the API from the actually frigate server. I will test and verify tomorrow... But I'm pretty sure the API for proxy would be the actually frigate external server.

[twister36]

OK, I have opened Frigate Proxy issue, to see if this a Frigate Proxy issue, as I can not access Frigate through HA on port 5000
blakeblackshear/frigate-hass-addons#203

[Frazou1]

Sometime the clip work but they dont summarize only said Person seen. Those i missing something ?

[andyp05]

Don't know if it is related to this specific issue, but when I don't get a clip or LLM description sometimes I see:

Error: Error: all the input array dimensions except for the concatenation axis must match exactly, but along dimension 1, the array at index 0 has size 2985984 and the array at index 1 has size 5308416

[valentinfrlch]

The larger array should now be trimmed to match the shape of the smaller array, in case there's a mismatch. I can't replicate the issue so I can't directly test the fix either.
I will release another beta tomorrow and would be happy to hear your feedback!

[andyp05]

using new 1.3.7 and get the same error when looking at the trace

Executed: January 29, 2025 at 10:30:04 AM
Error: Failed to fetch frigate clip 1738164603.418317-lo6ady
Result:
choice: 0