Local buffering for Influxdb2 to cloud #147
-
|
Here's the English translation: I have installed the plugin "signalk-to-influxdb-v2-buffering". In the plugin instructions, I read that you need to specify the "full path to directory where the buffer should be stored (note no at end of dir)", but in the plugin configuration, this is not possible, as the option simply does not exist. Is this perhaps a plugin still in development? Or has it been abandoned? Thank you! |
Beta Was this translation helpful? Give feedback.
Replies: 8 comments 1 reply
-
|
Hello, I have the same needs. I tried many ways, and the last one, which seems working, is using signalk-prometheus, and VictoriaMetrics. With VictoriaMetrics, you have a leightweight program, vmagent, that can cache data in case of connections loss. You have to install this agent locally through CLI to do that. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you very much for the suggestion. However, I have two problems:
So, I ask: is it possible to update one of these two plugins so that the buffering function works? I'm ready to make a donation.
Or, dear Macjl, could you kindly provide me with a step-by-step guide to have a working Signalk - VmAgent - Prometheus system? Thanks again! |
Beta Was this translation helpful? Give feedback.
-
|
It’s not easy to make a step by step guide. But I can help you to experiment this solution. On what platform will you run SignalK and the remote server ? Have you knowledge on docker ? |
Beta Was this translation helpful? Give feedback.
-
|
I tried to look into how signalk-to-influxdb-v2-buffering is supposed to do buffering, but it seems the author has removed the code that actually was supposed to do buffering, leaving documentation and configuration options related to it in place 🤷 . I am not aware of any out of the mechanism for buffering writes to a cloud Influxdb instance. A method that I've used successfully in the past is to use an MQTT bridge from the local Signal K server to another running in the cloud. The idea is that you use the mqtt plugin on the local SK server to publish all SK data to a local MQTT Mosquitto server. That server is configured to work as a store-and-forward bridge to another Mosquitto server running in the cloud. There you have an SK Server reading from the MQTT server and you have the full capabilities of SK server at your disposal, including any db plugins such as the InfluxDb plugins. So when there is no connectivity the local Mosquitto buffers the data on disk and when connectivity is restored the data is published to the cloud. I created a demo of this setup a few years ago. If there's interest I can update it to work with the current MQTT version plugin, it is now outdated but nevertheless demonstrates the setup. The setup is more cumbersome and you need to run the secondary SK server on the server than simply having the Influxdb plugin do the buffering, but the upside is that the bridge is 100% generic - if you choose to switch to another db backend (Timescale for example) the buffering bridge setup can stay as is, no changes needed onboard the vessels. If you have several vessels each can have their independent MQTT credentials. Another option would be to forego the cloud SK server and shovel data from MQTT directly to Influxdb, for example with InfluxDB native MQTT collector. |
Beta Was this translation helpful? Give feedback.
-
|
This was my solution for buffering to the cloud. I quite like the approach
of batched files on disk. It will optionally upload the files to S3,
including working through a backlog. It requires some work to stitch
together, but I've found it very reliable over an unreliable network
connection from the boat.
It can be operated in a mode similar to what Teppo described, where a
signalk in the cloud consumes the files and ingests into a local signalk.
From there you can go to influxdb (I have a cloud instance doing this).
However, the history API for signalk is not implemented yet, so there will
be gaps in signalk to signalk after a network blip. I don't see gaps when
going signalk -> S3 -> Lambda -> Timestream. I don't have the time to
implement the signalk history API, but if someone is motivated, I'm happy
to receive a pull request.
https://github.com/c33howard/signalk-to-batch-format
https://github.com/c33howard/signalk-from-batch-format
https://github.com/c33howard/signalk-to-timestream-lambda
Craig
…On Sat, Oct 26, 2024 at 6:40 AM Teppo Kurki ***@***.***> wrote:
I tried to look into how signalk-to-influxdb-v2-buffering is supposed to
do buffering, but it seems the author has removed the code that actually
was supposed to do buffering, leaving documentation and configuration
options related to it in place 🤷 . I am not aware of any out of the
mechanism for buffering writes to a cloud Influxdb instance.
A method that I've used successfully in the past is to use an MQTT bridge
from the local Signal K server to another running in the cloud.
The idea is that you use the mqtt plugin
<https://github.com/tkurki/signalk-mqtt-gw> on the local SK server to
publish all SK data to a local MQTT Mosquitto server. That server is
configured to work as a store-and-forward bridge to another Mosquitto
server running in the cloud. There you have an SK Server reading from the
MQTT server and you have the full capabilities of SK server at your
disposal, including any db plugins such as the InfluxDb plugins.
So when there is no connectivity the local Mosquitto buffers the data on
disk and when connectivity is restored the data is published to the cloud.
I created a demo of this setup
<tkurki/signalk-mqtt-gw@master...cloud-bridge>
a few years ago. If there's interest I can update it to work with the
current MQTT version plugin, it is now outdated but nevertheless
demonstrates the setup.
The setup is more cumbersome and you need to run the secondary SK server
on the server than simply having the Influxdb plugin do the buffering, but
the upside is that the bridge is 100% generic - if you choose to switch to
another db backend (Timescale for example) the buffering bridge setup can
stay as is, no changes needed onboard the vessels.
If you have several vessels each can have their independent MQTT
credentials.
Another option would be to forego the cloud SK server and shovel data from
MQTT directly to Influxdb, for example with InfluxDB native MQTT collector
<https://www.influxdata.com/blog/faster-mqtt-data-collection-influxdb/>.
—
Reply to this email directly, view it on GitHub
<#147 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAN33OUOWBDWRNHO5YMDIOTZ5OLVXAVCNFSM6AAAAABQHRUGV6VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMBWGA2DOOI>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
My solution is to use InfluxDB replication. So on my boat computer, signalk
writes to local Influxdb, then the replication updates the replica at home
server.
This has some buffering ( 7days) built in so data is not lost if the
network is not available.
https://docs.influxdata.com/influxdb/v2/write-data/replication/replicate-data/
This is a great guide for setting this up.
https://www.youtube.com/watch?v=Lb1vIBNxrik
The other benefit is i can run Grafana also on the home server referencing
the local replica so performance is great!
It did take me about 12months to "get around to doing this". The
replication does not replicate existing data which meant i needed to do a
full backup and restore to get all previous data.
…On Sun, 27 Oct 2024 at 18:06, Craig Howard ***@***.***> wrote:
This was my solution for buffering to the cloud. I quite like the approach
of batched files on disk. It will optionally upload the files to S3,
including working through a backlog. It requires some work to stitch
together, but I've found it very reliable over an unreliable network
connection from the boat.
It can be operated in a mode similar to what Teppo described, where a
signalk in the cloud consumes the files and ingests into a local signalk.
From there you can go to influxdb (I have a cloud instance doing this).
However, the history API for signalk is not implemented yet, so there will
be gaps in signalk to signalk after a network blip. I don't see gaps when
going signalk -> S3 -> Lambda -> Timestream. I don't have the time to
implement the signalk history API, but if someone is motivated, I'm happy
to receive a pull request.
https://github.com/c33howard/signalk-to-batch-format
https://github.com/c33howard/signalk-from-batch-format
https://github.com/c33howard/signalk-to-timestream-lambda
Craig
On Sat, Oct 26, 2024 at 6:40 AM Teppo Kurki ***@***.***>
wrote:
> I tried to look into how signalk-to-influxdb-v2-buffering is supposed to
> do buffering, but it seems the author has removed the code that actually
> was supposed to do buffering, leaving documentation and configuration
> options related to it in place 🤷 . I am not aware of any out of the
> mechanism for buffering writes to a cloud Influxdb instance.
>
> A method that I've used successfully in the past is to use an MQTT
bridge
> from the local Signal K server to another running in the cloud.
>
> The idea is that you use the mqtt plugin
> <https://github.com/tkurki/signalk-mqtt-gw> on the local SK server to
> publish all SK data to a local MQTT Mosquitto server. That server is
> configured to work as a store-and-forward bridge to another Mosquitto
> server running in the cloud. There you have an SK Server reading from
the
> MQTT server and you have the full capabilities of SK server at your
> disposal, including any db plugins such as the InfluxDb plugins.
>
> So when there is no connectivity the local Mosquitto buffers the data on
> disk and when connectivity is restored the data is published to the
cloud.
>
> I created a demo of this setup
> <tkurki/signalk-mqtt-gw@master...cloud-bridge>
> a few years ago. If there's interest I can update it to work with the
> current MQTT version plugin, it is now outdated but nevertheless
> demonstrates the setup.
>
> The setup is more cumbersome and you need to run the secondary SK server
> on the server than simply having the Influxdb plugin do the buffering,
but
> the upside is that the bridge is 100% generic - if you choose to switch
to
> another db backend (Timescale for example) the buffering bridge setup
can
> stay as is, no changes needed onboard the vessels.
>
> If you have several vessels each can have their independent MQTT
> credentials.
>
> Another option would be to forego the cloud SK server and shovel data
from
> MQTT directly to Influxdb, for example with InfluxDB native MQTT
collector
> <https://www.influxdata.com/blog/faster-mqtt-data-collection-influxdb/>.
>
> —
> Reply to this email directly, view it on GitHub
> <
#147 (comment)>,
> or unsubscribe
> <
https://github.com/notifications/unsubscribe-auth/AAN33OUOWBDWRNHO5YMDIOTZ5OLVXAVCNFSM6AAAAABQHRUGV6VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMBWGA2DOOI>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>
—
Reply to this email directly, view it on GitHub
<#147 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABSI5SICYTVKTDLAOXUS5S3Z5SGF5AVCNFSM6AAAAABQHRUGV6VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMBWGQ2TQMY>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Many thanks to all for these solutions that you propose. I believe that a plugin dedicated to this need will be much appreciated! |
Beta Was this translation helpful? Give feedback.
-
|
An option not covered in this thread yet is InfluxDB's Edge data (db to db) replication, see https://docs.influxdata.com/influxdb/cloud/write-data/replication/replicate-data/ |
Beta Was this translation helpful? Give feedback.
-
|
Hello, |
Beta Was this translation helpful? Give feedback.
Hello,
after several tests, I have obtained the best result with the replication data to a second Influxdb2 bucket.
Following the instructions here:
https://www.youtube.com/watch?v=Lb1vIBNxrik
the server on the boat send data to a server on the cloud, even one week later without any lack.
Very good and simple solution, thank you to everyone!