Skip to content

Commit

Permalink
acquisition/victorialogs: add new datasource (crowdsecurity#3310)
Browse files Browse the repository at this point in the history
* acquisition/victorialogs: add new datasource

Data source supports:
- cat mode with automatic adjustment of poll interval (same as one at Loki datasource)
- tail mode by using tailing API
  • Loading branch information
zekker6 authored Jan 29, 2025
1 parent 5178a91 commit 172d6c6
Show file tree
Hide file tree
Showing 9 changed files with 1,310 additions and 14 deletions.
11 changes: 11 additions & 0 deletions .github/workflows/go-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,17 @@ jobs:
--health-retries 5
--health-start-period 30s
victorialogs:
image: victoriametrics/victoria-logs:v1.5.0-victorialogs
ports:
- "9428:9428"
options: >-
--name=victorialogs1
--health-cmd "wget -q -O - http://0.0.0.0:9428"
--health-interval 30s
--health-timeout 10s
--health-retries 5
--health-start-period 30s
steps:

- name: Check out CrowdSec repository
Expand Down
6 changes: 6 additions & 0 deletions .golangci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,7 @@ linters-settings:
- "!**/pkg/acquisition/modules/kubernetesaudit/k8s_audit.go"
- "!**/pkg/acquisition/modules/loki/loki.go"
- "!**/pkg/acquisition/modules/loki/timestamp_test.go"
- "!**/pkg/acquisition/modules/victorialogs/victorialogs.go"
- "!**/pkg/acquisition/modules/s3/s3.go"
- "!**/pkg/acquisition/modules/syslog/syslog.go"
- "!**/pkg/acquisition/modules/wineventlog/wineventlog_windows.go"
Expand Down Expand Up @@ -398,6 +399,11 @@ issues:
path: pkg/acquisition/modules/loki/internal/lokiclient/loki_client.go
text: "confusing-naming: Method 'QueryRange' differs only by capitalization to method 'queryRange' in the same source file"

- linters:
- revive
path: pkg/acquisition/modules/victorialogs/internal/vlclient/vl_client.go
text: "confusing-naming: Method 'QueryRange' differs only by capitalization to method 'queryRange' in the same source file"

- linters:
- revive
path: cmd/crowdsec-cli/copyfile.go
Expand Down
1 change: 1 addition & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,7 @@ COMPONENTS := \
datasource_journalctl \
datasource_kinesis \
datasource_loki \
datasource_victorialogs \
datasource_s3 \
datasource_syslog \
datasource_wineventlog \
Expand Down
12 changes: 12 additions & 0 deletions pkg/acquisition/modules/victorialogs/internal/vlclient/types.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
package vlclient

import (
"time"
)

// Log represents a VictoriaLogs log line
// See: https://docs.victoriametrics.com/victorialogs/querying/#querying-logs
type Log struct {
Message string `json:"_msg"`
Time time.Time `json:"_time"`
}
Loading

0 comments on commit 172d6c6

Please sign in to comment.