Skip to content

Commit

Permalink
Initial
Browse files Browse the repository at this point in the history
Ildar Minaev committed Dec 26, 2024
1 parent 34d3fca commit f68683a
Showing 45 changed files with 5,868 additions and 1 deletion.
45 changes: 45 additions & 0 deletions .github/workflows/push.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
name: Build Artifacts
on:
release:
types: [created]
push:
branches:
- '**'
env:
TAG_NAME: ${{ github.event.release.tag_name || (github.ref == 'refs/heads/main' && 'main' || ( inputs.postfix != '' && format('{0}-{1}', github.ref, inputs.postfix) || 'none' )) }}

jobs:
multiplatform_build:
strategy:
fail-fast: false
matrix:
component:
- name: qubership-open-telemetry-collector
file: Dockerfile
- name: qubership-open-telemetry-collector-transfer
file: docker-transfer/Dockerfile
context: ""
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Login to Docker Hub
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${GITHUB_ACTOR}
password: ${{secrets.GITHUB_TOKEN}}
- name: Build and push
uses: docker/build-push-action@v5
with:
no-cache: true
context: ${{ matrix.component.context }}
file: ${{ matrix.component.file }}
platforms: linux/amd64,linux/arm64
push: true
tags: ghcr.io/netcracker/${{ matrix.component.name }}:${{ env.TAG_NAME }}
provenance: false
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
/.idea/
build/_output/*
.vscode/
73 changes: 73 additions & 0 deletions CODE-OF-CONDUCT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
# Code of Conduct

This repository is governed by following code of conduct guidelines.

We put collaboration, trust, respect and transparency as core values for our community.
Our community welcomes participants from all over the world with different experience,
opinion and ideas to share.

We have adopted this code of conduct and require all contributors to agree with that to build a healthy,
safe and productive community for all.

The guideline is aimed to support a community where all people should feel safe to participate,
introduce new ideas and inspire others, regardless of:

* Age
* Gender
* Gender identity or expression
* Family status
* Marital status
* Ability
* Ethnicity
* Race
* Sex characteristics
* Sexual identity and orientation
* Education
* Native language
* Background
* Caste
* Religion
* Geographic location
* Socioeconomic status
* Personal appearance
* Any other dimension of diversity

## Our Standards

We are welcoming the following behavior:

* Be respectful for different ideas, opinions and points of view
* Be constructive and professional
* Use inclusive language
* Be collaborative and show the empathy
* Focus on the best results for the community

The following behavior is unacceptable:

* Violence, threats of violence, or inciting others to commit self-harm
* Personal attacks, trolling, intentionally spreading misinformation, insulting/derogatory comments
* Public or private harassment
* Publishing others' private information, such as a physical or electronic address, without explicit permission
* Derogatory language
* Encouraging unacceptable behavior
* Other conduct which could reasonably be considered inappropriate in a professional community

## Our Responsibilities

Project maintainers are responsible for clarifying the standards of the Code of Conduct
and are expected to take appropriate actions in response to any instances of unacceptable behavior.

Project maintainers have the right and responsibility to remove, edit, or reject comments,
commits, code, wiki edits, issues, and other contributions that are not aligned
to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors
that they deem inappropriate, threatening, offensive, or harmful.

## Reporting

If you believe you’re experiencing unacceptable behavior that will not be tolerated as outlined above,
please report to `opensourcegroup@netcracker.com`. All complaints will be reviewed and investigated and will result in a response
that is deemed necessary and appropriate to the circumstances. The project team is obligated to maintain confidentiality
with regard to the reporter of an incident.

Please also report if you observe a potentially dangerous situation, someone in distress, or violations of these guidelines,
even if the situation is not happening to you.
12 changes: 12 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# Contribution Guide

We'd love to accept patches and contributions to this project.
Please, follow these guidelines to make the contribution process easy and effective for everyone involved.

## Contributor License Agreement

You must sign the [Contributor License Agreement](https://pages.netcracker.com/cla-main.html) in order to contribute.

## Code of Conduct

Please make sure to read and follow the [Code of Conduct](CODE-OF-CONDUCT.md).
23 changes: 23 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
FROM golang:1.23.4-alpine3.21 AS builder

ARG GOPROXY=""
ENV GOSUMDB=off \
GO111MODULE=on

WORKDIR /workspace

COPY go.mod go.mod
COPY go.sum go.sum

COPY *.go ./
COPY connector/ connector/
COPY exporter/ exporter/
COPY receiver/ receiver/
COPY utils/ utils/

RUN go mod download -x

RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o qubership-open-telemetry-collector .

FROM alpine:3.21.0
COPY --from=builder --chown=10001:0 /workspace/qubership-open-telemetry-collector /otec
1 change: 1 addition & 0 deletions LICENSE
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@

Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
34 changes: 33 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1 +1,33 @@
# qubership-open-telemetry-collector
# Open-telemetry-collector

## Introduction

The implementation is based on [basic](https://github.com/open-telemetry/opentelemetry-collector) and [contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib) versions of open-telemetry-collector.
For more details about open-telemetry approach for processing traces, metrics and logs see [here](https://opentelemetry.io/docs/)
Open-telemetry-collector approach for configuration see [here](https://opentelemetry.io/docs/collector/configuration/)
New modules development approach see [here](https://opentelemetry.io/docs/collector/building/)

## Supported modules
The following third-party modules are supported by this implementation of open-telemetry-collector:
- [spanmetricsconnector](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/connector/spanmetricsconnector/README.md)
- [debugexporter](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/debugexporter/README.md)
- [otlpexporter](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/otlpexporter/README.md)
- [prometheusexporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/prometheusexporter/README.md)
- [loggingexporter](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/loggingexporter/README.md)
- [jaegerexporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.85.0/exporter/jaegerexporter)
- [healthcheckextension](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/extension/healthcheckextension/README.md)
- [pprofextension](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/extension/pprofextension/README.md)
- [batchprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/extension/pprofextension/README.md)
- [filterprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/filterprocessor/README.md)
- [probabilisticsamplerprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/probabilisticsamplerprocessor/README.md)
- [transformprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/transformprocessor/README.md)
- [otlpreceiver](https://github.com/open-telemetry/opentelemetry-collector/blob/main/receiver/otlpreceiver/README.md)
- [jaegerreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/jaegerreceiver/README.md)
- [zipkinreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/zipkinreceiver)

Also there are custom implementations for
- [sentryreceiver](receiver/sentryreceiver), see also the [document](docs/sentry-receiver.md#sentry-envelope-mapping-to-jaeger-traces)
- [sentrymetricsconnector](connector/sentrymetricsconnector), see also the [document](docs/sentry-receiver.md#sentry-envelope-to-metrics)
- [logtcpexporter](exporter/logtcpexporter), see also the [document](docs/sentry-receiver.md#sentry-envelope-to-logs-records-graylog-mapping)

All third-party and custom modules are listed in [componets.go](components.go)
15 changes: 15 additions & 0 deletions SECURITY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Security Reporting Process

Please, report any security issue to `opensourcegroup@netcracker.com` where the issue will be triaged appropriately.

If you know of a publicly disclosed security vulnerability please IMMEDIATELY email `opensourcegroup@netcracker.com`
to inform the team about the vulnerability, so we may start the patch, release, and communication process.

# Security Release Process

If the vulnerability is found in the latest stable release, then it would be fixed in patch version for that release.
E.g., issue is found in 2.5.0 release, then 2.5.1 version with a fix will be released.
By default, older versions will not have security releases.

If the issue doesn't affect any existing public releases, the fix for medium and high issues is performed
in a main branch before releasing a new version. For low priority issues the fix can be planned for future releases.
118 changes: 118 additions & 0 deletions components.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
// Copyright 2024 Qubership
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package main

import (
"go.opentelemetry.io/collector/component"
"go.opentelemetry.io/collector/connector"
"go.opentelemetry.io/collector/exporter"
"go.opentelemetry.io/collector/extension"
"go.opentelemetry.io/collector/otelcol"
"go.opentelemetry.io/collector/processor"
"go.opentelemetry.io/collector/receiver"
spanmetricsconnector "github.com/open-telemetry/opentelemetry-collector-contrib/connector/spanmetricsconnector"
sentrymetricsconnector "otec/connector/sentrymetricsconnector"
debugexporter "go.opentelemetry.io/collector/exporter/debugexporter"
otlpexporter "go.opentelemetry.io/collector/exporter/otlpexporter"
prometheusexporter "github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter"
loggingexporter "go.opentelemetry.io/collector/exporter/loggingexporter"
logtcpexporter "otec/exporter/logtcpexporter"
healthcheckextension "github.com/open-telemetry/opentelemetry-collector-contrib/extension/healthcheckextension"
pprofextension "github.com/open-telemetry/opentelemetry-collector-contrib/extension/pprofextension"
batchprocessor "go.opentelemetry.io/collector/processor/batchprocessor"
filterprocessor "github.com/open-telemetry/opentelemetry-collector-contrib/processor/filterprocessor"
probabilisticsamplerprocessor "github.com/open-telemetry/opentelemetry-collector-contrib/processor/probabilisticsamplerprocessor"
transformprocessor "github.com/open-telemetry/opentelemetry-collector-contrib/processor/transformprocessor"
otlpreceiver "go.opentelemetry.io/collector/receiver/otlpreceiver"
jaegerreceiver "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/jaegerreceiver"
zipkinreceiver "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/zipkinreceiver"
sentryreceiver "otec/receiver/sentryreceiver"
)

func components() (otelcol.Factories, error) {
var err error
factories := otelcol.Factories{}

factories.Extensions, err = extension.MakeFactoryMap(
healthcheckextension.NewFactory(),
pprofextension.NewFactory(),
)
if err != nil {
return otelcol.Factories{}, err
}
factories.ExtensionModules = make(map[component.Type]string, len(factories.Extensions))
factories.ExtensionModules[healthcheckextension.NewFactory().Type()] = "go.opentelemetry.io/collector/extension/github.com/open-telemetry/opentelemetry-collector-contrib/extension/healthcheckextension v0.106.1"
factories.ExtensionModules[pprofextension.NewFactory().Type()] = "go.opentelemetry.io/collector/extension/pprofextension v0.106.1"

factories.Receivers, err = receiver.MakeFactoryMap(
otlpreceiver.NewFactory(),
jaegerreceiver.NewFactory(),
zipkinreceiver.NewFactory(),
sentryreceiver.NewFactory(),
)
if err != nil {
return otelcol.Factories{}, err
}
factories.ReceiverModules = make(map[component.Type]string, len(factories.Receivers))
factories.ReceiverModules[otlpreceiver.NewFactory().Type()] = "go.opentelemetry.io/collector/receiver/otlpreceiver v0.106.1"
factories.ReceiverModules[jaegerreceiver.NewFactory().Type()] = "go.opentelemetry.io/collector/receiver/jaegerreceiver v0.106.1"
factories.ReceiverModules[zipkinreceiver.NewFactory().Type()] = "go.opentelemetry.io/collector/receiver/zipkinreceiver v0.106.1"
factories.ReceiverModules[sentryreceiver.NewFactory().Type()] = "otec/receiver/sentryreceiver v1.1.17"

factories.Exporters, err = exporter.MakeFactoryMap(
debugexporter.NewFactory(),
otlpexporter.NewFactory(),
prometheusexporter.NewFactory(),
loggingexporter.NewFactory(),
logtcpexporter.NewFactory(),
)
if err != nil {
return otelcol.Factories{}, err
}
factories.ExporterModules = make(map[component.Type]string, len(factories.Exporters))
factories.ExporterModules[debugexporter.NewFactory().Type()] = "go.opentelemetry.io/collector/exporter/debugexporter v0.106.1"
factories.ExporterModules[otlpexporter.NewFactory().Type()] = "go.opentelemetry.io/collector/exporter/otlpexporter v0.106.1"
factories.ExporterModules[prometheusexporter.NewFactory().Type()] = "go.opentelemetry.io/collector/exporter/prometheusexporter v0.106.1"
factories.ExporterModules[loggingexporter.NewFactory().Type()] = "go.opentelemetry.io/collector/exporter/loggingexporter v0.106.1"
factories.ExporterModules[logtcpexporter.NewFactory().Type()] = "otec/exporter/logtcpexporter v1.1.17"

factories.Processors, err = processor.MakeFactoryMap(
batchprocessor.NewFactory(),
filterprocessor.NewFactory(),
transformprocessor.NewFactory(),
probabilisticsamplerprocessor.NewFactory(),
)
if err != nil {
return otelcol.Factories{}, err
}
factories.ProcessorModules = make(map[component.Type]string, len(factories.Processors))
factories.ProcessorModules[batchprocessor.NewFactory().Type()] = "go.opentelemetry.io/collector/processor/batchprocessor v0.106.1"
factories.ProcessorModules[filterprocessor.NewFactory().Type()] = "go.opentelemetry.io/collector/processor/filterprocessor v0.106.1"
factories.ProcessorModules[transformprocessor.NewFactory().Type()] = "go.opentelemetry.io/collector/processor/transformprocessor v0.106.1"
factories.ProcessorModules[probabilisticsamplerprocessor.NewFactory().Type()] = "go.opentelemetry.io/collector/processor/probabilisticsamplerprocessor v0.106.1"

factories.Connectors, err = connector.MakeFactoryMap(
spanmetricsconnector.NewFactory(),
sentrymetricsconnector.NewFactory(),
)
if err != nil {
return otelcol.Factories{}, err
}
factories.ConnectorModules = make(map[component.Type]string, len(factories.Connectors))
factories.ConnectorModules[spanmetricsconnector.NewFactory().Type()] = "go.opentelemetry.io/collector/connector/spanmetricsconnector v0.106.1"
factories.ConnectorModules[sentrymetricsconnector.NewFactory().Type()] = "otec/connector/sentrymetricsconnector v1.1.17"

return factories, nil
}
39 changes: 39 additions & 0 deletions connector/sentrymetricsconnector/config.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
// Copyright 2024 Qubership
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package sentrymetricsconnector

type Config struct {
SentryMeasurementsCfg SentryMeasurementsConfig `mapstructure:"sentry_measurements"`
SentryEventCountCfg SentryEventCountConfig `mapstructure:"sentry_events"`
}

type SentryMeasurementsConfig struct {
DefaultBuckets []float64 `mapstructure:"default_buckets"`
DefaultLabels map[string]string `mapstructure:"default_labels"`
Custom map[string]*CustomSentryMeasurementsConfig `mapstructure:"custom"`
}

type CustomSentryMeasurementsConfig struct {
Buckets []float64 `mapstructure:"buckets"`
Labels *map[string]string `mapstructure:"labels"`
}

type SentryEventCountConfig struct {
Labels map[string]string `mapstructure:"labels"`
}

func (c *Config) Validate() error {
return nil
}
340 changes: 340 additions & 0 deletions connector/sentrymetricsconnector/connector.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,340 @@
// Copyright 2024 Qubership
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package sentrymetricsconnector

import (
"context"

"go.opentelemetry.io/collector/component"
"go.opentelemetry.io/collector/consumer"

"otec/connector/sentrymetricsconnector/metrics"
"otec/receiver/sentryreceiver/models"

"go.opentelemetry.io/collector/pdata/pcommon"
"go.opentelemetry.io/collector/pdata/pmetric"
"go.opentelemetry.io/collector/pdata/ptrace"
"go.opentelemetry.io/collector/connector"

"go.uber.org/zap"
)

const scopeName = "otelcol/sentrymetricsconnector"

// sentrymetrics can evaluate metrics based on sentry traces
// and emit them to a metrics pipeline.
type sentrymetrics struct {
config *Config
metricsConsumer consumer.Metrics
component.StartFunc
component.ShutdownFunc
logger *zap.Logger
counter int64
measurementsHist *metrics.CustomHistogram
measurementsBuckets map[string][]float64
defaultMeasurementsBuckets []float64
measurementsLabels map[string]map[string]string
defaultMeasurementsLabels map[string]string
}

func CreateSentryMetricsConnector(config *Config, metricsConsumer consumer.Metrics, set connector.Settings) *sentrymetrics {
result := sentrymetrics{}
result.config = config
result.metricsConsumer = metricsConsumer
result.logger = set.Logger
result.measurementsHist = metrics.NewCustomHistogram(set.Logger)
result.defaultMeasurementsBuckets = config.SentryMeasurementsCfg.DefaultBuckets
result.measurementsBuckets = make(map[string][]float64)
for k, v := range config.SentryMeasurementsCfg.Custom {
result.measurementsBuckets[k] = v.Buckets
}
result.defaultMeasurementsLabels = config.SentryMeasurementsCfg.DefaultLabels
result.logger.Sugar().Infof("DefaultMeasurementsLabels %+v", result.defaultMeasurementsLabels)
result.measurementsLabels = make(map[string]map[string]string)
for k, v := range config.SentryMeasurementsCfg.Custom {
if v.Labels != nil {
result.measurementsLabels[k] = *v.Labels
result.logger.Sugar().Infof("Custom measurementsLabels for %v : %+v", k, result.measurementsLabels[k])
}
}
return &result
}

func (c *sentrymetrics) Capabilities() consumer.Capabilities {
return consumer.Capabilities{MutatesData: false}
}

func (c *sentrymetrics) ConsumeTraces(ctx context.Context, td ptrace.Traces) error {
countMetrics := pmetric.NewMetrics()
resourceMetric := countMetrics.ResourceMetrics().AppendEmpty()
scopeMetrics := resourceMetric.ScopeMetrics().AppendEmpty()
c.calculateSessionCountMetric(scopeMetrics.Metrics().AppendEmpty(), td)
c.calculateEventCountMetric(scopeMetrics.Metrics().AppendEmpty(), td)
c.calculateMeasurementsMetric(scopeMetrics.Metrics().AppendEmpty(), td)
return c.metricsConsumer.ConsumeMetrics(ctx, countMetrics)
}

func (c *sentrymetrics) calculateSessionCountMetric(metric pmetric.Metric, td ptrace.Traces) error {
metric.SetName("sentry_session_exited_count")
metric.SetDescription("The metric counts total number of sessions")
sum := metric.SetEmptySum()
sum.SetAggregationTemporality(1)
sum.SetIsMonotonic(true)
dataPoints := sum.DataPoints()

rss := td.ResourceSpans()
for i := 0; i < rss.Len(); i++ {
rs := rss.At(i)
ilss := rs.ScopeSpans()
for j := 0; j < ilss.Len(); j++ {
spans := ilss.At(j).Spans()
for k := 0; k < spans.Len(); k++ {
span := spans.At(k)
var envelopTypeInt int64
var sessionStatusStr string
envelopType, ok := span.Attributes().Get("sentry.envelop.type.int")
if ok {
envelopTypeInt = envelopType.Int()
}
if envelopTypeInt != models.ENVELOP_TYPE_SESSION {
continue
}
sessionStatus, ok := span.Attributes().Get("session.status")
if ok {
sessionStatusStr = sessionStatus.AsString()
}
if sessionStatusStr != "exited" {
continue
}
var serviceNameStr string
serviceName, ok := span.Attributes().Get("service.name")
if ok {
serviceNameStr = serviceName.AsString()
}
dataPoint := dataPoints.AppendEmpty()
dataPoint.Attributes().PutStr("service_name", serviceNameStr)
dataPoint.SetDoubleValue(1.0)
}
}
}

return nil
}

func (c *sentrymetrics) calculateEventCountMetric(metric pmetric.Metric, td ptrace.Traces) error {
metric.SetName("sentry_event_count")
metric.SetDescription("The metric counts total number of events by level")
sum := metric.SetEmptySum()
sum.SetAggregationTemporality(1)
sum.SetIsMonotonic(true)
dataPoints := sum.DataPoints()
labelsToExtract := c.config.SentryEventCountCfg.Labels

rss := td.ResourceSpans()
for i := 0; i < rss.Len(); i++ {
rs := rss.At(i)
ilss := rs.ScopeSpans()
for j := 0; j < ilss.Len(); j++ {
spans := ilss.At(j).Spans()
for k := 0; k < spans.Len(); k++ {
span := spans.At(k)
var envelopTypeInt int64
envelopType, ok := span.Attributes().Get("sentry.envelop.type.int")
if ok {
envelopTypeInt = envelopType.Int()
}
if envelopTypeInt != models.ENVELOP_TYPE_EVENT {
continue
}
dataPoint := dataPoints.AppendEmpty()
dataPoint.SetDoubleValue(1.0)
labels := c.getLabels(span, labelsToExtract)
for labelName, labelValue := range labels {
dataPoint.Attributes().PutStr(labelName, labelValue)
}
}
}
}

return nil
}

func (c *sentrymetrics) calculateMeasurementsMetric(metric pmetric.Metric, td ptrace.Traces) error {
rss := td.ResourceSpans()
for i := 0; i < rss.Len(); i++ {
rs := rss.At(i)
ilss := rs.ScopeSpans()
for j := 0; j < ilss.Len(); j++ {
spans := ilss.At(j).Spans()
for k := 0; k < spans.Len(); k++ {
span := spans.At(k)
var envelopTypeInt int64
envelopType, ok := span.Attributes().Get("sentry.envelop.type.int")
if ok {
envelopTypeInt = envelopType.Int()
}
if envelopTypeInt != models.ENVELOP_TYPE_TRANSACTION {
continue
}
var measurementsCommonMap pcommon.Map
measurements, ok := span.Attributes().Get("measurements")
if ok {
measurementsCommonMap = measurements.Map()
}
configurableLabels := c.getConfigurableMeasurementLabels(span, "")
c.logger.Sugar().Debugf("SentryMetricsConnector : GOT TRANSACTION with measurements size=%v, configurableLabels=%+v", measurementsCommonMap.Len(), configurableLabels)
measurementsCommonMap.Range(func(k string, v pcommon.Value) bool {
labels := make(map[string]string)
labels["type"] = k
if c.measurementsLabels[k] != nil {
customConfigurableLabels := c.getConfigurableMeasurementLabels(span, k)
for k, v := range customConfigurableLabels {
labels[k] = v
}
} else {
for k, v := range configurableLabels {
labels[k] = v
}
}
var measurementFloat float64
var unitStr string
val, okVal := v.Map().Get("value")
if okVal {
measurementFloat = val.Double()
}
unit, okUnit := v.Map().Get("unit")
if okUnit {
unitStr = unit.AsString()
}

c.logger.Sugar().Debugf("SentryMetricsConnector : Measurements datapoint : labels=%+v, value=%v, unitStr=%v", labels, measurementFloat, unitStr)
if okVal {
c.measurementsHist.ObserveSingle(normalizeUnit(measurementFloat, unitStr), c.getMeasurementBuckets(k), labels)
} else {
c.logger.Error("SentryMetricsConnector : Error reading measurements value")
}
return true
})
labels := make(map[string]string)
labels["type"] = "transaction_duration"
if c.measurementsLabels["transaction_duration"] != nil {
customConfigurableLabels := c.getConfigurableMeasurementLabels(span, "transaction_duration")
for k, v := range customConfigurableLabels {
labels[k] = v
}
} else {
for k, v := range configurableLabels {
labels[k] = v
}
}
var unitStr string
durationFloat := float64(span.EndTimestamp().AsTime().Sub(span.StartTimestamp().AsTime()).Milliseconds())
c.logger.Sugar().Debugf("SentryMetricsConnector : Measurements datapoint : labels=%+v, value=%v, unitStr=%v", labels, durationFloat, unitStr)
c.measurementsHist.ObserveSingle(normalizeUnit(durationFloat, unitStr), c.getMeasurementBuckets("transaction_duration"), labels)
}
}
}
c.measurementsHist.UpdateDataPoints(metric)

return nil
}

func (c *sentrymetrics) getConfigurableMeasurementLabels(span ptrace.Span, measurementType string) map[string]string {
var labelsToExtract map[string]string
if measurementType == "" {
labelsToExtract = c.defaultMeasurementsLabels
} else {
labelsToExtract = c.measurementsLabels[measurementType]
}
if labelsToExtract == nil {
labelsToExtract = c.defaultMeasurementsLabels
c.measurementsLabels[measurementType] = labelsToExtract
c.logger.Sugar().Debugf("Set default labelsToExtract %+v for measurement %v", labelsToExtract, measurementType)
}

return c.getLabels(span, labelsToExtract)
}

func (c *sentrymetrics) getLabels(span ptrace.Span, labelsToExtract map[string]string) map[string]string {
result := make(map[string]string)
for labelName, labelPath := range labelsToExtract {
labelValue, ok := span.Attributes().Get(labelPath)
if ok {
result[labelName] = labelValue.AsString()
} else {
result[labelName] = ""
}
}

return result
}

func (c *sentrymetrics) getMeasurementBuckets(measurementType string) []float64 {
buckets := c.measurementsBuckets[measurementType]
if len(buckets) == 0 {
buckets = c.defaultMeasurementsBuckets
c.measurementsBuckets[measurementType] = buckets
c.logger.Sugar().Debugf("Set default buckets %+v for measurement %v", buckets, measurementType)
}
return buckets
}

func normalizeUnit(val float64, unit string) float64 {
switch unit {
case "millisecond", "byte", "none", "ratio", "":
return val
case "percent":
return val/100
case "microsecond":
return val/1000
case "nanosecond":
return val/1000_000
case "second", "kilobyte":
return val * 1000
case "minute":
return val * 1000 * 60
case "hour":
return val * 1000 * 60 * 60
case "day":
return val * 1000 * 60 * 60 * 24
case "week":
return val * 1000 * 60 * 60 * 24 * 7
case "bit":
return val / 8
case "megabyte":
return val * 1000_000
case "gigabyte":
return val * 1000_000_000
case "terabyte":
return val * 1000_000_000_000
case "petabyte":
return val * 1000_000_000_000_000
case "exabyte":
return val * 1000_000_000_000_000_000
case "kibibyte":
return val * 1024
case "mebibyte":
return val * 1024 * 1024
case "gibibyte":
return val * 1024 * 1024 * 1024
case "tebibyte":
return val * 1024 * 1024 * 1024 * 1024
case "pebibyte":
return val * 1024 * 1024 * 1024 * 1024 * 1024
case "exbibyte":
return val * 1024 * 1024 * 1024 * 1024 * 1024 * 1024
}
return val
}
53 changes: 53 additions & 0 deletions connector/sentrymetricsconnector/factory.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
// Copyright 2024 Qubership
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package sentrymetricsconnector

import (
"context"

"go.opentelemetry.io/collector/component"
"go.opentelemetry.io/collector/connector"
"go.opentelemetry.io/collector/consumer"
)

const (
typeStr = "sentrymetrics"
)

func NewFactory() connector.Factory {
return connector.NewFactory(
component.MustNewType(typeStr),
createDefaultConfig,
connector.WithTracesToMetrics(createTracesToMetrics, component.StabilityLevelDevelopment),
)
}

func createDefaultConfig() component.Config {
return &Config{
SentryMeasurementsCfg: SentryMeasurementsConfig{
DefaultBuckets: []float64{100, 1000, 5000},
},
}
}

// createTracesToMetrics creates a traces to metrics connector based on provided config
func createTracesToMetrics(
_ context.Context,
set connector.Settings,
cfg component.Config,
nextConsumer consumer.Metrics,
) (connector.Traces, error) {
return CreateSentryMetricsConnector(cfg.(*Config), nextConsumer, set), nil
}
93 changes: 93 additions & 0 deletions connector/sentrymetricsconnector/metrics/histogram.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
// Copyright 2024 Qubership
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package metrics

import (
"otec/utils"
"sync"
"go.opentelemetry.io/collector/pdata/pmetric"
"go.uber.org/zap"
)

type CustomHistogram struct {
sync.RWMutex
stateMap map[string]*CurrentHistogramState
logger *zap.Logger
}

type CurrentHistogramState struct {
Sum float64
Cnt uint64
Buckets map[float64]uint64
BucketList []float64
Labels map[string]string
}

func NewCustomHistogram(logger *zap.Logger) *CustomHistogram {
customHistogram := CustomHistogram{}
customHistogram.stateMap = make(map[string]*CurrentHistogramState)
customHistogram.logger = logger
return &customHistogram
}

func (h *CustomHistogram) ObserveSingle(val float64, bucketList []float64, labels map[string]string) {
h.Lock()
defer h.Unlock()
histKey := utils.MapToString(labels)
if h.stateMap[histKey] == nil {
histState := &CurrentHistogramState{
Sum: 0,
Cnt: 0,
Labels: labels,
BucketList: bucketList,
Buckets: make(map[float64]uint64),
}
for _, b := range bucketList {
histState.Buckets[b] = 0
}
h.stateMap[histKey] = histState
}

h.stateMap[histKey].Sum += val
h.stateMap[histKey].Cnt += 1
for _,b := range bucketList {
if val <= b {
h.stateMap[histKey].Buckets[b]++
break
}
}
}

func (h *CustomHistogram) UpdateDataPoints(metric pmetric.Metric) {
h.Lock()
defer h.Unlock()
metric.SetName("sentry_measurements_statistic")
metric.SetDescription("The metric shows sentry measurements statistic")
metric.SetUnit("millisecond")
hist := metric.SetEmptyHistogram()
hist.SetAggregationTemporality(2)
dataPoints := hist.DataPoints()

for _, v := range h.stateMap {
dataPoint := dataPoints.AppendEmpty()
dataPoint.SetSum(v.Sum)
dataPoint.SetCount(v.Cnt)
dataPoint.ExplicitBounds().FromRaw(v.BucketList)
dataPoint.BucketCounts().FromRaw(utils.GetOrderedMapValuesFloat64Uint64(v.Buckets, v.BucketList))
for label, labelValue := range v.Labels {
dataPoint.Attributes().PutStr(label, labelValue)
}
}
}
5 changes: 5 additions & 0 deletions docker-transfer/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
FROM scratch

# Collect helm charts and documentation
COPY helm-templates/ /helm-templates
COPY docs/ /docs
Binary file added docs/img/Architecture.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
42 changes: 42 additions & 0 deletions docs/installation-notes.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
## Deployment Configuration

See below the full list of parameters which defines the Open Telemetry Collector deployment mode and is used during the installation. Mandatory parameters from the list must be specified for Open Telemetry Collector in helm.

| Parameter | Mandatory | Default | Value Example |Description |
|---------------------------|------------|-----------------------------|-----------------------------|---------------------------------------------------------------------------------|
| OTEC_INSTALLATION_MODE | N | SPAN_METRICS_PROCESSOR | `SPAN_METRICS_PROCESSOR` | The type of OOB configuration (SPAN_METRICS_PROCESSOR or SENTRY_ENVELOPES_PROCESSING) |
| SERVICE_NAME | N | open-telemetry-collector | `open-telemetry-collector` | The Kubernetes service name for Open Telemetry Collector. |
| MONITORING_ENABLED | N | false | `false` | Disables and enables the service monitor. |
| resources | N | - | | Set up this object to use custom profiles configuration. |
| imagePullSecrets | N | - | | K8s imagePullSecrets for private registry access. |
| labels | N | - | | The Kubernetes labels setup. |
| entrypoint | N | /otelcol-contrib | /otelcol-contrib | The path to the opentelemetry binary. |
| JAEGER_COLLECTOR_HOST | N | - | jaeger-app-collector.jaeger | The Jaeger hostname or IP. |
| JAEGER_COLLECTOR_<br>OTLP_PORT | N | 4317 | 4317 | The port for the Jaeger service OpenTelemetry Protocol span reporting API. |
| TRACING_HOST | N | - | jaeger-app-collector.jaeger | Jaeger URL. It is used if JAEGER_COLLECTOR_HOST is not set. |
| GRAYLOG_COLLECTOR_HOST | N | https://graylog-logging.${CLOUD_PUBLIC_HOST} | | The Graylog hostname or IP to which OTeC sends data from traces |
| GRAYLOG_COLLECTOR_PORT | N | 12201 | 12201 | The port for the Graylog service. |
| GRAYLOG_UI_URL | N | https://graylog-logging.${CLOUD_PUBLIC_HOST} | | The Graylog hostname or IP. It is used if GRAYLOG_COLLECTOR_HOST is not set. |
| CONFIG_MAP | N | - | | Config map customization. |
| SERVICE_PORTS | N | - | | Customization for service.ports. |
| SERVICE_MONITOR_PORT_NAME | N | exporter-prom | exporter-prom | Customization for ServiceMonitor port. |
| LATENCY_HISTOGRAM_BUCKETS | N | [2ms, 4ms, 6ms, 8ms, 10ms, 50ms, 100ms, 200ms, 400ms, 800ms, 1s, 1400ms, 2s, 5s, 10s, 15s] | [100ms, 1s, 10s] | The list of durations defining the latency histogram buckets. If it is not set, the default list is used. |
| CLOUD_TOPOLOGIES | N | - | | Array of topology settings for topologySpreadConstraints. Each array item must contain at least 'topologyKey' attribute. Other supported attributes are 'maxSkew' and 'whenUnsatisfiable', which are optional. This parameter has higher priority over CLOUD_TOPOLOGY_KEY. Should not be an empty list. |
| CLOUD_TOPOLOGY_KEY | N | kubernetes.io/hostname | | Defines topologyKey in topologySpreadConstraints. This is a BWC parameter. |
| DEPLOYMENT_STRATEGY_<br>TYPE | N | | | The Kubernetes rolling update deployment strategy. Possible values are "recreate", "best_effort_controlled_rollout", "ramped_slow_rollout", and "custom_rollout". |
| DEPLOYMENT_STRATEGY_<br>MAXSURGE | N | 25% | 25% | The parameter sets maxSurge if DEPLOYMENT_STRATEGY_TYPE is "custom_rollout". |
| DEPLOYMENT_STRATEGY_<br>MAXUNAVAILABLE | N | 25% | 25% | The parameter sets maxUnavailable if DEPLOYMENT_STRATEGY_TYPE is "custom_rollout". |
| LOG_LEVEL | N | info | info | The parameter indicates the OTeC log level. The possible values are "debug", "info", "warn", and "error". |
| OTEC_LOG_FORMAT | N | json | text | The parameter allows to specify log format. It might be convenient to use text format for dev purposes. Json is strongly recommended on prod. |
| OTEC_SENTRY_<br>ENVELOPES_INGRESS_<br>ENABLED | N | false | true | The parameter allows to enable the default sentry ingress. |
| OTEC_SENTRY_<br>ENVELOPES_INGRESS_<br>ANNOTATIONS | N | | string map in yaml format | The parameter allows to specify the annotations map for the sentry ingress. |
| OTEC_SENTRY_<br>RECEIVER_PARAMETERS | N | | Object | The parameter allows to customize sentry receiver parameters. |
| OTEC_SENTRY_<br>EVENT_METRICS_PARAMETERS | N | | Object | The parameter allows to customize setry event metrics parameters. |
| OTEC_SENTRY_<br>MEASUREMENTS_METRICS_PARAMETERS | N | | Object | The parameter allows to customize setry measurements metrics parameters. |
| OTEC_LOGTCPEXPORTER<br>_PARAMETERS | N | | Object | The parameter allows to customize logtcpexporter parameters. |
| OTEC_HEALTH_<br>CHECK_PORT | N | 13133 | 13133 | The parameter allows to customize OTeC health check port wich is used for liveness and readiness probes |
| OTEC_ENABLE_<br>ARBITRARY_TRACES_<br>LOGGING | N | false | false | The parameter allows to enable arbitrary traces logging. |
| OTEC_ARBITRARY_<br>TRACES_LOGGING_<br>CONFIG | N | | Object | The parameter allows to customize arbitrary traces logging configuration. |

**Note:** Open Telemetry Collector preprocesses traces before they come to the Jaeger storage. For proper work, a trace generating application must be configured to send the traces not to Jaeger directly, but to the Open Telemtry Collector endpoint. Open Telemetry Collector receives the traces, generates the metrics on their basis, then resends the traces to Jaeger. The generated Prometheus metrics are later available for pulling via the service monitor.
Jaeger endpoint configuration is the essential part of the OTeC configuration in helm. Note, that if the parameter JAEGER_COLLECTOR_HOST is not empty, the traces are sent to JAEGER_COLLECTOR_HOST:JAEGER_COLLECTOR_PORT URL. If the helm parameter JAEGER_COLLECTOR_HOST is empty, URL is taken from the helm parameter TRACING_HOST. If the helm parameters TRACING_HOST and JAEGER_COLLECTOR_HOST are both empty, then the traces are sent to the default URL jaeger-app-collector.jaeger:JAEGER_COLLECTOR_PORT. The default value for JAEGER_COLLECTOR_PORT is 14250.
147 changes: 147 additions & 0 deletions docs/sentry-receiver.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
# Sentry receiver specification

## Envelope

**Envelope** - http/grpc/etc request with body in [ndjson](https://ndjson.org/libraries.html) format.

Usually it has structure like:

```json5
{ ... } // header - contains information who and how send that event.
{ "type": "event|transaction|session..." } // message type (json with one field)
{ ... } // content of envelope.
```

### Envelope Types

Sentry uses 3 types of messages (envelopes) to deliver metrics, traces, exceptions and etc.:

- **`session`** - envelope of this type means that user started a `session` on the page. This event just indicates that a User-Agent opened the page.
- **`transaction`** - this envelope contains `spans`, `breadcrumbs`, `measurements` and many other information. Usually transaction has start time and end time.
- **`event`** - this envelope contains information about errors, exceptions or manually triggered `events` in the single point of time.

## Response types

Sentry SDK produces _Envelopes_ to the http endpoint. `open-telemetry-collector` should respond with correct response

- `type: "session"`

```json
{}
```

- `type: "event|transaction"`

```json
{ "id": "d73ca72181e440ee94ff7782ceca65c5" } // event_id from envelope header
```

## Sentry Envelope mapping to Jaeger traces

In the table below you can find mapping for fields of Sentry envelopes of types **event** and **transaction** to the opentelemetry trace attributes.

| _Event_ or _Transaction_ field/HTTP | Otel Trace | Description | Envelope Event | Comment |
| ----------------------------------- | ------------------------------------------------------ | ----------------------------- | -------------- | -------------------------------------------------------------------- |
| `{transaction} {context.trace.op}` | `rootSpan.name` | The trace name | `transaction` | |
| `request.headers['x-service-name']` | `"service.name"` | Service name | any | supposed that `x-service-id` will contain a name of the microservice |
| `environment` | `environment` | The name of the telemetry SDK | any | |
| `measurements.*` | `measurements.{measurements.value} {mesurements.unit}` | - | any | |
| `start_timestamp` | `rootSpan.start_time_unix_nano` | - | `transaction` | |
| `timestamp` | `end_time_unix_nano` | - | any | |
| `context.trace.trace_id` | `trace_id` | - | any | |
| `context.trace.span_id` | `span_id` | - | any | |
| `transaction` | `transaction` | - | any | |
| `dist` | `dist` | - | any | |

In the table below you can find mapping of Sentry **spans** fields to the attributes of opentelemetry spans:

| Sentry `span` | Otel Span | Description | Comment |
| ---------------------- | --------------------------- | ----------- | ------- |
| `span.start_timestamp` | `span.start_time_unix_nano` | - | |
| `span.timestamp` | `span.end_time_unix_nano` | - | |
| `span.trace_id` | `span.trace_id` | | |
| `span.span_id` | `span.span_id` | | |
| `span.parent_span_id` | `span.parent_span_id` | | |
| `span.op` | `span.name` | - | |
| `span.status` | `span.status.code` | - | |
| `span.data[*]` | `span.[*]` | - | |
| `span.origin` | `span.origin` | - | |
| `span.description` | `span.description` | - | |

## Sentry Envelope to Logs records (Graylog mapping)

LogTCP Exporter allows to log certain data from sentry envelopes to the Graylog. For now only sentry envelopes of event type can be logged:

### `type: "event"`

| Event field | Graylog field | Description | Comment |
| ------------------------------------- | ------------------------------ | --------------------------------------------------------------------------------------- | -------------------------------------------------------- |
| `contexts.trace.span_id` | `span_id` | | |
| `contexts.trace.trace_id` | `trace_id` | | |
| `'frontend'` | `component` | | |
| `level` (need to map to the level_id) | `level` | | |
| `'open-telemetry-collector'` | `facility` | | |
| `{sdk.name}@{sdk.version}` | `sdk`? | | |
| `message` or `context.Error.*` or `exception.values` or constant `empty_message` | `message` | | |
| `exception.values` | `stacktrace` or `full_message` | `exception.values` - list of chained exceptions, they are should be joined to one line. | https://develop.sentry.dev/sdk/event-payloads/exception/ |
| `timestamp` | `time`,`timestamp` | | |
| `event_id` | `event_id` | | |
| `release` or constant `empty_version` | `version` | | |
| `req.headers.x-service-id` | `name` | | |
| `platform` | `platform` | | |
| `user.id` | `user_id` | | _new field_ |
| `tags.transaction` | `transaction` | | _new field_ |
| `logger` | `category` | if logger present, use it in category field. Otherwise use `frontend-event` value | _new field_ |
| `request.url` | `url` | | _new field_ |
| `request.headers.User-Agent` | `browser` | | _new field_ |

In case **`event.level == "error"`** the `breadcrumbs` of `event` are also logged as separate log records.

All fields from `event` should be the same, but they can be overridden by the `breadcrumb` field.

#### `breadcrumb type: "http"`

| Event field | Graylog field | Description | Comment |
| ------------------------------------------------ | ------------------ | --------------- | ------- |
| `breadcrumb?.level` | `level` | only if present | |
| `breadcrumb.timestamp` | `time`,`timestamp` | | |
| `breadcrumb.category` | `category` | | |
| `{breadcrumb.data.method} {breadcrumb.data.url}` | `message` | | |
| `breadcrumb.data.status_code` | `status` | | |

#### `breadcrumb category: "navigation"`

| Event field | Graylog field | Description | Comment |
| -------------------------------------------------------------------------- | ------------------ | ----------- | ------- |
| `breadcrumb.timestamp` | `time`,`timestamp` | | |
| `breadcrumb.category` | `'navigation'` | | |
| `Browser navigation from: {breadcrumb.data.from} to: {breadcrumb.data.to}` | `message` | | |

#### `breadcrumb category: "console"`

| Event field | Graylog field | Description | Comment |
| ----------------------------- | ------------------ | ----------- | ------------------------- |
| `breadcrumb?.level` | `level` | | Maybe skip `info` levels? |
| `breadcrumb.timestamp` | `time`,`timestamp` | | |
| `breadcrumb.category` | `'console'` | | |
| `breadcrumb.message` | `message` | | |

### `type: "session"`

Envelopes with type `session` are not logged to the logging system.

## Sentry Envelope to Metrics

SentryMetrics Connector allows to generate metrics for each type of sentry envelopes below:

### `type: "session"`

- sentry_session_exited_count - allows to monitor amount of unique sessions with `status: "exited"` when session ends.

### `type: "transaction"`

- sentry_measurements_statistic - allows to monitor Browser Web Vitals - measurements and duration of transactions - for each `{transaction} {context.trace.op}`.

### `type: "event"`

- sentry_event_count - allows to monitor amount of sentry events by `event.level`
90 changes: 90 additions & 0 deletions docs/user-guide.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
# Open-telemetry-collector

## Introduction

The implementation is based on [basic](https://github.com/open-telemetry/opentelemetry-collector) and [contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib) versions of open-telemetry-collector.
For more details about open-telemetry approach for processing traces, metrics and logs see [here](https://opentelemetry.io/docs/)
Open-telemetry-collector approach for configuration see [here](https://opentelemetry.io/docs/collector/configuration/)
New modules development approach see [here](https://opentelemetry.io/docs/collector/building/)

## Supported modules
The following third-party(contrib) modules are supported by this implementation of open-telemetry-collector:
- [spanmetricsconnector](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/connector/spanmetricsconnector/README.md)
- [debugexporter](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/debugexporter/README.md)
- [otlpexporter](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/otlpexporter/README.md)
- [prometheusexporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/prometheusexporter/README.md)
- [loggingexporter](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/loggingexporter/README.md)
- [jaegerexporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.85.0/exporter/jaegerexporter)
- [healthcheckextension](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/extension/healthcheckextension/README.md)
- [pprofextension](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/extension/pprofextension/README.md)
- [batchprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/extension/pprofextension/README.md)
- [filterprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/filterprocessor/README.md)
- [probabilisticsamplerprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/probabilisticsamplerprocessor/README.md)
- [transformprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/transformprocessor/README.md)
- [otlpreceiver](https://github.com/open-telemetry/opentelemetry-collector/blob/main/receiver/otlpreceiver/README.md)
- [jaegerreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/jaegerreceiver/README.md)
- [zipkinreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/zipkinreceiver)

Also there are custom implementations for
- [sentryreceiver](receiver/sentryreceiver), see also the [document](sentry-receiver.md#sentry-envelope-mapping-to-jaeger-traces)
- [sentrymetricsconnector](connector/sentrymetricsconnector), see also the [document](sentry-receiver.md#sentry-envelope-to-metrics)
- [logtcpexporter](exporter/logtcpexporter), see also the [document](sentry-receiver.md#sentry-envelope-to-logs-records-graylog-mapping)

All third-party and custom modules are listed in [componets.go](../components.go)

# Use cases
## Span Metrics Connector
### Overview
[Span Metrics Connector](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/spanmetricsconnector) :
Aggregates Request, Error and Duration (R.E.D) metrics from span data. Earlier this module was implemented as a processor, so the module can be named as "Span Metrics Processor"

### Architecture
Traces source send traces to [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/). OpenTelemetry Collector later send traces to Jaeger, generate metrics for Prometheus and write logs to the Graylog with use of contrib and custom components, see the diagram below:

![Open-telemetry-collector Architecture](img/Architecture.png)

## Sentry Envelopes Processing
### Overview
[Sentry Envelopes Processing](../receiver/sentryreceiver) : OTeC receives Sentry envelops via custom receiver, aggregates Request, Error and Duration (R.E.D) metrics from envelopes data, transforms sentry envelopes to Jaeger traces and pushes them to Jaeger backend.
### Architecture
Sentry SDK collects browser events and sends them as Sentry envelopes to custom sentry-receiver of the open-telemetry-collector-contrib. Custom sentry receiver transforms the received data to the inner open-telemetry-collector format. Events are aggregated by Custom SentryMetrics Connector. Evaluated metrics are exported to the :8889/metrics endpoint by contrib Prometheus Exporter. Traces, evaluated on the basis of the events are sent to the Jaeger, which URL is configured by JAEGER_COLLECTOR_HOST and JAEGER_COLLECTOR_PORT helm parameters via contrib Jaeger Exporter. Logs, which are also evaluated on the basis of the traces, are sent to the graylog via custom LogTCP Exporter to the destination, which can be configured in helm via parameters GRAYLOG_COLLECTOR_HOST and GRAYLOG_COLLECTOR_PORT.

![Open-telemetry-collector Architecture](img/Architecture_SentryEnvelopesProcessing.PNG )

## Custom modules configuration
General approach for the configuration is described [here](https://opentelemetry.io/docs/collector/configuration/).
See below the custom modules configuration description.

### Sentry Receiver

* `endpoint` (`required`) - Contains a string with the port number on which sentry-receiver is listening for input sentry-envelopes.
* `http-query-param-values-to-attrs` (`optional`) - list of the url's http query parameters which must be used as open-telemetry-collector attributes named http.qparam.<query_parameter_name>. The value of the attribiute is the value of http query parameter. The URL is taken from sentry envelope by jsonpath "request.url".
* `http-query-param-existence-to-attrs` (`optional`) - list of the url's http query parameters which must be used as open-telemetry-collector attributes named http.qparam.<query_parameter_name>. The value of the attribiute is the `true`, if the http query parameter exists or `false`, if this parameter doesn't exist. The URL is taken from sentry envelope by jsonpath "request.url".
* `level-evaluation-strategy` (`optional`) - a string with possible value `breadcrumb`. If the value is `breadcrumb`, the values from sentry envelope breadcrumbs are taken into consideration during level evaluation. By default, level is taken from sentry envelope by "level" jsonpath.
* `context-span-attributes-list` (`optional`) - a list of strings, each string is used as a key to the entity inside the context of the sentry envelop. The sentry envelop context is the map which contains arbitrary datastractures, filled in by the browser application. The entity, which is referred in the list must be a string or a map with a key string and key value. If the context entity is a string, this string is put to the value of contexts.<context_name> attribute. If the context entity is a map with string key and string value, each value of the map is put to the value of contexts.<context_name>.<map_key> attribute.

### Sentrymetrics Connector

* `sentry_measurements` (`optional`) - Contains settings for sentry_measurements Prometheus metric
* `default_buckets` (`optional`) - Contains a list of float values which are defining default buckets for the mesurements histograms. For the particular measurement buckets can be overwritten in custom section.
* `default_labels` (`optional`) - Contains a map, in which a key is the label name and a value is the name of the open-telemetry-collector attribute, from which the label value must be taken. For the particular measurement the map can be overwritten in custom section.
* `custom` (`optional`) - Contains a map, in which a key is the measurement name and a value is the data structure which contains `buckets` list and `labels` map for this particular measurement. This map allows to overwrite the `default_buckets` and `default_labels` settings respectively for the particular measurement.
* `sentry_events` (`optional`) - Contains settings for sentry_events Prometheus metric
* `labels` (`optional`) - Contains a map, in which a key is the label name and a value is the name of the open-telemetry-collector attribute, from which the label value must be taken.

### Logtcp Exporter

* `endpoint` (`required`) - Contains host and port for the Graylog TCP destination
* `arbitrary-traces-logging` (`optional`) - Contains settings for arbitrary traces logging (ATL). ATL allows to log the traces with certain attributes. ATL can work in two modes : span mode and trace mode. In the first mode certain attributes of the spans are logging to the graylog. In the second mode the whole trace is logged to the graylog.
* `span-filters` (`optional`) - Contains a list of ATL filters for spans. Spans which satisfy at least one filter will be sent to the graylog. Filter consists of 2 conditions and a mapping below:
* `service-names` - a list of service names. A span must be produced by one of the services in the list
* `tags` - a map of attributes and values which span must have to be logged. If at least one attribute is not matched, the span is not logged.
* `mapping` - map with string key and list of strings value. The parameter contains a mapping between graylog field name and a list of span attributes which must be recorded in the field. Specific naming for graylog fields is supported: `__message__`, `__host__`, `__timestamp__`. Specific names for non-attribute data inside spans are also supported: `__spanId__`, `__traceId__`, `__name__`, `__end_timestamp__`, `__start_timestamp__`, `__kind__`, `__parentSpanId__`, `__startTime__`, `__endTime__`.
* `trace-filters` (`optional`) - Contains a list of ATL filters for traces. Trace which satisfy at least one filter will be sent to the graylog. Filter consists of two conditions below.
* `service-names` - a list of service names. A trace must be produced by one of the services in the list
* `tags` - a map of attributes and values which trace must have to be logged. If at least one attribute is not matched, the trace is not logged.
* `connection-pool-size` (`optional`) - Connection pool size for the graylog. Default value is 1.
* `queue-size` (`optional`) - Size of the queue of messages before sendining them to the graylog.
* `max-message-send-retry-count` (`optional`) - message is skipped after this number of retries to send it to the graylog. Default value is 1.
* `max-successive-send-error-count` (`optional`) - the number of successive send errors to the graylog after which open-telemetry-collector stops sending messages to the graylog for a `successive-send-error-freeze-time` time period. Default value is 5.
* `successive-send-error-freeze-time` (`optional`) - The time period for which open-telemetry-collector stops sending messages to the graylog after `max-successive-send-error-count` successive send errors to the graylog. The time perod is set in golang duration format. Default value is "1m" - 1 minute.
142 changes: 142 additions & 0 deletions exporter/logtcpexporter/atl/atlmarshaller/tracestologsmarshaller.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
// Copyright 2024 Qubership
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package atlmarshaller

import (
"bytes"
"fmt"
"strings"

"go.opentelemetry.io/collector/pdata/pcommon"
"go.opentelemetry.io/collector/pdata/ptrace"
)

type dataBuffer struct {
buf bytes.Buffer
}

func (b *dataBuffer) logEntry(format string, a ...any) {
b.buf.WriteString(fmt.Sprintf(format, a...))
b.buf.WriteString("\n")
}

func (b *dataBuffer) logAttr(attr string, value any) {
b.logEntry(" %-15s: %s", attr, value)
}

func (b *dataBuffer) logAttributes(header string, m pcommon.Map) {
if m.Len() == 0 {
return
}

b.logEntry("%s:", header)
attrPrefix := " ->"

headerParts := strings.Split(header, "->")
if len(headerParts) > 1 {
attrPrefix = headerParts[0] + attrPrefix
}

m.Range(func(k string, v pcommon.Value) bool {
b.logEntry("%s %s: %s", attrPrefix, k, valueToString(v))
return true
})
}

func (b *dataBuffer) logInstrumentationScope(il pcommon.InstrumentationScope) {
b.logEntry(
"InstrumentationScope %s %s",
il.Name(),
il.Version())
b.logAttributes("InstrumentationScope attributes", il.Attributes())
}

func (b *dataBuffer) logEvents(description string, se ptrace.SpanEventSlice) {
if se.Len() == 0 {
return
}

b.logEntry("%s:", description)
for i := 0; i < se.Len(); i++ {
e := se.At(i)
b.logEntry("SpanEvent #%d", i)
b.logEntry(" -> Name: %s", e.Name())
b.logEntry(" -> Timestamp: %s", e.Timestamp())
b.logEntry(" -> DroppedAttributesCount: %d", e.DroppedAttributesCount())
b.logAttributes(" -> Attributes:", e.Attributes())
}
}

func (b *dataBuffer) logLinks(description string, sl ptrace.SpanLinkSlice) {
if sl.Len() == 0 {
return
}

b.logEntry("%s:", description)

for i := 0; i < sl.Len(); i++ {
l := sl.At(i)
b.logEntry("SpanLink #%d", i)
b.logEntry(" -> Trace ID: %s", l.TraceID())
b.logEntry(" -> ID: %s", l.SpanID())
b.logEntry(" -> TraceState: %s", l.TraceState().AsRaw())
b.logEntry(" -> DroppedAttributesCount: %d", l.DroppedAttributesCount())
b.logAttributes(" -> Attributes:", l.Attributes())
}
}

func valueToString(v pcommon.Value) string {
return fmt.Sprintf("%s(%s)", v.Type().String(), v.AsString())
}

func MarshalTraces(td ptrace.Traces) ([]byte, error) {
buf := dataBuffer{}
rss := td.ResourceSpans()
for i := 0; i < rss.Len(); i++ {
buf.logEntry("ResourceSpans #%d", i)
rs := rss.At(i)
buf.logEntry("Resource SchemaURL: %s", rs.SchemaUrl())
buf.logAttributes("Resource attributes", rs.Resource().Attributes())
ilss := rs.ScopeSpans()
for j := 0; j < ilss.Len(); j++ {
buf.logEntry("ScopeSpans #%d", j)
ils := ilss.At(j)
buf.logEntry("ScopeSpans SchemaURL: %s", ils.SchemaUrl())
buf.logInstrumentationScope(ils.Scope())

spans := ils.Spans()
for k := 0; k < spans.Len(); k++ {
buf.logEntry("Span #%d", k)
span := spans.At(k)
buf.logAttr("Trace ID", span.TraceID())
buf.logAttr("Parent ID", span.ParentSpanID())
buf.logAttr("ID", span.SpanID())
buf.logAttr("Name", span.Name())
buf.logAttr("Kind", span.Kind().String())
buf.logAttr("Start time", span.StartTimestamp().String())
buf.logAttr("End time", span.EndTimestamp().String())

buf.logAttr("Status code", span.Status().Code().String())
buf.logAttr("Status message", span.Status().Message())

buf.logAttributes("Attributes", span.Attributes())
buf.logEvents("Events", span.Events())
buf.logLinks("Links", span.Links())
}
}
}

return buf.buf.Bytes(), nil
}
65 changes: 65 additions & 0 deletions exporter/logtcpexporter/config.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
// Copyright 2024 Qubership
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package logtcpexporter

import (
"fmt"
"time"
"go.opentelemetry.io/collector/component"
"go.opentelemetry.io/collector/config/confignet"
)

type Config struct {
confignet.TCPAddrConfig `mapstructure:",squash"`
ATLCfg ATLConfig `mapstructure:"arbitrary-traces-logging"`
ConnPoolSize int `mapstructure:"connection-pool-size"`
QueueSize int `mapstructure:"queue-size"`
MaxMessageSendRetryCnt int `mapstructure:"max-message-send-retry-count"`
MaxSuccessiveSendErrCnt int `mapstructure:"max-successive-send-error-count"`
SuccessiveSendErrFreezeTime string `mapstructure:"successive-send-error-freeze-time"`
}

var _ component.Config = (*Config)(nil)

func (cfg *Config) Validate() error {
if cfg.ConnPoolSize < 1 {
return fmt.Errorf("connection-pool-size can not be less than 1 (actual value is %v)", cfg.ConnPoolSize)
}
if cfg.QueueSize < 1 {
return fmt.Errorf("queue-size can not be less than 1 (actual value is %v)", cfg.QueueSize)
}
if cfg.MaxMessageSendRetryCnt < 0 {
return fmt.Errorf("max-message-send-retry-count can not be negative (actual value is %v)", cfg.MaxMessageSendRetryCnt)
}
if cfg.MaxSuccessiveSendErrCnt < 0 {
return fmt.Errorf("max-successive-send-error-count can not be negative (actual value is %v)", cfg.MaxSuccessiveSendErrCnt)
}
_, err := time.ParseDuration(cfg.SuccessiveSendErrFreezeTime)
if err != nil {
return fmt.Errorf("successive-send-error-freeze-time is not parseable : %+v", err)
}
return nil
}

type ATLConfig struct {
SpanFilters []ATLFilter `mapstructure:"span-filters"`
TraceFilters []ATLFilter `mapstructure:"trace-filters"`
}

type ATLFilter struct { // ArbitraryTracesLoggingFilter
ServiceNames []string `mapstructure:"service-names"`
Tags map[string]string `mapstructure:"tags"`
Mapping map[string][]string `mapstructure:"mapping"`
}
73 changes: 73 additions & 0 deletions exporter/logtcpexporter/factory.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
// Copyright 2024 Qubership
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package logtcpexporter

import (
"context"
"errors"

"go.opentelemetry.io/collector/component"
"go.opentelemetry.io/collector/config/confignet"
"go.opentelemetry.io/collector/exporter"
"go.opentelemetry.io/collector/exporter/exporterhelper"

)

const (
typeStr = "logtcpexporter"
defaultBindEndpoint = "0.0.0.0:12201"
)

func NewFactory() exporter.Factory {
return exporter.NewFactory(
component.MustNewType(typeStr),
createDefaultConfig,
exporter.WithTraces(createTracesExporter, component.StabilityLevelAlpha))
}

func createDefaultConfig() component.Config {
return &Config{
TCPAddrConfig: confignet.TCPAddrConfig{
Endpoint: defaultBindEndpoint,
},
ConnPoolSize: 1,
QueueSize: 1024,
MaxMessageSendRetryCnt: 1,
MaxSuccessiveSendErrCnt: 5,
SuccessiveSendErrFreezeTime: "1m",
}
}

func createTracesExporter(
ctx context.Context,
set exporter.Settings,
cfg component.Config,
) (exporter.Traces, error) {
ltec := cfg.(*Config)

if ltec.Endpoint == "" {
return nil, errors.New("exporter config requires a non-empty 'endpoint'")
}

lte := createLogTcpExporter(ltec, set)
return exporterhelper.NewTracesExporter(
ctx,
set,
cfg,
lte.pushTraces,
exporterhelper.WithStart(lte.start),
exporterhelper.WithTimeout(exporterhelper.TimeoutSettings{Timeout: 0}),
)
}
196 changes: 196 additions & 0 deletions exporter/logtcpexporter/graylog/graylog.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,196 @@
// Copyright 2024 Qubership
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package graylog

import (
"encoding/json"
"runtime/debug"
"fmt"
"net"
"time"
"context"

"go.uber.org/zap"
"github.com/Jeffail/gabs"
)

type Transport string

const (
UDP Transport = "udp"
TCP Transport = "tcp"
)

type Endpoint struct {
Transport Transport
Address string
Port uint
}

type GraylogSender struct {
ctx context.Context
endpoint Endpoint
msgQueue chan *Message
logger *zap.Logger
maxMessageSendRetryCnt int
maxSuccessiveSendErrCnt int
successiveSendErrFreezeTime time.Duration
}

type Message struct {
Version string `json:"version"`
Host string `json:"host"`
ShortMessage string `json:"short_message"`
FullMessage string `json:"full_message,omitempty"`
Timestamp int64 `json:"timestamp,omitempty"`
Level uint `json:"level,omitempty"`
Extra map[string]string `json:"-"`
}

func NewGraylogSender(
endpoint Endpoint,
logger *zap.Logger,
connPullSize int,
queueSize int,
maxMessageSendRetryCnt int,
maxSuccessiveSendErrCnt int,
successiveSendErrFreezeTime time.Duration,
) *GraylogSender {
result := GraylogSender{
endpoint: endpoint,
logger: logger,
msgQueue: make(chan *Message, queueSize),
maxMessageSendRetryCnt: maxMessageSendRetryCnt,
maxSuccessiveSendErrCnt: maxSuccessiveSendErrCnt,
successiveSendErrFreezeTime: successiveSendErrFreezeTime,
ctx: context.Background(),
}

for i := 0 ; i < connPullSize; i++ {
i := i
go result.tcpConnGoroutine(i)
}

return &result
}

func (gs *GraylogSender) tcpConnGoroutine(connNumber int) {
defer gs.logger.Sugar().Infof("GraylogTcpConnection : Goroutine #%v is finished", connNumber)
defer func() {
if rec := recover(); rec != nil {
gs.logger.Sugar().Errorf("GraylogTcpConnection : Panic in goroutine #%v : %+v ; Stacktrace of the panic : %v", connNumber, rec, string(debug.Stack()))
time.Sleep(time.Second * 5)
gs.logger.Sugar().Infof("GraylogTcpConnection : Starting gouroutine #%v again ...", connNumber)
go gs.tcpConnGoroutine(connNumber)
}
}()
tcpAddress := fmt.Sprintf("%s:%d", gs.endpoint.Address, gs.endpoint.Port)
gs.logger.Sugar().Infof("GraylogTcpConnection : Goroutine #%v for %v started", connNumber, tcpAddress)
successiveGraylogErrCnt := 0
MAX_SUCCESSIVE_SEND_ERR_CNT := gs.maxSuccessiveSendErrCnt
messageRetryCnt := 0
MAX_RETRY_CNT := gs.maxMessageSendRetryCnt
FREEZE_TIME := gs.successiveSendErrFreezeTime
var retryData *[]byte
for {
gs.logger.Sugar().Infof("GraylogTcpConnection : Creating tcp connection #%v to the graylog", connNumber)
tcpConn, err := net.Dial(string(gs.endpoint.Transport), tcpAddress)
if err != nil {
gs.logger.Sugar().Errorf("GraylogTcpConnection : Error creating tcp connection #%v to the graylog : %+v", connNumber, err)
time.Sleep(time.Second * 5)
continue
}
for {
var data []byte
if messageRetryCnt > MAX_RETRY_CNT {
gs.logger.Sugar().Errorf("GraylogTcpConnection : Message %+v is skipped after %v retries in the goroutine #%v", retryData, messageRetryCnt - 1, connNumber)
retryData = nil
}
if retryData != nil {
data = *retryData
gs.logger.Sugar().Infof("GraylogTcpConnection : Retry %v sending message in the goroutine #%v", messageRetryCnt, connNumber)
} else {
msg, ok := <- gs.msgQueue
if !ok {
gs.logger.Sugar().Errorf("GraylogTcpConnection : Message chan is closed, stopping the goroutine #%v", connNumber)
return
}
if msg == nil {
gs.logger.Sugar().Errorf("GraylogTcpConnection : Nil message received from chan in the goroutine #%v", connNumber)
continue
}
data, err = prepareMessage(msg)
if err != nil {
gs.logger.Sugar().Errorf("GraylogTcpConnection : Message %+v is skipped in the goroutine #%v, because error has happened during message preparation : %+v", msg, connNumber, err)
continue
}
}
_, err = tcpConn.Write(data)
if err != nil {
gs.logger.Sugar().Errorf("GraylogTcpConnection : Message %v has not been sent to the graylog, connection #%v to the graylog will be recreated : %+v\n", string(data), connNumber, err)
err2 := tcpConn.Close()
if err2 != nil {
gs.logger.Sugar().Errorf("GraylogTcpConnection : Error closing tcp connection #%v to the graylog : %+v", connNumber, err2)
}
retryData = &data
messageRetryCnt++
successiveGraylogErrCnt++
if successiveGraylogErrCnt > MAX_SUCCESSIVE_SEND_ERR_CNT {
gs.logger.Sugar().Errorf("GraylogTcpConnection : %v successive errors recieved from the graylog. Connection #%v is freezed for %s", successiveGraylogErrCnt, connNumber, FREEZE_TIME)
time.Sleep(FREEZE_TIME)
}
break
} else {
messageRetryCnt = 0
successiveGraylogErrCnt = 0
retryData = nil
gs.logger.Sugar().Debugf("GraylogTcpConnection : Message %+v has been sent successfully to the graylog in the goroutine #%v\n", string(data), connNumber)
}
}
}
}

func (gs *GraylogSender) SendToQueue(m *Message) error {
select {
case gs.msgQueue <- m:
return nil
default:
err := fmt.Errorf("Chan is full")
return err
}
}

func prepareMessage(m *Message) ([]byte, error) {
jsonMessage, err := json.Marshal(m)
if err != nil {
return []byte{}, err
}

c, err := gabs.ParseJSON(jsonMessage)
if err != nil {
return []byte{}, err
}

for key, value := range m.Extra {
_, err = c.Set(value, fmt.Sprintf("_%s", key))
if err != nil {
return []byte{}, err
}
}

data := append(c.Bytes(), '\n', byte(0))

return data, nil
}
654 changes: 654 additions & 0 deletions exporter/logtcpexporter/logtcpexporter.go

Large diffs are not rendered by default.

186 changes: 186 additions & 0 deletions go.mod
Original file line number Diff line number Diff line change
@@ -0,0 +1,186 @@
// Code generated by "go.opentelemetry.io/collector/cmd/builder". DO NOT EDIT.

module otec

go 1.22.0

toolchain go1.22.5

require (
github.com/Jeffail/gabs v1.4.0
github.com/open-telemetry/opentelemetry-collector-contrib/connector/spanmetricsconnector v0.110.0
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter v0.110.0
github.com/open-telemetry/opentelemetry-collector-contrib/extension/healthcheckextension v0.110.0
github.com/open-telemetry/opentelemetry-collector-contrib/extension/pprofextension v0.110.0
github.com/open-telemetry/opentelemetry-collector-contrib/processor/filterprocessor v0.110.0
github.com/open-telemetry/opentelemetry-collector-contrib/processor/probabilisticsamplerprocessor v0.110.0
github.com/open-telemetry/opentelemetry-collector-contrib/processor/transformprocessor v0.110.0
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/jaegerreceiver v0.110.0
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/zipkinreceiver v0.110.0
go.opentelemetry.io/collector/component v0.110.0
go.opentelemetry.io/collector/config/confighttp v0.110.0
go.opentelemetry.io/collector/config/confignet v1.16.0
go.opentelemetry.io/collector/connector v0.110.0
go.opentelemetry.io/collector/consumer v0.110.0
go.opentelemetry.io/collector/exporter v0.110.0
go.opentelemetry.io/collector/exporter/debugexporter v0.110.0
go.opentelemetry.io/collector/exporter/loggingexporter v0.110.0
go.opentelemetry.io/collector/exporter/otlpexporter v0.110.0
go.opentelemetry.io/collector/extension v0.110.0
go.opentelemetry.io/collector/otelcol v0.110.0
go.opentelemetry.io/collector/pdata v1.16.0
go.opentelemetry.io/collector/processor v0.110.0
go.opentelemetry.io/collector/processor/batchprocessor v0.110.0
go.opentelemetry.io/collector/receiver v0.110.0
go.opentelemetry.io/collector/receiver/otlpreceiver v0.110.0
go.opentelemetry.io/collector/semconv v0.110.0
go.uber.org/zap v1.27.0
golang.org/x/sys v0.25.0
)

require (
go.opentelemetry.io/collector/confmap v1.16.0
go.opentelemetry.io/collector/confmap/converter/expandconverter v0.110.0
go.opentelemetry.io/collector/confmap/provider/envprovider v1.16.0
go.opentelemetry.io/collector/confmap/provider/fileprovider v1.16.0
go.opentelemetry.io/collector/confmap/provider/httpprovider v0.110.0
go.opentelemetry.io/collector/confmap/provider/httpsprovider v0.110.0
go.opentelemetry.io/collector/confmap/provider/yamlprovider v0.110.0
)

require (
cloud.google.com/go v0.65.0 // indirect
github.com/alecthomas/participle/v2 v2.1.1 // indirect
github.com/apache/thrift v0.20.0 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/cenkalti/backoff/v4 v4.3.0 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/elastic/go-grok v0.3.1 // indirect
github.com/elastic/lunes v0.1.0 // indirect
github.com/expr-lang/expr v1.16.9 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/fsnotify/fsnotify v1.7.0 // indirect
github.com/go-logr/logr v1.4.2 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-ole/go-ole v1.2.6 // indirect
github.com/go-viper/mapstructure/v2 v2.1.0 // indirect
github.com/gobwas/glob v0.2.3 // indirect
github.com/goccy/go-json v0.10.3 // indirect
github.com/gogo/googleapis v1.4.1 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/snappy v0.0.4 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/gorilla/mux v1.8.1 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.22.0 // indirect
github.com/hashicorp/go-version v1.7.0 // indirect
github.com/hashicorp/golang-lru v1.0.2 // indirect
github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect
github.com/iancoleman/strcase v0.3.0 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/jaegertracing/jaeger v1.60.0 // indirect
github.com/jonboulle/clockwork v0.4.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/klauspost/compress v1.17.9 // indirect
github.com/knadh/koanf/maps v0.1.1 // indirect
github.com/knadh/koanf/providers/confmap v0.1.0 // indirect
github.com/knadh/koanf/v2 v2.1.1 // indirect
github.com/lightstep/go-expohisto v1.0.0 // indirect
github.com/lufia/plan9stats v0.0.0-20220913051719-115f729f3c8c // indirect
github.com/magefile/mage v1.15.0 // indirect
github.com/mitchellh/copystructure v1.2.0 // indirect
github.com/mitchellh/reflectwalk v1.0.2 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/mostynb/go-grpc-compression v1.2.3 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/open-telemetry/opentelemetry-collector-contrib/internal/common v0.110.0 // indirect
github.com/open-telemetry/opentelemetry-collector-contrib/internal/coreinternal v0.110.0 // indirect
github.com/open-telemetry/opentelemetry-collector-contrib/internal/filter v0.110.0 // indirect
github.com/open-telemetry/opentelemetry-collector-contrib/internal/pdatautil v0.110.0 // indirect
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl v0.110.0 // indirect
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/pdatautil v0.110.0 // indirect
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/resourcetotelemetry v0.110.0 // indirect
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/sampling v0.110.0 // indirect
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/jaeger v0.110.0 // indirect
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/prometheus v0.110.0 // indirect
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/zipkin v0.110.0 // indirect
github.com/openzipkin/zipkin-go v0.4.3 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/power-devops/perfstat v0.0.0-20220216144756-c35f1ee13d7c // indirect
github.com/prometheus/client_golang v1.20.4 // indirect
github.com/prometheus/client_model v0.6.1 // indirect
github.com/prometheus/common v0.59.1 // indirect
github.com/prometheus/procfs v0.15.1 // indirect
github.com/rs/cors v1.11.1 // indirect
github.com/shirou/gopsutil/v4 v4.24.8 // indirect
github.com/shoenig/go-m1cpu v0.1.6 // indirect
github.com/spf13/cobra v1.8.1 // indirect
github.com/spf13/pflag v1.0.5 // indirect
github.com/stretchr/testify v1.9.0 // indirect
github.com/tklauser/go-sysconf v0.3.12 // indirect
github.com/tklauser/numcpus v0.6.1 // indirect
github.com/ua-parser/uap-go v0.0.0-20240611065828-3a4781585db6 // indirect
github.com/yusufpapurcu/wmi v1.2.4 // indirect
go.opentelemetry.io/collector v0.110.0 // indirect
go.opentelemetry.io/collector/client v1.16.0 // indirect
go.opentelemetry.io/collector/component/componentprofiles v0.110.0 // indirect
go.opentelemetry.io/collector/component/componentstatus v0.110.0 // indirect
go.opentelemetry.io/collector/config/configauth v0.110.0 // indirect
go.opentelemetry.io/collector/config/configcompression v1.16.0 // indirect
go.opentelemetry.io/collector/config/configgrpc v0.110.0 // indirect
go.opentelemetry.io/collector/config/configopaque v1.16.0 // indirect
go.opentelemetry.io/collector/config/configretry v1.16.0 // indirect
go.opentelemetry.io/collector/config/configtelemetry v0.110.0 // indirect
go.opentelemetry.io/collector/config/configtls v1.16.0 // indirect
go.opentelemetry.io/collector/config/internal v0.110.0 // indirect
go.opentelemetry.io/collector/connector/connectorprofiles v0.110.0 // indirect
go.opentelemetry.io/collector/consumer/consumerprofiles v0.110.0 // indirect
go.opentelemetry.io/collector/consumer/consumertest v0.110.0 // indirect
go.opentelemetry.io/collector/exporter/exporterprofiles v0.110.0 // indirect
go.opentelemetry.io/collector/extension/auth v0.110.0 // indirect
go.opentelemetry.io/collector/extension/experimental/storage v0.110.0 // indirect
go.opentelemetry.io/collector/extension/extensioncapabilities v0.110.0 // indirect
go.opentelemetry.io/collector/featuregate v1.16.0 // indirect
go.opentelemetry.io/collector/internal/globalgates v0.110.0 // indirect
go.opentelemetry.io/collector/internal/globalsignal v0.110.0 // indirect
go.opentelemetry.io/collector/pdata/pprofile v0.110.0 // indirect
go.opentelemetry.io/collector/pdata/testdata v0.110.0 // indirect
go.opentelemetry.io/collector/pipeline v0.110.0 // indirect
go.opentelemetry.io/collector/processor/processorprofiles v0.110.0 // indirect
go.opentelemetry.io/collector/receiver/receiverprofiles v0.110.0 // indirect
go.opentelemetry.io/collector/service v0.110.0 // indirect
go.opentelemetry.io/contrib/config v0.10.0 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.55.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.55.0 // indirect
go.opentelemetry.io/contrib/propagators/b3 v1.30.0 // indirect
go.opentelemetry.io/otel v1.30.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp v0.6.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.30.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.30.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.30.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.30.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.30.0 // indirect
go.opentelemetry.io/otel/exporters/prometheus v0.52.0 // indirect
go.opentelemetry.io/otel/exporters/stdout/stdoutlog v0.6.0 // indirect
go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v1.30.0 // indirect
go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.30.0 // indirect
go.opentelemetry.io/otel/log v0.6.0 // indirect
go.opentelemetry.io/otel/metric v1.30.0 // indirect
go.opentelemetry.io/otel/sdk v1.30.0 // indirect
go.opentelemetry.io/otel/sdk/log v0.6.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.30.0 // indirect
go.opentelemetry.io/otel/trace v1.30.0 // indirect
go.opentelemetry.io/proto/otlp v1.3.1 // indirect
go.uber.org/multierr v1.11.0 // indirect
golang.org/x/exp v0.0.0-20240506185415-9bf2ced13842 // indirect
golang.org/x/net v0.29.0 // indirect
golang.org/x/text v0.18.0 // indirect
gonum.org/v1/gonum v0.15.1 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20240903143218-8af14fe29dc1 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240903143218-8af14fe29dc1 // indirect
google.golang.org/grpc v1.66.2 // indirect
google.golang.org/protobuf v1.34.2 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)
976 changes: 976 additions & 0 deletions go.sum

Large diffs are not rendered by default.

5 changes: 5 additions & 0 deletions helm-templates/open-telemetry-collector/Chart.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
apiVersion: v2
appVersion: "1.0"
description: A Helm chart for qubership-open-telemetry-collector
name: qubership-open-telemetry-collector
version: 0.0.1
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: '{{ coalesce .Values.DEPLOYMENT_RESOURCE_NAME .Values.SERVICE_NAME }}'
labels:
app.kubernetes.io/name: '{{ .Values.SERVICE_NAME }}'
app.kubernetes.io/part-of: open-telemetry-collector
app.kubernetes.io/managed-by: saasDeployer
spec:
scaleTargetRef:
{{- if eq .Values.PAAS_PLATFORM "KUBERNETES" }}
apiVersion: apps/v1
kind: Deployment
{{- else if eq .Values.PAAS_PLATFORM "OPENSHIFT" }}
apiVersion: apps.openshift.io/v1
kind: DeploymentConfig
{{- end }}
name: '{{ coalesce .Values.DEPLOYMENT_RESOURCE_NAME .Values.SERVICE_NAME }}'
{{- if .Values.HPA_ENABLED }}
minReplicas: {{ coalesce .Values.HPA_MIN_REPLICAS .Values.REPLICAS }}
maxReplicas: {{ .Values.HPA_MAX_REPLICAS }}
{{- else }}
minReplicas: 1
maxReplicas: 9999
{{- end }}
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: {{ int (divf (mulf (default 75 .Values.HPA_AVG_CPU_UTILIZATION_TARGET_PERCENT) (include "to_millicores" .Values.CPU_LIMIT)) (include "to_millicores" .Values.CPU_REQUEST)) }}
behavior:
scaleUp:
stabilizationWindowSeconds: {{ default 0 .Values.HPA_SCALING_UP_STABILIZATION_WINDOW_SECONDS }}
selectPolicy: {{ if .Values.HPA_ENABLED }}{{ default "Max" .Values.HPA_SCALING_UP_SELECT_POLICY }}{{ else }}Disabled{{ end }}
policies:
{{- if and .Values.HPA_SCALING_UP_PERCENT_VALUE (ge (int .Values.HPA_SCALING_UP_PERCENT_PERIOD_SECONDS) 0) }}
- type: Percent
value: {{ .Values.HPA_SCALING_UP_PERCENT_VALUE }}
periodSeconds: {{ .Values.HPA_SCALING_UP_PERCENT_PERIOD_SECONDS }}
{{- end }}
{{- if and .Values.HPA_SCALING_UP_PODS_VALUE (ge (int .Values.HPA_SCALING_UP_PODS_PERIOD_SECONDS) 0) }}
- type: Pods
value: {{ .Values.HPA_SCALING_UP_PODS_VALUE }}
periodSeconds: {{ .Values.HPA_SCALING_UP_PODS_PERIOD_SECONDS }}
{{- end }}
scaleDown:
stabilizationWindowSeconds: {{ default 300 .Values.HPA_SCALING_DOWN_STABILIZATION_WINDOW_SECONDS }}
selectPolicy: {{ if .Values.HPA_ENABLED }}{{ default "Max" .Values.HPA_SCALING_DOWN_SELECT_POLICY }}{{ else }}Disabled{{ end }}
policies:
{{- if and .Values.HPA_SCALING_DOWN_PERCENT_VALUE (ge (int .Values.HPA_SCALING_DOWN_PERCENT_PERIOD_SECONDS) 0) }}
- type: Percent
value: {{ .Values.HPA_SCALING_DOWN_PERCENT_VALUE }}
periodSeconds: {{ .Values.HPA_SCALING_DOWN_PERCENT_PERIOD_SECONDS }}
{{- end }}
{{- if and .Values.HPA_SCALING_DOWN_PODS_VALUE (ge (int .Values.HPA_SCALING_DOWN_PODS_PERIOD_SECONDS) 0)}}
- type: Pods
value: {{ .Values.HPA_SCALING_DOWN_PODS_VALUE }}
periodSeconds: {{ .Values.HPA_SCALING_DOWN_PODS_PERIOD_SECONDS }}
{{- end }}
34 changes: 34 additions & 0 deletions helm-templates/open-telemetry-collector/templates/Ingress.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
{{- if .Values.OTEC_SENTRY_ENVELOPES_INGRESS_ENABLED }}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: '{{ .Values.SERVICE_NAME }}'
namespace: '{{ .Values.NAMESPACE }}'
labels:
application: '{{ .Values.SERVICE_NAME }}'
name: '{{ .Values.SERVICE_NAME }}'
tier: backend
app.kubernetes.io/name: '{{ .Values.SERVICE_NAME }}'
app.kubernetes.io/instance: '{{ cat .Values.SERVICE_NAME .Values.DELIMITER .Values.NAMESPACE | nospace | trunc 63 }}'
app.kubernetes.io/version: '{{ .Values.ARTIFACT_DESCRIPTOR_VERSION }}'
app.kubernetes.io/component: monitoring
app.kubernetes.io/part-of: open-telemetry-collector
app.kubernetes.io/managed-by: saasDeployer
{{- if .Values.OTEC_SENTRY_ENVELOPES_INGRESS_ANNOTATIONS }}
annotations:
{{- toYaml (.Values.OTEC_SENTRY_ENVELOPES_INGRESS_ANNOTATIONS) | nindent 4 }}
{{- end }}
spec:
ingressClassName: nginx
rules:
- host: "{{ .Values.SERVICE_NAME}}-{{ .Values.NAMESPACE}}.{{ .Values.CLOUD_PUBLIC_HOST}}"
http:
paths:
- path: /
pathType: ImplementationSpecific
backend:
service:
name: '{{ .Values.SERVICE_NAME }}'
port:
name: sentry-receiver
{{- end }}
32 changes: 32 additions & 0 deletions helm-templates/open-telemetry-collector/templates/_helpers.tpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
{{- define "to_millicores" -}}
{{- $value := toString . -}}
{{- if hasSuffix "m" $value -}}
{{ trimSuffix "m" $value }}
{{- else -}}
{{ mulf $value 1000 }}
{{- end -}}
{{- end -}}

{{/* Define new HPA parameters based on old if old parameters presented */}}
{{- define "hpa_bwc" -}}
{{- $valuesMap := .Values -}}
{{ $paramValue := int $valuesMap.HPA_SCALING_UP_INTERVAL -}}
{{- if and (hasKey .Values "HPA_SCALING_UP_INTERVAL") (ge $paramValue 0) -}}
{{- $noop := set $valuesMap "HPA_SCALING_UP_STABILIZATION_WINDOW_SECONDS" $paramValue -}}
{{- $noop := set $valuesMap "HPA_SCALING_UP_SELECT_POLICY" "Max" -}}
{{- $noop := set $valuesMap "HPA_SCALING_UP_PERCENT_VALUE" 95 -}}
{{- $noop := set $valuesMap "HPA_SCALING_UP_PERCENT_PERIOD_SECONDS" $paramValue -}}
{{- $noop := unset $valuesMap "HPA_SCALING_UP_PODS_VALUE" -}}
{{- $noop := unset $valuesMap "HPA_SCALING_UP_PODS_PERIOD_SECONDS" -}}
{{- end -}}
{{ $paramValue := int $valuesMap.HPA_SCALING_DOWN_INTERVAL -}}
{{- if and (hasKey .Values "HPA_SCALING_DOWN_INTERVAL") (ge $paramValue 0) -}}
{{- $paramValue := get $valuesMap "HPA_SCALING_DOWN_INTERVAL" -}}
{{- $noop := set $valuesMap "HPA_SCALING_DOWN_STABILIZATION_WINDOW_SECONDS" $paramValue -}}
{{- $noop := set $valuesMap "HPA_SCALING_DOWN_SELECT_POLICY" "Max" -}}
{{- $noop := set $valuesMap "HPA_SCALING_DOWN_PERCENT_VALUE" 95 -}}
{{- $noop := set $valuesMap "HPA_SCALING_DOWN_PERCENT_PERIOD_SECONDS" $paramValue -}}
{{- $noop := unset $valuesMap "HPA_SCALING_DOWN_PODS_VALUE" -}}
{{- $noop := unset $valuesMap "HPA_SCALING_DOWN_PODS_PERIOD_SECONDS" -}}
{{- end -}}
{{- end -}}
216 changes: 216 additions & 0 deletions helm-templates/open-telemetry-collector/templates/deployment.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,216 @@
kind: Deployment
apiVersion: apps/v1
metadata:
name: {{ .Values.SERVICE_NAME }}
labels:
{{- if .Values.labels }}
{{- toYaml (.Values.labels | default dict) | nindent 4 }}
{{- end }}
name: '{{ .Values.SERVICE_NAME }}'
app: open-telemetry-collector
app.kubernetes.io/instance: '{{ cat (coalesce .Values.DEPLOYMENT_RESOURCE_NAME .Values.SERVICE_NAME) "-" .Values.NAMESPACE | nospace | trunc 63 | trimSuffix "-" }}'
app.kubernetes.io/name: '{{ .Values.SERVICE_NAME }}'
app.kubernetes.io/component: backend
app.kubernetes.io/version: '{{ .Values.ARTIFACT_DESCRIPTOR_VERSION }}'
app.kubernetes.io/managed-by: saasDeployer
app.kubernetes.io/part-of: open-telemetry-collector
app.kubernetes.io/technology: 'go'
spec:
strategy:
{{- if eq (default "" .Values.DEPLOYMENT_STRATEGY_TYPE) "recreate" }}
type: Recreate
{{- else if eq (default "" .Values.DEPLOYMENT_STRATEGY_TYPE) "best_effort_controlled_rollout" }}
type: RollingUpdate
rollingUpdate:
maxSurge: 0
maxUnavailable: 80%
{{- else if eq (default "" .Values.DEPLOYMENT_STRATEGY_TYPE) "ramped_slow_rollout" }}
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
{{- else if eq (default "" .Values.DEPLOYMENT_STRATEGY_TYPE) "custom_rollout" }}
type: RollingUpdate
rollingUpdate:
maxSurge: {{ .Values.DEPLOYMENT_STRATEGY_MAXSURGE | default "25%" }}
maxUnavailable: {{ .Values.DEPLOYMENT_STRATEGY_MAXUNAVAILABLE | default "25%" }}
{{- else }}
type: RollingUpdate
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
{{- end }}
replicas: 1
selector:
matchLabels:
app: open-telemetry-collector
template:
metadata:
creationTimestamp: null
labels:
name: '{{ coalesce .Values.DEPLOYMENT_RESOURCE_NAME .Values.SERVICE_NAME }}'
app: open-telemetry-collector
app.kubernetes.io/instance: '{{ cat (coalesce .Values.DEPLOYMENT_RESOURCE_NAME .Values.SERVICE_NAME) "-" .Values.NAMESPACE | nospace | trunc 63 | trimSuffix "-" }}'
app.kubernetes.io/name: '{{ coalesce .Values.DEPLOYMENT_RESOURCE_NAME .Values.SERVICE_NAME }}'
app.kubernetes.io/component: backend
app.kubernetes.io/version: '{{ .Values.ARTIFACT_DESCRIPTOR_VERSION }}'
app.kubernetes.io/managed-by: saasDeployer
app.kubernetes.io/part-of: open-telemetry-collector
annotations:
checksum/config: {{ include (print $.Template.BasePath "/metrics_collector_config.yaml") . | sha256sum }}
spec:
topologySpreadConstraints:
{{- if .Values.CLOUD_TOPOLOGIES }}
{{- range $v := .Values.CLOUD_TOPOLOGIES }}
- topologyKey: {{ $v.topologyKey }}
maxSkew: {{ $v.maxSkew | default 1 }}
whenUnsatisfiable: {{ $v.whenUnsatisfiable | default "ScheduleAnyway" }}
labelSelector:
matchLabels:
name: "{{ coalesce $.Values.DEPLOYMENT_RESOURCE_NAME $.Values.SERVICE_NAME }}"
{{- end }}
{{- else }}
- maxSkew: 1
topologyKey: "{{ .Values.CLOUD_TOPOLOGY_KEY }}"
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
name: "{{ coalesce .Values.DEPLOYMENT_RESOURCE_NAME .Values.SERVICE_NAME }}"
{{- end }}
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
volumes:
- name: opentelemetry-collector-vol
configMap:
name: opentelemetry-collector-config
defaultMode: 420
{{- if .Values.READONLY_CONTAINER_FILE_SYSTEM_ENABLED }}
- name: tmp
emptyDir: {}
- name: env
emptyDir: {}
- name: ncdiag
emptyDir: {}
- name: glowroot
emptyDir: {}
- name: nss
emptyDir: {}
- name: java-certs
emptyDir: {}
{{- end }}
containers:
- name: opentelemetry-collector
command: ["{{ .Values.entrypoint }}","--config", "/conf/otel.yaml"]
image: '{{ .Values.IMAGE_REPOSITORY }}:{{ .Values.TAG }}'
ports:
- name: jaeger-compact
containerPort: 6831
protocol: UDP
- name: jaeger-grpc
containerPort: 14250
protocol: TCP
- name: c-tchan-trft
containerPort: 14267
protocol: TCP
- name: jaeger-thrift
containerPort: 14268
protocol: TCP
- name: otlp
containerPort: 4317
protocol: TCP
- name: otlp-http
containerPort: 4318
protocol: TCP
- name: zipkin
containerPort: 9411
protocol: TCP
- name: exporter-prom
containerPort: 8889
protocol: TCP
- name: sentry-receiver
containerPort: 8080
protocol: TCP
- name: pprof
containerPort: 1777
protocol: TCP
- name: self-telemetry
containerPort: 8888
protocol: TCP
resources:
{{- if .Values.resources }}
{{- toYaml .Values.resources | nindent 12 }}
{{- else }}
limits:
cpu: '{{ .Values.CPU_LIMIT }}'
memory: '{{ .Values.MEMORY_LIMIT }}'
requests:
cpu: '{{ .Values.CPU_REQUEST }}'
memory: '{{ .Values.MEMORY_REQUEST }}'
{{- end }}
volumeMounts:
- name: opentelemetry-collector-vol
mountPath: /conf/
{{- if .Values.READONLY_CONTAINER_FILE_SYSTEM_ENABLED }}
- name: tmp
mountPath: /tmp
- name: env
mountPath: /etc/env
- name: ncdiag
mountPath: /app/ncdiag
- name: glowroot
mountPath: /app/glowroot
- name: nss
mountPath: /app/nss
- name: java-certs
mountPath: /etc/ssl/certs/java
{{- end }}
env:
- name: OTEC_LOG_FORMAT
value: "{{ .Values.OTEC_LOG_FORMAT }}"
livenessProbe:
httpGet:
path: /
port: {{ .Values.OTEC_HEALTH_CHECK_PORT }}
scheme: HTTP
timeoutSeconds: 1
periodSeconds: 10
successThreshold: 1
failureThreshold: 3
readinessProbe:
httpGet:
path: /
port: {{ .Values.OTEC_HEALTH_CHECK_PORT }}
scheme: HTTP
timeoutSeconds: 1
periodSeconds: 10
successThreshold: 1
failureThreshold: 3
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
securityContext:
{{- if eq .Values.PAAS_PLATFORM "KUBERNETES" }}
runAsGroup: 10001
{{- end }}
readOnlyRootFilesystem: true
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
restartPolicy: Always
terminationGracePeriodSeconds: 30
dnsPolicy: ClusterFirst
securityContext: {}
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 25%
maxSurge: 25%
revisionHistoryLimit: 10
progressDeadlineSeconds: 600
Original file line number Diff line number Diff line change
@@ -0,0 +1,164 @@
kind: ConfigMap
apiVersion: v1
metadata:
name: opentelemetry-collector-config
labels:
app.kubernetes.io/instance: '{{ cat .Values.SERVICE_NAME .Values.DELIMITER .Values.NAMESPACE | nospace | trunc 63 }}'
app.kubernetes.io/name: '{{ .Values.SERVICE_NAME }}'
app.kubernetes.io/part-of: open-telemetry-collector
app.kubernetes.io/managed-by: saasDeployer
data:
otel.yaml: |
{{- if .Values.CONFIG_MAP }}
{{- toYaml (.Values.CONFIG_MAP) | nindent 4 }}
{{- else }}
{{- if eq .Values.OTEC_INSTALLATION_MODE "SPAN_METRICS_PROCESSOR" }}
receivers:
jaeger:
protocols:
thrift_http:
endpoint: ":14268"
grpc:
endpoint: ":14250"
otlp:
protocols:
grpc:
endpoint: ":4317"
exporters:
prometheus:
endpoint: ":8889"
otlp:
endpoint: {{- if .Values.JAEGER_COLLECTOR_HOST }} "{{ .Values.JAEGER_COLLECTOR_HOST }}:{{ .Values.JAEGER_COLLECTOR_OTLP_PORT }}" {{- else if .Values.TRACING_HOST }} "{{ .Values.TRACING_HOST }}:{{ .Values.JAEGER_COLLECTOR_OTLP_PORT }}" {{- else }} "jaeger-app-collector.jaeger:{{ .Values.JAEGER_COLLECTOR_OTLP_PORT }}" {{- end }}
tls:
insecure: true
{{- if .Values.OTEC_ENABLE_ARBITRARY_TRACES_LOGGING | default false }}
logtcpexporter:
endpoint: {{- if .Values.GRAYLOG_COLLECTOR_HOST }} {{ printf "%s:%s" ( .Values.GRAYLOG_COLLECTOR_HOST | replace "https://" "" | replace "http://" "") ( .Values.GRAYLOG_COLLECTOR_PORT | toString ) }} {{- else }} {{ printf "%s:%s" ( .Values.GRAYLOG_UI_URL | replace "https://" "" | replace "http://" "") ( .Values.GRAYLOG_COLLECTOR_PORT | toString ) }} {{- end }}
{{- toYaml (.Values.OTEC_LOGTCPEXPORTER_PARAMETERS) | nindent 8 }}
arbitrary-traces-logging:
{{- toYaml (.Values.OTEC_ARBITRARY_TRACES_LOGGING_CONFIG) | nindent 10 }}
{{- end }}
processors:
batch:
connectors:
spanmetrics:
histogram:
explicit:
buckets:
{{- toYaml (.Values.LATENCY_HISTOGRAM_BUCKETS) | nindent 14 }}
dimensions:
- name: destinationPageUrl
default: UNKNOWN
- name: error
default: UNKNOWN
- name: spanStatus
default: UNKNOWN
dimensions_cache_size: 5000
aggregation_temporality: "AGGREGATION_TEMPORALITY_CUMULATIVE"
metrics_flush_interval: 15s
extensions:
health_check:
endpoint: "0.0.0.0:{{ .Values.OTEC_HEALTH_CHECK_PORT }}"
pprof:
endpoint: ":1777"
service:
extensions: [health_check, pprof]
pipelines:
traces:
receivers: [jaeger, otlp]
processors: [batch]
{{- if .Values.OTEC_ENABLE_ARBITRARY_TRACES_LOGGING | default false }}
exporters: [otlp, spanmetrics, logtcpexporter]
{{- else }}
exporters: [otlp, spanmetrics]
{{- end }}
metrics:
receivers: [spanmetrics]
exporters: [prometheus]
telemetry:
logs:
level: "{{ .Values.LOG_LEVEL }}"
metrics:
address: ":8888"
{{- else if eq .Values.OTEC_INSTALLATION_MODE "SENTRY_ENVELOPES_PROCESSING" }}
receivers:
sentryreceiver:
{{- if .Values.OTEC_SENTRY_RECEIVER_PARAMETERS }}
{{- toYaml (.Values.OTEC_SENTRY_RECEIVER_PARAMETERS) | nindent 8 }}
{{- else }}
endpoint: ":8080"
{{- end }}
exporters:
prometheus:
endpoint: ":8889"
metric_expiration: 336h
otlp:
endpoint: {{- if .Values.JAEGER_COLLECTOR_HOST }} "{{ .Values.JAEGER_COLLECTOR_HOST }}:{{ .Values.JAEGER_COLLECTOR_OTLP_PORT }}" {{- else if .Values.TRACING_HOST }} "{{ .Values.TRACING_HOST }}:{{ .Values.JAEGER_COLLECTOR_OTLP_PORT }}" {{- else }} "jaeger-app-collector.jaeger:{{ .Values.JAEGER_COLLECTOR_OTLP_PORT }}" {{- end }}
tls:
insecure: true
logtcpexporter:
endpoint: {{- if .Values.GRAYLOG_COLLECTOR_HOST }} {{ printf "%s:%s" ( .Values.GRAYLOG_COLLECTOR_HOST | replace "https://" "" | replace "http://" "") ( .Values.GRAYLOG_COLLECTOR_PORT | toString ) }} {{- else }} {{ printf "%s:%s" ( .Values.GRAYLOG_UI_URL | replace "https://" "" | replace "http://" "") ( .Values.GRAYLOG_COLLECTOR_PORT | toString ) }} {{- end }}
{{- toYaml (.Values.OTEC_LOGTCPEXPORTER_PARAMETERS) | nindent 8 }}
processors:
batch:
filter/removesessiontraces:
error_mode: ignore
traces:
span:
- attributes["sentry.envelop.type"] == "session"
connectors:
sentrymetrics:
sentry_events:
{{- if .Values.OTEC_SENTRY_EVENT_METRICS_PARAMETERS }}
{{- toYaml (.Values.OTEC_SENTRY_EVENT_METRICS_PARAMETERS) | nindent 10 }}
{{- else }}
labels:
level: level
service_name: service.name
{{- end }}
sentry_measurements:
{{- if .Values.OTEC_SENTRY_MEASUREMENTS_METRICS_PARAMETERS }}
{{- toYaml (.Values.OTEC_SENTRY_MEASUREMENTS_METRICS_PARAMETERS) | nindent 10 }}
{{- else }}
default_buckets: [100, 1000, 10000]
default_labels:
operation: operation
transaction: transaction_path
service_name: service.name
custom:
transaction_duration:
buckets: [1000, 10000, 100000]
ttfb:
buckets: [100, 1000]
ttfb.requestTime:
buckets: [100, 1000]
fp:
buckets: [1000, 3000]
fcp:
buckets: [1000, 3000]
{{- end }}
extensions:
health_check:
endpoint: "0.0.0.0:{{ .Values.OTEC_HEALTH_CHECK_PORT }}"
pprof:
endpoint: ":1777"
service:
extensions: [health_check, pprof]
pipelines:
traces/tometrics:
receivers: [sentryreceiver]
exporters: [sentrymetrics]
metrics:
receivers: [sentrymetrics]
exporters: [prometheus]
traces/tojaegerandgraylog:
receivers: [sentryreceiver]
processors: [filter/removesessiontraces]
exporters: [otlp, logtcpexporter]
telemetry:
logs:
level: "{{ .Values.LOG_LEVEL }}"
metrics:
address: ":8888"
{{- end }}
{{- end }}
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
{{- if .Values.MONITORING_ENABLED }}
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: {{ .Values.SERVICE_NAME }}-service-monitor
labels:
app.kubernetes.io/component: monitoring
app.kubernetes.io/name: {{ .Values.SERVICE_NAME }}
app: open-telemetry-collector
spec:
endpoints:
- interval: 30s
port: {{ .Values.SERVICE_MONITOR_PORT_NAME }}
path: /metrics
scheme: http
- interval: 30s
port: self-telemetry
path: /metrics
scheme: http
selector:
matchLabels:
app: open-telemetry-collector
{{- end }}
64 changes: 64 additions & 0 deletions helm-templates/open-telemetry-collector/templates/service.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
kind: Service
apiVersion: v1
metadata:
name: {{ .Values.SERVICE_NAME }}
labels:
name: {{ .Values.SERVICE_NAME }}
app.kubernetes.io/name: {{ .Values.SERVICE_NAME }}
app.kubernetes.io/part-of: open-telemetry-collector
app.kubernetes.io/managed-by: saasDeployer
app: open-telemetry-collector
spec:
ports:
{{- if .Values.SERVICE_PORTS }}
{{- toYaml (.Values.SERVICE_PORTS) | nindent 4 }}
{{- else }}
- name: jaeger-compact
protocol: UDP
port: 6831
targetPort: jaeger-compact
- name: http-zipkin
protocol: TCP
port: 9411
targetPort: zipkin
- name: grpc
protocol: TCP
port: 14250
targetPort: jaeger-grpc
- name: c-tchan-trft
protocol: TCP
port: 14267
targetPort: c-tchan-trft
- name: http-c-binary-trft
protocol: TCP
port: 14268
targetPort: jaeger-thrift
- name: otlp
protocol: TCP
port: 4317
targetPort: otlp
- name: otlp-http
protocol: TCP
port: 4318
targetPort: otlp-http
- name: exporter-prom
protocol: TCP
port: 8889
targetPort: exporter-prom
- name: sentry-receiver
protocol: TCP
port: 8080
targetPort: sentry-receiver
- name: pprof
protocol: TCP
port: 1777
targetPort: pprof
- name: self-telemetry
protocol: TCP
port: 8888
targetPort: self-telemetry
{{- end }}
selector:
app: open-telemetry-collector
sessionAffinity: None
type: ClusterIP
Loading

0 comments on commit f68683a

Please sign in to comment.