Live debugging service with prometheus.relabel #797

wildum · 2024-05-08T14:18:23Z

THIS PR WILL BE MERGED TO A FEATURE BRANCH WHICH WILL RECEIVE A FEW OTHER PRs BEFORE BEING READY TO BE MERGED ON MAIN

This PR is the first part of the live debugging functionality.

It creates live debugging as a service and connects the prometheus.relabel component to it (an easy first candidate).

A stream to a component can be opened via http://localhost:12345/api/v0/web/debug/ + componentID.
This adds a callback to the x-ray service that the component can retrieve and use to pass data as a string which will be directly flushed to the HTTP client.

It supports components that are inside modules.

Fixes #790

internal/service/livedebugging/service.go

internal/service/livedebugging/debug_stream_handler.go

internal/service/livedebugging/debug_stream_manager.go

internal/web/api/api.go

mattdurham · 2024-05-14T13:36:34Z

Looks reasonable, tested both wide 1,000 streams and deep 1 stream with 1m metrics. Long term some changes around when to flush and the channel would be nice. But performance is totally reasonable.

internal/component/prometheus/relabel/relabel.go

internal/service/livedebugging/debug_stream_manager.go

thampiotr

Good start! some comments around readability and naming.

internal/alloy/componenttest/componenttest.go

internal/alloycli/cmd_run.go

internal/component/prometheus/relabel/relabel.go

internal/service/ui/ui.go

internal/web/api/api.go

internal/service/livedebugging/debug_stream_manager.go

…void expensive string computation

wildum · 2024-05-24T14:08:37Z

@ptodev @thampiotr thanks for your feedback!
I made some changes to try to improve naming and readability. I also split the interface in two: DebugCallbackRegistry and DebugDataPublisher.
I'm now using two interfaces to clearly separate the roles between the debug data publishers (the components) and the API that sets the callbacks. I'm not sure where these interfaces should be defined so I left them in the livedebugging package.
Please have another look at it when you have time :)

internal/service/livedebugging/livedebugging.go

ptodev

Thank you, LGTM!

thampiotr · 2024-06-03T10:05:20Z

internal/web/api/api.go

+		}
+
+		// Buffer of 1000 entries to handle load spikes and prevent this functionality from eating up too much memory.
+		// TODO: in the future we may want to make this value configurable to handle heavy load


Perhaps we should track these TODOs somewhere in a GitHub issue, cause after we're done reviewing these partial PRs, we may miss them when merging to main.

Good point, I created this issue to track the todos: #974
Not sure if we do the optimizations before we merge it to main. It might be better to have more components that supports the feature to identify and solve the pain points

* create xray service to manage the debug streams * add xray support to the prometheus.relabel component * add xray endpoint to the api * rename handler to manager * defer cleanup func * rework to add multi-streams support * more tests * additional multi delete test * add todo comment for buffer size * add error checks * improve naming and readability, split interfaces and add a check to avoid expensive string computation * move register and isregistered to a new debugRegistry interface

* Live debugging service with prometheus.relabel (#797) * create xray service to manage the debug streams * add xray support to the prometheus.relabel component * add xray endpoint to the api * rename handler to manager * defer cleanup func * rework to add multi-streams support * more tests * additional multi delete test * add todo comment for buffer size * add error checks * improve naming and readability, split interfaces and add a check to avoid expensive string computation * move register and isregistered to a new debugRegistry interface * Live debugging front (#977) * add live debugging page in UI * change button size * move icon in front of text in buttons * add warning if the number of incoming lines is greater than the buffer size * Live debugging doc (#987) * add live debugging doc * Update docs/sources/tasks/debug.md Co-authored-by: Clayton Cornell <[email protected]> * Update docs/sources/tasks/debug.md Co-authored-by: Clayton Cornell <[email protected]> * Update docs/sources/tasks/debug.md Co-authored-by: Clayton Cornell <[email protected]> * improve doc * Update docs/sources/tasks/debug.md Co-authored-by: Clayton Cornell <[email protected]> --------- Co-authored-by: Clayton Cornell <[email protected]> * changelog * add list of supported components in debug.md * switch to pull based registration * switch stability to GA * update changelog * use monospace font for log lines * copy to clipboard now copies the filtered data * disable livedebugging by default and add a config block to enable it. The stability is also updated to experimental * Update docs/sources/reference/config-blocks/livedebugging.md Co-authored-by: Clayton Cornell <[email protected]> * Update docs/sources/reference/config-blocks/livedebugging.md Co-authored-by: Clayton Cornell <[email protected]> * replace debugging data by live debugging data in the doc * update screenshots * Update docs/sources/reference/config-blocks/livedebugging.md Co-authored-by: Clayton Cornell <[email protected]> * Update docs/sources/reference/config-blocks/livedebugging.md Co-authored-by: Clayton Cornell <[email protected]> * small doc fix --------- Co-authored-by: Clayton Cornell <[email protected]>

wildum force-pushed the xray-service branch from 05a5376 to 2f87657 Compare May 8, 2024 14:26

create xray service to manage the debug streams

35abfc2

wildum force-pushed the xray-service branch from 2f87657 to 31474cd Compare May 8, 2024 14:41

wildum changed the title ~~Xray service with prometheus.relabel~~ Live debugging service with prometheus.relabel May 8, 2024

wildum force-pushed the xray-service branch 2 times, most recently from da55944 to 47650ee Compare May 8, 2024 14:55

wildum added 2 commits May 8, 2024 17:01

add xray support to the prometheus.relabel component

290b5a1

add xray endpoint to the api

e864cc2

wildum force-pushed the xray-service branch from 47650ee to e864cc2 Compare May 8, 2024 15:01

wildum commented May 13, 2024

View reviewed changes

internal/service/livedebugging/service.go Outdated Show resolved Hide resolved

wildum commented May 13, 2024

View reviewed changes

internal/service/livedebugging/debug_stream_handler.go Outdated Show resolved Hide resolved

wildum commented May 13, 2024

View reviewed changes

internal/service/livedebugging/debug_stream_manager.go Outdated Show resolved Hide resolved

wildum commented May 13, 2024

View reviewed changes

internal/web/api/api.go Outdated Show resolved Hide resolved

wildum added 5 commits May 14, 2024 14:30

rename handler to manager

46d859c

defer cleanup func

c5d6c45

rework to add multi-streams support

fe2e47c

more tests

3409b92

additional multi delete test

0f5ca8e

rfratto added the backport-to-agent:no PR should NOT be backported to the agent repo. label May 14, 2024

wildum added 2 commits May 14, 2024 18:52

add todo comment for buffer size

3fcd62f

add error checks

347c140

wildum force-pushed the xray-service branch from 7727de5 to 347c140 Compare May 22, 2024 09:54

ptodev reviewed May 22, 2024

View reviewed changes

internal/component/prometheus/relabel/relabel.go Outdated Show resolved Hide resolved

internal/service/livedebugging/debug_stream_manager.go Outdated Show resolved Hide resolved

internal/service/livedebugging/debug_stream_manager.go Outdated Show resolved Hide resolved

thampiotr reviewed May 22, 2024

View reviewed changes

improve naming and readability, split interfaces and add a check to a…

231e992

…void expensive string computation

thampiotr reviewed May 28, 2024

View reviewed changes

internal/service/livedebugging/livedebugging.go Outdated Show resolved Hide resolved

thampiotr reviewed May 28, 2024

View reviewed changes

internal/service/livedebugging/livedebugging.go Outdated Show resolved Hide resolved

move register and isregistered to a new debugRegistry interface

5bfc8cf

ptodev approved these changes May 30, 2024

View reviewed changes

thampiotr approved these changes Jun 3, 2024

View reviewed changes

wildum merged commit 1523af0 into xray Jun 3, 2024
15 checks passed

wildum deleted the xray-service branch June 3, 2024 10:22

wildum mentioned this pull request Jun 5, 2024

Run benchmark and optimise if necessary #832

Closed

wildum mentioned this pull request Jun 7, 2024

Live debugging #999

Merged

5 tasks

github-actions bot added the frozen-due-to-age label Jul 8, 2024

github-actions bot locked as resolved and limited conversation to collaborators Jul 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Live debugging service with prometheus.relabel #797

Live debugging service with prometheus.relabel #797

wildum commented May 8, 2024 •

edited

Loading

mattdurham commented May 14, 2024

thampiotr left a comment

wildum commented May 24, 2024 •

edited

Loading

ptodev left a comment

thampiotr Jun 3, 2024

wildum Jun 3, 2024

Live debugging service with prometheus.relabel #797

Live debugging service with prometheus.relabel #797

Conversation

wildum commented May 8, 2024 • edited Loading

mattdurham commented May 14, 2024

thampiotr left a comment

Choose a reason for hiding this comment

wildum commented May 24, 2024 • edited Loading

ptodev left a comment

Choose a reason for hiding this comment

thampiotr Jun 3, 2024

Choose a reason for hiding this comment

wildum Jun 3, 2024

Choose a reason for hiding this comment

wildum commented May 8, 2024 •

edited

Loading

wildum commented May 24, 2024 •

edited

Loading