Merge pull request #78 from intergral/metrics_support

chore(update): update deep proto

CHANGELOG.md

@@ -1,11 +1,12 @@
-<!-- 1.0.5 START -->
-# 1.0.5 (xx/xx/2024)
-- **[BUGFIX]**: compactor/retention - missing loop in retention and compactor [#76](https://github.com/intergral/deep/pull/76) [@Umaaz](https://github.com/Umaaz)
-<!-- 1.0.5 END -->
+<!-- main START -->
+# main (unreleased)
+- **[ENHANCEMENT]**: Update deep proto version to support new properties on tracepoints [#78](https://github.com/intergral/deep/pull/78) [@Umaaz](https://github.com/Umaaz) 
+- **[ENHANCEMENT]**: change(builds): change builds to use goreleaser [#79](https://github.com/intergral/deep/pull/79) [@Umaaz](https://github.com/Umaaz)
+<!-- main START -->
 
 <!-- 1.0.4 START -->
 # 1.0.4 (04/12/2023)
-- **[BUGFIX]**: compactor/retention - missing loop in retention and compactor [#76](https://github.com/intergral/deep/pull/76) [@Umaaz](https://github.com/Umaaz)
+- **[BUGFIX]**: compactor loop was not running [#76](https://github.com/intergral/deep/pull/76) [@Umaaz](https://github.com/Umaaz)
 <!-- 1.0.4 END -->
 
 <!-- 1.0.3 START -->

docs/docs/.pages

@@ -0,0 +1,2 @@
+nav:
+    - ... | regex=^[^_][A-z0-9]+$

docs/docs/_sections/client_docs.md

@@ -0,0 +1,2 @@
+- [github](intergral/deep-python-client)
+- [github](intergral/deep-java-client)

docs/docs/features/.pages

@@ -0,0 +1,5 @@
+nav:
+    - Tracepoints: tracepoints.md
+    - Snapshot: snapshots.md
+    - Method Entry: method_entry.md
+    - ...

docs/docs/features/logging.md

@@ -6,7 +6,7 @@ To add a log message simply attach the argument 'log_msg' to the tracepoint.
 ```json
 {
   "path": "some/file.py",
-  "line": 22,
+  "line_number": 22,
   "args": {
     "log_msg": "This log message will be injected"
   }

docs/docs/features/method_entry.md

@@ -0,0 +1,18 @@
+# Method Entry
+
+This feature is linked to the [span features](./traces.md). Essentially this allows Deep to create snapshots or spans on
+methods rather than lines, by setting the argument 'method_name'.
+
+```json
+{
+  "path": "some/file.py",
+  "line_number": -1,
+  "args": {
+    "method_name": "session_created"
+  }
+}
+```
+
+This would essentially ignore the line number and instead create a snapshot when the method/function 'session_created'
+in the file 'some/file.py' is called. This can be useful if the line numbers of the running code is not known, but you
+know the method/function name. 

docs/docs/features/metrics.md

@@ -0,0 +1,87 @@
+# Metrics
+
+Deep offers the ability to dynamically create metrics at arbitrary points in the code. This allows you to create metrics
+about specific actions within your applications without having to change the code.
+
+To attach a metric
+a '[Metric](https://github.com/intergral/deep-proto/blob/master/deepproto/proto/tracepoint/v1/tracepoint.proto#L45)'
+must be attached to the tracepoint config.
+
+```json
+{
+  "path": "some/file.py",
+  "line_number": 22,
+  "metrics": [
+    {
+      "name": "session_created",
+      "labelExpressions": [
+        {
+          "key": "user_id",
+          "value": {
+            "expression": "user.id"
+          }
+        }
+      ]
+    }
+  ]
+}
+```
+
+The above example would create a metric called `session_created` with the labels `user_ud: 167252` (the value of the
+label evaluated at runtime). The value of the labels can be either expressions (as above) or static values. The value of
+the metric is optional, if not set then the value will be `1`, if set the value should be a valid expression for the
+point it is executed.
+
+# Supported providers
+
+The specific supported providers is dependent on the client that is being used. In all cases custom providers can be set
+using the plugin architecture of the client. As a generalisation the following providers will be detected and used automatically:
+
+ - Prometheus
+ - Otel
+
+For more info on what providers are supported by which clients see the client docs.
+
+{!_sections/client_docs.md!}
+
+# Example (User Session Duration)
+
+Let us imagine a scenario where we want to track when a user session is destroyed. Let us imagine the follow code is
+called when a user session is destroyed.
+
+```py title="users/session.py" linenums="22"
+def delete_session(self, user, session):
+    self.process_session_end(session)
+    session = self.db.delete_session(user.id)
+    return True
+```
+
+In this example we can create the tracepoint as follows:
+
+```json
+{
+  "path": "users/sessions.py",
+  "line_number": 24,
+  "metrics": [
+    {
+      "name": "session_duration",
+      "type": "HISTOGRAM",
+      "labelExpressions": [
+        {
+          "key": "user_id",
+          "value": {
+            "expression": "user.id"
+          }
+        }
+      ],
+      "expression": "time.time() - session.start_ts"
+    }
+  ]
+}
+```
+
+This tracepoint would result in the Deep intercepting line 24 and creating a new histogram metric using the value of the
+expression `time.time() - session.start_ts` as the value. This metric will then be pushed to metric provider and
+captured by your existing metric services (e.g. Prometheus).
+
+{!_sections/expression.md!}

docs/docs/features/snapshots.md

@@ -0,0 +1,26 @@
+# Snapshots
+
+Deeps primary feature is the ability to capture data at any point in your code. This allows you to gather any data at
+any point to help speed up diagnosis of application issues.
+
+The way Deep triggers controls its data collection is with '[Tracepoints](./tracepoints.md)'. A tracepoint is a
+definition of a point of code (file and line number), with some customisations of how to collect data. The default
+behaviour is to create a snapshot that contains the stack and variable data at that point in the code.
+
+```json
+{
+  "path": "some/file.py",
+  "line_number": 22
+}
+```
+
+## Data Collection
+
+The data that is collected by a snapshot is controlled by the config of the tracepoint but can include:
+
+ - Stack Frames - The list of frames that where executed for the application to get to the tracepoints location, this includes file name, class name, line number etc.
+ - Variables - The variable data that is available at that point of the code, this includes local variables, method parameters (arguments), defining object (this/self)
+ - Watches - User specified values that are evaluated at the point the code is executed
+
+It is possible to customise the data collection by using plugins.
+

docs/docs/features/tracepoints.md

@@ -0,0 +1,62 @@
+# Tracepoints
+
+Tracepoints are the way Deep defines the actions it will take. A tracepoint is primarily a code location (file name and
+line number), however, there can also be other configurations that control how the data is collected and any
+restrictions the client should make.
+
+For details of the structure of a tracepoint see
+the [proto definition](https://github.com/intergral/deep-proto/blob/master/deepproto/proto/tracepoint/v1/tracepoint.proto).
+
+## Tracepoint Types
+
+There are a few different types of tracepoints supported by Deep, each with slightly different config options and use
+cases.
+
+### Line Tracepoints
+
+A line tracepoint is the basic form of a tracepoint and allows tracepoints to be installed on a line of code. There are
+a couple of options that change the behaviour of line tracepoints.
+
+- **Line Start**: This is the default behaviour for line tracepoints and will trigger the tracepoint at the start of the
+  line.
+- **Line End**: This will tell Deep to collect data at the end of the line, and capture the time spent on the line as
+  well as any exceptions from the execution of the line.
+- **Line Capture**: This will combine the two, collecting the data at the start of the line, but also capturing the line
+  execution time and exception.
+
+### Method Tracepoints
+
+A method tracepoint is a form of tracepoint that will wrap a method (or function) and capture the start, or end of a
+method call. There are a few options for this tracepoint type:
+
+- **Method Start**: This is the default behaviour of a method tracepoint, and will capture a snapshot at the start of a
+  method. Capturing only the method parameters (or arguments) and the declaring object (this/self)
+- **Method End**: This type allows the end of a method call to be captured, capturing the result of the call or
+  exception to be captured
+- **Method Capture**: This will combine the two, capturing the data at the start of the method, and capturing the
+  returned data or exception.
+
+## Dynamic Monitoring
+
+Deep provides the ability to create Spans, Metrics and Logs dynamically by configuring the tracepoint appropriately.
+
+- [Logging](./logging.md)
+- [Spans](./traces.md)
+- [Metrics](./metrics.md)
+
+It is also possible to set the tracepoint to not capture any data and only create Spans, Logs or Metrics.
+
+### Logging
+
+All tracepoints can define a log message that will be evaluated at the point the tracepoint is executed and result in a
+log message in the application logs.
+
+### Metrics
+
+All tracepoints can define metrics that will be evaluated at the point the tracepoint is executed and the result will be
+processed by the configured metric processor.
+
+## Span Options
+
+It is also possible to start a span when a tracepoint is captured. The Span will have the name of the method, or
+file/line number it was created from.

docs/docs/features/traces.md

@@ -0,0 +1,34 @@
+# Traces / Spans
+
+Deep offers the ability to dynamically create Spans at arbitrary points in the code. This allows you to augment your
+exising monitoring with specific Spans when you need additional data while diagnosing an issue.
+
+To create a Span the `span` argument should be attached to the tracepoint. This argument can have the values of:
+
+- `method` - This will create a span around the method the tracepoint is located in
+- `line` - This will create a span around the line the tracepoint is on
+
+Additionally, the argument `method_name` can be set (to the name of the method) to crate a span around any method in the
+file with that name. If `method_name` is set then the `line_number` property is ignored for the span. This method also
+allows for '[method entry tracepoints](./method_entry.md)'  
+
+```json
+{
+  "path": "some/file.py",
+  "line_number": 22,
+  "args": {
+    "span": "method"
+  }
+}
+```
+
+# Supported providers
+
+The specific supported providers is dependent on the client that is being used. In all cases custom providers can be set
+using the plugin architecture of the client. As a generalisation the following providers will be detected and used automatically:
+
+- Otel
+
+For more info on what providers are supported by which clients see the client docs.
+
+{!_sections/client_docs.md!}

docs/docs/index.md

@@ -33,5 +33,4 @@ These plugins are planned for release, pending review from Grafana.
 To set up the client you will need to pick the appropriate language and follow the instructions for that client. The
 available clients are:
 
-- [github](intergral/deep-python-client)
-- [github](intergral/deep-java-client)
+{!_sections/client_docs.md!}

docs/mkdocs.yml

@@ -36,6 +36,7 @@ markdown_extensions:
       base_path: docs
 
 plugins:
+  - awesome-pages:
   - glightbox:
   - mkdocs_gitlinks:
       show_docs: true

docs/requirements.txt

@@ -2,3 +2,4 @@ markdown-include
 mkdocs-material
 mkdocs-glightbox
 mkdocs-gitlinks
+mkdocs-awesome-pages-plugin

go.mod

@@ -80,7 +80,7 @@ require (
 	github.com/gogo/protobuf v1.3.2
 	github.com/googleapis/gax-go/v2 v2.6.0
 	github.com/grpc-ecosystem/go-grpc-middleware v1.3.0
-	github.com/intergral/go-deep-proto v1.0.2
+	github.com/intergral/go-deep-proto v1.0.5
 	golang.org/x/exp v0.0.0-20221002003631-540bb7301a08
 )
 

go.sum

@@ -500,6 +500,10 @@ github.com/imdario/mergo v0.3.12 h1:b6R2BslTbIEToALKP7LxUvijTsNI9TAe80pLWN2g/HU=
 github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
 github.com/intergral/go-deep-proto v1.0.2 h1:ezpNLMusTjpQ2yvXUYT9aLmAL3s+/CRYdsMyiKN5ckg=
 github.com/intergral/go-deep-proto v1.0.2/go.mod h1:kYiHIYAykVITDekpSAu1y9Z8Hr6OKpoYJGorm9uaeak=
+github.com/intergral/go-deep-proto v1.0.3 h1:p0fmzX7Qn9ZlDkM1FEGzmy3vXkbzqEN9gUIVghPlknw=
+github.com/intergral/go-deep-proto v1.0.3/go.mod h1:kYiHIYAykVITDekpSAu1y9Z8Hr6OKpoYJGorm9uaeak=
+github.com/intergral/go-deep-proto v1.0.5 h1:j5s2ovw7l+zrNrhjZteP8pE5kZe/fJXhgtwfn2JATeg=
+github.com/intergral/go-deep-proto v1.0.5/go.mod h1:kYiHIYAykVITDekpSAu1y9Z8Hr6OKpoYJGorm9uaeak=
 github.com/ionos-cloud/sdk-go/v6 v6.1.3 h1:vb6yqdpiqaytvreM0bsn2pXw+1YDvEk2RKSmBAQvgDQ=
 github.com/jedib0t/go-pretty/v6 v6.2.4 h1:wdaj2KHD2W+mz8JgJ/Q6L/T5dB7kyqEFI16eLq7GEmk=
 github.com/jedib0t/go-pretty/v6 v6.2.4/go.mod h1:+nE9fyyHGil+PuISTCrp7avEdo6bqoMwqZnuiK2r2a0=