v0.4.0

This release contains a few cleanup that was way overdue:

1. Reset the author on all my commits, hence no changelog, since it changed all the commit hashes, and the tags are invalid up to this point.
2. Replace the goreleaser config and release flow with good old bash and make files (#143).
3. Extract plugin types, constants, protobuf files, buf configs and stubs to gatewayd-plugin-sdk (gatewayd-io/gatewayd-plugin-sdk#1).
4. Enable dependabot to help with upgrading dependencies.
5. A few other fixes and cleanups.

v0.3.5

This release officially marks the end of milestone 0.3.x. 🚀 The focus of the milestone was on reliability and most of the tickets belong to that overarching goal. These are overall the changes:

1. Expose Prometheus metrics from the plugins over HTTP via Unix domain socket (UDS)
2. Collect, aggregate, relabel and merge metrics from the core with the plugins' metrics and expose them over HTTP
3. Add version command to get information the running version of GatewayD core (for now), commit, time, Go version, architecture and OS
4. Multiple logger outputs can be chosen in the config
5. Add logging output to syslog and rsyslog
6. Remove use of embedded PostgreSQL in tests, that caused tests to randomly fail
7. Initialize logger right after configs are loaded and before any plugins and remove use of DefaultLogger
8. Add Sentry for collecting exceptions (can be disable via ./gatewayd run --sentry=false)
9. Add build and publish workflows with goreleaser and GitHub Actions (see attached files below)
10. Write a minimal README
11. Lots of refactoring, fixes and clean-ups

GatewayD core seems mature enough for handling multiple plugins and the focus of the next milestones will be on creating new plugins to harness the power of the core. Plus, the core will be adjusted based on plugins' needs.

Full Changelog: v0.3.4...v0.3.5

v0.3.4

This release was made to test the CD workflow and subsequently the goreleaser. Changelogs from this release on will be published like below with an introduction.

What's Changed

• Add build and release workflow by @mostafa in #135

Full Changelog: v0.3.3...v0.3.4

v0.3.3

This release contain two major changes:

1. The logger is now created and initialized after reading configuration from the file, and before loading plugins. This removes the need to have two loggers, one DefaultLogger for plugin registry and the rest of the code, right before the other logger is actually initialized.
2. Sentry is added for capturing panics and reporting important errors, so they can be fixed in later versions.

What's Changed

• Load logger first by @mostafa in #133
• Add sentry by @mostafa in #134 and 4bdee07

Full Changelog: v0.3.2...v0.3.3

v0.3.2

In this release, the focus was on improving the logging feature and the overall stability of the tests:

1. Multiple logger outputs can chosen in the global config.
2. Syslog and rsyslog outputs are added.
3. Remove embedded postgres in favor of PostgreSQL service in the GitHub Actions.

What's Changed

• Remove embedded postgres by @mostafa in #130
• Use multiple outputs in logger by @mostafa in #129
• Add logging to (r)syslog by @mostafa in #131

Full Changelog: v0.3.1...v0.3.2

v0.3.1

This release contains slight refactoring of the metrics merger functionality. The version command is also added to get the information about the running GatewayD executable. All configuration handling code has been moved from the run command into the config package, which makes the code more readable and easier to debug and maintain.

There was an issue with the CI jobs, which resulted in tests failing unexpectedly. My initial guess is because of running PostgreSQL both as a service in the test-plugin job and as embedded server in the test job. I made them serial, so that test-plugin job runs after test job succeeds. This should be fixed in the future and there should only be one PostgreSQL service for all the tests.

What's Changed

• Refactor merger slightly by @mostafa in #126
• Add version subcommand by @mostafa in #127
• Reorganize config by @mostafa in #128

Full Changelog: v0.3.0...v0.3.1

v0.3.0

This release contains a very useful feature: exposing metrics from GatewayD and collecting, merging, relabeling and exposing metrics from plugins along with it. The plugins can expose their metrics on HTTP via a Unix domain socket, like /tmp/gatewayd-plugin-test.sock. Upon start GatewayD loads the plugins and registers their UDS path by retrieving their compile-time configuration, which happens on plugin registration by calling the plugin's GetPluginConfig method. Later in the initialization process, an async scheduler runs to periodically scrap metrics from plugins that expose metrics.
Upon each round of scraping metrics, GatewayD will merge and relabel the plugins' metrics with the name of the plugin. That is, gatewayd_test_metric_total becomes gatewayd_test_metric_total{plugin="test"}. Also, for the ease of access and the uniformity of metrics, the namespace must be set to "gatewayd". This is not a requirement, and it isn't enforced anywhere, but if not set, it makes metrics hard to find on Prometheus UI and other visualization platforms, especially if they have non-uniform names.
The scheduler runs periodically and the interval between each runs can be set using the metricsMergerPeriod parameter in the plugins' configuration file, gatewayd_plugins.yaml. It is set to 5s by default, but can be adjusted based on demand. Don't try to set it to a very low value unless you absolutely have to. Note that the scheduler runs in singleton mode, and prevents new jobs to be run, if the previous hasn't finished processing, which might be caused by setting the parameter very low.

What's Changed

• Add Prometheus metrics by @mostafa in #121
• Collect and merge Prometheus metrics from plugins by @mostafa in #125

Full Changelog: v0.2.5...v0.3.0

v0.2.5

This release officially marks the end of milestone v0.2.x. This milestone had a lot of changes to the core and the plugin subsystem. The core is more robust and stable and the plugin subsystem can handle a lot of edge cases and issues manifested in the previous milestone and the features needed to be added. Here are the changes:

1. Add command-line arguments (flags) to plugins via gatewayd_plugin.yaml config file (#75)

2. Pass environment variables to plugins via gatewayd_plugin.yaml config file (#77)

   verificationPolicy: "passdown"
   compatibilityPolicy: "strict"
   acceptancePolicy: "accept"
   
   plugins:
     - name: gatewayd-plugin-test
       enabled: True
       localPath: ./gatewayd-plugin-test
       args: ["--parameter", "value"]
       env:
         - key=value
       checksum: b1cfe3c6b268d904abe405b106ef737da5373912586ff8a16695939c38caf30d

   The plugins can optionally opt-in to receive the required MAGIC_COOKIE_KEY and MAGIC_COOKIE_VALUE via environment variables, which can be passed via the above config file. GatewayD needs the following constants to be passed, so it verifies the plugin is valid:

   MAGIC_COOKIE_KEY=GATEWAYD_PLUGIN
   MAGIC_COOKIE_VALUE=5712b87aa5d7e9f9e9ab643e6603181c5b796015cb1c09d6f5ada882bf2a1872

3. Add requires field to plugin config to check for loaded plugins required by the current plugin (#81)

4. Add a new config parser for global and plugin config (#84)
   The global config file, gatewayd.yaml, contains a default object for each object except server (for now). This is the foundation to have multiple objects of the same type with different configuration for running multi-tenant servers (TBD), clients, loggers and proxy and to connect to multiple database servers at the same time for aggregations, load-balancing and related functions.

5. Use multiple sources for configuring GatewayD including environment variables (#86)
   Multiple sources calls for order of precedence in how different sources affect the result of configurations in GatewayD. This will be the precedence order for loading and applying config:

   1. Plugin default values (lowest precedence)
   2. Plugin config file
   3. Environment variables for plugins
   4. Global default values
   5. Environment variables for global config
   6. Global config updated by plugins via OnConfigLoaded hooks: plugins override the global configs and has the highest precedence overall.

6. Add two new hooks for controlling traffic (#94, #96 and #98):

   hook type	description
   OnTraffic	Notification of incoming traffic from the client.
   OnTrafficFromClient	Traffic is received from the client. This can terminate the connection.1️⃣
   OnTrafficToServer	Traffic is sent to the database server.
   OnTrafficFromServer	Traffic is received from the database server.
   OnTrafficToClient	Traffic is sent to the client.

   1️⃣ As explained here, it is now possible to terminate the connection before the request even reaches the database server. To achieve this, the plugins needs to have the OnTrafficFromClient RPC endpoint and also expose the hook in their config. Then they can achieve connection termination by setting two variables in the return value of the RPC endpoint:

   return structpb.NewStruct(map[string]interface{}{
       "terminate": true,
       "response":  base64.StdEncoding.EncodeToString([]byte{'X', 0, 0, 0, 4}),
   })

   The terminate variable instructs the proxy to stop sending the received data (from client) to the server. The response variable will be sent back to the client, instead. If there is no response in the return value, the connection from the client will close abruptly.
   The use case of the second change mentioned above is to let plugins stop requests from reaching the server, for example in a cache plugin scenario, the plugin should decide whether to return the cached value (of a query, which is the previous server's response) or let the message reach the server (and cache it afterwards). This is now possible using connection termination.

7. Recycle available connections using an internal in-process scheduler (#101) to avoid having stale and timed-out connections in the pool of available connections

8. Add custom hooks and the OnHook method for plugin to process those custom hooks #110

9. Add log file rotation #115 (current release)

10. Other changes, fixes and cleanups: #80, #82, #104, #105, #106, #109 and #117.

Full Changelog: v0.2.4...v0.2.5

v0.2.4

In this release, I've cleaned up the code as best as I can. This release also contains a new hook, OnHook, and the introduction of custom hook names to be mapped with OnHook. This way, any hook name can be used to call the OnHook method of the plugins and then return the results based on the hook name in the method's args.

What's Changed

• Minor fixes by @mostafa in #104
• Fix time.Duration parsing by @mostafa in #105
• Clean-ups for CLI flag parsing by @mostafa in #106
• Hooks cleanup and merger by @mostafa in #109
• Make hooks more generic with generic gRPC methods by @mostafa in #110

Full Changelog: v0.2.3...v0.2.4

v0.2.3

This release is considered a stable release, because it contains fixes to the connection handling. So far, GatewayD had the issue of connections becoming stale over time, which was is partially due to how GatewayD works and mainly due to the authentication timeout of TCP connections made to PostgreSQL server. PostgreSQL server has a timeout for TCP connections that connect, but do not authenticate. After the timeout, the connection is alive, and sending data seems to work, however, receiving data from the database server causes io.EOF and connection timeouts. This issue was really hard to detect and it happened often. I tried so many different ways to overcome this, but to no avail. In PR #101 I have introduced an in-process task scheduler to run a cleanup, aka. recycle or retry, job every 60 seconds (default, configurable). The job will iterate through all the available connections in the proxy's connection pool and runs this sequence:

1. Close the connection.
2. Removes the connection from the pool.
3. Create a new connection.
4. Add the new connection to the pool.

And manual testing shows that the issue is fixed. On top of the 60s interval between running the above loop (job), the scheduled job starts with a delay of 60s. That is, new connections won't be recycled; only stale ones. The rest of the changes are cleanups, fixes, slight refactoring and disabling of the test plugin in favor of the cache plugin.

What's Changed

• Implement onTrafficToClient by @mostafa in #98
• Recycle connection on timeout by @mostafa in #101

Full Changelog: v0.2.2...v0.2.3