Merge pull request #20 from amazeeio/support-ip-useragent-blocking

feature: support useragent and ip allow/blocking, and namespace interval idling overrides

.github/workflows/aergia-controller.yaml

@@ -13,10 +13,12 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        kindest_node_version: [v1.21.12, v1.22.9, v1.23.6, v1.24.7]
+        kindest_node_version: [v1.23.6, v1.24.7, v1.25.3]
         experimental: [false]
         include:
-          - kindest_node_version: v1.25.0
+          - kindest_node_version: v1.26.3
+            experimental: true
+          - kindest_node_version: v1.27.3
             experimental: true
     steps:
     - name: Checkout
@@ -28,7 +30,7 @@ jobs:
     - name: Setup correct Go version
       uses: actions/setup-go@v2
       with:
-        go-version: '1.18'
+        go-version: '1.21'
     - name: Install kustomize, kubebuilder, helm
       run: |
         #kubebuilder

Dockerfile

@@ -1,5 +1,5 @@
 # Build the manager binary
-FROM golang:1.18-alpine as builder
+FROM golang:1.21-alpine as builder
 
 WORKDIR /workspace
 # Copy the Go Modules manifests

README.md

@@ -21,6 +21,44 @@ To force scale a namespace, you can label the namespace using `idling.amazee.io/
 ### Unidle
 To unidle a namespace, you can label the namespace using `idling.amazee.io/unidle=true`. This will cause the environment to be scaled back up to its previous state.
 
+### Idled
+A label `idling.amazee.io/idled` is set that will be true or false depending on if the environment is idled. This ideally should not be modified as Aergia will update it as required.
+
+### Namespace Idling Overrides
+If you want to change a namespaces interval check times outside of the globally applied intervals, the following annotations can be added to the namespace
+* `idling.amazee.io/prometheus-interval` - set this to the time interval for prometheus checks, the format must be in [30m|4h|1h30m](https://pkg.go.dev/time#ParseDuration) notation
+* `idling.amazee.io/pod-interval` - set this to the time interval for pod uptime checks, the format must be in [30m|4h|1h30m](https://pkg.go.dev/time#ParseDuration) notation
+
+### IP Allow/Block Lists
+It is possible to add global IP allow and block lists, the helm chart will have support for handling this creation
+* allowing IP addresses via `/lists/allowedips` file which is a single line per entry of ip address to allow
+* blocking IP addresses via `/lists/blockedips` file which is a single line per entry of ip address to block
+
+There are also annotations that can be added to the namespace, or individual `Kind: Ingress` objects that allow for ip allow or blocking.
+* `idling.amazee.io/ip-allow-list` - a comma separated list of ip addresses to allow, will be checked against x-forward-for, but if true-client-ip is provided it will prefer this.
+* `idling.amazee.io/ip-block-list` - a comma separated list of ip addresses to allow, will be checked against x-forward-for, but if true-client-ip is provided it will prefer this.
+
+### UserAgent Allow/Block Lists
+It is possible to add global UserAgent allow and block lists, the helm chart will have support for handling this creation
+* allowing user agents via a `/lists/allowedagents` file which is a single line per entry of useragents or regex patterns to match against. These must be `go` based regular expressions.
+* blocking user agents via a `/lists/blockedagents` file which is a single line per entry of useragents or regex patterns to match against. These must be `go` based regular expressions.
+
+There are also annotations that can be added to the namespace, or individual `Kind: Ingress` objects that allow for user agent allow or blocking.
+* `idling.amazee.io/allowed-agents` - a comma separated list of user agents or regex patterns to allow.
+* `idling.amazee.io/blocked-agents` - a comma separated list of user agents or regex patterns to block.
+
+### Verify Unidling Requests
+It is possible to start Aergia in a mode where it will require unidling requests to be verified. The way this works is by using HMAC and passing the signed version of the requested namespace back to the user when the initial request to unidle the environment is received. When a client loads this page, it will execute a javascript query back to the requested ingress which is then verified by Aergia. If verification suceeds, it proceeds to unidle the environment. This functionality can be useful to prevent bots and other systems that don't have the ability to execute javascript from unidling environments uncessarily. The signed namespace value will only work for the requested namespace.
+
+To enable this functionality, set the following:
+- `--verified-unidling=true` or envvar `VERIFIED_UNIDLING=true`
+- `--verify-secret=use-your-own-secret` or envvar `VERIFY_SECRET=use-your-own-secret`
+
+If the verification featuer is enabled, and you need to unidle environments using tools that can't execute javascript, then it is possible to allow a namespace to override the feature by adding the following annotation to the namespace. Using the other allow/blocking mechanisms can then be used to restrict how the environment can unidle if required.
+* `idling.amazee.io/disable-request-verification=true` - set this to disable the hmac verification on a namespace if Aergia has unidling request verification turned on.
+
+If you're using custom template overrides and enable this functionality, you will need to extend your `unidle.html` template with the additional changes to allow it to to perform the call back function or else environments will never unidle. See the bundled `unidle.html` file to see how this may differ from your custom templates.
+
 ## Change the default templates
 
 By using the environment variable `ERROR_FILES_PATH`, and pointing to a location that contains the three templates `error.html`, `forced.html`, and `unidle.html`, you can change what is shown to the end user.

config/rbac/role.yaml

@@ -54,12 +54,6 @@ rules:
   - list
   - patch
   - watch
-- apiGroups:
-  - '*'
-  resources:
-  - '*'
-  verbs:
-  - '*'
 - apiGroups:
   - '*'
   resources:

controllers/idling_controller.go

@@ -41,9 +41,6 @@ type IdlingReconciler struct {
 	Unidler *unidler.Unidler
 }
 
-// all the things
-// +kubebuilder:rbac:groups=*,resources=*,verbs=*
-
 func (r *IdlingReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
 	opLog := r.Log.WithValues("idler", req.NamespacedName)
 
@@ -88,7 +85,7 @@ func (r *IdlingReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctr
 
 	if val, ok := namespace.ObjectMeta.Labels["idling.amazee.io/unidle"]; ok && val == "true" {
 		opLog.Info(fmt.Sprintf("Unidling environment %s", namespace.Name))
-		r.Unidler.UnIdle(ctx, namespace.Name, opLog)
+		r.Unidler.Unidle(ctx, &namespace, opLog)
 		nsMergePatch, _ := json.Marshal(map[string]interface{}{
 			"metadata": map[string]interface{}{
 				"labels": map[string]*string{

go.mod

@@ -1,74 +1,71 @@
 module github.com/amazeeio/aergia-controller
 
-go 1.18
+go 1.21
 
 require (
-	github.com/go-logr/logr v1.2.3
-	github.com/prometheus/client_golang v1.12.2
-	github.com/prometheus/common v0.32.1
+	github.com/go-logr/logr v1.2.4
+	github.com/prometheus/client_golang v1.15.1
+	github.com/prometheus/common v0.42.0
 	github.com/uselagoon/machinery v0.0.8
 	gopkg.in/robfig/cron.v2 v2.0.0-20150107220207-be2e0b0deed5
 	gopkg.in/yaml.v2 v2.4.0
-	k8s.io/api v0.25.3
-	k8s.io/apimachinery v0.25.3
-	k8s.io/client-go v0.25.3
-	sigs.k8s.io/controller-runtime v0.13.0
+	k8s.io/api v0.28.4
+	k8s.io/apimachinery v0.28.4
+	k8s.io/client-go v0.28.4
+	sigs.k8s.io/controller-runtime v0.15.0
 )
 
 require (
-	cloud.google.com/go v0.97.0 // indirect
-	github.com/PuerkitoBio/purell v1.1.1 // indirect
-	github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
-	github.com/cespare/xxhash/v2 v2.1.2 // indirect
+	github.com/cespare/xxhash/v2 v2.2.0 // indirect
 	github.com/davecgh/go-spew v1.1.1 // indirect
-	github.com/emicklei/go-restful/v3 v3.8.0 // indirect
+	github.com/emicklei/go-restful/v3 v3.9.0 // indirect
 	github.com/evanphx/json-patch/v5 v5.6.0 // indirect
-	github.com/fsnotify/fsnotify v1.5.4 // indirect
-	github.com/go-logr/zapr v1.2.3 // indirect
-	github.com/go-openapi/jsonpointer v0.19.5 // indirect
-	github.com/go-openapi/jsonreference v0.19.5 // indirect
-	github.com/go-openapi/swag v0.19.14 // indirect
+	github.com/fsnotify/fsnotify v1.6.0 // indirect
+	github.com/go-logr/zapr v1.2.4 // indirect
+	github.com/go-openapi/jsonpointer v0.19.6 // indirect
+	github.com/go-openapi/jsonreference v0.20.2 // indirect
+	github.com/go-openapi/swag v0.22.3 // indirect
 	github.com/gogo/protobuf v1.3.2 // indirect
 	github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
-	github.com/golang/protobuf v1.5.2 // indirect
-	github.com/google/gnostic v0.5.7-v3refs // indirect
+	github.com/golang/protobuf v1.5.3 // indirect
+	github.com/google/gnostic-models v0.6.8 // indirect
 	github.com/google/go-cmp v0.5.9 // indirect
-	github.com/google/gofuzz v1.1.0 // indirect
+	github.com/google/gofuzz v1.2.0 // indirect
 	github.com/google/uuid v1.3.0 // indirect
 	github.com/imdario/mergo v0.3.12 // indirect
 	github.com/josharian/intern v1.0.0 // indirect
 	github.com/json-iterator/go v1.1.12 // indirect
-	github.com/mailru/easyjson v0.7.6 // indirect
-	github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369 // indirect
+	github.com/mailru/easyjson v0.7.7 // indirect
+	github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect
 	github.com/moby/spdystream v0.2.0 // indirect
 	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
 	github.com/modern-go/reflect2 v1.0.2 // indirect
 	github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
 	github.com/pkg/errors v0.9.1 // indirect
-	github.com/prometheus/client_model v0.2.0 // indirect
-	github.com/prometheus/procfs v0.7.3 // indirect
+	github.com/prometheus/client_model v0.4.0 // indirect
+	github.com/prometheus/procfs v0.9.0 // indirect
 	github.com/spf13/pflag v1.0.5 // indirect
 	go.uber.org/atomic v1.7.0 // indirect
 	go.uber.org/multierr v1.6.0 // indirect
-	go.uber.org/zap v1.21.0 // indirect
-	golang.org/x/net v0.0.0-20220722155237-a158d28d115b // indirect
-	golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8 // indirect
-	golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f // indirect
-	golang.org/x/term v0.0.0-20210927222741-03fcf44c2211 // indirect
-	golang.org/x/text v0.3.7 // indirect
-	golang.org/x/time v0.0.0-20220609170525-579cf78fd858 // indirect
-	gomodules.xyz/jsonpatch/v2 v2.2.0 // indirect
+	go.uber.org/zap v1.24.0 // indirect
+	golang.org/x/net v0.17.0 // indirect
+	golang.org/x/oauth2 v0.8.0 // indirect
+	golang.org/x/sys v0.13.0 // indirect
+	golang.org/x/term v0.13.0 // indirect
+	golang.org/x/text v0.13.0 // indirect
+	golang.org/x/time v0.3.0 // indirect
+	gomodules.xyz/jsonpatch/v2 v2.3.0 // indirect
 	google.golang.org/appengine v1.6.7 // indirect
-	google.golang.org/protobuf v1.28.0 // indirect
+	google.golang.org/protobuf v1.31.0 // indirect
 	gopkg.in/inf.v0 v0.9.1 // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect
-	k8s.io/apiextensions-apiserver v0.25.0 // indirect
-	k8s.io/component-base v0.25.0 // indirect
-	k8s.io/klog/v2 v2.70.1 // indirect
-	k8s.io/kube-openapi v0.0.0-20220803162953-67bda5d908f1 // indirect
-	k8s.io/utils v0.0.0-20220728103510-ee6ede2d64ed // indirect
-	sigs.k8s.io/json v0.0.0-20220713155537-f223a00ba0e2 // indirect
+	k8s.io/apiextensions-apiserver v0.27.2 // indirect
+	k8s.io/component-base v0.27.2 // indirect
+	k8s.io/klog/v2 v2.100.1 // indirect
+	k8s.io/kube-openapi v0.0.0-20230717233707-2695361300d9 // indirect
+	k8s.io/utils v0.0.0-20230406110748-d93618cff8a2 // indirect
+	sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd // indirect
 	sigs.k8s.io/structured-merge-diff/v4 v4.2.3 // indirect
 	sigs.k8s.io/yaml v1.3.0 // indirect
 )