This chapter describes the features and possible implementation ideas/solutions of those functionalities that make up a real Dogu in the first place and solve recurring problems.
Therefore, the following sections deal with recurring functionalities on how a Dogu can embed itself into the Cloudogu EcoSystem landscape.
Given a Dogu provides this, users can enjoy the benefits of enterprise-level single sign-on (SSO) and single log-out (SLO). Authentication is enabled by the Central Authentication Service (CAS). Dogu developers have several authentication options at their disposal. The following three sections deal with methods that have already been successfully used, namely:
- Authentication using CAS protocol
- Authentication using OAuth 2.0 protocol
- Authentication using OpenID Connect protocol
The basis for CAS is always a correct configuration of a user/group directory, be it the LDAP dogu or an external LDAP-compatible service.
Authentication within the Cloudogu EcoSystem is done using the CAS protocol, which enables single sign-on and single log-out. Different CAS protocol versions are supported (2.0 and 3.0). When implementing using CAS protocol, it is recommended to use version 3.0 because (compared to version 2.0) it can return important user attributes after a successful authentication.
The following diagram shows the parties involved in the authentication configuration. Before a dogu (here using Redmine as an example) can participate in this process, the dogu must internally configure a set of CAS URLs:
- CAS log-in URL (for redirecting users to the web form).
- CAS validation URL (for validating service tickets)
- CAS log-out URL (to invalidate an SSO session)
- The service recognition in CAS against the Dogus happens automatically during the Dogu installation and will not be considered further in the following.
The actual authentication happens via a sequence of HTTP redirects and the exchange of session cookies in the background, of which users are unaware. The display of the CAS login form as an active login step stands out strongly.
The following figure provides a rough overview of the login process with the parties involved.
The SSO of the CAS significantly reduces this process when logging in to additional dogus.
For more information and a more detailed illustration before, during and after an authentication, see the CAS documentation.
entry for a normal CAS client:
{
"ServiceAccounts": [
{
"Type": "cas",
"Params": [
"cas"
]
}
]
}
CAS offers OAuth/OIDC as a protocol for authentication including SSO/SLO. The following describes the specification of OAuth 2.0 protocol in CAS.
In order for a Dogu to use the CAS's OAuth endpoints, it must log in to the CAS as a client. To do this, the OAuth-specific CAS service can be requested via the dogu.json
of the respective dogu.
entry for an OAuth client:
{
"ServiceAccounts": [
{
"Type": "cas",
"Params": [
"oauth"
]
}
]
}
The service account credentials are randomly generated (see create-sa.sh) and stored encrypted in the registry under the path /config/<dogu>/sa-cas/oauth
and /config/<dogu>/sa-cas/oauth_client_secret
.
The credentials are composed of the CLIENT_ID
and the CLIENT_SECRET
. For the CAS, the CLIENT_SECRET
is stored as a hash in the Cloudogu EcoSystem Registry under the path /config/cas/service_accounts/oauth/<CLIENT_ID>/secret
.
The following steps describe a successful OAuth authentication flow.
- request a short term token: see section "OAuth authorize endpoint" below.
- swap short-term token for long-term token: see section below "AccessToken endpoint".
- long term token can now be used to authenticate against resources. Currently, CAS only offers the user's profile as a resource: See section OAuth user profile.
Note: Resource is a part of CAS.
This endpoint serves as the initial start of OAuth authorization.
The authorize
endpoint is used to request a short-lived token from the CAS.
URL : <fqdn>/oauth2.0/authorize
Method : GET
Condition of the data
response_type = code
client_id = Valid clientID from the dogu
state = Any string
redirect_url = <URL to which the short-term token will be redirected upon successful authentication>
Data example
response_type = code
client_id = portainer
state = b8c57125-9281-4b67-b857-1559cdfcdf31
redirect_url = http://local.cloudogu.com/portainer/
call example
https://local.cloudogu.com/cas/oauth2.0/authorize?client_id=portainer&redirect_uri=http%3A%2F%2Flocal.cloudogu.com%2Fportainer%2F&response_type=code&state=b8c57125-9281-4b67-b857-1559cdfcdf31
Automatically redirects the user to the CAS login screen.
After successful login the redirect_url
is passed as code
GET parameter.
Example for code
: ST-1-wzG237MUOvfjfZrvRH5s-cas.ces.local
This endpoint is used to exchange a short term token (code
) for a long term token (access_token
).
URL : <fqdn>/oauth2.0/accessToken
Method : GET
Data constraints
grant_type = authorization_code
code = Valid code from `authorize` endpoint
client_id = Valid clientID from the dogu
client_secret = Valid secret from the dogu
redirect_url = <URL to which the long term token will be sent upon successful authentication>
Data example
grant_type = authorization_code
code = ST-1-wzG237MUOvfjfZrvRH5s-cas.ces.local
client_id = portainer
client_secret = sPJtcNrmROZ3sZu3
redirect_url = https://local.cloudogu.com/portainer/
Request example
https://local.cloudogu.com/cas/oauth2.0/accessToken?grant_type=authorization_code&code=ST-1-wzG237MUOvfjfZrvRH5s-cas.ces.local&client_id=portainer&client_secret=sPJtcNrmROZ3sZu3&redirect_uri=https%3A%2F%2Flocal.cloudogu.com%2Fportainer%2F
Status: 200 OK
Example response:
{
"access_token": "TGT-1-m2gUNJwEqXyV7aAEXekihcVnFc5iI4mpfdZGOTSiiHzEbwr1cr-cas.ces.local",
"expires_in": "7196",
"token_type": "Bearer"
}
Error: The short-term token is invalid or has already expired.
Status: 500 Bad request
Example response:
{
"message": "invalid_grant"
}
This endpoint is used to retrieve the user profile from the logged in user using a long term token (access_token
).
URL : <fqdn>/oauth2.0/profile
Method : GET
Request header
authorization = Valid Access Token as Bearer
Request header example
authorization: Bearer TGT-1-m2gUNJwEqXyV7aAEXekihcVnFc5iI4mpfdZGOTSiiHzEbwr1cr-cas.ces.local
Status: 200 OK
Example response:
{
"id": "cesadmin",
"attributes": {
"username": "cesadmin",
"cn": "admin",
"mail": "[email protected]",
"givenName": "admin",
"surname": "admin",
"displayName": "admin",
"groups": [
"cesManager",
"cesadmin"
]
}
}
Error: The long-term token is invalid or has already expired.
Status: 500 Bad request
Example response:
{
"message": "expired_accessToken"
}
This endpoint is used to terminate the OAuth session. The logout endpoint is used to invalidate the long term token from CAS.
URL : <fqdn>/cas/logout
CAS provides OAuth/OpenID Connect (OIDC) as a protocol for authentication including SSO/SLO. The following describes the specification of the OpenID Connect protocol in CAS.
In order for a dogu to use the CAS's OIDC endpoints, it must log in to the CAS as a client. To do this, the request for an OIDC-specific CAS service account can be stored in the dogu.json
of the dogu in question.
entry for an OIDC client:
{
"ServiceAccounts": [
{
"Type": "cas",
"Params": [
"oidc"
]
}
]
}
The service account credentials are randomly generated (see create-sa.sh) and stored encrypted in the registry under the path /config/<dogu>/sa-cas/oidc
and /config/<dogu>/sa-cas/oidc_client_secret
.
The credentials are composed of the CLIENT_ID
and the CLIENT_SECRET
. For the CAS, the CLIENT_SECRET
is stored as a hash in the Cloudogu EcoSystem Registry under the path /config/cas/service_accounts/oidc/<CLIENT_ID>/secret
.
This endpoint serves as the initial start of the OpenID Connect authorization.
The authorize
endpoint is used to request a short-lived token from the CAS.
URL : <fqdn>/oidc/authorize
Method : GET
Condition of the data
response_type = code
client_id = Valid clientID from the dogu
state = Any string
redirect_url = <URL to which the short term token of successful authentication will be redirected>.
Data example
response_type = code
client_id = teamscale
state = b8c57125-9281-4b67-b857-1559cdfcdf31
redirect_url = http://local.cloudogu.com/teamscale/
Request example
https://local.cloudogu.com/cas/oidc/authorize?client_id=portainer&redirect_uri=http%3A%2F%2Flocal.cloudogu.com%2Fteamscale%2F&response_type=code&state=b8c57125-9281-4b67-b857-1559cdfcdf31
Automatically redirects one to the CAS login screen.
After successful login the redirect_url
is passed with a code
as GET parameter.
Example for code
: ST-1-wzG237MUOvfjfZrvRH5s-cas.ces.local
This endpoint is used to exchange a short term token (code
) for a long term token (access_token
).
URL : <fqdn>/oidc/accessToken
Method : GET
Data constraints
grant_type = authorization_code
code = Valid code from `authorize` endpoint
client_id = Valid clientID from the dogu
client_secret = Valid secret from the dogu
redirect_url = <URL to which the long term token of successful authentication is sent>
Data example
grant_type = authorization_code
code = ST-1-wzG237MUOvfjfZrvRH5s-cas.ces.local
client_id = teamscale
client_secret = sPJtcNrmROZ3sZu3
redirect_url = https://local.cloudogu.com/teamscale/
Request example
https://local.cloudogu.com/cas/oidc/accessToken?grant_type=authorization_code&code=ST-1-wzG237MUOvfjfZrvRH5s-cas.ces.local&client_id=teamscale&client_secret=sPJtcNrmROZ3sZu3&redirect_uri=https%3A%2F%2Flocal.cloudogu.com%2Fteamscale%2F
Status: 200 OK
Example response:
{
"access_token": "TGT-1-m2gUNJwEqXyV7aAEXekihcVnFc5iI4mpfdZGOTSiiHzEbwr1cr-cas.ces.local",
"expires_in": "7196",
"token_type": "Bearer"
}
Error: The short-term token is invalid or has already expired.
Status: 500 Bad request
Example response:
{
"message": "invalid_grant"
}
This endpoint is used to terminate the OpenID Connect session. The logout endpoint is used to invalidate the long term token from the CAS.
URL : <fqdn>/oidc/logout
Method : GET
Condition of the data
response_type = code
client_id = Valid clientID from the dogu
state = Any string
redirect_url = <URL to which the short term token of successful authentication will be redirected>.
Data example
response_type = code
client_id = teamscale
state = b8c57125-9281-4b67-b857-1559cdfcdf31
redirect_url = http://local.cloudogu.com/teamscale/
Request example
https://local.cloudogu.com/cas/oidc/authorize?client_id=teamscale&redirect_uri=http%3A%2F%2Flocal.cloudogu.com%2Fteamscale%2F&response_type=code&state=b8c57125-9281-4b67-b857-1559cdfcdf31
Automatically redirects to the CAS login screen.
After successful login the redirect_url
is passed with a code
as GET parameter.
Example for code
: ST-1-wzG237MUOvfjfZrvRH5s-cas.ces.local
The Cloudogu EcoSystem Registry is a key-value database that is also a core element for Dogus. The configuration of a Dogu is performed via the Registry. In addition, values that are also of a global nature are stored in the registry.
The following image focuses on parts that play a role in the communication between Dogu (that is: Redmine) and the Registry:
- Registry: It runs beyond the Dogu container and can be reached via an IP address from the container network.
- After installation, the registry contains the public key of each dogu among other keys.
- File
/etc/ces/node_master
.- This is automatically mounted in the dogu during the dogu installation. It contains the IP address of the registry so that the dogu can access the registry.
- File
/private/private.pem
- This file contains the private key of the dogu. This is used, among other things, to decrypt encrypted registry values.
- Often these are service accounts to other Dogus.
The Dogu specific configuration is located in the registry path /config/<dogu>/
. Registry keys should be written in snake_case
, i.e. lowercase with underscores.
A valuable help is the command line tool doguctl
among other things also in the start phase of the Dogu container. This tool simplifies access to the registry, e.g. by automatically reading the node_master
file or simply addressing Dogu's own registry keys. A list of the exact commands can be found under The use of doguctl.
The dogu.json
allows you to define your own configuration values, which can also be validated automatically.
# reads configuration value from the key /config/<dogu>/my_key
$ doguctl config my_key
my old value
# reads global configuration value from /config/_global/fqdn
$ doguctl config -g fqdn
your-ecosystem.example.com
There are other areas beyond the dogu-specific registry values that are of interest in the dogu startup phase.
The global configuration is located in the registry path /config/_global/
and can be managed with doguctl.
- Global values
/config/_global
/config/_global/fqdn
- the FQDN of this Cloudogu EcoSystem instance.
- e.g. eco.example.com
- This value may change during operation. See also the section on Changeability of the FQDN.
/config/_global/domain
- the domain portion of this Cloudogu EcoSystem instance, often identical with the mail domain.
- e.g. example.com
/config/_global/mail_address
- the email address of the instance administrator
- e.g. [email protected]
/config/_global/admin_group
- the current name of the LDAP group whose members administer the Cloudogu EcoSystem instance in the UI.
- This value may change during operation. See also the section on admin group changeability.
- Dogu states
/state/<dogu>
- if the dogu defines a HealthCheck of type
state
, then it allows administrators and other dogus to get health hints on the dogu. - In your own dogu e.g. set
doguctl state installing
when a longer installation routine is started. Just before the main process is started then usedoguctl state ready
to indicate a proper operating state. - In the EcoSystem host this can be checked with
cesapp healthy <dogu>
. - In other dogus this can be checked with
doguctl healthy <dogu>
.- With the
--timeout <seconds>
switch, you can thus wait for dogus on which your own dogu depends. - A non-zero exit code (0) indicates that either the dogu did not finish its startup process in time or the registry communication failed.
- With the
- if the dogu defines a HealthCheck of type
A simple Dogu actually only needs a dogu.json
and a container image. But what happens when the reality of continuous operation differs from the assumptions made during Dogu development?
At Cloudogu, we quickly discovered that it is more clever to be able to deal with such changes during operation. The spectrum here is broad. It can be waiting times compared to other Dogus, or changing originally fixed identifiers to glitches in the container engine.
Recurring tasks at container startup can be found in the section "Typical Dogu Features" in the sections:
In order to dynamically respond to these conditions, Cloudogu has adopted the practice of not starting the main process in the container directly. Instead, error cases are first checked in a startup script, new configuration values are set if necessary, and only then the main process is started, e.g. like this:
- container starts
startup.sh
. startup.sh
- sets own options
- calls own commands to implement specific functionalities, e.g. to
- wait for dogu dependencies
- perform one-time installation processes
- generate a temporary admin account
- apply the current dogu log level to the dogu application
- customize the registration the Cloudogu EcoSystem's TLS certificates in the container
- Prepare system changes to the registry with API accesses to prepare the software
- Replication of LDAP groups/permissions
- Configure the software (log level, other configuration, e.g. by
doguctl template
) - React on system changes of the registry: e.g. admin-group has changed, which needs further action in the Dogu
- set the Dogu state to
ready
(appropriate HealthCheck provided) - start the main process
This section therefore addresses findings and best practices related to such startup scripts: The startup.sh
.
By the way, diligent developers can gather inspiration in Cloudogu`s own startup scripts e.g. in the Redmine dogu.
In order to run a script in a dogu, a script interpreter must exist in the container image. This can be an official package (like bash
), but there is nothing against using your own script interpreter.
There is a broad landscape of available scripting languages. While interpreters like Python, Ruby, Groovy, or Kotlin provide a certain developer experience, they also bring a larger binary size to the table. Since the Bash interpreter is widely used in the Linux community and because of its small footprint regarding the container image size, Bash syntax is used here.
Depending on the dogu, errors can occur in different places and (stupidly) be swallowed again. It is a good practice to abort the script with an appropriate exit code when errors occur, in order to identify the cause of the error more quickly.
The first thing to do is to set the following options
#!/bin/bash
set -o errexit # terminate the whole script (and thus the container) on an uncaught error
set -o nounset # find uninitialized variables
set -o pipefail # don't ignore errors on pipe usage
How an error is caught is an implementation detail. Sometimes it is convenient to test an error case and send a separate message, sometimes the error itself is sufficient.
If you already expect errors, then this construct can be helpful:
yourCommandExitCode=0
your-command || yourCommandExitCode=$?
if [[ ${yourCommandExitCode} -ne 0 ]]; then
echo "ERROR: Oh no. An error occurred during the execution of your-command.";
doguctl state "ErrorYourFunctionality".
sleep 300
exit 1
fi
The error of the command your-command
is caught. A separate error text is printed. The dogu-state is set to a non-ready
value, which could be useful for debugging. Finally, five minutes (= 300 seconds) are waited until exit 1
restarts the dogu as faulty.
The longer you develop an application, the more functionality you add to it. The same can happen with startup.sh
. Apart from easier testability (e.g. with BATS), smaller functions and components are easier to understand.
It is therefore a good idea to design Dogu functions in startup.sh
in the same way from the point of view of testability, readability or refactorability.
Bash functions help here:
function setDoguLogLevel() {
echo "Mapping dogu specific log level..."
currentLogLevel=$(doguctl config --default "WARN" "logging/root")
# map here the log level to the log configuration of the dogu
}
Recurring functions can be abstracted and parameterized:
function deleteSetupArtifact() {
local artifactFile="${1}"
rm -f "${artifactFile}"
}
function cleanUpSetup() {
deleteSetupArtifact "/tmp/tracker-extract-3-files.127" }
deleteSetupArtifact "/usr/share/lib/intermediateFile.bin"
}
As complexity increases, it is worth an idea to add a echo
to relevant steps to speed up a search for the error in log output in case of an error.
At the same time, the Dogu-State-HealthCheck provides the possibility to mark custom execution phases during Dogu startup. For example doguctl state 'aSpecialInstallPhase'
at the beginning or doguctl state 'ready'
at the end of the script.
A HealthCheck of type state
will only succeed if the state contains the string ready
.
Another possibility is to outsource functions to other script files, which are then included in the actual startup script using source
. It should be noted, however, that even source
can fail and should be handled.
sourcingExitCode=0
source /resources/util.sh || sourcingExitCode=$?
if [[ ${sourcingExitCode} -ne 0 ]]; then
echo "ERROR: An error occurred while sourcing /resources/util.sh: Exit code ${sourcingExitCode}"
doguctl state "ErrorSourcingUtilSh"
sleep 300
exit 1
fi
For more complex tasks, a startup script such as startup.sh
may gain complexity. Like any other program, it should be tested and analyzed for robustness. There are several ways to do this. BATS can assist in unit testing shell functions.
Analysis tools such as Shellcheck can detect errors in the use of shell calls.
In the multinode CES, etcd
has been replaced by ConfigMaps and SecretMaps as the storage location
for the Dogus configuration. Config and SecretMaps are made available to a pod as a mounted volume via Kubernetes.
These volumes are readonly. The keys of SecretsMaps are decrypted by Kubernetes before mounting.
This means that doguctl
cannot write a configuration and sensitive data no longer needs to be encrypted.
If a configuration value is set with doguctl
, it is written to the LocalConfig. This is a file that is located
in a writable volume. Sensitive configuration values are not encrypted and are also written to the LocalConfig.
If configuration values are read, sensitive values have the highest priority, followed by non-sensitive values. Values in the LocalConfig have the lowest priority. For example, if a key is in the ConfigMap and in the LocalConfig, the value from the ConfigMap is used.
The section about registry access has already touched on the subject of doguctl
. doguctl
is a command line tool that bundles and simplifies recurring interactions with the Cloudogu EcoSystem registry. This section describes possible calls.
With --help
, each subcommand of doguctl
prints a help page.
This call reads and writes configuration values.
# reads configuration value from the key /config/<dogu>/my_key
$ doguctl config my_key
my old value
# reads global configuration value from /config/_global/fqdn
$ doguctl config -g fqdn
your-ecosystem.example.com
# reads configuration value from /config/<dogu>/my_key and returns default value if not set
$ doguctl config --default NOVALUE my_key2
NOVALUE
# reads encrypted DB name /config/<dogu>/sa-postgresql/db_name
$ doguctl config -e sa-postgresql/db_name
your-dogu-database-1234
# writes configuration value into /config/<dogu>/my_key
$ doguctl config my_key 'my new value'
# writes encrypted secret /config/<dogu>/secret/credential
$ doguctl config -e secret/credential '$up3r$3cre7'
# deletes the key /config/<dogu>/delete_me
$ doguctl config --rm delete_me
This call validates registry configuration values. The command validate
returns an exit code == 0 if all health checks of the dogus under consideration return a positive result. Otherwise it returns an exit code == 1.
$ doguctl validate logging/root # validate single value, good case: value=WARN from ERROR,WARN,INFO,DEBUG
Key logging/root: OK
$ echo $?
0
$ doguctl validate logging/root # validates single value, error case: value=foo from ERROR,WARN,INFO,DEBUG
Key logging/root: INVALID
at least one key was invalid
$ echo $?
1
$ doguctl validate --all # validates all values with output
...
Key container_config/memory_limit: OK
Key container_config/swap_limit: OK
Key container_config/java_max_ram_percentage: OK
Key container_config/java_min_ram_percentage: OK
Key logging/root: INVALID
Key caching/workdir/size: OK (No Validator)
at least one key was invalid
$ echo $?
1
$ doguctl validate --silent # validates values without output (can be combined with --all), good case
$ echo $?
0
$ doguctl validate --silent # validates values without output (can be combined with --all), error case
at least one key was invalid
echo $?
1
This call checks whether a given dogu is healthy. The command healthy
returns an exit code == 0 if the given dogu is healthy. Otherwise, it returns an exit code == 1.
if ! doguctl healthy --wait --timeout 120 postgresql; then
echo "timeout reached by waiting of ldap to get healthy"
exit 1
fi
This call reads and writes dogu state values. It is used in combination with a HealthCheck.
It is lost upon restart of the container. For persistent values that need to be present even after restarts, doguctl config
should be used.
$ doguctl state "installing" # write the value to the state
# Do installation
$ doguctl state # reads the value from the state
installing
$ doguctl state "ready" # set the state to the default healthy value
# start the application
This call generates random strings, suitable for generating passwords or other credentials.
$ doguctl random # generates random string with length 16
9HoF4nYmlLYtf6Ju
$ doguctl random -l 5 # generates random string with length 5
W6Wmj
This call creates a file from a golang template. Different registry values and environment variables can be processed directly in this.
The call takes one or two parameters:
$ doguctl template <template file> [output file]
If the template
command is used without an output file, then the rendered output is printed to stdout
.
Supported template parameters:
- Environment variables
.Env.Get <environment variable>
- uses previously exported environment variable "ADMIN_USERNAME"
- Dogu configuration
.Config.Get <Dogu configuration key>
- uses unencrypted Dogu configuration value under/config/<dogu>/<key>
.Config.GetOrDefault <Dogu configuration key> <default value>
- uses unencrypted Dogu configuration value or provided default value.Config.GetAndDecrypt <encrypted Dogu configuration key>
- decrypts Dogu configuration value and uses it.Config.Exists <Dogu configuration key>
- returns abool
value if a Dogu configuration value exists under/config/<dogu>/<key>
- Global configuration
.GlobalConfig.Get <global configuration key>
- uses global configuration value/config/_global/<key>
.GlobalConfig.GetOrDefault <global configuration key> <default value>
- uses unencrypted global configuration value or provided default value.GlobalConfig.Exists <Dogu configuration key>
- returns abool
value if a global configuration value exists at/config/_global/<key>
- Dogu queries
Example of a template file:
# my-pseudo-config.conf
[Global]
admin.username = {{ .Env.Get "ADMIN_USERNAME" }}
funny.name = {{ .Config.Get "something_funny" }}
log.level = {{ .Config.GetOrDefault "log_level" "WARN" }}
url.jdbc = jdbc:postgresql://postgresql:5432/{{ .Config.GetAndDecrypt "sa-postgresql/database" }}
url.fqdn = https://{{ .GlobalConfig.Get "fqdn" }}/my-dogu
{{ if .Config.Exists "notification" }}
notification = mail
{{ end }}
{{ if .Dogus.IsEnabled "redmine" }}
ticketsystem.url = https://{{ .GlobalConfig.Get "fqdn" }}/redmine
{{ end }}
...
With this call doguctl
waits until a given TCP port is open or a timeout (in seconds) occurs. An exit code != 0 signals an error.
if ! doguctl wait-for-tcp --timeout 120 8080; then
echo "Received timeout after waiting 120 seconds for port 8080. Exiting."
exit 1
fi
With this call doguctl
waits until a given HTTP(s) URL is ready or a timeout (in seconds) occurs. An exit code != 0 signals an error.
if ! doguctl wait-for-tcp --host postgresql --timeout 120 5432; then
echo "Received timeout after waiting 120 seconds for port 5432 of host postgresql. Exiting."
exit 1
fi
Service Accounts represent a mechanism for access accounts that Dogus need to secure or store their data, but do not want to provide this functionality themselves. Dogus can represent themselves as producers and/or consumers of Service Accounts. In each case, both are optional features of a Dogu.
Generally, the nature of the application in the dogu itself dictates whether or which of the two makes the most sense. For example, a database management dogu such as PostgreSQL that does not offer other dogus access to its database would not be useful. On the other hand, it would be fatal for a dogu like Redmine not to have a dogu into which it can store its data via SQL.
Dogus like PostgreSQL that can provide access accounts to other dogus are also called service account producers. Dogus like Redmine that require an access account to another dogu are called service account consumers.
Service accounts are not requested by a service account consumer itself, because a dogu should not care about this form of orchestration. This is instead the job of a client like cesapp
or k8s-dogu-operator
during the installation of the service account consumer. The ExposedCommand
for this is service-account-create
.
Similarly, the deletion of service accounts is handled by a client like cesapp
or k8s-dogu-operator
during the uninstallation of a dogu to delete its requested service accounts. The ExposedCommand
for this is service-account-remove
.
In order for a dogu to act as a service account producer, it must provide two executables in its dogu.json
:
{
"ExposedCommands": [
{
"Name": "service-account-create",
"Description": "Creates a new service account",
"Command": "/create-sa.sh"
},
{
"Name": "service-account-remove",
"Description": "Removes a service account",
"Command": "/remove-sa.sh"
}
]
}
The terms service-account-create
and service-account-remove
are protected properties and are used by clients interpreting dogu.json
as commands to create and delete service accounts.
The following two sections discuss the creation and deletion of service accounts. More information about exposed commands can be found in the ExposedCommands section of the compendium.
The service-account-create
script manages the creation of a new service account. It is always executed when a new consumer is installed. In any case, during the call, the service-account-create
script receives the name of consumer dogu as its first argument. A second argument to fine-tune the service account is optional and depends on the support of the producer dogu.
A service-account-create
script consists of three steps:
- create the access data
- creating an account for our software with the information
- returning the access data
- This results in them being automatically written in encrypted form to the consumer's dogu-specific configuration area.
An example of the service-account-create
script could look like this:
create-sa.sh
#!/bin/bash
set -o errexit
set -o nounset
set -o pipefail
doguName="${1}"
{
#1) Create the credentials
USER="${doguName}-$(doguctl random)"
PASSWORD=$(doguctl random)
#2) Creates an account for our software with the information
# ... Implement the dogu specific logic here
} >/dev/null 2>&1 # suppress other output on stdout, because only credentials or errors are expected
#3) Return credentials. The client takes care of storage and encryption.
echo "username: ${USER}"
echo "password: ${PASSWORD}"
In step 1, credentials are randomly generated. The name of the consumer dogus can be included in the credentials, but this is not mandatory.
In step 2, an account must be created in the Dogu software with the generated information.
In step 3, the credentials get printed in the format <registrykey>: <registryvalue>
. registrykey
and registryvalue
must be separated by a colon and a space. The whole line must be terminated by a newline.
These credentials are automatically read by the processing client and stored in the consumer's dogu-specific configuration area in encrypted form. Following the example above, the consumer dogu would set up two registry entries containing the respective secret as a value:
/config/<consumer>/sa-<producer>/username
/config/<consumer>/sa-<producer>/password
The consumer dogu can now read and decrypt these e.g. by doguctl config -e sa-<producer>/username
.
The service-account-remove
script deletes an existing service account of a consumer dogu.
It is executed whenever the consumer dogu gets uninstalled. During the call, the service-account-remove
script receives the name of the consumer dogu as its first argument. Since no output from the script is processed here, error messages can also be printed to stdout
.
A service-account-remove
script consists of two steps:
- identify the consumer
- delete the data and the service account of the consumer
An example of the service-account-remove
script might look like this:
remove-sa.sh
#!/bin/bash
set -o errexit
set -o nounset
set -o pipefail
#1) identify the consumer
doguName="${1}"
if [ X"${doguName}" = X"" ]; then
echo "usage remove-sa.sh dogu-name"
exit 1
fi
#2) Delete the service account of the consumer
# ... Implement the Dogu specific logic here
In step 1, the name of the consumer dogu is read (e.g. "redmine") to identify the service account of the dogu.
In step 2 the account is deleted from the producer's database including access data.
It is very easy to request a service account from a producer dogu, since the main work is done by the client and the producer. A consumer dogu just needs to name the desired producer dogu as service account in its dogu.json
:
{
"ServiceAccounts": [
{
"Type": "your-producer-dogu",
"Kind": "dogu"
}
]
}
A service account is a special form of dependency. Therefore, it makes sense to keep the producer dogu in its dependency list. In the different phases of a Dogu installation, this ensures that the producer dogu is really available for use:
{
"Dependencies": [
{
"Name": "your-producer-dogu",
"Type": "dogu"
}
],
"ServiceAccounts": [
{
"Type": "your-producer-dogu",
"Kind": "dogu"
}
]
}
Dogus can be upgraded in the Cloudogu EcoSystem using simple means (e.g. cesapp upgrade dogu <dogu name>
). The entire upgrade process must be automated beforehand during dogu development and can then be run through by the CES tools (here: cesapp
) without much assistance from the admins.
The upgrade process includes the following steps:
- Displaying upgrade notifications, if any
- Downloading the new Dogu image
- Ensuring that the Dogu is healthy
- Ensuring that all dependent dogus are available
- Running the pre-upgrade script on the old dogu, if available
- Replacing the old dogu with the new one
- Starting the new dogu
- Running the post-upgrade script in the new dogu, if any
If it is necessary to perform pre- or post-upgrade steps during a dogu update, optional pre- and post-upgrade scripts can be created. As an example, consider software that requires a different database schema after an update. In such a case it is possible to write upgrade scripts to start a migration of the database after a Dogu update.
In addition, it is possible to be notified by an upgrade notification script about important changes when upgrading Dogus.
All upgrade scripts described below should be placed in the resources
folder (besides e.g. the startup.sh
script).
When developing a Dogu, it must be ensured that upgrading a Dogu also works across multiple versions. For this purpose, the upgrade scripts must be designed in such a way that they determine the upgrade path and automatically perform any necessary migration steps.
For this purpose, all upgrade scripts can read the previous (FROM
) and new (TO
) Dogu version:
FROM_VERSION="${1}"
TO_VERSION="${2}"
The pre-upgrade
script can be defined as Exposed Command in the dogu.json
of a Dogu:
{
"ExposedCommands": [
{
"Name": "pre-upgrade",
"Command": "/pre-upgrade.sh"
}
]
}
This script is executed before the actual upgrade of the Dogu in the old Dogu container.
- The script is not necessarily executed from the same path where it is located in the new Dogu container. Accordingly, absolute paths must be used in the script.
- The complete pre-upgrade must be a single script, which can be copied accordingly into the old container. If the script uses additional files, these must already be present on the old container.
The post-upgrade
script can be defined as Exposed Command in the dogu.json
of a dogu:
{
"ExposedCommands": [
{
"Name": "post-upgrade",
"Command": "/post-upgrade.sh"
}
]
}
As soon as it boots up, this script will be started in the new dogu container after the dogu has been upgraded. So normally both the startup.sh
and the post-upgrade
script will run at the same time. If it should be necessary that the post-upgrade
script is executed before the startup.sh
script, the startup.sh
script must be provided with a waiting mechanism. This can be done, for example, by waiting for a registry key to be set by the post-upgrade
script once it has run completely.
The upgrade-notification
script can be defined as Exposed Command in the dogu.json
of a dogus:
{
"ExposedCommands": [
{
"Name": "upgrade-notification",
"Command": "/upgrade-notification.sh"
}
]
}
This script is executed before the upgrade process and should only output information (via echo
) that may be relevant for the administrator before the upgrade. For example, it may indicate breaking changes or that a backup should be made before the upgrade.
This chapter describes features that integrate Dogus deeper into the Cloudogu EcoSystem and make them uniformly administrable.
The resource requirements for a Dogu can be applied to the following resources:
- Memory (RAM)
- Swap
- CPU cores
- Ephemeral storage
A distinction is made between resource requests and limits:
- Resource Requests: Specify the minimum resources (CPU cores, memory, ephemeral storage) required by a dogu for it to be functional. In the Multinode-EcoSystem, the Kubernetes scheduler ensures that the Dogu is started on a node with sufficient resources.
- Resource Limits: Specify the maximum amount of resources a Dogu is allowed to use.
Memory limits can be used to restrict the memory consumption (RAM) of Dogus.
If a Dogu exceeds its memory limit, the largest process in the container is terminated. This is usually the main process of the Dogus and causes the container to be restarted. If no value is set for the memory limit, this will not happen.
Memory requests can be used to specify the minimum memory requirement for Dogus to be fully functional.
Important: Memory-Requests are only applied in the Multinode-EcoSystem.
Swap limits can be used to restrict the memory consumption (swap space) of Dogus.
If a Dogu exceeds its swap limit, the largest process in the container is terminated.
This is usually the main process of the Dogus and causes the container to be restarted.
If no value is set on swap limiting, it will not take place.
For swap limiting 0b
is the default value and thus does not provide swap.
Important: Swap limits are NOT applied in the Multinode-EcoSystem!
In order for the ClouDogu EcoSystem host to limit memory and swap, the following settings must be made beforehand:
- open the file
/etc/default/grub
. - the following value must be added to the variable
GRUB_CMDLINE_LINUX_DEFAULT
:cgroup_enable=memory swapaccount=1
. - save the changes
- execute the command
sudo update-grub
orsudo update-bootloader --refresh
. - reboot the machine
Warning!
The above activation with cgroup_enable=memory swapaccount=1
is expected to result in a memory overhead of 1% and a performance penalty of 10% even if Docker is not running.
CPU limits can be used to restrict the maximum CPU cores available to Dogus. When the CPU cores limit is exceeded, the container runtime throttles the available CPU resources for the container.
CPU requests can be used to specify the minimum CPU cores required for a Dogu to be fully functional.
Important: CPU limits and requests are only applied in the Multinode-EcoSystem.
Ephemeral storage limits can be used to limit the maximum available ephemeral storage (see Kubernetes-Ephemeral-Storage) of Dogus. When the limit is exceeded, the container is restarted.
Ephemeral storage requests can be used to specify the minimum ephemeral storage required for a Dogu to be fully functional.
Important: Ephemeral-Storage-Limits and -Requests are only applied in the Multinode-EcoSystem.
To be able to set resource requests, the dogu.json
of the Dogus must contain the following entries:
{
"Configuration": [
{
"Name": "container_config/memory_limit",
"Description":"Limits the container's memory usage. Use a positive integer value followed by one of these units [b,k,m,g] (byte, kibibyte, mebibyte, gibibyte).",
"Optional": true,
"Validation": {
"Type": "BINARY_MEASUREMENT"
}
},
{
"Name": "container_config/memory_request",
"Description":"Requests the container's minimal memory requirement. Use a positive integer value followed by one of these units [b,k,m,g] (byte, kibibyte, mebibyte, gibibyte). This configuration is only applicable to the Multinode-EcoSystem.",
"Optional": true,
"Validation": {
"Type": "BINARY_MEASUREMENT"
}
},
{
"Name": "container_config/swap_limit",
"Description":"Limits the container's swap memory usage. Use zero or a positive integer value followed by one of these units [b,k,m,g] (byte, kibibyte, mebibyte, gibibyte). 0 will disable swapping.",
"Optional": true,
"Validation": {
"Type": "BINARY_MEASUREMENT"
}
},
{
"Name": "container_config/cpu_core_limit",
"Description":"Limits the container's CPU core usage. Use a positive floating value describing a fraction of 1 CPU core. When you define a value of '0.5', you are requesting half as much CPU time compared to if you asked for '1.0' CPU. This configuration is only applicable to the Multinode-EcoSystem.",
"Optional": true
},
{
"Name": "container_config/cpu_core_request",
"Description":"Requests the container's minimal CPU core requirement. Use a positive floating value describing a fraction of 1 CPU core. When you define a value of '0.5', you are requesting half as much CPU time compared to if you asked for '1.0' CPU. This configuration is only applicable to the Multinode-EcoSystem.",
"Optional": true
},
{
"Name": "container_config/storage_limit",
"Description":"Limits the container's ephemeral storage usage. Use a positive integer value followed by one of these units [b,k,m,g] (byte, kibibyte, mebibyte, gibibyte). This configuration is only applicable to the Multinode-EcoSystem.",
"Optional": true,
"Validation": {
"Type": "BINARY_MEASUREMENT"
}
},
{
"Name": "container_config/storage_request",
"Description":"Requests the container's minimal ephemeral storage requirement. Use a positive integer value followed by one of these units [b,k,m,g] (byte, kibibyte, mebibyte, gibibyte). This configuration is only applicable to the Multinode-EcoSystem.",
"Optional": true,
"Validation": {
"Type": "BINARY_MEASUREMENT"
}
}
]
}
This creates the registry entries for the resource requirements in the respective Dogu configuration.
Memory
- key for request:
config/<DOGU_NAME>/container_config/memory_request
- Key for limit:
config/<DOGU_NAME>/container_config/memory_limit
- Optional
- Description: Sets the memory resource requirement of the Dogus.
- Format: The configurable values for the keys are each a string of the form
<number value><unit>
and describe the maximum amount of memory that can be used by the Dogu. Note here that there must be no space between the numeric value and the unit. Available units areb
,k
,m
andg
(for byte, kibibyte, mebibyte and gibibyte).
Swap
- Key for limit:
config/<DOGU_NAME>/container_config/swap_limit
- Optional
- Description: Sets the swap limit of the dogus.
- Format: The configurable values for the keys are each a string of the form
<number value><unit>
and describe the maximum amount of memory that can be used by the dogu. Note here that there must be no space between the numeric value and the unit. Available units areb
,k
,m
andg
(for byte, kibibyte, mebibyte and gibibyte).
CPU cores
- Key for request:
config/<DOGU_NAME>/container_config/cpu_core_request
. - Key for limit:
config/<DOGU_NAME>/container_config/cpu_core_limit
- Optional
- Description: Sets the CPU resource request of the dogus.
- Format: See https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#resource-units-in-kubernetes
Ephemeral-Storage
- Key for request:
config/<DOGU_NAME>/container_config/storage_request
- Key for limit:
config/<DOGU_NAME>/container_config/storage_limit
- Optional
- Description: Sets the ephemeral storage resource requirement of the Dogus.
- Format: The configurable values for the keys are each a string of the form
<number value><unit>
and describe the maximum amount of storage that can be used by the Dogu. Note here that there must be no space between the numeric value and the unit. Available units areb
,k
,m
andg
(for byte, kibibyte, mebibyte and gibibyte).
Setting the values can be done in the following ways:
doguctl config container_config/memory_limit 1g
cesapp edit-config <doguname>
(only from the host)etcdctl set /config/<doguname>/container_config/memory_limit "1g"
(only from the host)
To apply the limits in the Singlenode-EcoSystem, the Dogu must be recreated (cesapp recreate <doguname>
) and then restarted (cesapp start <doguname>
).
In the Multinode-EcoSystem, the global etcd key config/_global/sync_resource_requirements
must be created/modified/deleted.
Any change to the key will start an automatic update process for all dogus.
In this update process resource requirements are applied to all dogus and dogus are restarted if new resource requirements are set.
Unchanged dogus are not restarted.
In general, the update process can be started with the following command:
etcdctl set /config/_global/sync_resource_requirements true
Singlenode-EcoSystem:
The limitation of memory (RAM only, no swap) can be checked using docker stats <doguname>
.
The MEM USAGE / LIMIT
column should correctly show the memory limit set.
Multinode-EcoSystem: The resource requirements are contained in the Kubernetes pod resource and can be checked there. For example, with:
kubectl get pod ldap-6ccb6c78fd-rp86t -o yaml | yq -r '.spec.containers[0].resources'
A special case is the limiting of a Java process. If a Dogu contains a Java process, the following additional entries can be added to dogu.json
:
{
"Configuration": [
{
"Name": "container_config/java_max_ram_percentage",
"Description": "Limits the heap stack size of the Java process to the configured percentage of the available physical memory when the container has more than approx. 250 MB of memory available. Is only considered when a memory_limit is set. Use a valid float value with decimals between 0 and 100 (f. ex. 55.0 for 55%). Default value: 25%",
"Optional": true,
"Default": "25.0",
"Validation": {
"Type": "FLOAT_PERCENTAGE_HUNDRED"
}
},
{
"Name": "container_config/java_min_ram_percentage",
"Description": "Limits the heap stack size of the Java process to the configured percentage of the available physical memory when the container has less than approx. 250 MB of memory available. Is only considered when a memory_limit is set. Use a valid float value with decimals between 0 and 100 (f. ex. 55.0 for 55%). Default value: 50%",
"Optional": true,
"Default": "50.0",
"Validation": {
"Type": "FLOAT_PERCENTAGE_HUNDRED"
}
}
]
}
Notes:
- The percentages always refer to the limited memory of a container. If no limit is set, the percentages are also ignored.
- A Java process should not be allocated too large a share of the memory. One should always consider any other processes that also need memory. If this does not happen, the actual program may crash.
- The values that can be configured with this must be given as parameters to the corresponding Java process in the start scripts of the Dogus. A reference implementation can be found in the Nexus Dogu.
One feature of the Cloudogu EcoSystem is that Dogus can be backed up and restored via a central backup system. For this to work, all Dogus must offload their mutable data to volumes. This procedure is already described in the Compendium.
When developing a Dogu, care must be taken to ensure that the Dogu continues to function normally after a successful backup and restore operation and that all data and functions are available. To do this, all volumes in the dogu.json
that contain production data must first be marked with the NeedsBackup flag. After that the Dogu should be built and filled with test data.
After that, one performs a backup and a restore of the system. The commands cesapp backup and cesapp restore can be used for this.
Once all Dogus have been restarted, test whether your own Dogu is running normally and all test data is still available.
When installing the Cloudogu EcoSystem, a global admin group is set. All members of this group should be given admin rights in all Dogus. This must be ensured during the development of the Dogus. The name of the globally defined admin group can be queried from within the Dogu using doguctl
: doguctl config --global 'admin_group'
.
Additionally, the admin group can be changed afterwards. The Dogu must then react (after a reboot, if necessary) in such a way that only members of the new admin group are granted admin rights. Users in the old admin group will only receive normal user rights unless they are also in the new admin group.
The Fully Qualified Domain Name (FQDN) of the Cloudogu EcoSystem is stored globally in the registry and can be read by the Dogus using doguctl
. If necessary, it can be integrated into the Dogu configuration (see also Creating an exemplary application. In addition, care must be taken during Dogu development to ensure that the FQDN is changeable. So the Dogu should be able (after a restart if necessary) to read a new FQDN and adapt its configuration to this new FQDN.
This section shows how to create a uniform, abstracted setting option that can be used to set the log behavior in all Dogus in the same way. Four different log levels that can be set via the registry are used for this purpose: ERROR
, WARN
, INFO
and DEBUG
.
These abstract log levels help the administrator as he does not need to know what a dogu-specific log level is called, but can instead fall back on four basic values and reuse them safely in all Dogus.
Dogu developers must take care that these four log levels are mapped meaningfully to the log levels of the software in the Dogu, which may have different names (e.g. TRACE
or FATAL
).
Nobody reads logs. If someone reads them, it is not with pleasure, for two reasons:
- standard logs contain mostly unimportant information for the viewer
- logs are only read when a problem occurs.
Log level control addresses these circumstances.
If a dogu logs too many lines by default and permanently, then administrators or support staff have their lives made harder than necessary. This is because they have to work through a mountain of lines and separate the logs into relevant and irrelevant information. In addition, excessive logging (e.g. long stack traces combined with container crash loops) can lead to so much data that the hard disk fills up and log rotation is no longer possible.
The log levels are sorted by level of detail from coarse to fine and contain the respective level above. For example, log outputs of the level WARN
thus also contain log outputs of the level ERROR
.
This gives the developer a great deal of freedom as to which logs are output at which log level.
To distinguish between relevant and irrelevant, there is of course room for interpretation. Cloudogu as a Dogu provider supports the customer in making logging as low and as concise as possible for their productive systems. This means, based on our experience as both software developers and Dogu users, we decide for the customer, which log sources, log lines etc. are appropriate per Dogu and log level.
Log levels analogous to ERROR
are very often found on production systems. They are used so that operations or support do not have to go through superfluous additional information, but can efficiently get to the core of problems.
Log output of this level should only contain output that indicates acute errors, operational prevention, or security violations, etc.
Examples:
- Repair of corrupt data in a database
- application error
- unplanned stop or restart of services or applications
- acute connectivity errors
- e.g. Postgresql dogu is not reachable even after several minutes
- repeated, failed login attempts
- acutely threatening or acutely existing lack of resources
- messages with
FATAL
orSEVERE
log level
Log levels analogous to WARN
are frequently found on production or staging systems.
Log outputs of this level are very similar to those of ERROR
. Unlike these, however, they contain output indicating imminent errors.
Examples:
- faulty or optimizable configuration was detected
- update of application or libraries is suggested
- imminent lack of resources in the medium term
- singular connectivity errors, which are limited in time
- e.g. Postgresql dogu is not reachable at the first attempt, but after a few minutes it is reachable again
- single failed login attempts
- scheduled start and stop of services or applications
Outputs that might be of interest in special but non-critical situations. A catch-all for all the output that is not important enough for WARN
, but does not yet go deep into the details of the application.
Examples:
- a subsystem was completely loaded
- Error-free result of background jobs
- Successful start/stop of a component
Output that may be of interest in critical situations and/or used for debugging.
Examples:
- a relevant decision for a configuration was made
- Messages with
TRACE
log level
These four abstract log levels occasionally have no direct relationship to concrete log levels in the tools or plugins in the Dogu and must be mapped manually in Dogu development. The reason for this is that both the tools and the associated plugins and frameworks have log levels that differ too greatly in quantity and quality.
The mapping of an abstract log level is thus in the hands of the Dogu development. Here the developers can decide, for example, to map the TRACE
log level from the software to the DEBUG
log level of the Dogu.
This mapping from abstract to concrete log levels has the advantage of flexibility, e.g. if several configuration files have to be filled with different log levels.
To set the log level, an optional key /config/${dogu}/logging/root
is introduced per Dogu, which can take one of the following abstract log level values.
ERROR
WARN
(default value)INFO
DEBUG
If no /config/${dogu}/logging/root
key exists, then doguctl
will automatically use the WARN
value if properly configured as default value.
doguctl
supports dogu developers defining a log level for a dogu in a sturdy way. This leaves only the mapping of a found log level to the respective logging framework in the dogu.
In the dogu.json
a log level configuration value can be set, which both can be validated and also provides a default value on the Dogu.
{
"Configuration": [
{
"Name": "logging/root",
"Description": "Set the root log level to one of ERROR, WARN, INFO, DEBUG.",
"Optional": true,
"Default": "WARN",
"Validation": {
"Type": "ONE_OF",
"Values": [
"WARN",
"DEBUG",
"INFO",
"ERROR"
]
}
}
]
}
This value can be checked with doguctl validate
:
if [[ ! doguctl validate logging/root --silent ]]; then
echo "WARNING: Found invalid value in logging/root. Resetting it to WARN"
doguctl config logging/root WARN
fi
The set or default value from dogu.json
can be retrieved as usual. doguctl
takes care of getting a default value if no value has been set:
rootLogLevel=$(doguctl config logging/root)
If there is a need for additional Dogu-specific log levels, the Dogu log levels can be supplemented with additional keys (also under /config/<doguname>/logging/
).
Such log levels are very specific by their nature.
Therefore, different rules may apply to the use of such specific keys.
In the Cloudogu EcoSystem, communication protocols play a key role in data exchange, bei it the connection between a user and a dogu, or a dogu to another dogu or another actively running service. The protocols being used strongly depend on each dogu. This chapter enumerates protocols that are supported and have been successfully tested.
Further information about troubleshooting network communication can be found in the troubleshooting document
There are two ways to make a connection from a user client to a dogu:
- the nginx reverse-proxy dogu
- generally, all network traffic from the user's web client to the target dogu is reverse-proxied by the nginx dogu
- nginx takes care of the TLS termination and the internal URL rewriting
- exposed dogu ports
- a dogu might also expose its ports to the outside
- the nginx dogu will not handle or re-route any requests made to this port
- generally speaking, this is a dangerous matter because the dogu must solve all security measures by itself
- f. i. authorization, SSO/SLO, request throttling, access logging, etc.
Currently, these communication protocols have been successfully tested from outside into a Cloudogu EcoSystem:
- via nginx:
- HTTP 1 and 2
- Websockets
- via port exposing
- SSH
Of course, dogus can communicate with other dogus over the internal network, too! Usually this is done in an unsecured fashion, i. e. HTTP (and not HTTPS).
Currently, these communication protocols have been successfully tested from inside the Cloudogu EcoSystem:
- HTTP 1 and 2
- Websockets
- gRPC