improve guard performance and expose additional config options

This release contains two changes:

guard, reqctx: added cache to guard for current request and reqctx package to store data for the current request;

We can do a lot of queries for the same policies if your response contains multiple objects you have to do access control on (e.g., returning a list of entities). To help here, we introduce a cache for the current request to avoid retrieving the same policies again and again.

cloud/aws: exposed additional options to config;

This exposes additional options for AWS clients to the config files. These include:

• Disabling request compression and min compression request size for cloudwatch
• Disabling response checksum validation and accepting gzip compression for dynamodb
• The usePathStyle setting was documented for S3 (but it was already exposed before)

These settings are needed when talking to services implementing the AWS API like localstack or ScyllaDB as these services sometimes can't map all features exactly.

redis: fix lifecycle deadlock issue

This release fixes a deadlock issue which occurs under certain circumstances while using redis clients.

mdlsub: types provider util for transformers

This release adds a small helper for mdlsub transformers.
Instead of implementing the GetInput and GetModel functions, one can embed the TypesProvider.

before

type TestTransformer struct{}

func (t TestTransformer) GetInput() any {
	return &TestInput{}
}

func (t TestTransformer) GetModel() any {
	return &TestModel{}
}

after

type TestTransformer struct {
	mdlsub.TypesProvider[TestInput, TestModel]
}

reslife: introduction of advanced resource lifecycle management

This release works towards unifying creation and purging of external resources.
Currently, all resources like SQS queues or DynamoDb tables are created "in place" during instantiation of their corresponding services.

This change adds 4 life cycles for any type of resources: create, init, register and purge.
For SQS queues this means:

1. create: creating the queue if not existing
2. init: fetching properties like URL and ARN to initialize the queue service
3. register: adding metadata information to the metadata server
4. purge: purging any remaining messages in the queue

These cycles can be enabled / disabled via configuration and have the following defaults:

resource_lifecycles:
    create:
        enabled: false
    init:
        enabled: true
    register:
        enabled: true
    purge:
        enabled: false

In addition, while running with the dev or test environment, create will be enabled by default, too, if not disabled explicitly. This replaces the former functionality provided by dx.auto_create.

breaking changes

1. The mdlsub.ModelTransformer interface is extended by GetModel() any which should return an instance of the output type. Make sure to adjust your subscribers.
2. existing implementations of the fixture writer interface like for ddb, mysql, redis or kvstore have moved from the fixtures package to their corresponding packages. You will have to adjust your import paths

What's Changed

• resource lifecycles by @j4k4 in #1196

Full Changelog: v0.36.1...v0.37.0

etc: several improvements and smaller additions

This release consists of several smaller improvements and additions.
The biggest new feature is the possibility to set configuration values via command line flags with the new application option WithConfigFlags:

application.New(application.WithConfigFlags(os.Args, &struct {
	ServerPort int `short:"p" long:"server-port" cfg:"httpserver.default.port" default:"8080" description:"port of the default http server"`
}{}))

Providing this option would enable to set / overwrite the config value httpserver.default.port with providing -p 8888 or --server-port 8888 as command line arguments.

What's Changed

• justdev changes by @j4k4 in #1199

Full Changelog: v0.36.0...v0.36.1

fixtures: added post processors to loaders

Major change of this release is the additions of post processors to the fixture package. These allow additional handling after loading the fixture data, e.g. populating internal caches. For this change, the signatures of adding fixtures has changed slightly:

func main() {
	application.RunHttpDefaultServer(
		api.DefineRouter,
		application.WithFixtureSetFactory("default", service.FixtureSetsFactory, service.PostProcessor),
	)
}

or if you want to add multiple groups with the same post processor:

func main() {
	application.RunHttpDefaultServer(
		api.DefineRouter,
		application.WithFixtureSetFactories(map[string]fixtures.FixtureSetsFactory{
			"group1": service.FixtureSetsFactory1,
			"group2": service.FixtureSetsFactory2,
		}, service.PostProcessor),
	)
}

The options for creating test suites have been adjusted accordingly:

func (s *TestSuite) SetupSuite() []suite.Option {
	return []suite.Option{
		suite.WithLogLevel(log.LevelWarn),
		suite.WithConfigFile("./config.test.yml"),
		suite.WithFixtureSetFactory(service.FixtureSetsFactory, service.PostProcessors...),
	}
}

and

func (s *TestSuite) SetupSuite() []suite.Option {
	return []suite.Option{
		suite.WithLogLevel("warn"),
		suite.WithConfigFile("../config.test.yml"),
		suite.WithFixtureSetFactories([]fixtures.FixtureSetsFactory{
			common.FixtureSetsFactory,
			service.FixtureSetsFactory,
		}, service.PostProcessor),
	}
}

Post processors have to implement a simple interface:

type PostProcessor interface {
	Process(ctx context.Context) error
}

What's Changed

• fixtures: added post processors to loaders by @j4k4 in #1193

Full Changelog: v0.35.1...v0.36.0

Emit stream metrics

Metrics

The Messages per runner calculator for streams now also re-emits the Visible and Sent messages from AWS SQS Queues.

This is to reduce complexity and ensuring you don't have to import cloudwatch metrics via an exporter, which would require another component in the system where 2 additional emitted metrics would be enough.

The newly emitted metrics are:

• StreamMessagesAvailable (sum of ApproximateNumberOfMessagesVisible of all input streams)
• StreamMessagesSent (sum of NumberOfMessagesSent of all input streams)

Improve retry behavior

Client configuration

There are now five places where you can configure settings for your AWS client:

1. Client-specific settings for the given service (e.g., cloud.aws.dynamodb.clients.someClientName)
2. Client-specific settings from the defaults (e.g., cloud.aws.defaults.clients.someClientName)
3. Service-wide default settings for the given service (e.g., cloud.aws.dynamodb.clients.default)
4. Global defaults for all clients (e.g., cloud.aws.defaults.clients.default)
5. Global defaults for all AWS clients (e.g., cloud.aws.defaults)

Breaking change: Besides affecting the configuration of the clients itself, naming patterns also fall back to the naming patterns of the default client of the service. Thus, we use cloud.aws.sqs.clients.default.naming.pattern if cloud.aws.sqs.clients.yourClient.naming.pattern was not provided. This was done for all services (SQS, SNS, Kinesis, DDB)

DDB Locks

DDB locks now extend the context to unlock the lock until the lock expires when unlocking the lock.

Producer daemons

Producer daemons now configure their own AWS clients using the name of the producer (i.e., producer-{eventName}). These clients are then configured with infinite backoff, thus, even if your application is running as a gateway (which only retry for 10s by default), the producer daemon will retry indefinitely as it is running in the background to avoid losing data.

Kinesis outputs

Writing to Kinesis now takes the setting of the client used for writing into account (previously, we would have used the default backoff settings, i.e., stop after 10s for gateways) for the outer write loop (when writing to Kinesis, we might have a call to write 5 records which gets retried as well as an outer loop which would try writing 2 records in the next iteration if Kinesis accepted 3 and rejected 2 records in our example).

add kinsumer autoscale module

This release adds an optional module that will periodically check the current number of tasks of you kinsumer and the number of shards of the corresponding kinesis stream and set the task count to the number of shards if they are not equal.
You can add the module to your kinsumer app using the WithKinsumerAutoscaleModule option.

Since the module is using the DdbLeaderElection you will have to provide a DynamoDB table with a hash key named groupId of type String , a TTL attribute named leadingUntil and the table name has to match the pattern {env}-kinsumer-autoscale-leaders by default but can be configured.

For now, the module only supports ECS as the orchestrator.
You will have to give your app permissions for the ecs:DescribeServices and ecs:UpdateService actions, which you can do by using version 3.2.0 of our terraform-aws-ecs-app module and setting the ecs_access_policy_enabled variable to true.

treat request cancellations not as errors and make dispatcher errors more obvious

HTTP Server
The standard http server treats external context cancellations and disconnects as errors; even though the client may have closed or canceled the request on purpose.
As a follow up to #1178 (targeting the crud handlers), this PR is adjusting the standard behaviour of any other handler similarly.
The most common use cases:

• access forbidden transformed into HTTP 403
• context cancellation / disconnects are transformed into HTTP 499
• any other error being transformed into a HTTP 500

The logger middleware is also adjusted appropriately, logging connection and request cancelations as level info. Bind errors and render errors are continously logged as warnings; by default any other error is logged as an error.

Exec

• fixing bugs which causes the code to panic on nil errors

Dispatcher
The dispatcher is hiding some errors which may happen during execution of the callbacks. They're now propagated back to the callee, making them obvious and necessary to deal with.
In some cases, you may have to wrap adding callbacks into the appctx to prevent them from being added multiple times.

This is a breaking change, as the dispatcher will now stop execution if any error is encountered