Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add pgbouncer #114

Merged
merged 5 commits into from
Jan 24, 2025
Merged

feat: add pgbouncer #114

merged 5 commits into from
Jan 24, 2025

Conversation

hrodmn
Copy link
Contributor

@hrodmn hrodmn commented Jan 3, 2025

Add option to set up a PgBouncer server that can efficiently route traffic to the actual database

⚠️ Checklist if your PR is changing anything else than documentation

Merge request description

Add option to set up a pgbouncer server that can manage traffic to the actual database.

Adding a connection pooler like PgBouncer to eoAPI drastically improves performance by preventing the database instance from being overwhelmed by connections from eager clients like titiler-pgstac. AWS offers the RDS Proxy as a managed version of this setup, but it is quite expensive and can double the cost of running an RDS instance. By setting up an EC2 instance running PgBouncer, we can get the same impact for a fraction of the RDS Proxy cost (<$10/mo).

Most of the work to get PgBouncer running was done in MAAP-Project/maap-eoapi#39. This PR builds off of that work and does a more complete job of integrating the PgBouncer server with the rest of the eoAPI applications (not just titiler-pgstac).

I did that by creating a new secret that contains the same values as the original PgstacSecret but with the PgBouncer EC2 instance private or public IP address (depending on the networking setup) substituted for the host key. If the user specifies addPgBouncer: true in the PgStacDatabase construct, an EC2 instance running PgBouncer will be created and the pgstac_secret attribute will be set to the PgBouncer secret instead of the original PgStac secret. This makes it easy to configure client apps to connect to the database via PgBouncer instead of making a direct connection to the database.

The PgBouncer infrastructure is slightly more complicated than I want it to be because we cannot create the PgBouncerSecret until the EC2 instance has been created and assigned an IP address. To get around this, I set up a Lambda function to create the PgBouncer secret with nothing in it, then update it with the correct values after the EC2 instance is running.

Breaking change

The db argument in the TiPgApiLambda, PgStacApiLambda, and TitilerPgstacApiLambda constructs has been replaced withconnectionTarget. This object was/is used to add the Lambdas to the list of acceptable connections and now it can be either an RDS Instance or an EC2 instance (running PgBouncer). I made the breaking change because it would be very easy for an existing user to add PgBouncer to the PgStacDatabase construct then fail to use it when creating the downstream application constructs. By making the breaking change it should force users to at least re-specify the argument name (but since the argument can be either an RDS instance or an EC2 Instance they could easily still mistakenly specify the RDS instance :/). Maybe it is not worth a breaking change - what do you think??

I opted to make it a non-breaking change and will make some updates in eoapi-template to show how to use the new pgbouncer feature.

@hrodmn hrodmn force-pushed the feat/pgbouncer branch 5 times, most recently from 9c69644 to 352bea8 Compare January 3, 2025 17:47
Add option to set up a pgbouncer server that can manage traffic to the
actual database

BREAKING CHANGE: The `db` argument in the TiPgApiLambda, PgStacApiLambda,
and TitilerPgstacApiLambda constructs has been replaced with
`connectionTarget`.
This object is used to add the Lambdas to the list of acceptable
connections and can be either an RDS Instance or an EC2 instance
running pgbouncer.

also install new pgbouncer-secret-updater package in ci
@hrodmn hrodmn changed the title feat: add pgbouncer feat!: add pgbouncer Jan 4, 2025
@hrodmn hrodmn marked this pull request as ready for review January 6, 2025 11:10
@hrodmn hrodmn marked this pull request as draft January 8, 2025 02:25
@hrodmn hrodmn marked this pull request as ready for review January 22, 2025 02:46
@hrodmn hrodmn requested a review from vincentsarago January 22, 2025 02:47
@hrodmn
Copy link
Contributor Author

hrodmn commented Jan 22, 2025

@vincentsarago I have it set up to work with Lambdas in or out of the VPC. Now the pgbouncer instance will advertise its public IP address if the database instance is in a public subnet and use its private ip address if the database is in a private subnet. Lambdas in the vpc can use the private ip address while Lambdas outside the vpc need to use the public ip address.

I added some more notes about networking setups over here: #116.

@hrodmn hrodmn changed the title feat!: add pgbouncer feat: add pgbouncer Jan 24, 2025
@hrodmn hrodmn merged commit 5952858 into main Jan 24, 2025
6 checks passed
@hrodmn hrodmn deleted the feat/pgbouncer branch January 24, 2025 15:02
github-actions bot pushed a commit that referenced this pull request Jan 24, 2025
# [7.4.0](v7.3.0...v7.4.0) (2025-01-24)

### Features

* add pgbouncer ([#114](#114)) ([5952858](5952858))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants