Skip to content

Commit

Permalink
Describe the main idea in readme or contribution guide (#545)
Browse files Browse the repository at this point in the history
* Add standard test

* Fix test
  • Loading branch information
mfvanek authored Dec 21, 2024
1 parent 3e9a63c commit 15b081c
Show file tree
Hide file tree
Showing 13 changed files with 375 additions and 219 deletions.
51 changes: 47 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,8 @@ For **Java 8** compatible version take a look at release [0.7.0](https://github.
All checks can be divided into 2 groups:

1. Runtime checks (those that make sense to perform only on a production database with real data and statistics).
Runtime checks usually [require aggregating data from all nodes in the cluster](https://github.com/mfvanek/pg-index-health/blob/d1473dc68975ebe932d92c9e43ceebde657d0cc7/pg-index-health-core/src/main/java/io/github/mfvanek/pg/common/maintenance/Diagnostic.java#L162).
This necessitated creating [our own abstraction over the database connection](https://github.com/mfvanek/pg-index-health/tree/master/pg-index-health-jdbc-connection).
Runtime checks usually [require aggregating data from all nodes in the cluster](https://github.com/mfvanek/pg-index-health/blob/3e9a63cc2a04799f3e97c9bec9b684ababca8db7/pg-index-health-core/src/main/java/io/github/mfvanek/pg/core/checks/common/Diagnostic.java#L162).
This necessitated creating [our own abstraction over the database connection](https://github.com/mfvanek/pg-index-health/blob/3e9a63cc2a04799f3e97c9bec9b684ababca8db7/pg-index-health-jdbc-connection/src/main/java/io/github/mfvanek/pg/connection/HighAvailabilityPgConnection.java#L22).
2. Static checks (those can be run in tests on an empty database).
All static checks can be performed at runtime as well.

Expand Down Expand Up @@ -81,14 +81,15 @@ For raw sql queries see [pg-index-health-sql](https://github.com/mfvanek/pg-inde
### Static checks

Static checks are based on [information schema](https://www.postgresql.org/docs/current/information-schema.html)/[system catalogs](https://www.postgresql.org/docs/current/catalogs.html).
They work with finite database state (after all migrations are applied).

### Runtime checks

**pg_index_health** utilizes [the Cumulative Statistics System](https://www.postgresql.org/docs/current/monitoring-stats.html)
(formerly known as [PostgreSQL's statistics collector](https://www.postgresql.org/docs/14/monitoring-stats.html)).

You can call `pg_stat_reset()` on each host to reset all statistics counters for the current database to zero
but the best way to do it is to use [DatabaseManagement::resetStatistics()](https://github.com/mfvanek/pg-index-health/blob/d1473dc68975ebe932d92c9e43ceebde657d0cc7/pg-index-health/src/main/java/io/github/mfvanek/pg/common/management/DatabaseManagement.java#L33) method.
but the best way to do it is to use [DatabaseManagement::resetStatistics()](https://github.com/mfvanek/pg-index-health/blob/3e9a63cc2a04799f3e97c9bec9b684ababca8db7/pg-index-health/src/main/java/io/github/mfvanek/pg/health/checks/management/DatabaseManagement.java#L33) method.

## Installation

Expand Down Expand Up @@ -116,12 +117,15 @@ Using Maven:

## Articles and publications

### In Russian
* [Index health in PostgreSQL through the eyes of a Java developer](https://habr.com/ru/post/490824/)
* [DBA: finding useless indexes](https://habr.com/ru/companies/tensor/articles/488104/)
* [The series of articles "Static analysis of the database structure"](https://habr.com/ru/articles/800121/)

## How to use

There are three main scenarios of using **pg-index-health** in your projects:
* unit\functional testing;
* unit\functional testing (see **standard test** in section below);
* collecting indexes health data and monitoring bloat;
* analysis of database configuration.

Expand Down Expand Up @@ -161,6 +165,45 @@ Using Maven:
</dependency>
```

### Standard test

Add a standard test to your project as shown below. Ideally, all checks should work and return an empty result.

```java
import io.github.mfvanek.pg.core.checks.common.DatabaseCheckOnHost;
import io.github.mfvanek.pg.core.checks.common.Diagnostic;
import io.github.mfvanek.pg.model.dbobject.DbObject;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.ActiveProfiles;

import java.util.List;

import static org.assertj.core.api.Assertions.assertThat;

@SpringBootTest
@ActiveProfiles("test")
class DatabaseStructureStaticAnalysisTest {

@Autowired
private List<DatabaseCheckOnHost<? extends DbObject>> checks;

@Test
void checksShouldWork() {
assertThat(checks)
.hasSameSizeAs(Diagnostic.values());

checks.stream()
.filter(DatabaseCheckOnHost::isStatic)
.forEach(c ->
assertThat(c.check())
.as(c.getDiagnostic().name())
.isEmpty());
}
}
```

### Spring Boot compatibility

| Spring Boot | Min JDK | pg-index-health-test-starter |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,8 @@

import static org.assertj.core.api.Assertions.assertThat;

@ActiveProfiles("test")
@SpringBootTest
@ActiveProfiles("test")
class H2DemoApplicationTest {

@Autowired
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
/*
* Copyright (c) 2019-2024. Ivan Vakhrushev and others.
* https://github.com/mfvanek/pg-index-health
*
* This file is a part of "pg-index-health" - a Java library for
* analyzing and maintaining indexes health in PostgreSQL databases.
*
* Licensed under the Apache License 2.0
*/

package io.github.mfvanek.pg.spring.postgres.kt.custom.ds

import io.github.mfvanek.pg.core.checks.common.DatabaseCheckOnHost
import io.github.mfvanek.pg.core.checks.common.Diagnostic
import io.github.mfvanek.pg.model.context.PgContext
import io.github.mfvanek.pg.model.dbobject.DbObject
import io.github.mfvanek.pg.model.predicates.SkipLiquibaseTablesPredicate
import io.github.mfvanek.pg.model.table.Table
import org.assertj.core.api.Assertions.assertThat
import org.junit.jupiter.api.Test
import org.springframework.beans.factory.annotation.Autowired
import org.springframework.boot.test.context.SpringBootTest
import org.springframework.test.context.ActiveProfiles

@SpringBootTest
@ActiveProfiles("test")
internal class DatabaseStructureStaticAnalysisTest {

@Autowired
private lateinit var checks: List<DatabaseCheckOnHost<out DbObject?>>

@Test
fun checksShouldWork() {
assertThat(checks)
.hasSameSizeAs(Diagnostic.entries.toTypedArray())

checks
.filter { it.isStatic }
.forEach { check ->
val ctx = PgContext.of("custom_ds_schema")
// Due to the use of spring.liquibase.default-schema, all names are resolved without a schema
val listAssert = assertThat(check.check(ctx, SkipLiquibaseTablesPredicate.ofPublic()))
.`as`(check.diagnostic.name)

when (check.diagnostic) {
Diagnostic.TABLES_NOT_LINKED_TO_OTHERS ->
listAssert
.hasSize(1)
.containsExactly(Table.of("warehouse"))

else -> listAssert.isEmpty()
}
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@ internal class PostgresCustomDataSourceDemoApplicationKtRunTest {
assertThat(output.all)
.contains("Reading from custom_ds_schema.databasechangelog")
.contains("Starting PostgresCustomDataSourceDemoApplicationKt using Java")
.contains("Container is started (JDBC URL: jdbc:postgresql://localhost:")
.contains("Started PostgresCustomDataSourceDemoApplicationKt in")
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -2,29 +2,20 @@ package io.github.mfvanek.pg.spring.postgres.kt.custom.ds

import com.zaxxer.hikari.HikariDataSource
import io.github.mfvanek.pg.connection.PgConnection
import io.github.mfvanek.pg.core.checks.common.DatabaseCheckOnHost
import io.github.mfvanek.pg.core.checks.common.Diagnostic
import io.github.mfvanek.pg.model.context.PgContext
import io.github.mfvanek.pg.model.dbobject.DbObject
import io.github.mfvanek.pg.model.predicates.SkipLiquibaseTablesPredicate
import io.github.mfvanek.pg.model.table.Table
import org.assertj.core.api.Assertions.assertThat
import org.junit.jupiter.api.Test
import org.springframework.beans.factory.annotation.Autowired
import org.springframework.boot.test.context.SpringBootTest
import org.springframework.context.ApplicationContext
import org.springframework.test.context.ActiveProfiles

@ActiveProfiles("test")
@SpringBootTest
@ActiveProfiles("test")
internal class PostgresCustomDataSourceDemoApplicationKtTest {

@Autowired
private lateinit var applicationContext: ApplicationContext

@Autowired
private lateinit var checks: List<DatabaseCheckOnHost<out DbObject?>>

@Test
fun contextLoadsAndContainsPgIndexHealthBeans() {
assertThat(applicationContext.getBean("pgihCustomDataSource"))
Expand All @@ -33,28 +24,4 @@ internal class PostgresCustomDataSourceDemoApplicationKtTest {
assertThat(applicationContext.getBean("pgConnection"))
.isInstanceOf(PgConnection::class.java)
}

@Test
fun checksShouldWork() {
assertThat(checks)
.hasSameSizeAs(Diagnostic.entries.toTypedArray())

checks
.filter { it.isStatic }
.forEach { check ->
val ctx = PgContext.of("custom_ds_schema")
// Due to the use of spring.liquibase.default-schema, all names are resolved without a schema
val listAssert = assertThat(check.check(ctx, SkipLiquibaseTablesPredicate.ofPublic()))
.`as`(check.diagnostic.name)

when (check.diagnostic) {
Diagnostic.TABLES_NOT_LINKED_TO_OTHERS ->
listAssert
.hasSize(1)
.containsExactly(Table.of("warehouse"))

else -> listAssert.isEmpty()
}
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
/*
* Copyright (c) 2019-2024. Ivan Vakhrushev and others.
* https://github.com/mfvanek/pg-index-health
*
* This file is a part of "pg-index-health" - a Java library for
* analyzing and maintaining indexes health in PostgreSQL databases.
*
* Licensed under the Apache License 2.0
*/

package io.github.mfvanek.pg.spring.postgres.kt

import io.github.mfvanek.pg.core.checks.common.DatabaseCheckOnHost
import io.github.mfvanek.pg.core.checks.common.Diagnostic
import io.github.mfvanek.pg.model.dbobject.DbObject
import org.assertj.core.api.Assertions.assertThat
import org.junit.jupiter.api.Test
import org.springframework.beans.factory.annotation.Autowired
import org.springframework.boot.test.context.SpringBootTest
import org.springframework.test.context.ActiveProfiles

@SpringBootTest
@ActiveProfiles("test")
internal class DatabaseStructureStaticAnalysisTest {

@Autowired
private lateinit var checks: List<DatabaseCheckOnHost<out DbObject?>>

@Test
fun checksShouldWork() {
assertThat(checks)
.hasSameSizeAs(Diagnostic.entries.toTypedArray())

checks
.filter { it.isStatic }
.forEach {
assertThat(it.check())
.`as`(it.diagnostic.name)
.isEmpty()
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,6 @@ package io.github.mfvanek.pg.spring.postgres.kt

import com.zaxxer.hikari.HikariDataSource
import io.github.mfvanek.pg.connection.PgConnection
import io.github.mfvanek.pg.core.checks.common.DatabaseCheckOnHost
import io.github.mfvanek.pg.core.checks.common.Diagnostic
import io.github.mfvanek.pg.model.dbobject.DbObject
import org.assertj.core.api.Assertions.assertThat
import org.junit.jupiter.api.Test
import org.springframework.beans.factory.annotation.Autowired
Expand All @@ -23,8 +20,8 @@ import org.springframework.context.ApplicationContext
import org.springframework.core.env.Environment
import org.springframework.test.context.ActiveProfiles

@ActiveProfiles("test")
@SpringBootTest
@ActiveProfiles("test")
internal class PostgresDemoApplicationKtTest {

@Autowired
Expand All @@ -33,9 +30,6 @@ internal class PostgresDemoApplicationKtTest {
@Autowired
private lateinit var environment: Environment

@Autowired
private lateinit var checks: List<DatabaseCheckOnHost<out DbObject?>>

@Test
fun contextLoadsAndContainsPgIndexHealthBeans() {
assertThat(applicationContext.getBean("dataSource"))
Expand All @@ -49,18 +43,4 @@ internal class PostgresDemoApplicationKtTest {
.startsWith("jdbc:postgresql://localhost:")
.endsWith("/demo_for_pg_index_health_starter?loggerLevel=OFF")
}

@Test
fun checksShouldWork() {
assertThat(checks)
.hasSameSizeAs(Diagnostic.entries.toTypedArray())

checks
.filter { it.isStatic }
.forEach {
assertThat(it.check())
.`as`(it.diagnostic.name)
.isEmpty()
}
}
}
Loading

0 comments on commit 15b081c

Please sign in to comment.