Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ADBDEV-6936 Avoid reusing timeline for interrupted promotion #1173

Closed
wants to merge 3 commits into from

Conversation

Stolb27
Copy link
Collaborator

@Stolb27 Stolb27 commented Jan 4, 2025

Each Postgres instance (that GPDB segments are) should use own timeline
to exclude WAL segment names interference if derived from the common
ancestor. When we derive the new instance from backup or promote a
replica, the Postgres searches the next free timeline number to identify
the given reincarnation. Postgres persists information about selected
timeline in the Checkpoint record or associates it with End of Recovery
record in the controldata. Also, it remembers switch point to the
history file. These two entities helps the Postgres to handle timeline
switch in case of failure and consequent recovery. But Postgres used to
remove signal files earlier than persisting new timeline info. As a
result, the Postgres had no way to remember about promotion or to try
again, if something goes wrong soon. So, Postgres carried out the common
crash recovery and continued the previous timeline. As a result, the
instance may overwrite next WAL segments and push it to the archive with
the same name with unpredictable results depending on archive_command
implementation. But even without archiving, possible presence of this
orphaned timeline in the history file may lead to unexpected error
during the next replica switching because LSN of such switch point is
before latest checkpoint record created on the reused timeline.

The optimal solution is to retry interrupted promotion. FTS is ready to
do it, at least. To achieve this goal and fall into archive recovery
again, we should delay removing signal files until we persist the new
timeline information. The only insignificant negative outcome for this
approach is possible assignment of the one more timeline identity, if
we fall right after history file creation.

Each Postgres instance (that GPDB segments are) should use own timeline
to exclude WAL segment names interferension if derived from the common
ancestor. When we derive the new instance from backup or promote a
replica, the Postgres searches the next free timeline number to identify
the given reincarnation. Postgres perists information about selected
timeline in the Checkpoint record or associates it with End of Recovery
record in the controldata. Also, its remembers switch point to the
history file. These two entities helps the Postgres to handle timeline
switch in case of failure and consequent recovery. But Postgres used to
remove signal files earlier than persisting new timeline info. As a
result, the Postgres had no way to remember about promotion or to try
again if something goes wrong soon. So, Postgres carried out the common
crash recovery and continued the previous timeline. As a result, the
instance may overwrite next WAL segments and push it to the archive with
the same name with unpredictable results depending on archive_command
implementation. But even without archiving, possible presence of this
orphaned timeline in the history file may lead to unexpected error
during the next replica switching because LSN of such switch point is
before latest checkpoint record created on the reused timeline.

The optimal solution is to retry interrupted promotion. FTS is ready to
do it, at least. To achieve this goal and fall into archive recovery
again we should delay removing signal files until we persist the new
timeline information. The only insignificant negative outcome for this
approach is possible assignment of the one more timeline identity, if
we fall right after history file creation.
@Stolb27
Copy link
Collaborator Author

Stolb27 commented Jan 17, 2025

Closed in favor of #1177

@Stolb27 Stolb27 closed this Jan 17, 2025
@Stolb27 Stolb27 deleted the ADBDEV-6936 branch January 17, 2025 15:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant