Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

save: Perform write process safe #3273

Open
wants to merge 33 commits into
base: master
Choose a base branch
from

Conversation

JoeKar
Copy link
Collaborator

@JoeKar JoeKar commented Apr 28, 2024

The target situation shall be, that micro checks if the file to be stored already exists and if so it shall work with a temporary file first, before the target file is overwritten respective the temporary file renamed to the target file.
Possible symlinks pointing to the target file will be resolved, before the save takes place. This shall guarantee that this symlink isn't renamed by the new approach.

TODOs:

Fixes #1916
Fixes #3148
Fixes #3196

@JoeKar JoeKar requested a review from dmaluka April 28, 2024 15:10
internal/buffer/save.go Outdated Show resolved Hide resolved
@JoeKar JoeKar force-pushed the fix/save-atomically branch 2 times, most recently from 7d663d8 to afdc7a1 Compare April 29, 2024 21:46
internal/buffer/save.go Outdated Show resolved Hide resolved
internal/buffer/save.go Show resolved Hide resolved
internal/buffer/save.go Show resolved Hide resolved
internal/buffer/save.go Outdated Show resolved Hide resolved
internal/buffer/save.go Outdated Show resolved Hide resolved
@JoeKar JoeKar marked this pull request as draft April 30, 2024 18:46
@JoeKar JoeKar force-pushed the fix/save-atomically branch 3 times, most recently from 1ea6aaf to 3510fcc Compare May 12, 2024 19:47
@JoeKar JoeKar marked this pull request as ready for review May 12, 2024 19:49
internal/buffer/buffer.go Outdated Show resolved Hide resolved
internal/buffer/save.go Outdated Show resolved Hide resolved
internal/buffer/save.go Outdated Show resolved Hide resolved
internal/buffer/save.go Outdated Show resolved Hide resolved
internal/util/util.go Outdated Show resolved Hide resolved
internal/buffer/save.go Outdated Show resolved Hide resolved
internal/buffer/buffer.go Outdated Show resolved Hide resolved
internal/action/bindings.go Outdated Show resolved Hide resolved
internal/buffer/buffer.go Outdated Show resolved Hide resolved
cmd/micro/micro.go Outdated Show resolved Hide resolved
internal/util/util.go Outdated Show resolved Hide resolved
internal/util/util.go Outdated Show resolved Hide resolved
internal/buffer/save.go Outdated Show resolved Hide resolved
@JoeKar
Copy link
Collaborator Author

JoeKar commented May 16, 2024

Still a lot of rework left, but thank your for your time reviewing this! 👍

@JoeKar JoeKar marked this pull request as draft May 23, 2024 18:39
@JoeKar JoeKar force-pushed the fix/save-atomically branch from 3510fcc to 97a4e83 Compare May 25, 2024 15:20
@JoeKar
Copy link
Collaborator Author

JoeKar commented May 26, 2024

I did a lot of (commit) refactoring, but still I'm not fully happy with it.
The backup approach is still messy (duplicated code, no single point of truth, etc.), but currently I'm running out of ideas.

internal/buffer/buffer.go Show resolved Hide resolved
internal/buffer/save.go Outdated Show resolved Hide resolved
internal/buffer/save.go Outdated Show resolved Hide resolved
internal/util/util.go Show resolved Hide resolved
@dmaluka
Copy link
Collaborator

dmaluka commented May 27, 2024

I did a lot of (commit) refactoring, but still I'm not fully happy with it. The backup approach is still messy (duplicated code, no single point of truth, etc.), but currently I'm running out of ideas.

IIUC, basically, all these troubles are due to the need to avoid circular dependencies between packages?

If so, it seems we could avoid these troubles by implementing overwrite & backup stuff as a part of the config package. OTOH, that sounds like an abuse of the config package: it is supposed to implement stuff related specifically to configuration, not stuff like buffer backups.

So maybe instead, simply provide separate implementations of overwrite & backup stuff for different kinds of files, in both buffer and config packages (especially since, as I noted in another comment, it seems better to have separate implementations of Overwrite() anyway). Maybe make them implement some common interface (if there is a need for that).

A very rough idea:

type FileWriter interface {
	Overwrite(name string) error
	BackupName(name string) string
	KeepBackup() bool
}

and a common Write() implementation using this interface:

func Write(name string, fwriter FileWriter) error

Or maybe even Write() implementations need to be different for different files.

P.S. Side note: maybe let's name it SafeWrite()?

@JoeKar JoeKar force-pushed the fix/save-atomically branch from ea4176c to 474f98d Compare June 1, 2024 14:56
@JoeKar
Copy link
Collaborator Author

JoeKar commented Jun 1, 2024

Yes, I know dc701a1 doesn't really belong to the PR in the first place, but I touched some WriteFile's anyway.
It was not a big deal to do this cleanup.

@JoeKar JoeKar marked this pull request as ready for review June 2, 2024 19:36
internal/buffer/backup.go Outdated Show resolved Hide resolved
@JoeKar JoeKar force-pushed the fix/save-atomically branch from 474f98d to 8c6835c Compare June 4, 2024 18:29
@JoeKar JoeKar force-pushed the fix/save-atomically branch 2 times, most recently from 42e896f to b92a6cd Compare September 3, 2024 16:30
JoeKar added 29 commits January 26, 2025 15:01
If the new URL encoded path is found then it has precedence over the '%' escaped
path. In case none of both is found the new URL approach is used.
This is fully handled within the buffers `save` domain.
…cess

SafeWrite() will create a temporary intermediate file.
- extract the open logic into `openFile()` and return a `wrappedFile`
- extract the closing logic into `Close()` and make a method of `wrappedFile`
- rename `writeFile()` into `Write()` and make a method of `wrappedFile`

This allows to use the split parts alone while keeping overwriteFile() as simple
interface to use all in a row.
As advantage we don't need to synchonize them any longer and
don't need further insufficient lock mechanisms.
Now the main go routine takes care of the backup synchronization.
@JoeKar JoeKar force-pushed the fix/save-atomically branch from 63cef41 to ca6d1e9 Compare January 26, 2025 14:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants