Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Item: remove .chunks_healthy, fixes #8559 #8568

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Item: remove .chunks_healthy, fixes #8559 #8568

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3bc0b04
Item: remove .chunks_healthy, fixes #8559
ThomasWaldmann Nov 25, 2024
c429dec
simplify fetch_many api
ThomasWaldmann Nov 25, 2024
69d504d
get/get_many: add raise_missing=True param
ThomasWaldmann Nov 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/internals/frontends.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -480,8 +480,8 @@ Refer to the *borg list* documentation for the available keys and their meaning.

Example (excerpt) of ``borg list --json-lines``::

{"type": "d", "mode": "drwxr-xr-x", "user": "user", "group": "user", "uid": 1000, "gid": 1000, "path": "linux", "healthy": true, "target": "", "flags": null, "mtime": "2017-02-27T12:27:20.023407", "size": 0}
{"type": "d", "mode": "drwxr-xr-x", "user": "user", "group": "user", "uid": 1000, "gid": 1000, "path": "linux/baz", "healthy": true, "target": "", "flags": null, "mtime": "2017-02-27T12:27:20.585407", "size": 0}
{"type": "d", "mode": "drwxr-xr-x", "user": "user", "group": "user", "uid": 1000, "gid": 1000, "path": "linux", "target": "", "flags": null, "mtime": "2017-02-27T12:27:20.023407", "size": 0}
{"type": "d", "mode": "drwxr-xr-x", "user": "user", "group": "user", "uid": 1000, "gid": 1000, "path": "linux/baz", "target": "", "flags": null, "mtime": "2017-02-27T12:27:20.585407", "size": 0}


Archive Differencing
Expand Down
134 changes: 33 additions & 101 deletions src/borg/archive.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -280,7 +280,7 @@
if filter is None or filter(item):
if "chunks" in item:
item.chunks = [ChunkListEntry(*e) for e in item.chunks]
if "chunks_healthy" in item:
if "chunks_healthy" in item: # legacy
item.chunks_healthy = [ChunkListEntry(*e) for e in item.chunks_healthy]
yield item

Expand Down Expand Up @@ -312,10 +312,22 @@
self.repository.preload([c.id for c in item.chunks])
return preload_chunks

def fetch_many(self, ids, is_preloaded=False, ro_type=None):
def fetch_many(self, chunks, is_preloaded=False, ro_type=None):
assert ro_type is not None
for id_, cdata in zip(ids, self.repository.get_many(ids, is_preloaded=is_preloaded)):
_, data = self.repo_objs.parse(id_, cdata, ro_type=ro_type)
ids = []
sizes = []
if all(isinstance(chunk, ChunkListEntry) for chunk in chunks):
for chunk in chunks:
ids.append(chunk.id)
sizes.append(chunk.size)
elif all(isinstance(chunk, bytes) for chunk in chunks):
ids = list(chunks)
sizes = [None] * len(ids)
else:
raise TypeError(f"unsupported or mixed element types: {chunks}")

Check warning on line 327 in src/borg/archive.py

View check run for this annotation

Codecov / codecov/patch

src/borg/archive.py#L327 

Added line #L327 was not covered by tests
for id, size, cdata in zip(ids, sizes, self.repository.get_many(ids, is_preloaded=is_preloaded)):
_, data = self.repo_objs.parse(id, cdata, ro_type=ro_type)
assert size is None or len(data) == size
yield data


Expand Down Expand Up @@ -762,7 +774,6 @@
# if a previous extraction was interrupted between setting the mtime and setting non-default flags.
return True

has_damaged_chunks = "chunks_healthy" in item
if dry_run or stdout:
with self.extract_helper(item, "", hlm, dry_run=dry_run or stdout) as hardlink_set:
if not hardlink_set:
Expand All @@ -771,9 +782,7 @@
# it would get stuck.
if "chunks" in item:
item_chunks_size = 0
for data in self.pipeline.fetch_many(
[c.id for c in item.chunks], is_preloaded=True, ro_type=ROBJ_FILE_STREAM
):
for data in self.pipeline.fetch_many(item.chunks, is_preloaded=True, ro_type=ROBJ_FILE_STREAM):
if pi:
pi.show(increase=len(data), info=[remove_surrogates(item.path)])
if stdout:
Expand All @@ -789,8 +798,6 @@
item_size, item_chunks_size
)
)
if has_damaged_chunks:
raise BackupError("File has damaged (all-zero) chunks. Try running borg check --repair.")
return

dest = self.cwd
Expand Down Expand Up @@ -824,8 +831,7 @@
with backup_io("open"):
fd = open(path, "wb")
with fd:
ids = [c.id for c in item.chunks]
for data in self.pipeline.fetch_many(ids, is_preloaded=True, ro_type=ROBJ_FILE_STREAM):
for data in self.pipeline.fetch_many(item.chunks, is_preloaded=True, ro_type=ROBJ_FILE_STREAM):
if pi:
pi.show(increase=len(data), info=[remove_surrogates(item.path)])
with backup_io("write"):
Expand All @@ -845,8 +851,6 @@
raise BackupError(
f"Size inconsistency detected: size {item_size}, chunks size {item_chunks_size}"
)
if has_damaged_chunks:
raise BackupError("File has damaged (all-zero) chunks. Try running borg check --repair.")
return
with backup_io:
# No repository access beyond this point.
Expand Down Expand Up @@ -1010,8 +1014,8 @@
path,
item1,
item2,
archive1.pipeline.fetch_many([c.id for c in item1.get("chunks", [])], ro_type=ROBJ_FILE_STREAM),
archive2.pipeline.fetch_many([c.id for c in item2.get("chunks", [])], ro_type=ROBJ_FILE_STREAM),
archive1.pipeline.fetch_many(item1.get("chunks", []), ro_type=ROBJ_FILE_STREAM),
archive2.pipeline.fetch_many(item2.get("chunks", []), ro_type=ROBJ_FILE_STREAM),
can_compare_chunk_ids=can_compare_chunk_ids,
)

Expand Down Expand Up @@ -1159,10 +1163,6 @@
return chunk_entry

item.chunks = []
# if we rechunkify, we'll get a fundamentally different chunks list, thus we need
# to get rid of .chunks_healthy, as it might not correspond to .chunks any more.
if self.rechunkify and "chunks_healthy" in item:
del item.chunks_healthy
for chunk in chunk_iter:
chunk_entry = chunk_processor(chunk)
item.chunks.append(chunk_entry)
Expand Down Expand Up @@ -1779,13 +1779,10 @@
if defect_chunks:
if self.repair:
# if we kill the defect chunk here, subsequent actions within this "borg check"
# run will find missing chunks and replace them with all-zero replacement
# chunks and flag the files as "repaired".
# if another backup is done later and the missing chunks get backed up again,
# a "borg check" afterwards can heal all files where this chunk was missing.
# run will find missing chunks.
logger.warning(
"Found defect chunks. They will be deleted now, so affected files can "
"get repaired now and maybe healed later."
"Found defect chunks and will delete them now. "
"Reading files referencing these chunks will result in an I/O error."
)
for defect_chunk in defect_chunks:
# remote repo (ssh): retry might help for strange network / NIC / RAM errors
Expand All @@ -1805,10 +1802,7 @@
else:
logger.warning("chunk %s not deleted, did not consistently fail.", bin_to_hex(defect_chunk))
else:
logger.warning(
"Found defect chunks. With --repair, they would get deleted, so affected "
"files could get repaired then and maybe healed later."
)
logger.warning("Found defect chunks. With --repair, they would get deleted.")
for defect_chunk in defect_chunks:
logger.debug("chunk %s is defect.", bin_to_hex(defect_chunk))
log = logger.error if errors else logger.info
Expand Down Expand Up @@ -1919,80 +1913,18 @@
self.repository.put(id_, cdata)

def verify_file_chunks(archive_name, item):
"""Verifies that all file chunks are present.

Missing file chunks will be replaced with new chunks of the same length containing all zeros.
If a previously missing file chunk re-appears, the replacement chunk is replaced by the correct one.
"""

def replacement_chunk(size):
chunk = Chunk(None, allocation=CH_ALLOC, size=size)
chunk_id, data = cached_hash(chunk, self.key.id_hash)
cdata = self.repo_objs.format(chunk_id, {}, data, ro_type=ROBJ_FILE_STREAM)
return chunk_id, size, cdata

"""Verifies that all file chunks are present. Missing file chunks will be logged."""
offset = 0
chunk_list = []
chunks_replaced = False
has_chunks_healthy = "chunks_healthy" in item
chunks_current = item.chunks
chunks_healthy = item.chunks_healthy if has_chunks_healthy else chunks_current
if has_chunks_healthy and len(chunks_current) != len(chunks_healthy):
# should never happen, but there was issue #3218.
logger.warning(f"{archive_name}: {item.path}: Invalid chunks_healthy metadata removed!")
del item.chunks_healthy
has_chunks_healthy = False
chunks_healthy = chunks_current
for chunk_current, chunk_healthy in zip(chunks_current, chunks_healthy):
chunk_id, size = chunk_healthy
for chunk in item.chunks:
chunk_id, size = chunk
if chunk_id not in self.chunks:
# a chunk of the healthy list is missing
if chunk_current == chunk_healthy:
logger.error(
"{}: {}: New missing file chunk detected (Byte {}-{}, Chunk {}). "
"Replacing with all-zero chunk.".format(
archive_name, item.path, offset, offset + size, bin_to_hex(chunk_id)
)
)
self.error_found = chunks_replaced = True
chunk_id, size, cdata = replacement_chunk(size)
add_reference(chunk_id, size, cdata)
else:
logger.info(
"{}: {}: Previously missing file chunk is still missing (Byte {}-{}, Chunk {}). "
"It has an all-zero replacement chunk already.".format(
archive_name, item.path, offset, offset + size, bin_to_hex(chunk_id)
)
)
chunk_id, size = chunk_current
if chunk_id not in self.chunks:
logger.warning(
"{}: {}: Missing all-zero replacement chunk detected (Byte {}-{}, Chunk {}). "
"Generating new replacement chunk.".format(
archive_name, item.path, offset, offset + size, bin_to_hex(chunk_id)
)
)
self.error_found = chunks_replaced = True
chunk_id, size, cdata = replacement_chunk(size)
add_reference(chunk_id, size, cdata)
else:
if chunk_current == chunk_healthy:
pass # normal case, all fine.
else:
logger.info(
"{}: {}: Healed previously missing file chunk! (Byte {}-{}, Chunk {}).".format(
archive_name, item.path, offset, offset + size, bin_to_hex(chunk_id)
)
logger.error(
"{}: {}: Missing file chunk detected (Byte {}-{}, Chunk {}).".format(
archive_name, item.path, offset, offset + size, bin_to_hex(chunk_id)
)
chunk_list.append([chunk_id, size]) # list-typed element as chunks_healthy is list-of-lists
)
self.error_found = True
offset += size
if chunks_replaced and not has_chunks_healthy:
# if this is first repair, remember the correct chunk IDs, so we can maybe heal the file later
item.chunks_healthy = item.chunks
if has_chunks_healthy and chunk_list == chunks_healthy:
logger.info(f"{archive_name}: {item.path}: Completely healed previously damaged file!")
del item.chunks_healthy
item.chunks = chunk_list
if "size" in item:
item_size = item.size
item_chunks_size = item.get_size(from_chunks=True)
Expand Down Expand Up @@ -2270,7 +2202,7 @@
return chunk_entry

def iter_chunks(self, archive, target, chunks):
chunk_iterator = archive.pipeline.fetch_many([chunk_id for chunk_id, _ in chunks], ro_type=ROBJ_FILE_STREAM)
chunk_iterator = archive.pipeline.fetch_many(chunks, ro_type=ROBJ_FILE_STREAM)
if target.recreate_rechunkify:
# The target.chunker will read the file contents through ChunkIteratorFileWrapper chunk-by-chunk
# (does not load the entire file into memory)
Expand Down
23 changes: 1 addition & 22 deletions src/borg/archiver/check_cmd.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,28 +168,7 @@ def build_parser_check(self, subparsers, common_parser, mid_common_parser):

2. When checking the consistency and correctness of archives, repair mode might
remove whole archives from the manifest if their archive metadata chunk is
corrupt or lost. On a chunk level (i.e. the contents of files), repair mode
will replace corrupt or lost chunks with a same-size replacement chunk of
zeroes. If a previously zeroed chunk reappears, repair mode will restore
this lost chunk using the new chunk.

Most steps taken by repair mode have a one-time effect on the repository, like
removing a lost archive from the repository. However, replacing a corrupt or
lost chunk with an all-zero replacement will have an ongoing effect on the
repository: When attempting to extract a file referencing an all-zero chunk,
the ``extract`` command will distinctly warn about it. The FUSE filesystem
created by the ``mount`` command will reject reading such a "zero-patched"
file unless a special mount option is given.

As mentioned earlier, Borg might be able to "heal" a "zero-patched" file in
repair mode, if all its previously lost chunks reappear (e.g. via a later
backup). This is achieved by Borg not only keeping track of the all-zero
replacement chunks, but also by keeping metadata about the lost chunks. In
repair mode Borg will check whether a previously lost chunk reappeared and will
replace the all-zero replacement chunk by the reappeared chunk. If all lost
chunks of a "zero-patched" file reappear, this effectively "heals" the file.
Consequently, if lost chunks were repaired earlier, it is advised to run
``--repair`` a second time after creating some new backups.
corrupt or lost. Borg will also report files that reference missing chunks.

If ``--repair --find-lost-archives`` is given, previously lost entries will
be recreated in the archive directory. This is only possible before
Expand Down
Loading
Loading