How to reliably write contiguous file to exFAT?

[snhirsch]

Title says it. I have searched Google extensively and cannot find a mention of how this might be done. Even when the SD card is 90% free the firmware complains that a newly added image is not contiguous. I've tried cp, dd and Midnight Commander file copy under Linux and File Explorer in Windows 10. Just seems to be hit or miss. Worse yet, I cannot find a Windows or Linux tool that shows whether the file is contiguous, so I have to waste the time transferring the card to the ZuluSCSI RP2040 to find out. Any advice would be appreciated!

[morio]

@snhirsch
Have you tried formatting without using the quick format option.

You can download the SD Memory Card formatter here: https://www.sdcard.org/downloads/formatter/

[snhirsch]

I haven't tried that, but it's unclear why it would help. In either situation (quick vs. complete), the directory and allocation tables are totally initialized and sequential writes should end up contiguous. Since my first post I have had great luck with Windows file explorer. Linux seems completely incapable of writing a contiguous file.

[MeTalOneOEight]

You could check with a defrag tool to visualize if the files are fragmented.
It's not recommend to use it on solid state media though.

[snhirsch]

After some experimentation it appears that Windows File Explorer does reliably produce contiguous exFAT files. Linux simply cannot, so something to keep in mind. 'Defraggler' from the CC Cleaner folks does report on file layout but as you pointed out the repair process probably shouldn't be run on an SSD.

[aperezbios]

@snhirsch UltraDefrag, which is free, can display fragmentation info, as well as defrag the file, if necessary. It's primarily Windows software, see https://sourceforge.net/projects/ultradefrag/

That said, there's a command-line port of it to Linux.

[aperezbios]

After some experimentation it appears that Windows File Explorer does reliably produce contiguous exFAT files. Linux simply cannot, so something to keep in mind.

I think you may have been running up against a known (and fixed) bug under Linux, but without knowing the exact version of the distribution and kernel versions you were using when you experienced the issue, we'll never know for certain. Thank you for the update.

[snhirsch]

FWIW, I'm using Ubuntu 18.04 with kernel 4.15.0-221. The link for the linux command line build is broken, so that is no longer available.

[MeTalOneOEight]

Maybe this helps on linux:
arter97/exfat-linux#25 (comment)

[snhirsch]

Interesting read, but no indication that anything was ever fixed.

[MeTalOneOEight]

Seems so, but maybe you try that
fallocate -xl 1G testimage
for creating a contiguous image with linux and exFAT.

[dranch]

I had an extensive discussion about this in Aug 2024 with @aperezbios. In that discussion, he said:

The issue with fallocate is that it only works on supported filesystems, which, it turns out, does not include (ex)FAT*

The fallocate man page states the following:

Not all filesystems support FALLOC_FL_PUNCH_HOLE; if a filesystem doesn't support the operation, an error is returned.
The operation is supported on at least the following filesystems:

• XFS (since Linux 2.6.38)
• ext4 (since Linux 3.0)
• Btrfs (since Linux 3.7)
• tmpfs(5) (since Linux 3.5)
• gfs2(5) (since Linux 4.16)

See #277 for some detail but I never had great luck with the "truncate" program either. Alex then later came back with:

Are you using the old exfat-fuse driver? See below feedback from my core developer.

Our documentation has now been updated with instructions to use truncate under Linux. See c4b7528

• Mounted using fuseblk: I think he may be using the old exfat-fuse driver.
  It has a bug preventing contiguous file writing that has been fixed since 2018 but the fix has not ended up in distribution packages.

• Linux kernel has native exfat read&write support since version 5.7, so you can remove the exfat-fuse package.
  After that the mount should show something like "/dev/sde1 on /mnt/sd type exfat".

  fallocate not working on exfat: Good catch, looks like exfat driver (neither fuse or kernel) does not support fallocate (it would create a sparse file).
  Apparently I had only tested it on main drive and copied the file over.
  I'll change README to recommend truncate instead.

  Drive geometry warning: The value is number of sectors, not bytes. Most hosts don't really care either way.
  My personal opinion is that maybe we should change the defaults to e.g. 32 sectors x 32 heads, to make a nice 1024.
  But I'm not sure whether that would affect some users negatively - then they would have override geometry in zuluscsi.ini.
  So far Sparcstation seems to be the most picky about the geometry settings, and it will probably refuse to access the old > images if reported geometry changes.
  "This makes sense with the math but it the ZuluSCSI still complains about it not being byte aligned BUT it seems to perform fast)":
  It's not about byte alignment, it is about number of sectors.
  It will not affect speed, like the warning says it just can cause notifications in some diagnostic tools when geometry doesn't > match disk size.
  Very few operating systems care, and those that do, refuse to format the drive.

--

Hey Alex,

Per that exfat-fuse URL you provided, I started looking around and strangely enough, Ubuntu 20.04 has BOTH the FUSE exfat but also the native in-kernel exfat.

#The OS does have the fuse drivers installed by default:

$ dpkg -l | grep exfat
ii exfat-fuse 1.3.0-1 amd64 read and write exFAT driver for FUSE
ii exfat-utils 1.3.0-1 amd64 utilities to create, check, label and dump exFAT filesystem

Googleing around, I found https://askubuntu.com/questions/1234755/how-to-enable-exfat-for-ubuntu-20-04 which seems like the user can safely remove the exfat-fuse package to use the in-kernel one. Unfortunately for me, there is a dependency:

$ sudo dpkg -r exfat-fuse
dpkg: dependency problems prevent removal of exfat-fuse:
libguestfs0:amd64 depends on exfat-fuse; however:
Package exfat-fuse is to be removed.

I do have an Ubuntu 22.04 machine so I guess the next step here would be to try using the "native" exfat driver and use truncate to see if it won't create a fragmented file.

--

Damn. Dependency hell. There\u2019s quite possibly a way to blacklist loading of the exfat-fuse userspace module.

You could also create a zero-byte file of the desired size, and then copy it (yes, all 10GB or whatever) to a freshly-formatted SD card on a non-Linux machine, if you have one.

[dranch]

Ultimately, the only method I found to create a non-fragmented file was using Windows. Yuk.

[MeTalOneOEight]

The stuff I linked suggest the -x to make it work which is

       -x, --posix
           Enable POSIX operation mode. In that mode allocation
           operation always completes, but it may take longer time when
           fast allocation is not supported by the underlying
           filesystem.

For posix_fallocate see:
https://man7.org/linux/man-pages/man3/posix_fallocate.3.html