Externally-visible metadata?

[sagebind]

I've been working on an alternate implementation to be compliant with the AppImage type 2 spec using Zip as the image file system (https://github.com/sagebind/appimagezip), but now I realize there is no way for external tools to know what file system an AppImage actually uses. Would it be a good idea to add some metadata to a new ELF section?

I'm thinking something like this:

$ readelf -p .appimage MyApp.AppImage
name = My App
builder = AppImageKit
filesystem = squashfs

Being able to identify the file system type could be useful, so that tools don't have to guess the format by scanning the file. For example, if we want to extract an AppImage or list its contents, we would need to know the file system format first. It may also be useful to record the application used to generate the AppImage.

We can't store this metadata in the .desktop file since we can't access it without reading the file system format first.

[probonopd]

Hi @sagebind,

welcome to AppImage!

The name you mention is in the Name= key of the .desktop file. The builder you mentioned is currently not available in the metadata but could be embedded inside the filesystem; I don't see a pressing need for it to be outside of the filesystem. The filesystem you are mentioning is represented by different magic bytes. AppImageSpec currently specifies two image types with two different filesystems for AppImage:

1. ISO9660 (type 1 images)
2. squashfs (type 2 images)

They have different magic numbers which can be used to determine the file type.

1. 0x414901 at offset 8
2. 0x414902 at offset 8

If you have a clear case why you need another filesystem, we could think about type 3 AppImages, with, you guessed it, 0x414903 at offset 8. But we should do so only if there is a very clear need and a very clear use case which type 2 cannot do. Also, we should do performance comparisons.

[sagebind]

Maybe I misunderstood the draft spec, but I did not think type 2 was specific to squashfs and that using squashfs was an implementation detail. From the draft:

Type 2 image format

An AppImage which conforms to the type 2 image format:
[...]

• MUST have appended to it a filesystem that the ELF part can mount

Also, in the announcement of the type 2 format: http://discourse.appimage.org/t/the-future-of-appimage-type-2-and-new-tools/65/2:

In contrast to the earlier ISO9660-based AppImage format, the new type 2 format:

• Is not tied to any particular filesystem format (so there might be others in the future); right now my implementation uses squashfs rather than ISO9660

If type 2 is squashfs-specific, then we should probably say so in the draft. Otherwise, I see the fs implementation as distinct from the spec version. We could have a different set of magic bytes at a different offset for that.

To be clear, I don't think that we should replace squashfs with something else, but add some means to identify if an image is squashfs or something else. A type 3 sounds like it supersedes type 2, which is not my goal.

[probonopd]

Actually you are right, the spec for type 2 allows any type of filesystem to be used. I forgot that I had intentionally made it that generic, so thanks for reminding me. An external application wishing to determine the filesystem format would, in this case, need to get the offset to the beginning of the filesystem (= the byte after the end of the ELF) and then determine from the magic bytes of that filesystem what it actually is.

[probonopd]

@sagebind still I'd be interested in your motivation to go with zip rather than squashfs.

[sagebind]

That's a good point, it shouldn't be too hard to read the magic bytes right after the offset of the ELF, that seems like a reasonable solution to me. I should have thought of that...

I'm mostly experimenting for the heck of it, but I decided to use zip for three reasons:

1. Just like squashfs has better, more available tooling than ISO9660, zip has even more tooling than squashfs.
2. The zip format is very simple, so it should be possible to implement a smaller runtime binary that uses either very small zip libraries or no library at all. With squashfs you need to pull in all of squashfuse.
3. You can extract and/or view the image without executing it without any special tools, since the zip format allows arbitrary prepended data. So for example, zipinfo UsingZip.AppImage works out-of-the-box, and renaming to a .zip lets you open it in a GUI.

I'm also concerned about performance; in theory zip lookup tables are quite fast, but in-file seeking is an issue. I think I can do some benchmarks once my implementation is more complete. The compression algorithm used might have a big effect on performance too.

[probonopd]

Yes @sagebind I'd be very interested in benchmarks, e.g., of LibreOffice launch times.