Memory mapping with files in the old format (Klusters)

[rossant]

There would be no more mandatory conversion before opening the files.

[nippoo]

How would we deal with saving? Would we have to continue supporting this code?

[rossant]

Hmm. I can see three options:

1. Full identical support of KWIK & Klusters (save multiple clusterings, cluster colors, groups, etc.). Complex since we need to define new file formats with these metadata.
2. Supporting saving the clusters only (no colors, groups). Only the .clu file is modified.
3. Read-only support: when loading files in the old format, the user would be prompted to either open the old files quickly in read-only mode, or to convert them to the new file format (slow conversion required first).

I'm leaning toward 3. I think 1 is too much work and not worth it, and 2 is not a good idea (you would want the cluster groups too, at least).

@kdharris101 what do you think? (the question was: whether to support the old file format in KlustaViewa without mandatory conversion, conversion which is a huge drawback according to many users who contacted me).

[kdharris101]

Hi Cyrille,

I’m sorry I didn’t understand the context (or missed a previous email). Can you explain in more detail?

k

From: Cyrille Rossant [mailto:[email protected]]
Sent: 06 December 2013 09:11
To: klusta-team/klustaviewa
Cc: Harris, Kenneth
Subject: Re: [klustaviewa] Memory mapping with files in the old format (Klusters) (#32)

Hmm. I can see three options:

1. Full identical support of KWIK & Klusters (save multiple clusterings, cluster colors, groups, etc.). Complex since we need to define new file formats with these metadata.
2. Supporting saving the clusters only (no colors, groups). Only the .clu file is modified.
3. Read-only support: when loading files in the old format, the user would be prompted to either open the old files quickly in read-only mode, or to convert them to the new file format (slow conversion required first).

I'm leaning toward 3. I think 1 is too much work and not worth it, and 2 is not a good idea (you would want the cluster groups too, at least).

@kdharris101https://github.com/kdharris101 what do you think? (the question was: whether to support the old file format in KlustaViewa without mandatory conversion, conversion which is a huge drawback according to many users who contacted me).

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/32#issuecomment-29973469.

[rossant]

@kdharris101 I'd like to add the possibility to open files in the old file format without any conversion. I recently discovered a very simple technique in Python that allows to do that efficiently, both in terms of loading times and memory consumption. Many people are complaining about the need to convert their terabytes of data before being able to open them in KlustaViewa. I think people will have files in the old format for still quite a few years...

A possibility would be to only support a read-only mode for the old files.