Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Store foldermap and it's dependencies at start of c4folder #70

Open
Somebodyisnobody opened this issue Feb 3, 2022 · 1 comment
Open

Store foldermap and it's dependencies at start of c4folder #70

Somebodyisnobody opened this issue Feb 3, 2022 · 1 comment

Comments

@Somebodyisnobody
Copy link
Contributor

Somebodyisnobody commented Feb 3, 2022 •
edited
Loading

In the current implementation of c4group, the foldermap and its corresponding dependencies are not stored at the beginning of a c4folder. This means that the whole file has to be read until all required contents have been loaded to display the content in the clonk menu.
In addition, there is an implementation weakness when reading the dependencies from the foldermap. So the dependencies are read after occurrences in the foldermap. If resource B is listed before resource A in the foldermap, but resource A has been saved before resource B, then resource B is loaded first before the c4folder is read from the beginning in order to load resource A.

Details 
Begin foldermap.txt                      Begin c4folder           First read   Second read
resource B needed                        c4s, c4d, other stuff         |             |  
resource C needed                        resource A                    |            <┘
resource A needed                        resource B                   <┤
End foldermap.txt                        resource C                   <┘
                                         End c4folder

These two characteristics of the c4folder cause a freezing GUI when loading large c4folders like ModernCombat.c4f.

Example with the current ModernCombat.c4f

Occurence in Foldermap:

OverlayCategories.png
OverlayCategories.png
OverlayCategories.png
OverlayAtlanticCrisis.png
OverlayBabelTower.png
OverlayBaseJumping.png
OverlayBase1337.png
OverlayBlackThunder.png
OverlayBreakthrough.png
OverlayColdCash.png
OverlayInfinityIsland.png
OverlayInterception.png
OverlayIslandStrike.png
OverlayMankelOverpass.png
OverlayMorayTowers.png
OverlayOasis.png
OverlayOperationCaveOffensive.png
OverlayPortArcford.png
OverlayTwinPeaks.png
OverlayWavebreaker.png

Actual ordering in c4folder:

                        Title.txt       54 Bytes 23.12.15 14:47:45 !FE1E31F9
                       DescDE.rtf      807 Bytes 10.04.21 22:42:44 !D724D587
                       DescUS.rtf      729 Bytes 10.04.21 22:42:55 !61CDC654
                        Title.png    55569 Bytes 10.12.20 14:56:44 !C18FE41E
                         Icon.png     1289 Bytes 10.12.20 14:55:44 !3B2FA2E6
                       Author.txt       18 Bytes 08.07.14 21:51:42 !905D789F
                      Version.txt        4 Bytes 12.10.20 23:34:03 !3E573EEA
           CMC_AtlanticCrisis.c4s   152124 Bytes 29.12.21 00:04:49 !210518E6 (Group)
               CMC_BabelTower.c4s   206482 Bytes 29.12.21 00:04:49 !8DD78565 (Group)
                 CMC_Base1337.c4s   187871 Bytes 29.12.21 00:04:49 !460AE531 (Group)
              CMC_BaseJumping.c4s   137437 Bytes 29.12.21 00:04:49 !EB8C8AEB (Group)
             CMC_BlackThunder.c4s   349523 Bytes 29.12.21 00:04:49 !489977C8 (Group)
             CMC_Breakthrough.c4s   232360 Bytes 29.12.21 00:04:50 !EDFF4499 (Group)
                 CMC_ColdCash.c4s   216069 Bytes 29.12.21 00:04:50 !F80F6C6B (Group)
           CMC_InfinityIsland.c4s   291915 Bytes 29.12.21 00:04:50 !5A872826 (Group)
             CMC_Interception.c4s   254149 Bytes 29.12.21 00:04:50 !ED4F29B4 (Group)
             CMC_IslandStrike.c4s   217980 Bytes 29.12.21 00:04:50 !C8F9B518 (Group)
           CMC_MankelOverpass.c4s   289901 Bytes 29.12.21 00:04:50 !D3F70243 (Group)
              CMC_MorayTowers.c4s   192966 Bytes 29.12.21 00:04:50 !DCE8E205 (Group)
                    CMC_Oasis.c4s   136865 Bytes 29.12.21 00:04:50 !6402F5B1 (Group)
   CMC_OperationCaveOffensive.c4s   352970 Bytes 29.12.21 00:04:50 !89621AE8 (Group)
              CMC_PortArcford.c4s   283751 Bytes 29.12.21 00:04:50 !E28B3EA6 (Group)
                CMC_TwinPeaks.c4s   205019 Bytes 29.12.21 00:04:50 !790CFBE2 (Group)
              CMC_Wavebreaker.c4s   913784 Bytes 29.12.21 00:04:50 !5665DFD6 (Group)
                  Loader_CMC1.png  1894883 Bytes 11.12.20 00:46:40 !F4F1DF40
                  Loader_CMC2.png  1799992 Bytes 11.12.20 00:47:38 !7560E503
                  Loader_CMC3.png  2516617 Bytes 11.12.20 00:48:32 !419B78D5
                    FolderMap.txt     2190 Bytes 14.08.21 23:14:37 !0B6ABE62
                    FolderMap.png  1706055 Bytes 15.08.21 12:30:44 !5358B858
                  Backgrounds.c4d  8204745 Bytes 29.12.21 00:04:46 !B2C2C909 (Group)
                      Classic.c4f 11164809 Bytes 29.12.21 00:04:49 !1C2682DD (Group)
                    Copyright.txt      292 Bytes 11.06.20 20:24:10 !2454894D
                      Credits.c4f  1785918 Bytes 29.12.21 11:56:30 !7628E026 (Group)
                     Graphics.c4g   457551 Bytes 29.12.21 00:04:50 !F22509A9 (Group)
                     Material.c4g  2300554 Bytes 29.12.21 00:04:51 !5450A609 (Group)
                        Music.c4g 37141957 Bytes 29.12.21 00:04:53 !01665E3D (Group)
        OverlayAtlanticCrisis.png    28053 Bytes 14.08.21 22:59:10 !25B6694B
            OverlayBabelTower.png    34490 Bytes 14.08.21 22:59:18 !B8D50131
              OverlayBase1337.png    37015 Bytes 14.08.21 22:59:23 !CC854C40
           OverlayBaseJumping.png    29482 Bytes 22.12.21 20:00:06 !60FBB99B
          OverlayBlackThunder.png    25187 Bytes 14.08.21 22:59:33 !335028B5
          OverlayBreakthrough.png    27939 Bytes 14.08.21 22:59:39 !B88D3DD5
            OverlayCategories.png      167 Bytes 11.06.20 17:06:52 !AF42C85C
              OverlayColdCash.png    35084 Bytes 14.08.21 22:59:51 !06C62801
        OverlayInfinityIsland.png    34428 Bytes 14.08.21 23:00:01 !C6A99214
          OverlayInterception.png    30846 Bytes 11.09.21 17:22:59 !22D629DC
          OverlayIslandStrike.png    30702 Bytes 14.08.21 23:00:09 !B037B6DF
        OverlayMankelOverpass.png    31228 Bytes 14.08.21 23:00:17 !532A311A
           OverlayMorayTowers.png    37208 Bytes 14.08.21 23:00:23 !073D98F7
                 OverlayOasis.png    33038 Bytes 14.08.21 23:00:28 !A10613AB
OverlayOperationCaveOffensive.png    31940 Bytes 14.08.21 23:00:32 !7A649B83
           OverlayPortArcford.png    35856 Bytes 14.08.21 23:00:36 !7AEDCA96
             OverlayTwinPeaks.png    37195 Bytes 14.08.21 23:00:41 !473679D3
           OverlayWavebreaker.png    27751 Bytes 22.12.21 20:00:25 !BBEE4D02
                      Special.c4f  2194845 Bytes 29.12.21 00:04:55 !E5F7F1AF (Group)
In this example the foldermap.txt and foldermap.png comes after all c4scenario groups. The dependencies of the foldermap.txt (png files) comes after the c4definitions and general c4groups, which can contain - with music for example - large files.

After sorting the c4folder manually with c4group.exe ModernCombat.c4f -s "Folder.txt|Title*.txt|Info.txt|Desc*.rtf|Title.png|Title.bmp|Icon.png|Icon.bmp|Author.txt|Version.txt|*.c4s|*.c4f|Loader*.bmp|Loader*.png|Loader*.jpeg|Loader*.jpg|FolderMap.txt|FolderMap.png|OverlayCategories.png|OverlayAtlanticCrisis.png|OverlayBabelTower.png|OverlayBaseJumping.png|OverlayBase1337.png|*.png" in pre-release v350 (v349 doesn't save the new file) the files are ordered much better. Thanks to @maxmitti for the command.

New file ordering of ModernCombat.c4f 
                        Title.txt       54 Bytes 23.12.15 14:47:45 !FE1E31F9
                       DescDE.rtf      807 Bytes 10.04.21 22:42:44 !D724D587
                       DescUS.rtf      729 Bytes 10.04.21 22:42:55 !61CDC654
                        Title.png    55569 Bytes 10.12.20 14:56:44 !C18FE41E
                         Icon.png     1289 Bytes 10.12.20 14:55:44 !3B2FA2E6
                       Author.txt       18 Bytes 08.07.14 21:51:42 !905D789F
                      Version.txt        4 Bytes 12.10.20 23:34:03 !3E573EEA
           CMC_AtlanticCrisis.c4s   152124 Bytes 29.12.21 00:04:49 !210518E6 (Group)
               CMC_BabelTower.c4s   206482 Bytes 29.12.21 00:04:49 !8DD78565 (Group)
                 CMC_Base1337.c4s   187871 Bytes 29.12.21 00:04:49 !460AE531 (Group)
              CMC_BaseJumping.c4s   137437 Bytes 29.12.21 00:04:49 !EB8C8AEB (Group)
             CMC_BlackThunder.c4s   349523 Bytes 29.12.21 00:04:49 !489977C8 (Group)
             CMC_Breakthrough.c4s   232360 Bytes 29.12.21 00:04:50 !EDFF4499 (Group)
                 CMC_ColdCash.c4s   216069 Bytes 29.12.21 00:04:50 !F80F6C6B (Group)
           CMC_InfinityIsland.c4s   291915 Bytes 29.12.21 00:04:50 !5A872826 (Group)
             CMC_Interception.c4s   254149 Bytes 29.12.21 00:04:50 !ED4F29B4 (Group)
             CMC_IslandStrike.c4s   217980 Bytes 29.12.21 00:04:50 !C8F9B518 (Group)
           CMC_MankelOverpass.c4s   289901 Bytes 29.12.21 00:04:50 !D3F70243 (Group)
              CMC_MorayTowers.c4s   192966 Bytes 29.12.21 00:04:50 !DCE8E205 (Group)
                    CMC_Oasis.c4s   136865 Bytes 29.12.21 00:04:50 !6402F5B1 (Group)
   CMC_OperationCaveOffensive.c4s   352970 Bytes 29.12.21 00:04:50 !89621AE8 (Group)
              CMC_PortArcford.c4s   283751 Bytes 29.12.21 00:04:50 !E28B3EA6 (Group)
                CMC_TwinPeaks.c4s   205019 Bytes 29.12.21 00:04:50 !790CFBE2 (Group)
              CMC_Wavebreaker.c4s   913784 Bytes 29.12.21 00:04:50 !5665DFD6 (Group)
                      Classic.c4f 11164809 Bytes 29.12.21 00:04:49 !1C2682DD (Group)
                      Credits.c4f  1785918 Bytes 29.12.21 11:56:30 !7628E026 (Group)
                      Special.c4f  2194845 Bytes 29.12.21 00:04:55 !E5F7F1AF (Group)
                  Loader_CMC1.png  1894883 Bytes 11.12.20 00:46:40 !F4F1DF40
                  Loader_CMC2.png  1799992 Bytes 11.12.20 00:47:38 !7560E503
                  Loader_CMC3.png  2516617 Bytes 11.12.20 00:48:32 !419B78D5
                    FolderMap.txt     2190 Bytes 14.08.21 23:14:37 !0B6ABE62
                    FolderMap.png  1706055 Bytes 15.08.21 12:30:44 !5358B858
            OverlayCategories.png      167 Bytes 11.06.20 17:06:52 !AF42C85C
        OverlayAtlanticCrisis.png    28053 Bytes 14.08.21 22:59:10 !25B6694B
            OverlayBabelTower.png    34490 Bytes 14.08.21 22:59:18 !B8D50131
           OverlayBaseJumping.png    29482 Bytes 22.12.21 20:00:06 !60FBB99B
              OverlayBase1337.png    37015 Bytes 14.08.21 22:59:23 !CC854C40
          OverlayBlackThunder.png    25187 Bytes 14.08.21 22:59:33 !335028B5
          OverlayBreakthrough.png    27939 Bytes 14.08.21 22:59:39 !B88D3DD5
              OverlayColdCash.png    35084 Bytes 14.08.21 22:59:51 !06C62801
        OverlayInfinityIsland.png    34428 Bytes 14.08.21 23:00:01 !C6A99214
          OverlayInterception.png    30846 Bytes 11.09.21 17:22:59 !22D629DC
          OverlayIslandStrike.png    30702 Bytes 14.08.21 23:00:09 !B037B6DF
        OverlayMankelOverpass.png    31228 Bytes 14.08.21 23:00:17 !532A311A
           OverlayMorayTowers.png    37208 Bytes 14.08.21 23:00:23 !073D98F7
                 OverlayOasis.png    33038 Bytes 14.08.21 23:00:28 !A10613AB
OverlayOperationCaveOffensive.png    31940 Bytes 14.08.21 23:00:32 !7A649B83
           OverlayPortArcford.png    35856 Bytes 14.08.21 23:00:36 !7AEDCA96
             OverlayTwinPeaks.png    37195 Bytes 14.08.21 23:00:41 !473679D3
           OverlayWavebreaker.png    27751 Bytes 22.12.21 20:00:25 !BBEE4D02
                  Backgrounds.c4d  8204745 Bytes 29.12.21 00:04:46 !B2C2C909 (Group)
                    Copyright.txt      292 Bytes 11.06.20 20:24:10 !2454894D
                     Graphics.c4g   457551 Bytes 29.12.21 00:04:50 !F22509A9 (Group)
                     Material.c4g  2300554 Bytes 29.12.21 00:04:51 !5450A609 (Group)
                        Music.c4g 37141957 Bytes 29.12.21 00:04:53 !01665E3D (Group)
The folder now load much faster than before.
with_optimization.mp4
without_optimization.mp4

I suggest that the dependencies of the foldermap and the foldermap itself are stored always near the start of the c4folder, before the other large c4 groups (c4s. c4d, c4g, c4f). This should be done by default trough c4group.exe

I also suggest to improve the reading of dependencies so that the order of dependencies in the foldermap.txt isn't a problem anymore and the c4folder must be read in only one time. @maxmitti is already aware of this issue.
@maxmitti
Copy link
Member

maxmitti commented Feb 3, 2022

abcb225 already improves the sorting order to match the command mentioned above, except for the alphabetic reordering of the overlay graphics.
It is easy enough to rearrange the folder map entries to be in alphabetical order. The order should match the output order of the command c4group Folder.c4f, as this shows the order as files are stored.

The most I would do is rewriting the graphics loading for folder map graphics to load them in group order instead of in FolderMap.txt order, although it is much better already.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@maxmitti @Fulgen301 @Somebodyisnobody