Add Models/Textures/Mapped Images Validation Checks

[DevGeniusCode]

This PR introduces comprehensive validation checks for graphical assets, including mapped images, textures, models, and language-specific assets. It also identifies unused assets and ensures that references are correctly accounted for, reporting any missing, duplicate, or invalid items. The changes include:

1. Mapped Image Parsing:

   • Scans .ini mapped files for texture and image references, ensuring that all referenced images are validated and duplicates are reported.

2. Missing Texture/Model Files (General):

   • Compares the list of texture and model files from the .ini files against a provided CSV file and reports any missing texture or model files.

3. Unused Graphical Assets (General):

   • Identifies texture, model, and image files present in the file system but without corresponding references in any .ini, .wnd, or .w3d file. These unused assets are reported, indicating potential dead assets.

4. Invalid Images in WND Files:

   • Scans .wnd files for references to images. Missing images that are referenced in WND files but not present in the mapped images are reported.

5. Invalid Textures in W3D Files:

   • Checks W3D files for invalid textures (textures referenced in W3D files but missing from the texture files list).

6. Language-Specific Asset Analysis:

   • Analyze assets within language-specific folders.
   • Identifies unused or invalid textures and models specific to each language based on the game's global INI files and language-specific asset lists.
   • Generates individual text reports for each language, detailing unused or invalid assets.

7. Interactive Main Menu:

   • Main script with a user-interactive dialog. Users can choose which function to execute.

8. Output Files:

   • Generates several output files, including:
     • mapped_images_list.txt: A list of all image mappings.
     • duplicate_mapped_images.txt: A list of duplicate image mappings.
     • invalid_textures_in_ini_files.txt: A list of invalid texture files found in INI files
     • invalid_mapped_images_in_ini_files.txt: A list of invalid mapped images found in INI files
     • invalid_models_in_ini_files.txt: A list of invalid model files found in INI files
     • invalid_textures_in_mapped_images_folder.txt: A list of invalid textures found in the mapped images folder
     • invalid_mapped_images_in_wnd_files.txt: A list of invalid mapped images found in WND files.
     • unused_textures.txt: A list of unused texture files.
     • unused_models.txt: A list of unused model files.
     • unused_mapped_images.txt: A list of unused image files.
     • invalid_textures_in_w3d_files.txt: A list of invalid textures found in W3D files.
     • Individual files for each language located in /languages/ folder.

9. Configurable Scanning Options:

   • Scan Original or Edited Files: Allows users to select whether to scan the original or edited files. In the case of edited files, the script includes also the original files for the validation check.

10. Documentation:

    • Results of research on relevant asset locations are detailed in Graphical_Assets_Location.md. If there are any additional locations or patterns not currently documented in that file, please let me know.

[DevGeniusCode]

Optimized the script and added a check for invalid textures in W3D files.
Additionally, I added the get_textures function in w3dfilemanager.py to extract the textures from W3D files.

[DevGeniusCode]

optimized the code and updated various functions/variables names for better clarity.

[DevGeniusCode]

Added validation for textures/images

1. Texture/Image Validation:

   • Implements a check for invalid textures and images. This identifies resources referenced in .ini files but not found in the file system, indicating broken or missing assets.
   • Adds a check for unused textures and images. This finds resource files present in the file system but without corresponding references in any .ini/.wnd/.w3d file, suggesting potential dead assets.

2. Interactive Main Menu:

   • Enhances the main script with a user-interactive dialog. Users can now choose which function to execute: either the new validation checks or the existing texture and image extraction process.

[DevGeniusCode]

I think I've now covered all the necessary validation for images and textures.

[xezon]

I think the txt file names need an improvement pass to clarify what they refer to.

New revision needs new approval

[DevGeniusCode]

Improvements and fixes, added checking for unused textures/images in the game.dat file, and added the SCPatchUserInterface512.INI file that was missing from the repo. the reports were updated and also include Generals 1.08 files.

Also, saw your YouTube review of my code. Yes, I know regex isn't everyone's cup of tea, but without it I'd be drowning in manual parsing. It's a necessary tool for this job, Oh, and a little secret: I'm also using AI to craft those regex monsters, so really, we are on the same boat! 😉

[DevGeniusCode]

Update:
Separation of textures and models, improved reports.

[xezon]

I wonder if we can classify ani files as cursors instead of textures to show them in separate lists.

[DevGeniusCode]

Update: fix script name and maps references and remove ani files

[xezon]

Note that we also have a GameFilesOptional folder that adds optional assets on top of the core assets (GameFilesEdited). Perhaps it would be a consideration to also add that, but that would add another layer of complexity.
https://github.com/TheSuperHackers/GeneralsGamePatch/tree/main/Patch104pZH/GameFilesOptional

Additionally, we also have assets that are generated via Mod Builder and defined in the json files:
https://github.com/TheSuperHackers/GeneralsGamePatch/blob/main/Patch104pZH/ModBundleCoreItems.json
https://github.com/TheSuperHackers/GeneralsGamePatch/blob/main/Patch104pZH/ModBundleOptionalItems.json

For example:

                            {
                                "source": "Art/Textures/GenerateMip/cbapt01_n.psd",
                                "target": "Art/Textures/cbapt01_sn.dds"
                            },
                            {
                                "source": "Art/Textures/GenerateMip/cbapt01_ng.psd",
                                "target": "Art/Textures/cbapt01_sng.dds"
                            },
                            {
                                "source": "Art/Textures/GenerateMip/cbapt01_dn.psd",
                                "target": "Art/Textures/cbapt01_dsn.dds"
                            },
                            {
                                "source": "Art/Textures/GenerateMip/cbapt01_dng.psd",
                                "target": "Art/Textures/cbapt01_dsng.dds"
                            },

[DevGeniusCode]

I think the additional lists will only contribute to unused lists, while our main concern is fixing broken references that can disrupt the game engine.

[xezon]

Yes ideally Optional files do not solely fix Core issues, but they could if we made mistakes (and I am sure we did).

[DevGeniusCode]

• Adds analyze_language_files to scan language folders for unused/invalid assets.

• Generates individual text reports for each language.

• Assumes language-specific assets are not present in global folders.
  [The language assets parameter is a dict containing models and textures that are only located inside the language folder.]

• Updated regex in Animation Models to correctly extract model names even when optional parameters are present.
  [Results of research on relevant value locations are detailed in Graphical_Assets_Location.md. If there are any additional locations or patterns not currently documented in that file, please let me know, as it will help improve the accuracy of the reports.]

[xezon]

Why is this not written as "EXthunderWave.tga" as per the original value in the INI file?

[DevGeniusCode]

Since the comparison was always without suffixes, so I did not capture them, I will include the suffixes for the reports.

[xezon]

Would a "Texture.psd" also match for a "Texture.tga" ?

[DevGeniusCode]

Indeed, assuming that Mod Builder converts it to dds/tga

[xezon]

Typically yes, but Mod Builder can also build a "Texture.psd" as "AnotherTexture.dds", and I assume the script will not catch that.

Perhaps it would be worthwhile to check against files in Patch104pZH\.Build\RawBundleItems (the files generated before archiving into big files), but this would require building (all languages) before running the script and likely make it more complicated.

[DevGeniusCode]

Will the ModBundleOptionalItems.json file be sufficient for processing?

[xezon]

Currently ModBundleCoreItems.json and ModBundleOptionalItems.json.

You could try search through all files listed in ModJsonFiles.json and find texture and model targets that are not *.ext. Perhaps that would suffice.

[xezon]

These 3 are textures, not models.

[DevGeniusCode]

True, but they are in the W3D folder instead of the Textures folder, so the function defines it as a model, if it is okay for texture files to be in the models folder, I will update the function to only look for w3d files.

[xezon]

It is a good question why textures are located in the w3d folder. I do not know what is up with that.

[DevGeniusCode]

[The current update only includes *.w3d files. So if there are texture files in the folder, they are not scanned at all.]

[xezon]

These are definitely texture files.

[DevGeniusCode]

I mean, do I need to scan them? Does the game engine scan them? Because right now my code only scans by folder paths.

[xezon]

I do not know that. We would need to test if the game can load textures from that folder.

[DevGeniusCode]

I took the file Data\English\Art\Textures\Defeated.tga and replaced it with the French version, it worked.
I moved it to the W3D folder, the English version appeared, meaning it didn't work.
I deleted it, now there is no Data\English\Art\Textures\Defeated.tga file at all, it still appeared in the English version, meaning the file is also in game.dat, but a hex scan yielded no results. I was confused.

[xezon]

The texture is not in game.dat

Defeated.tga is still inside original big file that is underlying Patch files.

Reads as if texture is not loaded from W3D folder, which means these 3 files are misplaced in the German language.

[DevGeniusCode]

I deleted all the big files after extracting.

[DevGeniusCode]

Added file extensions and improved naming.

Note that since we only catch with an extension, it was found that the ParticleSystem.ini file has an invalid parameter ParticleName = GlassShard01 because it does not include a file extension (it is currently not included in the files because it is not included in the regular expression)

[xezon]

Merge this or is anything left to do?

[DevGeniusCode]

Improve extract_art_files_from_ini_files and spelling

TODO:

• add ani assets.
• add json blender assets, currently is assuming that Mod Builder converts psd it to dds/tga