[5.x]: Applying project config changes will sometimes revert the config folder to an early state

[MoritzLost]

What happened?

Description

I'm still seeing the issue described in #15012. I can't reliably reproduce it, but sometimes, running up or project-config/apply will not apply the config, but will instead reset the config to an earlier state (essentially running project-config/write).

I've encountered this in two scenarios:

1. When switching between branches with different config states in a local dev environment (in dev mode with allowAdminChanges set to true).
2. When deploying to a live server (in production mode with allowAdminChanges set to false).

Not sure if those are the same issue, or two unrelated issues.

I don't know what causes this or how to reproduce it. Just now I saw this while deploying an update from Craft 4 to Craft 5. The environment doesn't allow admin changes, and the deployment script only applies the most recent state from GitHub, here's the relevant part:

# Flush all caches
$FORGE_PHP craft cache/flush-all --interactive=0
$FORGE_PHP craft clear-caches/all --interactive=0

# Update source code from repository
git pull --ff-only origin $FORGE_SITE_BRANCH

# Install Composer dependencies
$FORGE_COMPOSER install --no-dev --no-progress --no-interaction --prefer-dist --optimize-autoloader

# Apply project config and any pending migrations
$FORGE_PHP craft migrate/all --no-content --interactive=0
$FORGE_PHP craft project-config/apply
$FORGE_PHP craft migrate --track=content --interactive=0

Now I can see in the deployment log that the Composer installation ran successfully, and the first set of migrations (before the project-config/apply step) were also executed correctly. But after that, the project-config/apply command resulted in an error:

29 migrations were applied.

Migrated up successfully.
Your project config files were created for different versions of Craft and/or plugins than what’s currently installed.

Craft CMS is installed with schema version of 5.3.0.2 while 4.5.3.0 was expected.
Navigation is installed with schema version of 2.1.0 while 2.0.7 was expected.
Super Table is installed with schema version of 4.0.0 while 3.0.0 was expected.

Try running `composer install` from your terminal to resolve.

Checking the files on the server, I could see that the config/ folder was reset to the state before the deployment started. After running git reset --hard and git clean -df, I restarted the deployment. This time, it finished successfully.

So something caused Craft to write out the existing config instead of applying it. Either as part of the migrate/all --no-content step, or at the beginning of the project-config/apply step.

Steps to reproduce

Unclear.

Expected behavior

• Running project-config/apply should never, in any circumstances, change any files in the config/ folder. If the config can't be applied for any reason, it should throw an error, and possibly advice on how to fix the issue.
• While allowAdminChanges is set to false, Craft should never, ever write anything to the config/ folder.

Actual behavior

In some situations, running up or project-config/apply will have unexpectedly revert the config to the current state in the database, acting like project-config/write.

Craft CMS version

5.4.10

PHP version

8.3

Operating system and version

No response

Database type and version

MySQL 8

Image driver and version

No response

Installed plugins and versions

No response

[brandonkelly]

The next time this happens, can you capture the resulting (“original”) project config that gets output, and compare it with the actual original project config? I’m curious if there are any changes that stick around from the deployed project config changes.

[MoritzLost]

@brandonkelly Thanks for the reply, I'll try to check that next time it happens. Pretty sure there were no changes between the previous config and the resulting config, though. At the moment, we're seeing this issue frequently in dev – we'll switch branches, apply the config, and instead the config is reset to the state of the previous branch. Anything we can check in that state to see where it went wrong, or provide more debug info?

Though some instances of this were in outdated Craft versions, so this might also muddy the waters. I'm trying to get all our current projects updated to 5.5+ to see if it's still an issue there.

[MoritzLost]

@brandonkelly Some more context – I'm noticing that this happens at least every time there's any other error that causes up or project-config/apply to fail. Issues like #16150 or #16145 result in an error during this step. After that, everything looks fine and git doesn't show any changes. But running project-config/apply again after that will reset the config and write out the previous config to the config/ folder.

That's definitely not the only time this happens, but it's definitely a pain point.

[MoritzLost]

@brandonkelly Just had this happen again during a Craft 4 to 5 update on a production site. This time, we tested the deployment thoroughly on a local development environment – right before the update, we made a backup of the live database and files and installed those locally, so the dev environment had completely identical content to the live environment. Then we executed the update in the dev environment, which worked without problems. Deploying those exact changes in production again caused the project config to be reset to the previous version.

• The dev environment uses ddev and is set to CRAFT_ENVIRONMENT=dev.
• The production environment is a Hetzner VPS controlled through Laravel Forge, the deployment happens directly on the server. It's set to CRAFT_ENVIRONMENT=production.

You can see exactly where this happens in the deployment log (left some parts out, the log is thousands of lines):

Tue Jan 7 03:20:31 PM CET 2025
Backing up the database ... done
Backup file: ***--2025-01-07-142031--v4.13.6.sql.zip (12.91 MB)
The system is now offline.
The retry duration is now set to 15.
The following cache components were processed:

	* cache (craft\cache\FileCache)

Clearing cache: Asset caches
Clearing cache: Asset indexing data
Clearing cache: Asset transform index
Clearing cache: Compiled classes
Clearing cache: Compiled templates
Clearing cache: Control panel resources
Clearing cache: Data caches
Clearing cache: Hyper field element cache
Clearing cache: Link field element cache
Clearing cache: Retour redirect caches
Clearing cache: Temp files
From github.com:schwarzdesign/project
 * branch            main       -> FETCH_HEAD
   7281737..eb234b3  main       -> origin/main
Updating 7281737..eb234b3
Fast-forward
 composer.json                                      |   19 +-
 composer.lock                                      |  Bin 237872 -> 259208 bytes

[…]

Installing dependencies from lock file
Verifying lock file contents can be installed on current platform.
Package operations: 21 installs, 23 updates, 15 removals
  - Downloading dasprid/enum (1.0.6)
  - Downloading bacon/bacon-qr-code (2.0.8)
  - Downloading brick/math (0.12.1)
  - Downloading craftcms/cms (5.5.7)

[…]

Generating optimized autoload files
2 packages you are using are looking for funding.
Use the `composer fund` command to find out more!
> @php -r "file_exists('.env') || copy('.env.example', '.env');"
Checking for pending Craft and plugin migrations ...
Total 28 new Craft migrations to be applied:
    - m221101_115859_create_entries_authors_table
    - m221107_112121_add_max_authors_to_sections
    - m221205_082005_translatable_asset_alt_text

[…]

Backing up the database ... done
*** applying m221101_115859_create_entries_authors_table
    > dropping {{%entries_authors}} if it exists ... done (time: 0.003s)
    > create table {{%entries_authors}} ... done (time: 0.029s)
    > create index idx_djecjtzdhrunkynchcxqzhakfkkluxrngazd on {{%entries_authors}} (authorId) ... done (time: 0.029s)
    > create index idx_mdstwvlimqptegeksdfvkxnsuubytmvoykni on {{%entries_authors}} (entryId,sortOrder) ... done (time: 0.027s)

[… thousands of migrations and entry updates …]

    > drop table {{%supertableblocks_owners}} ... done (time: 0.023s)
    > drop table {{%supertableblocks}} ... done (time: 0.040s)
    > drop table {{%supertableblocktypes}} ... done (time: 0.025s)
    > drop table stc_11_entries_1 ... done (time: 0.041s)
    > drop table stc_15_social_media_1 ... done (time: 0.037s)
    > drop table stc_16_links_1 ... done (time: 0.032s)
    > drop table stc_19_teasers_1 ... done (time: 0.036s)
    > drop table stc_social_media ... done (time: 0.020s)
*** applied m240115_000000_craft5 (time: 38.615s)


30 migrations were applied.

Migrated up successfully.
Your project config files were created for different versions of Craft and/or plugins than what’s currently installed.

Craft CMS is installed with schema version of 5.5.0.0 while 4.5.3.0 was expected.
Navigation is installed with schema version of 2.1.0 while 2.0.7 was expected.
Super Table is installed with schema version of 4.0.0 while 3.0.0 was expected.

Try running `composer install` from your terminal to resolve.

This is the relevant part of the deployment script:

# Flush all caches
$FORGE_PHP craft cache/flush-all --interactive=0
$FORGE_PHP craft clear-caches/all --interactive=0

# Update source code from repository
git pull --ff-only origin $FORGE_SITE_BRANCH

# Install Composer dependencies
$FORGE_COMPOSER install --no-dev --no-progress --no-interaction --prefer-dist --optimize-autoloader

# Apply project config and any pending migrations
$FORGE_PHP craft migrate/all --no-content --interactive=0
$FORGE_PHP craft project-config/apply
$FORGE_PHP craft migrate --track=content --interactive=0

The git pull --ff-only will never reset the config to an early stage or perform a merge commit, and it would fail if it couldn't pull down the latest commit and apply it cleanly. So at this stage, the config/ folder was in the correct stage. So at some point during the composer install or craft migrate/all --no-content step, the config was reset. Once project-config/apply is executed, the config must already have reset, otherwise it wouldn't complain.

No idea what's causing this, and why it only happens in production (at least in this case). We tested the exact same steps and they worked correctly in dev. Only difference is that the dev environment uses ddev, and that the mode is set to dev instead of production. And of course the site may get live traffic during the deployment (shout-out to #9855). Maybe some form of race condition?

The next time this happens, can you capture the resulting (“original”) project config that gets output, and compare it with the actual original project config? I’m curious if there are any changes that stick around from the deployed project config changes.

As fas as I could tell, the contents of the config/ folder were completely reset to the config version before the Craft 5 update.

[brandonkelly]

@MoritzLost Can you try creating a new testing environment modeled after the production environment, and reproducing from there? And assuming you’re able to reliably reproduce it, can you give us access to that testing environment?