Player Save Data Corrupted on Restart

[leonardwohl]

Describe the bug

Player save data (at /Pal/Saved/SaveGames/0/643B916545D6886A839265911924FBEC/Players/) is occasionally corrupted on server restart.

To Reproduce

Steps to reproduce the behavior:

1. Have a server with multiple players
2. Reboot the server, either with the auto-restart functionality, or docker-compose down && docker-compose up -d
3. Have the players rejoin
4. See players' save data has been overwritten (timestamp on files are changed to time of server restart) and players see the new character creation screen.

Expected behavior

Restarting the server does not affect the player save data

Actual behavior

Restarting the server corrupts/overwrites the player save data

Screenshots

Restarted server at 10:29

Player data for those two characters is corrupted and has to be restored from backup.

OS information

• OS: Debian GNU/Linux 12 (bookworm) x86_64
• Palworld Version: v0.1.4.1
• Docker Image Version: latest

Hardware information

• CPU: Intel i3-10110U (4) @ 4.100GHz
• Memory: 16gb

docker-compose.yml contents

services:
  palworld:
    image: thijsvanloef/palworld-server-docker:latest
    restart: unless-stopped
    container_name: palworld-server
    stop_grace_period:
      30s # Set to however long you are willing to wait for the container to gracefully stop
    ports:
      - 8211:8211/udp
      - 27015:27015/udp
    volumes:
      - .:/palworld/
    env_file:
      - server.env

TZ=EST
PLAYERS=8
PORT=8211
MULTITHREADING=true
COMMUNITY=true
PUBLIC_IP=lenwohl.dev
PUBLIC_PORT=8211
SERVER_NAME=lenwohl.dev
SERVER_DESCRIPTION=
UPDATE_ON_BOOT=true
RCON_ENABLED=true
RCON_PORT=25575
QUERY_PORT=27015
BACKUP_ENABLED=true
DELETE_OLD_BACKUPS=true
OLD_BACKUP_DAYS=30
BACKUP_CRON_EXPRESSION=0 * * * *
AUTO_UPDATE_ENABLED=false
AUTO_UPDATE_CRON_EXPRESSION=0 * * * *
AUTO_UPDATE_WARN_MINUTES=30
AUTO_REBOOT_ENABLED=false
AUTO_REBOOT_EVEN_IF_PLAYERS_ONLINE=false
AUTO_REBOOT_WARN_MINUTES=5
AUTO_REBOOT_CRON_EXPRESSION=30 5 * * *
ENABLE_PLAYER_LOGGING=true
PLAYER_LOGGING_POLL_PERIOD=5

DIFFICULTY=None
DAYTIME_SPEEDRATE=1.000000
NIGHTTIME_SPEEDRATE=1.000000
EXP_RATE=1.000000
PAL_CAPTURE_RATE=1.000000
PAL_SPAWN_NUM_RATE=1.000000
PAL_DAMAGE_RATE_ATTACK=1.000000
PAL_DAMAGE_RATE_DEFENSE=1.000000
PLAYER_DAMAGE_RATE_ATTACK=1.000000
PLAYER_DAMAGE_RATE_DEFENSE=1.000000
PLAYER_STOMACH_DECREASE_RATE=1.000000
PLAYER_STAMINA_DECREASE_RATE=1.000000
PLAYER_AUTO_HP_REGEN_RATE=1.000000
PLAYER_AUTO_HP_REGEN_RATE_IN_SLEEP=1.000000
PAL_STOMACH_DECREASE_RATE=1.000000
PAL_STAMINA_DECREASE_RATE=1.000000
PAL_AUTO_HP_REGEN_RATE=1.000000
PAL_AUTO_HP_REGEN_RATE_IN_SLEEP=1.000000
BUILD_OBJECT_DAMAGE_RATE=1.000000
BUILD_OBJECT_DETERIORATION_DAMAGE_RATE=1.000000
COLLECTION_DROP_RATE=1.000000
COLLECTION_OBJECT_HP_RATE=1.000000
COLLECTION_OBJECT_RESPAWN_SPEED_RATE=1.000000
ENEMY_DROP_ITEM_RATE=1.000000
DEATH_PENALTY=All
ENABLE_PLAYER_TO_PLAYER_DAMAGE=False
ENABLE_FRIENDLY_FIRE=False
ENABLE_INVADER_ENEMY=False
ACTIVE_UNKO=False
ENABLE_AIM_ASSIST_PAD=True
ENABLE_AIM_ASSIST_KEYBOARD=True
DROP_ITEM_MAX_NUM=3000
DROP_ITEM_MAX_NUM_UNKO=100
BASE_CAMP_MAX_NUM=128
BASE_CAMP_WORKER_MAX_NUM=15
DROP_ITEM_ALIVE_MAX_HOURS=1.000000
AUTO_RESET_GUILD_NO_ONLINE_PLAYERS=False
AUTO_RESET_GUILD_TIME_NO_ONLINE_PLAYERS=72.00000
GUILD_PLAYER_MAX_NUM=20
PAL_EGG_DEFAULT_HATCHING_TIME=4.00000
WORK_SPEED_RATE=1.000000
IS_MULTIPLAY=False
IS_PVP=False
CAN_PICKUP_OTHER_GUILD_DEATH_PENALTY_DROP=False
ENABLE_NON_LOGIN_PENALTY=True
ENABLE_FAST_TRAVEL=True
IS_START_LOCATION_SELECT_BY_MAP=True
EXIST_PLAYER_AFTER_LOGOUT=False
ENABLE_DEFENSE_OTHER_GUILD_PLAYER=False
COOP_PLAYER_MAX_NUM=4
REGION=
USEAUTH=True
BAN_LIST_URL=https://api.palworldgame.com/api/banlist.txt

Container/Host Logs

****EXECUTING USERMOD****

usermod: no changes

****Checking for new update****

Current Version: 6370735655629434989

The Server is up to date!

****GENERATING CONFIG****

Using Env vars to create PalWorldSettings.ini

****Compiling PalWorldSettings.ini****

Compiling PalWorldSettings.ini done!

****GENERATING CRONTAB****

BACKUP_ENABLED=true

Adding cronjob for auto backups

Cronjobs started

****Starting Server****

./PalServer.sh -port=8211 -queryport=27015 EpicApp=PalServer -useperfthreads -NoAsyncLoadingThread -UseMultithreadForDS

time="2024-02-26T14:49:29-05:00" level=info msg="read crontab: /home/steam/server/crontab"

setrlimit() failed with error 22 (Invalid argument)

- Max per-process value allowed is 0 (we wanted infinity).

[S_API] SteamAPI_Init(): Loaded local 'steamclient.so' OK.

Shutdown handler: initalize.

Increasing per-process limit of core file size to infinity.

CAppInfoCacheReadFromDiskThread took 0 milliseconds to initialize

Setting breakpad minidump AppID = 2394010

[S_API FAIL] Tried to access Steam interface SteamUser021 before SteamAPI_Init succeeded.

[S_API FAIL] Tried to access Steam interface SteamFriends017 before SteamAPI_Init succeeded.

[S_API FAIL] Tried to access Steam interface STEAMAPPS_INTERFACE_VERSION008 before SteamAPI_Init succeeded.

[S_API FAIL] Tried to access Steam interface SteamNetworkingUtils004 before SteamAPI_Init succeeded.

Additional context

This happens very frequently, but not every time. It happens to some users more than others, but I can't figure out a pattern. Thanks for your help, I appreciate your maintenance of this fun project.

[MattDavies]

@leonardwohl I kept seeing this as well after restarting the container, not for all players. Eventually took a diff on the unpacked player save files (via palworld-save-tools) comparing to the working backups. For my case, the field being corrupted was save_game_class_name which was set to "None.PalWorldPlayerSaveGame" for the faulty saves and "/Script/Pal.PalWorldPlayerSaveGame" for the working saves. After patching that field, the players were able to login again. I've since found and incorporated this into my docker-compose.yml which fixes the issue on the files directly without converting to/from JSON: https://github.com/YDKK/palworld-save-repair

[leonardwohl]

You're the man!! It works!

[suiyu4073]

@leonardwohl I kept seeing this as well after restarting the container, not for all players. Eventually took a diff on the unpacked player save files (via palworld-save-tools) comparing to the working backups. For my case, the field being corrupted was save_game_class_name which was set to "None.PalWorldPlayerSaveGame" for the faulty saves and "/Script/Pal.PalWorldPlayerSaveGame" for the working saves. After patching that field, the players were able to login again. I've since found and incorporated this into my docker-compose.yml which fixes the issue on the files directly without converting to/from JSON: https://github.com/YDKK/palworld-save-repair

I tried your method and added this fix to the. yml file, but it didn't seem to work. I'm confused about why

[leonardwohl]

@suiyu4073 logs for the save_repair container?

[suiyu4073]

@suiyu4073save_repair 容器的日志？

@suiyu4073 logs for the save_repair container?

logs :

---
PalWorldSaveRepair v0.1.1 2024-02-18 dd5f298
https://github.com/YDKK/palworld-save-repair
---
failed to read dir: ./palworld/Pal/Saved/SaveGames/0/F1D09C3D4CE54F4CAA905CEAC4DBBF4A/Players

docker-compose.yml contents

services:
   palworld:
      image: thijsvanloef/palworld-server-docker:latest
      restart: unless-stopped
      container_name: palworld-server
      stop_grace_period: 30s # Set to however long you are willing to wait for the container to gracefully stop
      ports:
        - 8211:8211/udp
        - 27015:27015/udp
      environment:
         PUID: 1000
         PGID: 1000
         PORT: 8211 # Optional but recommended
         PLAYERS: 16 # Optional but recommended
         MULTITHREADING: true
         RCON_ENABLED: true
         RCON_PORT: 25575
         TZ: "UTC"
         ADMIN_PASSWORD: "adminPasswordHere"
         COMMUNITY: false  # Enable this if you want your server to show up in the community servers tab, USE WITH SERVER_PASSWORD!
         SERVER_NAME: "palworld-server-docker by Thijs van Loef"
         SERVER_DESCRIPTION: "palworld-server-docker by Thijs van Loef"
      volumes:
         - ./palworld:/palworld/
   save-repair:
      image: ydkk/palworld-save-repair:latest
      volumes:
         - ./palworld:/palworld/
      environment:
         - PLAYERS_SAVE_PATH=./palworld/Pal/Saved/SaveGames/0/F1D09C3D4CE54F4CAA905CEAC4DBBF4A/Players
      entrypoint: ""
      # check and repair save data every 1 hour
      command: ["/bin/sh", "-c", "while true; do sleep 3600; /app/pal_world_save_repair; done"]
      restart: unless-stopped

I can't read this folder whether I set the - PLAYERSSAVE-PATH=in the form of "./" or "/" here.

My absolute path is set to:

~/palworld_server/palworld/Pal/Saved/SaveGames/0/F1D09C3D4CE54F4CAA905CEAC4DBBF4A/Players

I don't know how to solve this problem, is it related to file read permissions?

[leonardwohl]

@suiyu4073 You're mounting ./palworld to /palworld so it should be PLAYERS_SAVE_PATH=/palworld/Pal/Saved/SaveGames/0/F1D09C3D4CE54F4CAA905CEAC4DBBF4A/Players. Make sure both containers are running with the same permissions, maybe try setting the same PUID and PGID in each?

[clyde0813]

I also faced the same issue, so whenever errors occurred, I restored the backup. It was cumbersome because I need to shut down the server using commands in the game and restart or update docker. But thankfully now, having a repair tool is a relief.

[mspieler1]

Hello,

I have the exact same issue.
I saved my Character save files at different points and compared them. It seems that the server stop (and possibly the /save command) have a problem which overwrites the value of "save_game_class_name" from "/Script/Pal.PalWorldPlayerSaveGame" to "None.PalWorldPlayerSaveGame".

Unfortunately I have a problem with the suggested workaround with this "save_file_repair" as every time I execute it and the files get repaired my character has no equipment, I can't interact with anything and can't even open the menu.

Edit: The shutdown Server does a "rcon save":

save_server() {
    local return_val=0
    if ! RCON save; then
        return_val=1
    fi
    return "$return_val"
}

# Saves then shutdowns the server
# Returns 0 if it is shutdown
# Returns 1 if it is not able to be shutdown
shutdown_server() {
    local return_val=0
    # Do not shutdown if not able to save
    if save_server; then
        if ! RCON "DoExit"; then
            return_val=1
        fi
    else
        return_val=1
    fi
    return "$return_val"
}

I would suspect an error there, would be good to know what the "rcon save" actually does.
According to the documentation it does the following:
"Save the world data to disk. Useful to ensure your Pal, player, and other data is saved before stopping the server or performing a risky gameplay option."
Not sure what it does exactly, but if it is only the player saves, this could be done manually maybe?

[MattDavies]

As of 1.5 I have the same issue with equipment, I suspect the repair tool might not be compatible with the latest game version. Planning to investigate over the weekend.

[suiyu4073]

@suiyu4073 You're mounting ./palworld to /palworld so it should be PLAYERS_SAVE_PATH=/palworld/Pal/Saved/SaveGames/0/F1D09C3D4CE54F4CAA905CEAC4DBBF4A/Players. Make sure both containers are running with the same permissions, maybe try setting the same PUID and PGID in each?

Thank you for your help. I have resolved the issue and do not need to set the same PID. You just need to pull it again and restart it

[ericxw]

Still running into this issue since maybe a week or so ago. I've had to disable daily restart. Anytime I have to restart, I need to take a backup manually, restart, then restore. I think shutdown calls backup. It's almost as if backup behavior has changed somehow that immediate shutdown after successful call to backup results in this.

[porochickenrye]

Hey @MattDavies, Thanks for investigating and providing a solution. I'm definitely still seeing save_game_class_name set to None.xxx from time to time. I can certainly repair with cheahjs' save tools, but am wondering if your save repair is still working automatically in docker in v1.5.1. Thoughts?

[thijsvanloef]

Hopefully new cases will be solved by this PR: #510

[porochickenrye]

Hey @thijsvanloef, I saw that just committed the change. Does the docker image get updated automatically at every start/reboot?

[thijsvanloef]

Hey @thijsvanloef, I saw that just committed the change. Does the docker image get updated automatically at every start/reboot?

https://palworld-server-docker.loef.dev/guides/update-the-container 😄

[porochickenrye]

Hey @thijsvanloef, I saw that just committed the change. Does the docker image get updated automatically at every start/reboot?

https://palworld-server-docker.loef.dev/guides/update-the-container 😄

Thanks. I did docker-compose down and up. How do I check if the container is running latest version of the help script (i.e. Shutdown instead of DoExit)?

By the way, in the doc link you sent, it should be docker-compose, not docker compose.

[thijsvanloef]

@porochickenrye If a pull has taken place once you start the docker-compose.yml you should probably be fine. You can test it by shutting down the server while you are in-game and seeing if a shutdown message is sent.

As for the docs, it should be docker compose, docker-compose is Compose V1 and deprecated since jul 2023 https://docs.docker.com/compose/migrate/

[porochickenrye]

Thanks @thijsvanloef on the docker compose migration note.

I have tested restarting the server many times but never seen an in-game shutdown message. I do get discord messages since I have those enabled.

Sending Discord json: {"embeds":[{"title":"Shutdown","description":"Server is shutting down...","color":15258703}]} 
Complete Save
Shutdown...
cli: execute: rcon: read packet size: EOF 
Player 1 has left 
cli: execute: auth: rcon: dial tcp 127.0.0.1:25575: connect: connection refused 
Shutdown handler: cleanup. 
Sending Discord json: {"embeds":[{"title":"Stop","description":"Server has been stopped!","color":14614528}]}

The version is see in the logs is 2815085007637542021 but I'm sure if that's the game version or the docker image version.

$ docker logs -f palworld-server
****EXECUTING USERMOD****
usermod: no changes
****Checking for new update****
Current Version: 2815085007637542021
The Server is up to date!
****GENERATING CONFIG****
Using Env vars to create PalWorldSettings.ini
****Compiling PalWorldSettings.ini****
Compiling PalWorldSettings.ini done!
****GENERATING CRONTAB****
BACKUP_ENABLED=true
Adding cronjob for auto backups
Cronjobs started
****Starting Server****

Update: I'm sure I'm running the latest image as I tried deploying a new contain from scratch. However, I still saw a corrupted save yesterday when I spun down use docker compose. Let me what logs I can provide.

[thijsvanloef]

@porochickenrye Do you still experience corrupted save files at restart with 0.35.1?