latest beta mariadb error #3651
Comments
adamaen87
added
bug
bw-unified-deploy
|
I'm getting the same issue. Here is the error in admin.log: |
|
I've also been running into this issue on an upgrade from 2023.12.0 to 2024.1.0. I just posted a comment in However, it looks like there are multiple problems occurring in the The first migration that is failing for me is running this query: ALTER TABLE `Grant` ADD CONSTRAINT `PK_Grant` PRIMARY KEY (`Id`);
Error: Duplicate entry '0' for key 'Grant.PRIMARY'This is because my Grant table is filled with rows and the previous migration runs this query: ALTER TABLE `Grant` ADD `Id` int NOT NULL DEFAULT 0;This sets all the values for the existing rows to 0, making it impossible to be a primary key as they have to be unique. Because of the failing migration, the admin process crashes and supervisor restarts it. However, this will not undo the migrations that were already executed. So now when it tries to re-execute them, the primary key is already gone ALTER TABLE `Grant` DROP PRIMARY KEY;
Error: Can't DROP 'PRIMARY'; check that column/key existsI'm still debugging some more, but this is what I have found so far. To me the issue appears to be that since there are already items in the |
|
@adamaen87 can you pull the latest from |
|
Hello, So, i just give a try and still have the same problem, but this time the container take much longer to start. Bitwarden logs: 12/01/2024 mariadb logs: 12/01/2024 Here is my compose file: version: "3.8" I have also provided my settings file. |
|
@adamaen87 I think the problem here is that your table is already been partially mutated. The migration executed the @withinfocus I think to fix the already mutated tables, they either have to be reverted to their pre-migration state ( |
|
The Entity Framework migration process shouldn't have applied a partial migration however and it'd roll back changes transactionally. Was something modified manually? |
|
I did not make manual modifications prior to the crashing admin panel other than updating the tag in my docker compose yaml from 2023.12.0 to 2024.1.0 and then doing: I'm also curious how exactly it would roll back as MySQL (and perhaps therefor also MariaDB) can't do a rollback on its own for DDL statements. https://dev.mysql.com/doc/refman/8.0/en/cannot-roll-back.html |
|
You're correct, and the ID application is what went awry here. Pomelo cannot apply auto-increment immediately. Dropping the |
|
Nothing manually modified either. |
|
@withinfocus I think so too. But what about the currently broken tables? CREATE TABLE `Grant` (
`Key` varchar(200) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL,
`Type` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL,
`SubjectId` varchar(200) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci DEFAULT NULL,
`SessionId` varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci DEFAULT NULL,
`ClientId` varchar(200) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL,
`Description` varchar(200) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci DEFAULT NULL,
`CreationDate` datetime(6) NOT NULL,
`ExpirationDate` datetime(6) DEFAULT NULL,
`ConsumedDate` datetime(6) DEFAULT NULL,
`Data` longtext CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL,
`Id` int NOT NULL DEFAULT '0'
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci |
|
We're gonna look into another migration edit that will inspect the state and partially patch as needed. |
|
FWIW I'm having the same issue. My production instance is no longer usable (users can't add new entries, can't login to bitwarden). Is there a workaround we can try? Or should we restore from backup? |
Hello, are you using docker? are you using the unfied-beta? |
Yes. I'm using docker / unified-beta. For now I restored from a backup (2024.1.1). It works fine and I'm planning to stay on this version until this issue is fixed. |
|
2024.1.2 includes #3668 that should resolve this issue for MySQL users. |
|
@withinfocus Are you sure this is included in latest |
Same here with latest beta, no change still getting exactly same errors. |
|
I am speaking about images that are at that pull request merge time or newer is all, so commit-based ones or |
|
Here is my experience. I noticed the issue when my docker container was at 2024.1.1. But I suspect the issue developed in one of the earlier builds (I update my docker images nightly in case of a 0-day fix). To resolve it, I restored the container data files (i.e. MariaDB data files, bitwarden data files, etc) to a backup from 1/3/2024. I did this while keeping the docker image at 2024.1.1. The restore fixed the problem. My suspicion is that one of the earlier builds left the database in a mangled state and could only be fixed by a db restore. @withinfocus, does your fix in 2024.1.2 correct a potentially-mangled database caused by bug in an upgrade script in an earlier version? Or does it correct the script so the issue doesn't happen for new upgrades? |
|
Given the bug in the library we use there should have been an erroneous |
I am experiencing the same issue. I pulled the |
|
I also don't think the latest dev docker image fixes a broken database. Steps:
Still unable to login. The following errors are observed
==> identity.log <== |
|
Based on that second log it looks like you're running newer code than what your migrations have caught up to. My belief is that you have that ALTER TABLE `Grant` DROP COLUMN `Id`;and see what happens. If your installation won't allow that, something a bit more advanced like: ALTER TABLE `Grant` DROP PRIMARY KEY, ADD PRIMARY KEY `PK_Grant` (`Key`); |
|
I dropped the Id column. Still not working. 2024-01-19 11:20:56.241 -06:00 [INF] Migrating database. ==> identity.log <== |
|
Did you run the second suggested command? |
|
You don't need anything but the user-specific MariaDB password you configured as that user is what performs the migrations and has the right to execute that SQL. |
|
@withinfocus awesome, it worked. I used |
|
Glad it worked. MySQL is a little crazy for us sometimes, thus the "beta". We hope the library issue that broke this will be resolved one day. |
|
@withinfocus will we have to do this for each additional release update to the image? |
|
Definitely not -- this was just an integrity fix for what a prior migration broke. That migration should be marked as complete and normal upgrades will continue to operate as expected. |
|
I can also confirm this works. I ran both commands against my MariaDB and I am back up and running. Thanks for the help! |
|
No luck for me. Stopped bitwarden container, ran both commands on mariadb, restarted bitwarden with latest beta and still the same errors. Reverted back to 2023.12.0-beta. |
@withinfocus Is the fix included in the latest 2024.1.2 self-host? (2024.1.2-beta)? I am still encountering the same error. I have not run any SQL manually because it is my understanding that a migration in your commits would fix the state of the database, and I felt like it would be beneficial to make sure the fix succeeded. I have only ever been on the Microsoft.EntityFrameworkCore.Database.Command[20102]
Failed executing DbCommand (6ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
CALL POMELO_BEFORE_DROP_PRIMARY_KEY(NULL, 'Grant');
ALTER TABLE `Grant` DROP PRIMARY KEY;
Unhandled exception. MySqlConnector.MySqlException (0x80004005): Can't DROP INDEX `PRIMARY`; check that it existsIf the commits are included, it appears they are not sufficient to fix the state of the database/migration and this issue should be re-opened. |
|
The fix is there, but difficult to predict for a small numbers of users with the "when" of the |
|
That is fine. I have decided to call it and execute the SQL above, which has solved it. If your point is that perhaps operating certain Hopefully those that stuck with the versioned releases are solved by your migrations - however, I feel like the percentage of users that fall into this category is quite low, if that is even the case. My feeling is that most people using the unified beta have probably updated like me and will run into this issue - requiring either running this SQL themselves or having an actual fixed migration implemented. I'm not sure that is the "small number[s] of users" you speak of, but you would know better than I. |
|
FWIW, I'm one of those that have encountered the issue which didn't get fixed by upgrading to the latest beta. I had to manually run the SQL commands. -From the beginning, I was running this: image: bitwarden/self-host:beta |
|
@withinfocus |
|
@adamaen87 Did you execute ALTER TABLE `Grant` ADD PRIMARY KEY `PK_Grant` (`Key`);or ALTER TABLE `Grant` DROP PRIMARY KEY, ADD PRIMARY KEY `PK_Grant` (`Key`);Make sure whatever you run doesn't result in an error. I would try adding the primary key again. |
|
Yes, sorry, i was stupid, just after my post i understood that i just need to add primary key, as dropping give me an error. |
|
Can confirm. Fixes it. |
How did you connect to the database? I get "mysql: command not found" when I try and connect to the database from the MariaDB container. |
Get into the container, then you can run |
|
That worked, I ran the commands and restarted the database container. The aborted connection errors are gone from the logs but the container is still unhealthy and I see the following via Portainer relating to Maria-DB container health: Last output/bin/sh: 1: mysqladmin: not found Is there something else I need to do as well? |
|
That I'm not sure about. MariaDB references that app here: https://mariadb.com/kb/en/mysqladmin/ but that sounds like an issue with how it's set up in portainer? |
|
@ruessda |
|
Thank you, changed DB health update and health check is now working and no errors in the log after running SQL commands. However I still cannot log in via web or mobile app. Fails on MFA. This stopped working prior to any troubleshooting and is what prompted me to start troubleshooting. I have a couple family members using this vault and they have the same issue. I can also create new users after changing disableUserRegistration and they cannot log in, fails on pw. Guessing this isn't related to the ID column issue. Anybody seen this issue? Edit 1 - Note the error I receive is "An error has occurred. An unexpected error has occurred." If I enter an incorrect password or MFA token I get "Username or password is incorrect." and similar error when I type the wrong MFA code so it does appear auth is working. Edit 2 - found resolution in BW community forum here: https://community.bitwarden.com/t/an-error-has-occurred-an-unexpected-error-has-occurred/62452/14 Edit 3 - going back image versions was only a temporary fix, issue is reoccurring (although I can still access mobile app). Still looking for a more permanent fix. |
Steps To Reproduce
I'am using bitwarden unified beta for month without problem. Since last beta there is a problem with mariadb connection.
Expected Result
everything fine :)
Actual Result
mariadb logs:
06/01/2024
17:47:24
2024-01-06 16:47:24 5 [Warning] Aborted connection 5 to db: 'bitwarden_vault' user: 'bitwarden' host: '172.29.0.4' (Got an error reading communication packets)
06/01/2024
17:47:24
2024-01-06 16:47:24 3 [Warning] Aborted connection 3 to db: 'unconnected' user: 'bitwarden' host: '172.29.0.4' (Got an error reading communication packets)
06/01/2024
17:47:47
and repeating every few seconds.
Same in bitwarden logs:
admin entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
exited: admin (terminated by SIGABRT (core dumped); not expected)
Screenshots or Videos
No response
Additional Context
I revert back to the 2023.12.0-beta image and no more error in both bitwarden and mariadb
Githash Version
b44cdc7-dirty
Environment Details
Debian 11 (OMV6 latest). Bitwarden and mariad in docker.
Database Image
mariadb:lts
Issue-Link
#2480
Issue Tracking Info
The text was updated successfully, but these errors were encountered: