Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

A fresh node announces ONLINE status for a database before completing full sync #10386

Open
ikysil opened this issue Jan 17, 2025 · 5 comments
Open

A fresh node announces ONLINE status for a database before completing full sync #10386

ikysil opened this issue Jan 17, 2025 · 5 comments

Comments

@ikysil
Copy link
Contributor

ikysil commented Jan 17, 2025 •
edited
Loading

OrientDB Version: 3.2.36, reproduced with 3.2.26 and 3.2.37

Java Version: Amazon Corretto 21.0.5

OS: Linux

Actual behavior

The fresh node announces ONLINE status for a database BEFORE completing full sync.
The OrientDBRemote clients connecting to this node and querying the synchronizing database are blocked waiting until full sync finishes.

Expected behavior

The fresh node announces ONLINE status for a database AFTER completing full sync.
The OrientDBRemote clients connecting to this node and querying the synchronizing database should receive an error.
@ikysil ikysil changed the title A fresh node announces ONLINE status for database before completing full sync A fresh node announces ONLINE status for a database before completing full sync Jan 17, 2025
@ikysil
Copy link
Contributor Author

ikysil commented Jan 17, 2025

Logs from live node:

odb1-1  | 2025-01-17 09:34:12:824 INFO  [odb1] Updated node status to 'ONLINE' [OHazelcastClusterMetadataManager]
odb1-1  | 2025-01-17 09:34:12:831 WARNI Authenticated clients can execute any kind of code into the server by using the following allowed languages: [sql] [OServerSideScriptInterpreter]
odb1-1  | 2025-01-17 09:34:12:832 INFO  [odb1] Distributed servers status (*=current):
odb1-1  |
odb1-1  | +-------+------+---------+-----+---------+---------------+---------------+-----------------------+
odb1-1  | |Name   |Status|Databases|Conns|StartedOn|Binary         |HTTP           |UsedMemory             |
odb1-1  | +-------+------+---------+-----+---------+---------------+---------------+-----------------------+
odb1-1  | |odb1(*)|ONLINE|         |0    |09:34:09 |172.18.0.2:2424|172.18.0.2:2480|127.44MB/4.00GB (3.11%)|
odb1-1  | +-------+------+---------+-----+---------+---------------+---------------+-----------------------+
odb1-1  |  [OHazelcastPlugin]
odb1-1  | 2025-01-17 09:34:12:872 INFO  OrientDB Studio available at http://172.18.0.2:2480/studio/index.html [OServer]
odb1-1  | 2025-01-17 09:34:12:872 INFO  OrientDB Server is active v3.2.37 (build ${buildNumber}, branch UNKNOWN). [OServer]
odb1-1  | 2025-01-17 09:34:27:818 INFO  Enterprise storage installed correctly. [OEnterpriseLocalPaginatedStorage]
odb1-1  | 2025-01-17 09:34:27:832 INFO  Page size for WAL located in /orientdb/databases/udb is set to 4096 bytes. [CASDiskWriteAheadLog]
odb1-1  | 2025-01-17 09:34:27:842 WARNI Error detecting block size ignoring [ONative]
odb1-1  | 2025-01-17 09:34:27:842 INFO  DWL:udb: block size = 4096 bytes, maximum segment size = 39318 MB [DoubleWriteLogGL]
odb1-1  | 2025-01-17 09:34:27:927 INFO  Storage 'plocal:/orientdb/databases/udb' is created under OrientDB distribution : 3.2.37 (build ${buildNumber}, branch UNKNOWN) [OEnterpriseLocalPaginatedStorage]
odb1-1  | 2025-01-17 09:34:28:983 INFO  {db=udb} [odb1] Loaded configuration for database 'udb' from disk: /orientdb/config/default-distributed-db-config.json [ODistributedConfigurationManager]
odb1-1  | 2025-01-17 09:34:28:985 INFO  {db=udb} [odb1] Broadcasting new distributed configuration for database: udb (version=1)
odb1-1  |  [OHazelcastClusterMetadataManager]
odb1-1  | 2025-01-17 09:34:28:989 INFO  {db=udb} [odb1] Adding node 'odb1' in partition: [*] db=udb v=3 [OHazelcastPlugin]
odb1-1  | 2025-01-17 09:34:28:989 INFO  {db=udb} [odb1] Setting new distributed configuration for database: udb (version=3)
odb1-1  |  [ODistributedConfigurationManager]
odb1-1  | 2025-01-17 09:34:28:995 INFO  {db=udb} [odb1] Broadcasting new distributed configuration for database: udb (version=3)
odb1-1  |  [OHazelcastClusterMetadataManager]
odb1-1  | 2025-01-17 09:34:28:997 INFO  {db=udb} [odb1] Publishing ONLINE status for database odb1.udb... [ODistributedDatabaseImpl]
odb1-1  | 2025-01-17 09:34:28:999 INFO  [odb1] Received new status odb1.udb=ONLINE [OHazelcastClusterMetadataManager]
odb1-1  | 2025-01-17 09:34:32:832 INFO  [odb1] Distributed servers status (*=current):
odb1-1  |
odb1-1  | +-------+------+-------------------+-----+---------+---------------+---------------+----------------------+
odb1-1  | |Name   |Status|Databases          |Conns|StartedOn|Binary         |HTTP           |UsedMemory            |
odb1-1  | +-------+------+-------------------+-----+---------+---------------+---------------+----------------------+
odb1-1  | |odb1(*)|ONLINE|udb=ONLINE (MASTER)|4    |09:34:09 |172.18.0.2:2424|172.18.0.2:2480|63.97MB/4.00GB (1.56%)|
odb1-1  | +-------+------+-------------------+-----+---------+---------------+---------------+----------------------+
odb1-1  |  [OHazelcastPlugin]
odb1-1  | 2025-01-17 09:37:22:430 INFO  [172.18.0.2]:2434 [orientdb] [3.12.13] Initialized new cluster connection between /172.18.0.2:2434 and /172.18.0.3:56653 [TcpIpConnection]
odb1-1  | 2025-01-17 09:37:28:431 INFO  [172.18.0.2]:2434 [orientdb] [3.12.13]
odb1-1  |
odb1-1  | Members {size:2, ver:2} [
odb1-1  |       Member [172.18.0.2]:2434 - ca9255a4-a79d-4f67-9a6b-b20382dc82b1 this
odb1-1  |       Member [172.18.0.3]:2434 - 90d574a7-7967-4cd6-adec-93340aadb336
odb1-1  | ]
odb1-1  |  [ClusterService]
odb1-1  | 2025-01-17 09:37:28:432 INFO  [odb1] Added new node id=Member [172.18.0.3]:2434 - 90d574a7-7967-4cd6-adec-93340aadb336 name=ext:90d574a7-7967-4cd6-adec-93340aadb336 [OHazelcastClusterMetadataManager]
odb1-1  | 2025-01-17 09:37:28:691 INFO  [172.18.0.2]:2434 [orientdb] [3.12.13] Re-partitioning cluster data... Migration queue size: 271 [MigrationManager]
odb1-1  | 2025-01-17 09:37:29:492 INFO  [odb1]<-[ext:90d574a7-7967-4cd6-adec-93340aadb336] Received updated about registered nodes [OHazelcastClusterMetadataManager]
odb1-1  | 2025-01-17 09:37:29:571 INFO  {db=udb} [odb1]->[odb2] Deploying database udb... [OSyncDatabaseTask]
odb1-1  | 2025-01-17 09:37:29:573 INFO  {db=udb} [odb1]->[odb2] Creating backup of database 'udb' (compressionRate=7) in directory: /tmp/orientdb/backup_udb.zip... [OSyncDatabaseTask]
odb1-1  | 2025-01-17 09:37:29:576 INFO  {db=udb} [odb1]->[odb2] Compressing database 'udb' 2161 clusters [<REDACTED>]... [OBackgroundBackup]
odb1-1  | 2025-01-17 09:37:29:587 INFO  {db=udb} [odb1]->[odb2] - transferring chunk #1 offset=0 size=52... [OSyncDatabaseTask]
odb1-1  | 2025-01-17 09:37:29:588 INFO  {db=udb} [odb1]->[odb2] Deploy database task completed [OSyncDatabaseTask]
odb1-1  | 2025-01-17 09:37:29:604 INFO  [odb1]<-[odb2] Added node configuration id=Member [172.18.0.3]:2434 - 90d574a7-7967-4cd6-adec-93340aadb336 name=odb2, now 2 nodes are configured [OHazelcastPlugin]
odb1-1  | 2025-01-17 09:37:29:605 INFO  [odb1] Distributed servers status (*=current):
odb1-1  |
odb1-1  | +-------+------+-------------------+-----+---------+---------------+---------------+-----------------------+
odb1-1  | |Name   |Status|Databases          |Conns|StartedOn|Binary         |HTTP           |UsedMemory             |
odb1-1  | +-------+------+-------------------+-----+---------+---------------+---------------+-----------------------+
odb1-1  | |odb2   |      |                   |     |         |               |               |                       |
odb1-1  | |odb1(*)|ONLINE|udb=ONLINE (MASTER)|5    |09:34:09 |172.18.0.2:2424|172.18.0.2:2480|367.83MB/4.00GB (8.98%)|
odb1-1  | +-------+------+-------------------+-----+---------+---------------+---------------+-----------------------+
odb1-1  |  [OHazelcastPlugin]
odb1-1  | 2025-01-17 09:37:29:606 INFO  [odb1] Distributed servers status (*=current):
odb1-1  |
odb1-1  | +-------+--------+--------------------------+-----+---------+---------------+---------------+-----------------------+
odb1-1  | |Name   |Status  |Databases                 |Conns|StartedOn|Binary         |HTTP           |UsedMemory             |
odb1-1  | +-------+--------+--------------------------+-----+---------+---------------+---------------+-----------------------+
odb1-1  | |odb2   |STARTING|udb=SYNCHRONIZING (MASTER)|0    |09:37:21 |172.18.0.3:2424|172.18.0.3:2480|138.90MB/4.00GB (3.39%)|
odb1-1  | |odb1(*)|ONLINE  |udb=ONLINE (MASTER)       |5    |09:34:09 |172.18.0.2:2424|172.18.0.2:2480|367.83MB/4.00GB (8.98%)|
odb1-1  | +-------+--------+--------------------------+-----+---------+---------------+---------------+-----------------------+
odb1-1  |  [OHazelcastPlugin]
odb1-1  | 2025-01-17 09:37:29:607 INFO  [odb1] Distributed servers status (*=current):
odb1-1  |
odb1-1  | +-------+------+--------------------------+-----+---------+---------------+---------------+-----------------------+
odb1-1  | |Name   |Status|Databases                 |Conns|StartedOn|Binary         |HTTP           |UsedMemory             |
odb1-1  | +-------+------+--------------------------+-----+---------+---------------+---------------+-----------------------+
odb1-1  | |odb2   |ONLINE|udb=SYNCHRONIZING (MASTER)|0    |09:37:21 |172.18.0.3:2424|172.18.0.3:2480|141.20MB/4.00GB (3.45%)|
odb1-1  | |odb1(*)|ONLINE|udb=ONLINE (MASTER)       |5    |09:34:09 |172.18.0.2:2424|172.18.0.2:2480|367.83MB/4.00GB (8.98%)|
odb1-1  | +-------+------+--------------------------+-----+---------+---------------+---------------+-----------------------+
odb1-1  |  [OHazelcastPlugin]
odb1-1  | 2025-01-17 09:37:29:607 INFO  [odb1] Received new status odb2.udb=SYNCHRONIZING [OHazelcastClusterMetadataManager]
odb1-1  | 2025-01-17 09:37:29:622 INFO  {db=udb} [odb1]->[odb2] - transferring chunk #2 offset=0 size=8171... [OCopyDatabaseChunkTask]
odb1-1  | 2025-01-17 09:37:30:167 INFO  [172.18.0.2]:2434 [orientdb] [3.12.13] All migration tasks have been completed. (lastRepartitionTime=Fri Jan 17 09:37:28 UTC 2025, completedMigrations=271, totalCompletedMigrations=271, elapsedMigrationTime=473ms, totalElapsedMigrationTime=473ms) [MigrationThread]
odb1-1  | 2025-01-17 09:37:31:641 INFO  [odb1]<-[odb2] Received updated status odb2.udb=ONLINE [OHazelcastClusterMetadataManager]
odb1-1  |
odb1-1  | +-------+------+-------------------+-----+---------+---------------+---------------+-----------------------+
odb1-1  | |Name   |Status|Databases          |Conns|StartedOn|Binary         |HTTP           |UsedMemory             |
odb1-1  | +-------+------+-------------------+-----+---------+---------------+---------------+-----------------------+
odb1-1  | |odb2   |ONLINE|udb=ONLINE (MASTER)|0    |09:37:21 |172.18.0.3:2424|172.18.0.3:2480|141.20MB/4.00GB (3.45%)|
odb1-1  | |odb1(*)|ONLINE|udb=ONLINE (MASTER)|5    |09:34:09 |172.18.0.2:2424|172.18.0.2:2480|367.83MB/4.00GB (8.98%)|
odb1-1  | +-------+------+-------------------+-----+---------+---------------+---------------+-----------------------+
odb1-1  |  [OHazelcastPlugin]
odb1-1  | 2025-01-17 09:37:32:841 INFO  {db=udb} [odb1] Setting new distributed configuration for database: udb (version=5)
odb1-1  |  [ODistributedConfigurationManager]
odb1-1  | 2025-01-17 09:37:33:566 INFO  {db=udb} [odb1]->[odb2] - transferring chunk #3 offset=0 size=2039742... [OCopyDatabaseChunkTask]
odb1-1  | 2025-01-17 09:37:33:776 INFO  {db=udb} Sending Enterprise backup (udb) for node sync [OBackgroundBackup]
odb1-1  | 2025-01-17 09:37:33:777 INFO  {db=udb} [odb1]->[odb2] Backup of database 'udb' completed. lastOperationId=1.0... [OBackgroundBackup]
odb1-1  | 2025-01-17 09:38:02:657 INFO  {db=udb} [odb1]->[odb2] - transferring chunk #4 offset=0 size=116150... [OCopyDatabaseChunkTask]
odb1-1  | 2025-01-17 09:38:09:532 INFO  {db=udb} [odb1] Setting new distributed configuration for database: udb (version=1080)

@ikysil
Copy link
Contributor Author

ikysil commented Jan 17, 2025

Logs from fresh node:

odb2-1  | 2025-01-17 09:37:22:431 INFO  [172.18.0.3]:2434 [orientdb] [3.12.13] Initialized new cluster connection between /172.18.0.3:56653 and /172.18.0.2:2434 [TcpIpConnection]
odb2-1  | 2025-01-17 09:37:28:440 INFO  [172.18.0.3]:2434 [orientdb] [3.12.13]
odb2-1  |
odb2-1  | Members {size:2, ver:2} [
odb2-1  |       Member [172.18.0.2]:2434 - ca9255a4-a79d-4f67-9a6b-b20382dc82b1
odb2-1  |       Member [172.18.0.3]:2434 - 90d574a7-7967-4cd6-adec-93340aadb336 this
odb2-1  | ]
odb2-1  |  [ClusterService]
odb2-1  | 2025-01-17 09:37:29:452 INFO  [172.18.0.3]:2434 [orientdb] [3.12.13] [172.18.0.3]:2434 is STARTED [LifecycleService]
odb2-1  | 2025-01-17 09:37:29:453 INFO  Starting distributed server 'odb2' (hzID=90d574a7-7967-4cd6-adec-93340aadb336)... [OHazelcastClusterMetadataManager]
odb2-1  | 2025-01-17 09:37:29:460 INFO  Distributed cluster time=Fri Jan 17 09:37:29 UTC 2025 (delta from local node=10)... [OHazelcastClusterMetadataManager]
odb2-1  | 2025-01-17 09:37:29:489 INFO  [odb2] Registered local server with nodeId=1 [OHazelcastClusterMetadataManager]
odb2-1  | 2025-01-17 09:37:29:499 INFO  [odb2] Servers in cluster: [odb2, odb1] [OHazelcastClusterMetadataManager]
odb2-1  | 2025-01-17 09:37:29:522 INFO  [odb2] Updated node status to 'ONLINE' [OHazelcastClusterMetadataManager]
odb2-1  | 2025-01-17 09:37:29:528 INFO  [odb2] Downloaded configuration for database 'udb' from the cluster [ODistributedConfigurationManager]
odb2-1  | 2025-01-17 09:37:29:529 INFO  [odb2] Setting new distributed configuration for database: udb (version=3)
odb2-1  |  [ODistributedConfigurationManager]
odb2-1  | 2025-01-17 09:37:29:529 WARNI Authenticated clients can execute any kind of code into the server by using the following allowed languages: [sql] [OServerSideScriptInterpreter]
odb2-1  | 2025-01-17 09:37:29:532 INFO  [odb2] Distributed servers status (*=current):
odb2-1  |
odb2-1  | +-------+------+-------------------+-----+---------+---------------+---------------+-----------------------+
odb2-1  | |Name   |Status|Databases          |Conns|StartedOn|Binary         |HTTP           |UsedMemory             |
odb2-1  | +-------+------+-------------------+-----+---------+---------------+---------------+-----------------------+
odb2-1  | |odb2(*)|ONLINE|                   |0    |09:37:21 |172.18.0.3:2424|172.18.0.3:2480|141.20MB/4.00GB (3.45%)|
odb2-1  | |odb1   |ONLINE|udb=ONLINE (MASTER)|5    |09:34:09 |172.18.0.2:2424|172.18.0.2:2480|367.83MB/4.00GB (8.98%)|
odb2-1  | +-------+------+-------------------+-----+---------+---------------+---------------+-----------------------+
odb2-1  |  [OHazelcastPlugin]
odb2-1  | 2025-01-17 09:37:29:536 INFO  [odb2] Adding node 'odb2' in partition: [*] db=udb v=5 [OHazelcastPlugin]
odb2-1  | 2025-01-17 09:37:29:536 INFO  [odb2] Current node is a MASTER for database 'udb' [OHazelcastPlugin]
odb2-1  | 2025-01-17 09:37:29:539 INFO  [odb2] Setting new distributed configuration for database: udb (version=5)
odb2-1  |  [ODistributedConfigurationManager]
odb2-1  | 2025-01-17 09:37:29:544 INFO  [odb2] Requesting full sync for database 'udb'... [OHazelcastPlugin]
odb2-1  | 2025-01-17 09:37:29:544 INFO  [odb2]->[[odb1]] Requesting deploy of database 'udb' on local server... [OHazelcastPlugin]
odb2-1  | 2025-01-17 09:37:29:571 INFO  [odb2]<-[odb1] Received new status odb2.udb=SYNCHRONIZING [OHazelcastClusterMetadataManager]
odb2-1  | 2025-01-17 09:37:29:572 INFO  OrientDB Studio available at http://172.18.0.3:2480/studio/index.html [OServer]
odb2-1  | 2025-01-17 09:37:29:572 INFO  OrientDB Server is active v3.2.37 (build ${buildNumber}, branch UNKNOWN). [OServer]
odb2-1  | 2025-01-17 09:37:29:573 INFO  [odb2] Distributed servers status (*=current):
odb2-1  |
odb2-1  | +-------+------+--------------------------+-----+---------+---------------+---------------+-----------------------+
odb2-1  | |Name   |Status|Databases                 |Conns|StartedOn|Binary         |HTTP           |UsedMemory             |
odb2-1  | +-------+------+--------------------------+-----+---------+---------------+---------------+-----------------------+
odb2-1  | |odb2(*)|ONLINE|udb=SYNCHRONIZING (MASTER)|0    |09:37:21 |172.18.0.3:2424|172.18.0.3:2480|141.20MB/4.00GB (3.45%)|
odb2-1  | |odb1   |ONLINE|udb=ONLINE (MASTER)       |5    |09:34:09 |172.18.0.2:2424|172.18.0.2:2480|367.83MB/4.00GB (8.98%)|
odb2-1  | +-------+------+--------------------------+-----+---------+---------------+---------------+-----------------------+
odb2-1  |  [OHazelcastPlugin]
odb2-1  | 2025-01-17 09:37:29:593 WARNI [odb2] Moving existent database 'udb' in '/orientdb/databases/udb' to '/orientdb/backup/databases/udb' and get a fresh copy from a remote node... [OHazelcastPlugin]
odb2-1  | 2025-01-17 09:37:29:596 SEVER Atomic moves not supported during database backup, will try not atomic move [OHazelcastPlugin]
odb2-1  | 2025-01-17 09:37:29:600 SEVER File rename not supported during database backup, will try coping files [OHazelcastPlugin]
odb2-1  | 2025-01-17 09:37:29:614 INFO  [odb2]<-[odb1] Installing database 'udb' to: /orientdb/databases/udb... [OHazelcastPlugin]
odb2-1  | 2025-01-17 09:37:29:614 INFO  [odb2] - writing chunk #1 offset=0 size=52b [OSyncReceiver]
odb2-1  | 2025-01-17 09:37:29:617 INFO  Enterprise storage installed correctly. [OEnterpriseLocalPaginatedStorage]
odb2-1  | 2025-01-17 09:37:29:624 INFO  [odb2] - writing chunk #2 offset=0 size=7.98KB [OSyncReceiver]
odb2-1  | 2025-01-17 09:37:29:628 INFO  Page size for WAL located in /orientdb/databases/udb is set to 4096 bytes. [CASDiskWriteAheadLog]
odb2-1  | 2025-01-17 09:37:29:650 WARNI Error detecting block size ignoring [ONative]
build ${buildNumber}, branch UNKNOWN) [OEnterpriseLocalPaginatedStorage]
odb2-1  | 2025-01-17 09:37:31:639 INFO  {db=udb} [odb2] Publishing ONLINE status for database odb2.udb... [ODistributedDatabaseImpl]
odb2-1  | 2025-01-17 09:37:31:642 INFO  [odb2] Received updated status odb2.udb=ONLINE [OHazelcastClusterMetadataManager]
odb2-1  | 2025-01-17 09:37:31:643 INFO  [odb2] Distributed servers status (*=current):
odb2-1  |
odb2-1  | +-------+------+-------------------+-----+---------+---------------+---------------+-----------------------+
odb2-1  | |Name   |Status|Databases          |Conns|StartedOn|Binary         |HTTP           |UsedMemory             |
odb2-1  | +-------+------+-------------------+-----+---------+---------------+---------------+-----------------------+
odb2-1  | |odb2(*)|ONLINE|udb=ONLINE (MASTER)|0    |09:37:21 |172.18.0.3:2424|172.18.0.3:2480|141.20MB/4.00GB (3.45%)|
odb2-1  | |odb1   |ONLINE|udb=ONLINE (MASTER)|5    |09:34:09 |172.18.0.2:2424|172.18.0.2:2480|367.83MB/4.00GB (8.98%)|
odb2-1  | +-------+------+-------------------+-----+---------+---------------+---------------+-----------------------+
odb2-1  |  [OHazelcastPlugin]
odb2-1  | 2025-01-17 09:37:33:568 INFO  [odb2] - writing chunk #3 offset=0 size=1.95MB [OSyncReceiver]
odb2-1  | 2025-01-17 09:38:02:658 INFO  [odb2] - writing chunk #4 offset=0 size=113.43KB [OSyncReceiver]
odb2-1  | 2025-01-17 09:38:04:338 INFO  [odb2] Database copied correctly, size=2.06MB [OSyncReceiver]
odb2-1  | 2025-01-17 09:38:04:344 INFO  Page size for WAL located in /orientdb/databases/udb/walIncrementalBackupRestoreDirectory is set to 4096 bytes. [CASDiskWriteAheadLog]
odb2-1  | 2025-01-17 09:38:04:353 WARNI Record com.orientechnologies.orient.core.storage.impl.local.paginated.wal.common.EmptyWALRecord{lsn =OLogSequenceNumber{segment=2, position=22}} will be skipped during data restore [OEnterpriseLocalPaginatedStorage]
odb2-1  | 2025-01-17 09:38:04:353 INFO  1 operations were processed, current LSN is OLogSequenceNumber{segment=2, position=22} last LSN is OLogSequenceNumber{segment=3, position=22} [OEnterpriseLocalPaginatedStorage]
odb2-1  | 2025-01-17 09:38:04:353 WARNI Record com.orientechnologies.orient.core.storage.impl.local.paginated.wal.common.EmptyWALRecord{lsn =OLogSequenceNumber{segment=3, position=22}} will be skipped during data restore [OEnterpriseLocalPaginatedStorage]
odb2-1  | 2025-01-17 09:38:09:391 INFO  {db=udb} [odb2] Publishing ONLINE status for database odb2.udb... [ODistributedDatabaseImpl]
odb2-1  | 2025-01-17 09:38:09:449 INFO  {db=udb} [odb2] Publishing ONLINE status for database odb2.udb... [ODistributedDatabaseImpl]
odb2-1  | 2025-01-17 09:38:09:453 INFO  {db=udb} [odb2] Reassigning ownership of clusters for database udb... [OHazelcastPlugin]
odb2-1  | 2025-01-17 09:38:09:456 INFO  [172.18.0.3]:2434 [orientdb] [3.12.13] The HealthMonitor has detected a high load on the system. For more detailed information,
odb2-1  | enable the Diagnostics by adding the property -Dhazelcast.diagnostics.enabled=true [HealthMonitor]
odb2-1  | 2025-01-17 09:38:09:459 INFO  [172.18.0.3]:2434 [orientdb] [3.12.13] processors=22, physical.memory.total=15.4G, physical.memory.free=7.9G, swap.space.total=4.0G, swap.space.free=4.0G, heap.memory.used=653.0M, heap.memory.free=3.4G, heap.memory.total=4.0G, heap.memory.max=4.0G, heap.memory.used/total=15.92%, heap.memory.used/max=15.92%, minor.gc.count=7, minor.gc.time=66ms, major.gc.count=0, major.gc.time=0ms, unknown.gc.count=6, unknown.gc.time=7ms, load.process=40.00%, load.system=2.14%, load.systemAverage=1.06, thread.count=111, thread.peakCount=112, cluster.timeDiff=-1, event.q
.size=0, executor.q.async.size=0, executor.q.client.size=0, executor.q.client.query.size=0, executor.q.client.blocking.size=0, executor.q.query.size=0, executor.q.scheduled.size=0, executor.q.io.size=0, executor.q.system.size=0, executor.q.operations.size=0, executor.q.priorityOperation.size=0, ope
rations.completed.count=1431, executor.q.mapLoad.size=0, executor.q.mapLoadAllKeys.size=0, executor.q.cluster.size=0, executor.q.response.size=0, operations.running.count=0, operations.pending.invocations.percentage=0.00%, operations.pending.invocations.count=0, proxy.count=0, clientEndpoint.count=0, connection.active.count=1, client.connection.count=0, connection.count=1 [HealthMonitor]
odb2-1  | 2025-01-17 09:38:09:473 INFO  {db=udb} [odb2] Reassignment of clusters for database 'udb' completed (classes=543) [OHazelcastPlugin]
odb2-1  | 2025-01-17 09:38:09:475 INFO  {db=udb} [odb2] Setting new distributed configuration for database: udb (version=1080)
odb2-1  |  [ODistributedConfigurationManager]
odb2-1  | 2025-01-17 09:38:09:510 INFO  [odb2] Broadcasting new distributed configuration for database: udb (version=1080)
odb2-1  |  [OHazelcastClusterMetadataManager]

@cschockaert
Copy link

cschockaert commented Jan 17, 2025 •
edited
Loading

hello @ikysil its perhaps linked to my issue
#10383

that behaviour trigger a save config and lock the backup / full sync process

issue appear on 3.2.27

@ikysil
Copy link
Contributor Author

ikysil commented Jan 17, 2025

hi @cschockaert

Thank you, the same functional area seems to be affected.

tglman added a commit that referenced this issue Jan 17, 2025
@tglman
fix: skip the health check if the database is on sync, issues #10383, #…
b62cc2f 
…10386
tglman added a commit that referenced this issue Jan 17, 2025
@tglman
fix: skip the health check if the database is on sync, issues #10383, #…
6ef186e 
…10386
@ikysil
Copy link
Contributor Author

ikysil commented Jan 27, 2025

The issue is still present in 3.2.38-SNAPSHOT (3.2.38-20250117.191042-3).

See the logs attached.

fresh-node-orientdb-3.2.38-SNAPSHOT.txt
live-node-orientdb-3.2.38-SNAPSHOT.txt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ikysil @cschockaert