Only add list capacity when 95% of active lists are fully assigned.

CHANGELOG.md

@@ -1,5 +1,10 @@
 # bedrock-vc-issuer ChangeLog
 
+## 29.0.2 - 2025-02-dd
+
+### Fixed
+- Only add list capacity when 95% of active lists are fully assigned.
+
 ## 29.0.1 - 2025-02-27
 
 ### Fixed

lib/ListManager.js

@@ -1,5 +1,5 @@
 /*!
- * Copyright (c) 2020-2024 Digital Bazaar, Inc. All rights reserved.
+ * Copyright (c) 2020-2025 Digital Bazaar, Inc. All rights reserved.
  */
 import * as bedrock from '@bedrock/core';
 import assert from 'assert-plus';
@@ -130,8 +130,12 @@ Note: See `IndexAllocationCache.js`.
 this._preferNewList(conflicts):
 
 1. If conflicts > 1 (50/50 chance of conflict is acceptable, but no more) and
-   all active BADs have 50% or more of their blocks assigned, return true,
-   otherwise return false.
+   all active BADs have 95% or more of their blocks assigned, return true,
+   otherwise return false. 95% is chosen to favor filling lists (over more
+   concurrency) as completely as possible before introducing new lists to
+   improve scaling for verifier requests in the aggregate. If all "active"
+   lists (see below) are just shy of 95% full then the conflict chance, which
+   is based on total remaining blocks, will be considerably smallers.
 
 this._tryAddCapacity(cache, target): Try to add another active BAD if necessary
   (from "inactive" set or a new one):
@@ -331,12 +335,25 @@ export class ListManager {
     "active" item (4). This is 4096 * 4 = 16384 total, so VC issuance
     concurrency without collision is ~sqrt(16384) = 128.
 
+    If list sizes are 2^26 (MAX_LIST_SIZE) = 67108864, then there are
+    2097152 blocks (67108864 / 32) to choose from per "active" item (4). This
+    is 2097152 * 4 = 8388608 total, so VC issuance concurrency without
+    collision is ~sqrt(8388608) = 2896.
+
     When a collision happens, another selection can still be made, so there is
     still more concurrency, but it takes more work. On average, every 128
-    attempts to find a block to assign indexes from will result in a collision
-    that requires additional work to select a different block, but 16384 total
-    blocks could actually be theoretrically serviced concurrently, enabling a
-    maximum of 16384 VCs to be issued concurrently.
+    (or 2896 for large lists) attempts to find a block to assign indexes from
+    will result in a collision that requires additional work to select a
+    different block, but 16384 total blocks (or 8388608 blocks for large lists)
+    could actually be theoretrically serviced concurrently, enabling a maximum
+    of 16384 VCs (or 8388608 VCs) to be issued concurrently.
+
+    Currently, no new active lists will be added until 95% of the existing
+    lists are consumed, which means that maximum concurrency w/o conflict
+    drops to ~sqrt(4096*.05) = 14 for a single small list, ~28 for 4 small
+    lists, ~323 for a single large list, and ~647 for 4 large lists. Clearly
+    using larger lists will ensure greater concurrency for larger scale use
+    cases.
 
     While setting the max "active" set size limit higher would allow more
     concurrency, it would also reduce the compactness of index assignment which
@@ -442,14 +459,14 @@ export class ListManager {
 
   _preferNewList() {
     // 1. If conflicts > 1 (50/50 chance of conflict is acceptable, but no more)
-    //   and all active BADs have 50% or more of their blocks assigned, return
+    //   and all active BADs have 95% or more of their blocks assigned, return
     //   true, otherwise return false.
     const {conflicts, activeCache} = this;
     if(conflicts < 2) {
       return false;
     }
     return activeCache.records.every(({blockAssignmentDoc: {content}}) =>
-      (content.assignedBlockCount / content.blockCount) >= 0.5);
+      (content.assignedBlockCount / content.blockCount) >= 0.95);
   }
 
   _canCreateNextStatusLists() {