-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #59 from pulibrary/43-hydration
43 hydration
- Loading branch information
Showing
30 changed files
with
1,618 additions
and
4 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,67 @@ | ||
When Producer spins up, initialize last_queried_marker from the ProcessorMarkers table, or set it to 1900. | ||
|
||
```mermaid | ||
sequenceDiagram title Figgy Producer | ||
Participant FiggyDatabase | ||
Participant Producer | ||
Participant Acknowledger | ||
Participant Consumer | ||
Participant Batcher as Batcher (batch size of 2) | ||
Participant HydrationCache | ||
Consumer->>Producer: Demand 3 records | ||
Producer->>FiggyDatabase: Query records since last_queried_marker (1900) | ||
FiggyDatabase->>Producer: Return 3 records | ||
Producer->>Producer: Set last_queried_marker to last updated_at | ||
Producer->>Producer: Add all 3 {record_id, updated_at} to pulled_records | ||
Producer->>Consumer: Deliver record[1,2,3] | ||
Consumer->>Consumer: Process record[1] | ||
Consumer->>Batcher: Deliver record[1] | ||
Consumer->>Consumer: Process record[3] | ||
Consumer->>Batcher: Deliver record[3] | ||
Batcher->>HydrationCache: Writes record[1,3] | ||
Batcher->>Acknowledger: Acknowledge record[1,3] | ||
Acknowledger->>Acknowledger: Do error handling | ||
Acknowledger->>Producer: Acknowledge record[1,3] | ||
Producer->>Producer: Update state: Append record[1,3] to acked_records | ||
Producer->>Producer: Update state: Sort acked_records by updated_at | ||
Producer->>Producer: Run Acknowledging Records algorithm | ||
Consumer->>Consumer: Process record[2] | ||
Consumer->>Batcher: Deliver record[2] | ||
Batcher->>Batcher: 1 minute passes | ||
Batcher->>HydrationCache: Writes record[2] | ||
Batcher->>Acknowledger: Acknowledge record[2] | ||
Acknowledger->>Acknowledger: Do error handling | ||
Acknowledger->>Producer: Acknowledge record[2] | ||
Producer->>Producer: Update state: Append record[2] to acked_records | ||
Producer->>Producer: Update state: Sort acked_records by updated_at | ||
Producer->>Producer: Run Acknowledging Records algorithm | ||
Consumer->>Producer: Demand 3 records | ||
Producer->>FiggyDatabase: Query records since last_queried_marker | ||
``` | ||
|
||
## Managing Producer State | ||
|
||
### Acknowledging Records | ||
|
||
When receiving acknowledgement for [1,3]: | ||
|
||
Start state: `{last_queried_marker: record[3].updated_at, pulled_records: [1,2,3], acked_records: [1,3]}` | ||
|
||
If the first element is the same in pulled_records and acked_records, then remove that element from both. Repeat until there's no match. Then write the timestamp from the last element that got removed from pulled_records. | ||
|
||
If the first marker in acked_records is less than the first marker in pulled_records, discard it. This means we must have acked that record already, and we don't need to ack it again. This could happen in certain producer crash scenarios. | ||
|
||
The processor will block during this acknowledgement, so you don't have to worry about race conditions here. | ||
|
||
End State: `{last_queried_marker: record[3].updated_at, pulled_records: [2,3], acked_records: [3]}` | ||
|
||
Write `1.updated_at` to `ProcessorMarkers` | ||
|
||
When receiving Acknowledgement for [2]: | ||
|
||
Start State: `{last_queried_marker: record[3].updated_at, pulled_records: [2,3], acked_records: [2,3]}` | ||
|
||
End State: `{last_queried_marker: record[3].updated_at, pulled_records: [], acked_records: []}` | ||
|
||
Write `3.updated_at` to `ProcessorMarkers` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
defmodule DpulCollections.FiggyRepo do | ||
use Ecto.Repo, | ||
otp_app: :dpul_collections, | ||
adapter: Ecto.Adapters.Postgres, | ||
read_only: true | ||
end |
Oops, something went wrong.