Added some readmes and doc comments #377
Conversation
| /// <summary> | ||
| /// Update event that executes when a orchestration work item is completed. | ||
| /// </summary> |
There was a problem hiding this comment.
based on my assumptions, I presume a batch refers to multiple WIs.
| /// <summary> | |
| /// Update event that executes when a orchestration work item is completed. | |
| /// </summary> | |
| /// <summary> | |
| /// Update event that executes when a orchestration work items are completed. | |
| /// </summary> |
|
|
||
| ### [Client events](./ClientEvents/) | ||
|
|
||
| Client events are events that are sent from the partition and *executed on the client*. In most cases, they are responses to client-initiated requests. For example, a client sends a [StateRequestReceived](./PartitionEvents/External/FromClients/StateRequestReceived.cs) event to a partition to request the state of an orchestration, and the partition responds back to the client with a [StateResponseReceived](./ClientEvents/StateResponseReceived.cs) event, containing that result. |
There was a problem hiding this comment.
Something confuses me here: This says that client events are sent from the partition and executed on the client, but then you explain that StateRequestReceived is sent from the client and executed in the partition which seems contradictory.
Perhaps the way to make sense of this is that StateRequestReceived is not considered a client event? If so - I recommend we call that out for clarity's sake :-) .
There was a problem hiding this comment.
Yeah, I was worried that readers might be confused by this, and you confirmed it. :) Let me try to clarify this...
| ### [Event fragments](./Fragments/) | ||
|
|
||
| (Appears to be a special type of event that allows taking large events and breaking them up into smaller events - more research is needed to confirm) |
There was a problem hiding this comment.
If I understand this correctly, I believe you're right. If I'm not mistaken - sometimes an event is split into multiple blobs and as such it's events are received in fragments that need to be re-assembled.
|
|
||
| ## Event processing | ||
|
|
||
| Netherite operations that modify state are often composed of multiple events in a sequence. This is typically done because partition state needs to be loaded into the cache before it can be modified. For example, the process of creating a new orchestration or entity (instance) involves three steps: |
There was a problem hiding this comment.
I'm intrigued by the word "cache" here - is that particularly meaningful here? We can't we just say "loaded into memory"?
There was a problem hiding this comment.
This was the terminology that Sebastian used pretty consistently throughout the part 2 presentation, so I was trying to be consistent. That said, I too don't know if there's any distinction between "cached" and "in-memory" data and prefer the latter if there isn't a useful distinction.
|
|
||
| * `IntakeWorker`: Assigns commit log position to each event and decides whether to send the event to a `LogWorker` or a `StoreWorker`. | ||
| * `LogWorker`: Continuously persists update events to the commit log using **FasterLog**. | ||
| * `StoreWorker`: Handles read/update/query events on the partition state, and periodically/asynchronously checkpoints to **FasterKV**. |
There was a problem hiding this comment.
Something worth calling out as we get into the nitty gritty - FasterKV also has an inner log (called the hybrid log) and this is important to remember in particularly hairy FASTER investigations. So when we talk about "the FASTER log" it's not always immediately clear if we mean FasterLog of FasterKV's "hybrid log". Just FYI, in case we want to sneak that info somewhere in here.
There was a problem hiding this comment.
It's interesting to know that FasterKV is internally implemented using a log, but I'll defer to Sebastian on whether we need additional clarity on the terminology here. I wouldn't expect the average Netherite developer to know the internals of FASTER, but then again, every Netherite contributor I know does know the internals of FASTER so the data already contradicts my assumptions. :)
These are some notes I took while watching the two recordings of the Netherite 101 sessions (both of which I unfortunately missed because I was on vacation). I thought it would be good to have them in the repo so that this additional context can be visible to anyone in an obvious location.
I suppose we could have put some of this content in the Netherite docs content, but I feel like this information is more suited for code maintainers rather than end users, hence it would make more sense to collocate it with the code.
@sebastianburckhardt I'm not 100% confident in the accuracy of the content I wrote, so it would be great if you could correct any mistakes. Feel free to suggest additional content as well. You can also just edit this PR directly if you want. :)