You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
state as visible by the source is just a dictionary (read and write via indexer)
each source iterator that requires state should operate on it's own namespace (Pipeline.state["namespace"] for explicit names or Pipeline.state for implicit names based ie. on name of calling module)
writing to state may only happen after iterator has stopped. that must be detected and extraction must be aborted
state is preserved atomically together with the results of the extraction
rudolfix
changed the title
[singer] implement state management
[pipeline] implement state management
Jun 15, 2022
rudolfix
changed the title
[pipeline] implement state management
[pipeline] implement state merge logic in case of parallel processing or state restoration from destination
Jan 20, 2023
implement proper state management in
Pipeline
Implementation
The text was updated successfully, but these errors were encountered: