You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
But ultimately we need to be able to handle massive data so we need to change the API to allow extract() to return an iterator, and then each source could return a db recordset, or it could open a csv and stream through it line by line etc
The text was updated successfully, but these errors were encountered:
The database source uses a recordset but then defeats the purpose by grinding through it and copying it all into an in memory array:
https://github.com/catalyst/moodle-tool_etl/blob/master/classes/source/source_database.php#L59-L66
But ultimately we need to be able to handle massive data so we need to change the API to allow extract() to return an iterator, and then each source could return a db recordset, or it could open a csv and stream through it line by line etc
The text was updated successfully, but these errors were encountered: