v0.12.5

This release introduces an important change on how intervals worked in ingestr.

Important - Closed Intervals

This release introduces a change in how --interval-start and --interval-end works in ingestr. Previously, these flags were treated as [interval-start, interval-end), meaning that the values with interval-start would be included, the ones with interval-end would be excluded, which means the ranges would not be overlapping.

In more tangible terms, the previous version would create a query like this to get the data based on an updated_at field:

SELECT *
FROM mytable
WHERE updated_at >= :interval_start AND updated_at < :interval_end

In the new version, this is how the query looks like:

SELECT *
FROM mytable
WHERE updated_at >= :interval_start AND updated_at <= :interval_end

This means certain bits of data will be fetched unnecessarily, but ingestr will deduplicate the data anyway. This is especially helpful for date-based incremental keys where there's no high-precision timestamp available.

Full Changelog: v0.12.4...v0.12.5

v0.12.4

What's Changed

• New source: Google Analytics by @sanjushahgupta in #70
• New source: GitHub by @sanjushahgupta in #61
• Added support for a new --columns flag that can be used to pass hints for the destination table structure.
• Fixed a bug with incremental load handling for date columns.

Full Changelog: v0.12.3...v0.12.4

v0.12.3

feature: custom SQL queries

New source: Kafka

This release introduces a new Kafka source to ingestr!

Using ingestr, you can ingest data from any Kafka topic into your DWH, see docs here.

Gorgias as a new source

This release adds Gorgias as a new source to ingestr!

The current release supports 4 resources at the moment:

• customers
• tickets
• ticket messages
• satisfaction surveys

Adding new resources would be pretty easy, feel free to contribute either directly to the code or creating an issue with your needs.

v0.6.0: Shopify as a new source

This release adds support for Shopify as a new source! 🚀

Give the docs a look, anda share your feedback.

What's Changed

• updated snowflake account identifier usage by @trakmaker in #17
• add shopify as a source by @karakanb in #20

New Contributors

• @trakmaker made their first contribution in #17
• @karakanb made their first contribution in #20

Full Changelog: v0.5.1...v0.6.0

SAP Hana as a source ✨

This release adds support for SAP Hana as a source. You can use the Hana databases as a source using the following URI structure:

hana://username:password@host:port/dbname

Enjoy!

Notion support 🚀

Along with some minor upgrades and maintenance work, this release introduces the support for a new source: Notion!

This is the first release where we step outside the db-only territory and integrate with external platforms, please feel free to suggest any other platform that might be helpful to you!

MongoDB Source

This release adds support for MongoDB as a source for ingestr!

ingestr ingest \
  --source-uri 'mongodb://localhost:27017' \
  --source-table 'mydb.mycollection' \
  --dest-uri $POSTGRES_URI \
  --dest-table 'testschema.mongofake'

Fixes #7

Oracle & CSV Support

This release is a quick addition with an experimental Oracle source support, and a new (quite hackily implemented) CSV destination support, which means that you can copy any table from any database into a CSV file locally!