v0.12.6

This release expands the closed intervals that were released for SQL sources in v0.12.5 to all the other platforms we support. In addition, it includes an improved Google Analytics source that can load past intervals as well.

What's Changed

• Made the access_token optional by @sanjushahgupta in #82
• Patch/closed intervals by @karakanb in #83

Full Changelog: v0.12.5...v0.12.6

v0.12.5

This release introduces an important change on how intervals worked in ingestr.

Important - Closed Intervals

This release introduces a change in how --interval-start and --interval-end works in ingestr. Previously, these flags were treated as [interval-start, interval-end), meaning that the values with interval-start would be included, the ones with interval-end would be excluded, which means the ranges would not be overlapping.

In more tangible terms, the previous version would create a query like this to get the data based on an updated_at field:

SELECT *
FROM mytable
WHERE updated_at >= :interval_start AND updated_at < :interval_end

In the new version, this is how the query looks like:

SELECT *
FROM mytable
WHERE updated_at >= :interval_start AND updated_at <= :interval_end

This means certain bits of data will be fetched unnecessarily, but ingestr will deduplicate the data anyway. This is especially helpful for date-based incremental keys where there's no high-precision timestamp available.

Full Changelog: v0.12.4...v0.12.5

v0.12.4

What's Changed

• New source: Google Analytics by @sanjushahgupta in #70
• New source: GitHub by @sanjushahgupta in #61
• Added support for a new --columns flag that can be used to pass hints for the destination table structure.
• Fixed a bug with incremental load handling for date columns.

Full Changelog: v0.12.3...v0.12.4

v0.12.3

feature: custom SQL queries

New source: Kafka

This release introduces a new Kafka source to ingestr!

Using ingestr, you can ingest data from any Kafka topic into your DWH, see docs here.

Gorgias as a new source

This release adds Gorgias as a new source to ingestr!

The current release supports 4 resources at the moment:

• customers
• tickets
• ticket messages
• satisfaction surveys

Adding new resources would be pretty easy, feel free to contribute either directly to the code or creating an issue with your needs.

v0.6.0: Shopify as a new source

This release adds support for Shopify as a new source! 🚀

Give the docs a look, anda share your feedback.

What's Changed

• updated snowflake account identifier usage by @trakmaker in #17
• add shopify as a source by @karakanb in #20

New Contributors

• @trakmaker made their first contribution in #17
• @karakanb made their first contribution in #20

Full Changelog: v0.5.1...v0.6.0

SAP Hana as a source ✨

This release adds support for SAP Hana as a source. You can use the Hana databases as a source using the following URI structure:

hana://username:password@host:port/dbname

Enjoy!

Notion support 🚀

Along with some minor upgrades and maintenance work, this release introduces the support for a new source: Notion!

This is the first release where we step outside the db-only territory and integrate with external platforms, please feel free to suggest any other platform that might be helpful to you!

MongoDB Source

This release adds support for MongoDB as a source for ingestr!

ingestr ingest \
  --source-uri 'mongodb://localhost:27017' \
  --source-table 'mydb.mycollection' \
  --dest-uri $POSTGRES_URI \
  --dest-table 'testschema.mongofake'

Fixes #7