Skip to content

Files

This branch is 878 commits behind datahub-project/datahub:master.

metadata-jobs

MXE Processing Jobs

DataHub uses Kafka as the pub-sub message queue in the backend. There are 2 Kafka topics used by DataHub which are MetadataChangeEvent and MetadataAuditEvent.

  • MetadataChangeEvent: This message is emitted by any data platform or crawler in which there is a change in the metadata.
  • MetadataAuditEvent: This message is emitted by DataHub GMS to notify that metadata change is registered.

To be able to consume from these two topics, there are two Spring jobs DataHub uses: