Add postcode data ingestion job

[ClaudiaGC1339]

Description of change

Dataflow currently exports postcode and region data from DataWorkspace to the Datahub S3 bucket with less frequency however this PR schedules tasks to ingest postcode data nightly.

Checklist

• Has this branch been rebased on top of the current main branch?

  Explanation

  The branch should not be stale or have conflicts at the time reviews are requested.

• Is the CircleCI build passing?

General points

Other things to check

• Make sure fixtures/test_data.yaml is maintained when updating models
• Consider the admin site when making changes to models
• Use select-/prefetch-related field lists in views and search apps, and update them when fields are added
• Make sure the README is updated e.g. when adding new environment variables

See docs/CONTRIBUTING.md for more guidelines.

[codecov]

Codecov Report

Attention: Patch coverage is 88.09524% with 10 lines in your changes missing coverage. Please review.

Project coverage is 96.62%. Comparing base (bebfa68) to head (7511281).

Files with missing lines	Patch %	Lines
datahub/metadata/tasks.py	78.57%	6 Missing and 3 partials ⚠️
datahub/metadata/test/factories.py	93.75%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5966      +/-   ##
==========================================
- Coverage   96.65%   96.62%   -0.03%     
==========================================
  Files        1081     1084       +3     
  Lines       25394    25474      +80     
  Branches     1676     1681       +5     
==========================================
+ Hits        24544    24614      +70     
- Misses        694      700       +6     
- Partials      156      160       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[oliverjwroberts]

@ClaudiaGC1339 this is great work so far! Ingesting postcode data (or any other reference data for that matter) in this manner is a very sensible approach. That is, compared to migrations that only ingest a snapshot.

[oliverjwroberts]

It feels like having both the name and postcode field is somewhat unnecessary, and we should choose one or the other.

However, I'd be keen to get others' opinions on this?

[baarkerlounger]

What is a name in the context of a postcode? Do we have an example of what this field would contain?

[oliverjwroberts]

It might be worth adding a test to check the serializer returns a Postcode instance in the intended format? For example, if you were to send a GET request to the metadata endpoint, that it returns a list of postcode instances.

[oliverjwroberts]

Because we aren't updating postcode records in the first instance, this update_or_create call is probably redundant. Instead you could do something like:

def _process_record(self, record: dict) -> None):
    serializer = self.serializer_class(data=record)
    if serializer.is_valid():
        serializer.validated_data.pop('id')  # because setting an id from the incoming data may raise an error when we've told Django to auto generate a UUID
        instance = Postcode.objects.create(**serializer.validated_data)
        self.created_ids.append(str(instance.id))
    else:
        self.errors.append({
            'record': record,
            'errors': serializer.errors,
        })

This may also fix some of the test coverage.

[baarkerlounger]

Why aren't we updating postcode records? Given the point of this exercise is to improve accuracy it seems like we probably should unless there's a reason not? I don't think the data size is prohibitive?

[oliverjwroberts]

Will the incoming records have an ID? Maybe we want to ignore these if we are setting our own UUID?

[oliverjwroberts]

The value passed into the factory.Faker() call is the name of the faker provider. I don't think there is one called postcode_region. It might be more appropriate to select a random UK region here instead?

[oliverjwroberts]

The updated ids may not be applicable here, as per the above comment, if we aren't updating postcode instance.

[oliverjwroberts]

To help with code coverage, it may also be worth adding a similar test calling the ingestion function directly and asserting the log messages appear, and an instance is created. See test_ingestion_task_success in test_ingest_eyb_marketing for an example.