Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: [8472] Migrate project point to GeoDjango and add a new serializer mixin to convert between gis and geojson #1765

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

goapunk
Copy link
Contributor

@goapunk goapunk commented Jan 28, 2025

This is a first attempt to move geographical locations from being stored as plain json/geojson to GeoDjango. This will allow us to filter for locations on a database level.

Tasks

  • PR name contains story or task reference
  • Documentation (docs and inline)
  • Tests (including n+1 and django_assert_num_queries where applicable)
  • Changelog

@goapunk goapunk force-pushed the jd-2025-01-custom-geometry-field branch 7 times, most recently from 8517541 to 3599699 Compare January 30, 2025 15:19
points as GeoDjango to the database for better filtering capabilites and
serializes them as geojson features.

- add PointInPolygon validator which validates that a given point is in
  the provided polygon.
@m4ra
Copy link
Contributor

m4ra commented Feb 10, 2025

@goapunk what's your concern at this point?

@goapunk
Copy link
Contributor Author

goapunk commented Feb 11, 2025

@goapunk what's your concern at this point?

@m4ra just that it's a rather big change. And we need to keep in mind this only migrates the points for projects. Points for ideas/proposals and other geometries (e.g. polygon) will stay as json fields.

Copy link
Contributor

@m4ra m4ra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this surgical operation :)
It needs a good docs, with payload example and an explanation on why we keep both paradigms, and maybe in the future all map based modules should migrate to django gis

if "properties" in point:
for property in properties:
if property in point["properties"]:
data[property] = point["properties"][property]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this if block (line 22 - 25) is hard to read, anyway to re-write it? or add some extra comments.

kwargs = super().get_form_kwargs()
if "data" in kwargs:
data = self.unpack_geojson(kwargs["data"])
kwargs["data"] = data
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here

self.polygon = polygon

def __call__(self, value):
""""""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what are these quotes for?

name="point",
),
migrations.AddField(
model_name="project",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think once data has migrated wth migrate_project_point_field and old point filed is removed you can just rename the field from geos_point back to point, but safer in a new migration. And no need for migrate_project_geos_point_field

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sadly the RenameField migration doesn't work. It just gives a cryptic error. I assume it's because the old field is a normal sqlite field and the new one is a spatialite gis field, but that's more of a guess.

+ ": "
+ str(geojson_point)
)
continue
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will this log warning be showing in sentry? otherwise i am afraid we will forget to check when deloying to prod/stage

street_name = models.CharField(null=True, blank=True, max_length=200)
house_number = models.CharField(null=True, blank=True, max_length=10)
zip_code = models.CharField(null=True, blank=True, max_length=20)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The properties of the existing geometry is strname etc, how do we convert between those and the different names you are using here?

Copy link
Contributor Author

@goapunk goapunk Feb 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point, I kind of ignored that for now since we don't actually use them anywhere. I've added these to not loose the information of the geojson points already in the database. But these won't be populated for new points at the moment. We'd either need a mapping or decide to get rid of them altogether

project.house_number = properties["hsnr"]
if "plz" in properties:
project.zip_code = properties["plz"]
project.save()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not clear where the geometry properties come from in your migration above. Needs docstrings.


serializer = FakePointSerializer()
data = serializer.to_representation(None)
assert data["point"] == geojson_point
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would move all fake classes and geojson points outside the tests, since they are hardcoded and re-used, and call them as fixtures from the tests.

@m4ra
Copy link
Contributor

m4ra commented Feb 11, 2025

@partizipation this PR, introducing django.gis for storing geopoints in the database

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants