[Fixes #12828] New remote datasets are not registered inside proxy al…

[mattiagiupponi]

…lowed hosts when GeoNode runs asynchoronously

Checklist

Reviewing is a process done by project maintainers, mostly on a volunteer basis. We try to keep the overhead as small as possible and appreciate if you help us to do so by completing the following items. Feel free to ask in a comment if you have troubles with any of them.

For all pull requests:

• Confirm you have read the contribution guidelines
• You have sent a Contribution Licence Agreement (CLA) as necessary (not required for small changes, e.g., fixing typos in the documentation)
• Make sure the first PR targets the master branch, eventual backports will be managed later. This can be ignored if the PR is fixing an issue that only happens in a specific branch, but not in newer ones.

The following are required only for core and extension modules (they are welcomed, but not required, for contrib modules):

• There is a ticket in https://github.com/GeoNode/geonode/issues describing the issue/improvement/feature (a notable exemption is, changes not visible to end-users)
• The issue connected to the PR must have Labels and Milestone assigned
• PR for bug fixes and small new features are presented as a single commit
• Commit message must be in the form "[Fixes #<issue_number>] Title of the Issue"
• PR title must be in the form "[Fixes #<issue_number>] Title of the PR"
• New unit tests have been added covering the changes, unless there is an explanation on why the tests are not necessary/implemented
• This PR passes all existing unit tests (test results will be reported by travis-ci after opening this PR)
• This PR passes the QA checks: black geonode && flake8 geonode
• Commits changing the settings, UI, existing user workflows, or adding new functionality, need to include documentation updates
• Commits adding new texts do use gettext and have updated .po / .mo files (without location infos)

Submitting the PR does not require you to check all items, but by the time it gets merged, they should be either satisfied or inapplicable.

[codecov]

Codecov Report

Attention: Patch coverage is 83.33333% with 2 lines in your changes missing coverage. Please review.

Project coverage is 67.97%. Comparing base (0258939) to head (5a8237a).
Report is 2 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #12829       +/-   ##
===========================================
+ Coverage   55.87%   67.97%   +12.10%     
===========================================
  Files         977      977               
  Lines       59035    59046       +11     
  Branches     6897     6899        +2     
===========================================
+ Hits        32984    40138     +7154     
+ Misses      24669    17264     -7405     
- Partials     1382     1644      +262     

[giohappy]

We should avoid using globals() like this.
And I don't understand why changes do not modify the proxy_urls_registry content, since it's a singleton instance at the module level.
Let's investigate this better @mattiagiupponi

[giohappy]

@mattiagiupponi we cannot rely on reading from memached (or, worst, the filesystem) on each request to the proxy. In the case of 3dtiles, for example, it would be a bottleneck that worsen the proxy performances even more.

To avoid implementing a signaling system between celery and django, we can take an optimistic approach: we add the URL to the proxy before the celery task is created by the importer, when we're still inside the django process.
In case the remote layer creation fails, for whatever reason, we will have a registered host that doesn't map to a layer, but I don't think it's a big problem.

We can have a mechanism that reloads the registry from the DB once in a while, to realign the registry to the DB.
For example, the time of the last registry initialization can be stored in the registry. On each request we can check the time and if the registry is stale (1 day?) we reinitialize the registry.

[mattiagiupponi]

backport label removed, in case is required the backport must be done manually