Merge pull request #1 from threedi/reinout-simpler-demo

Updating the demo info

README.md

@@ -6,107 +6,79 @@ Within the [EU TEMA project](https://tema-project.eu/), [Nelen & Schuurmans](htt
 
 Disaster management also means collaboration, of course. For that purpose, TEMA has provided infrastructure and technology suggestions as a framework for collaboration. The implementation plan in this document shows how we want to use the infrastructure to be able to show a nice realistic demo in cooperation with partners.
 
-I hope this document also helps other partners to get started. For that purpose, it is my intention to keep most of the code open source (and thus visible). Some project-internal IP adresses and URLs will probably be hidden away in a private repo of course. You can email [[email protected]](mailto:[email protected]) if you need access.
+I hope this document also helps other partners to get started. For that purpose, it is my intention to keep most of the code open source (and thus visible), though there's a https://github.com/HE-TEMA github team now, so some of the repositories will be visible there.
 
 
 ## Provided infrastructure
 
 - A **context broker** implemented in [NGSI-LD](https://en.wikipedia.org/wiki/NGSI-LD). Basically a semantic web storage. I've written a [summary on my blog](https://reinout.vanrees.org/weblog/2024/04/16/tema-workshop-json-ld.html) of a TEMA workshop on that topic that should give a pretty good introduction. The context broker is the central information exchange mechanism for the whole project. If we need weather forecasts, we'll ask the CB to send us new ones added by other project partners. If we have flood calculation results, we'll put their info into the CB for others to query.
 - **K3S**, a simple but powerful kubernetes cluster. According to what I gathered, you can connect your own kubernetes nodes but you can also use the provided infrastructure. You can use it to run servers or scripts, provided you "dockerize" them and know how to deploy those dockers to the cluster.
 
-Both context broker ("CB") and the K3S cluster are hidden away behind a firewall, btw.
+The K3S cluster is (rightfully so) hidden away behind a firewall, so you need VPN
+access. The context broker ("CB") is publicly accessible (I'm not mentioning the URL here as you have write access as anonoymous user :-) ).
 
+## Original setup idea from april
 
-## Setup
+Originally, I had a more elaborate setup in mind, based on the need to shuffle data to and from our regular data center. It turns out that everything can be much simpler, luckily.
 
-The sketch below shows the suggested setup, followed by the textual explanation of the various parts. Basically we use the context broker in combination with a custom vocabulary + three **small and simple** dockers inside the K3S cluster + our own S3 storage for shuttling info back and forth.
+See the [previous version](original-setup-from-april2024.md).
 
-![Sketch of the suggested setup](suggested-setup.png)
 
+## New simpler setup
 
-## Context broker: custom vocabulary
+The sketch below shows the suggested setup, followed by the textual explanation of the various parts. Basically we use the context broker in combination with two **small and simple** dockers inside the K3S cluster + the minio s3 storage.
 
-In the context broker, we'll need to store information. Information that needs terms and definitions because of its "linked data" nature. So if we want to talk about a `RegionAtRisk`, we'll need to define.
+![Sketch of the suggested setup](suggested-setup2.jpg)
 
-If there's already a suitable vocabulary, we can use it. But at first we're probably better off making our own "JSON-LD" vocabulary for our specific use case. That way we can store `https://our.vocabulary.org/v1/meta/RegionAtRisk=Ahrtal` in the context broker and add a subscription to that.
 
-We'll start such a bare-bones vocabulary and host it somewhere public. If handy, others can extend it or start their own.
+## Context broker: custom types
 
-Some initial ideas (all with proper urls instead of just words, of course):
+In the context broker, we'll need to store information. Information that needs terms and definitions because of its "linked data" nature. Initially, I thought a proper defined custom "JSON-LD" vocabulary was necessary, but it turns out the context broker just accepts entities with a new type, so at least initially we don't need one.
 
-    weather-occurrence
-        id=abcd
-        region = ahr
-        timestamp = 2021-07-xx
-        rainfall-data = s3://amaazon.com/xxx/something.tiff
+The three entity types that I expect to need or to write myself (just as a starting point for further discussion):
 
-    flood-calculation
-        weather-occurrence=http://tema.it/weather-occurrence/abcd
-        result=s3://amazon.com/s3/xxx/something.zip
 
-    disturbance
-        3di-scenario=https://www.3di.live/scenario/xyz
-        height=235
-        x1=...
-        y1=...
-        x2=...
-        y2=...
+    FloodRiskEvent
+        id=some:urn:1234
+        region = ahr  (hardcoded at the moment)
+        rainfall = ....  (in mm/h)
+        rainfall_duration = ...  (in h)
+        (and probably some timestamp which should be build in)
 
+    ElevationMap
+        id=some:urn:5678
+        flood_risk_event_id = some:urn:1234
+        minio_url = s3://....  (link to a geotiff)
 
-## K3S part 1a: *context broker* target 'flask' docker
+    FloodCalculationResult
+        id=some:urn:8901
+        flood_risk_event_id = some:urn:1234
+        water_depth_minio_url = s3://..... (link to a geotiff)
+        other_results_minio_url = s3://..... (link to a geotiff)
 
-You can subscribe at the context broker for certain specific items. For instance a new `RegionAtRisk`. You have to provide a URL that the context broker can send its notification to.
+- `FloodRiskEvent`: I've written a small form to fire off this event, but that might be another partner's job.
+- `ElevationMap`: DLR probably has to upload this after getting notified about a FloodRiskEvent. I'm planning to upload a coarse geotiff myself as a starting point just to get our use case to run initially. Afterwards a new ElevationMap can be uploaded and the calculation started with the new and improved data.
+- We'll react to the elevation map by starting a 3Di simulation and producing a `FloodCalculationResult` with some uploaded data in minio. Other partners can then use this for their calculations or visualisations.
 
-We'll make a docker that contains a small [flask (python)](https://pypi.org/project/Flask/) webserver that serves as the "target URL" for the context broker. Apparently there already is a small example app available within the project, but I've only seen the filenames in a screenshot :-)
 
-There's the technical detail of subscribing. It depends on the behaviour of the context broker. Perhaps we can just re-register ourselves when the flask server starts up?
+## Our flask docker: *context broker* target and task starter and overview website
 
-(The flask docker will be registered through a kubernetes `deployment` yaml file. The details will have to be in some non-public repository, but once it works we'll probably put an example here.)
+See https://github.com/HE-TEMA/flood-calculation-site
 
+We'll make a docker that contains a small [flask (python)](https://pypi.org/project/Flask/) webserver that serves as the "target URL" for the context broker. It is based on the small example app available within the project.
 
-## K3S part 1b: what the 'flask' docker does
+- CB subscription target for FloodRiskEvent and ElevationMap.
+- Small form to create FloodRiskEvent (DONE).
+- Some handy debug pages to browse the available entity types and entities in the CB (DONE).
+- Code to trigger the "task" docker below based on the two CB subscriptions.
 
-Custom scripting, installs of your own software: that's best handled on your own servers. At least, that's my line of thinking.
 
-So the 'flask' docker does this:
+## Our "task" docker
 
-- It gets the notifications from the context broker.
-- It converts them into some json instruction.
-- It uploads it to a company-specific S3 object store in an `inbox/` directory.
-- (Afterwards, our custom software will look in that directory and start up 3Di simulations and so).
+Internally we use "Prefect", which looks a bit like airflow. In the end, it are just simple python scripts.
 
-
-## K3S part 2: 'cronjob' script docker
-
-The next part will be a docker running some python script. It is started as a kubernetes `cronjob` resource, which means it is simply run once in a while (every minute for instance).
-
-The simple script will probably do this:
-
-- Look in our company-specific S3 object store in the `outbox/` directory.
-- Any messages/objects it finds there are posted to the context broker.
-
-Such messages can be things like "area to really watch with drones".
-
-This way, whatever we do or calculate or determine can be made available through the context broker. There is a possibility of storing blobs of data in an internal "minio" object store, but our guess is that it is handier to just provide the URL to our s3 store for the results we've calculated, as that will be a non-firewalled URL, btw.
-
-
-## K3S part 3: status overview docker
-
-To make the process visible/observable, we propose a simple web interface that shows the current status from our point of view:
-
-- It downloads a `status.json` from our company-specific S3 object store.
-- It shows what "our" infrastructure has received, what is being processed and what is ready.
-
-This docker is deployed as a `deployment` resource with a matching `service` and `ingress`. (In normal terms: a website with a url).
-
-
-## Summary
-
-Our main focus is improving the actual "3Di" flood calcuation software. What I'm describing here is what we (and probably lots of other partners) need to do to cooperate:
-
-- Find or make a good vocabulary for adding what we need to the *context broker*.
-- A webserver that the context broker can send targeted messages to.
-- A script that feeds the context broker (and possibly the META minio storage) with updated content.
+- One task uploads a simple elevation map geotiff + adds the event to the context broker.
+- The other task reacts to an ElevationMap, which means the FloodRiskEvent can now be converted into a 3di scenario. The scenario is uploaded to our regular 3Di data center in Amsterdam. Upon completion, we store the results in minio and in the context broker.
 
 
 ## Possible case study

original-setup-from-april2024.md

@@ -0,0 +1,99 @@
+# Original proposed setup from april 2024
+
+My original idea below could be simplified, but I've retained the documentation as it might be handy in the future.
+
+## Setup
+
+The sketch below shows the suggested setup, followed by the textual explanation of the various parts. Basically we use the context broker in combination with a custom vocabulary + three **small and simple** dockers inside the K3S cluster + our own S3 storage for shuttling info back and forth.
+
+![Sketch of the suggested setup](suggested-setup.png)
+
+
+## Context broker: custom vocabulary
+
+In the context broker, we'll need to store information. Information that needs terms and definitions because of its "linked data" nature. So if we want to talk about a `RegionAtRisk`, we'll need to define.
+
+If there's already a suitable vocabulary, we can use it. But at first we're probably better off making our own "JSON-LD" vocabulary for our specific use case. That way we can store `https://our.vocabulary.org/v1/meta/RegionAtRisk=Ahrtal` in the context broker and add a subscription to that.
+
+We'll start such a bare-bones vocabulary and host it somewhere public. If handy, others can extend it or start their own.
+
+Some initial ideas (all with proper urls instead of just words, of course):
+
+    weather-occurrence
+        id=abcd
+        region = ahr
+        timestamp = 2021-07-xx
+        rainfall-data = s3://amaazon.com/xxx/something.tiff
+
+    flood-calculation
+        weather-occurrence=http://tema.it/weather-occurrence/abcd
+        result=s3://amazon.com/s3/xxx/something.zip
+
+    disturbance
+        3di-scenario=https://www.3di.live/scenario/xyz
+        height=235
+        x1=...
+        y1=...
+        x2=...
+        y2=...
+
+
+## K3S part 1a: *context broker* target 'flask' docker
+
+You can subscribe at the context broker for certain specific items. For instance a new `RegionAtRisk`. You have to provide a URL that the context broker can send its notification to.
+
+We'll make a docker that contains a small [flask (python)](https://pypi.org/project/Flask/) webserver that serves as the "target URL" for the context broker. Apparently there already is a small example app available within the project, but I've only seen the filenames in a screenshot :-)
+
+There's the technical detail of subscribing. It depends on the behaviour of the context broker. Perhaps we can just re-register ourselves when the flask server starts up?
+
+(The flask docker will be registered through a kubernetes `deployment` yaml file. The details will have to be in some non-public repository, but once it works we'll probably put an example here.)
+
+
+## K3S part 1b: what the 'flask' docker does
+
+Custom scripting, installs of your own software: that's best handled on your own servers. At least, that's my line of thinking.
+
+So the 'flask' docker does this:
+
+- It gets the notifications from the context broker.
+- It converts them into some json instruction.
+- It uploads it to a company-specific S3 object store in an `inbox/` directory.
+- (Afterwards, our custom software will look in that directory and start up 3Di simulations and so).
+
+
+## K3S part 2: 'cronjob' script docker
+
+The next part will be a docker running some python script. It is started as a kubernetes `cronjob` resource, which means it is simply run once in a while (every minute for instance).
+
+The simple script will probably do this:
+
+- Look in our company-specific S3 object store in the `outbox/` directory.
+- Any messages/objects it finds there are posted to the context broker.
+
+Such messages can be things like "area to really watch with drones".
+
+This way, whatever we do or calculate or determine can be made available through the context broker. There is a possibility of storing blobs of data in an internal "minio" object store, but our guess is that it is handier to just provide the URL to our s3 store for the results we've calculated, as that will be a non-firewalled URL, btw.
+
+
+## K3S part 3: status overview docker
+
+To make the process visible/observable, we propose a simple web interface that shows the current status from our point of view:
+
+- It downloads a `status.json` from our company-specific S3 object store.
+- It shows what "our" infrastructure has received, what is being processed and what is ready.
+
+This docker is deployed as a `deployment` resource with a matching `service` and `ingress`. (In normal terms: a website with a url).
+
+
+## Summary
+
+Our main focus is improving the actual "3Di" flood calcuation software. What I'm describing here is what we (and probably lots of other partners) need to do to cooperate:
+
+- Find or make a good vocabulary for adding what we need to the *context broker*.
+- A webserver that the context broker can send targeted messages to.
+- A script that feeds the context broker (and possibly the META minio storage) with updated content.
+
+
+## Possible case study
+
+I'm throwing in [a possible case study](example-case-study/case-study.md) just to make the possible data flows a bit more concrete.