Skip to content

Commit

Permalink
Merge pull request #36 from aodn/features/5337-enhance-endpoints
Browse files Browse the repository at this point in the history 
Features/5337 enhance endpoints
  • Loading branch information
@HavierD
HavierD authored Apr 17, 2024
2 parents 0fe7ce3 + 292b643 commit 7bba833
Show file tree
Hide file tree
Showing 151 changed files with 2,496 additions and 3,055 deletions.
67 changes: 52 additions & 15 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,21 +13,28 @@ to alter the xml like what we did before plus we are using a Docker base image o
You need create a file call .env and put in the following attribute if you do not want the
default startup parameters

Assume you have installed docker and docker-compose, you can run ./startEsLocal.sh to run a elastic search 7 for
use by the geonetwork. It takes a while to start (3 mins+), so you can check http://localhost:5601 if system
started. Once started you can run

```shell
ES_HOST=ec2-3-25-64-248.ap-southeast-2.compute.amazonaws.com
INDEXER_HOST=ec2-3-25-163-152.ap-southeast-2.compute.amazonaws.com
INDEXER_PORT=8081
# If you run ./startEsLocal.sh, then you should point to localhost
ES_HOST=localhost

# We record changed, it will notify the indexer of update, check with dev on what is the value of API key
INDEXER_HOST=https://es-indexer-edge.aodn.org.au
INDEXER_PORT=443
INDEXER_APIKEY=THE_API_KEY_TO_CALL_INDEXER

# By default it runs in the h2 db, you can use postgis + postgres, so below config optional
GEONETWORK_DB_TYPE=postgres-postgis
GEONETWORK_DB_PORT=5432
GEONETWORK_DB_NAME=geonetwork
```

Then assume you have installed docker and docker-compose, then you can run ./startEsLocal.sh to run a elastic search 7 for
use by the geonetwork. It takes a while to start (3 mins+), so you can check http://localhost:5601 if system
started. Once started you can run
# Optional, by default use main branch to get the json config for GN4, however for development you may want to point
# to your own branch
GIT_BRANCH=xxx
```

```shell
# Start geonetwork4
Expand All @@ -40,6 +47,16 @@ docker-compose -f docker-gn-compose.yml down -v
Once elastic started you can run ./startGn4Local.sh to start the geonetwork. It is recommend to start
it like this because it will rebuild your images with the binary that you created from maven build install

## Debug
If you run the geonetwork using ./startGn4Local.sh, then you can setup a debug profile using
```shell
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:8000
```

and connect to the instance inside docker.

> Noted: You should run docker container prune and docker image prune periodically to free up disk space.
## Ssh to instance
You can login to the geonetwork4 instance to debug you setting by
```shell
Expand All @@ -59,14 +76,34 @@ endpoint to view the log file directly as cloud watch is not so easy to use.

### Endpoints:

| Description | Endpoints | Environment |
|--------------|--------------------------------------|-------------|
| Logfile | `/geonetwork/srv/api/manage/logfile` | Edge |
| Beans info | `/geonetwork/srv/api/manage/beans` | Edge |
| Env info | `/geonetwork/srv/api/manage/env` | Edge |
| Info | `/geonetwork/srv/api/manage/info` | Edge |
| Health check | `/geonetwork/srv/api/manage/health` | Edge |
| Setup | `/geonetwork/srv/api/setup` | Edge |
> You need to present X-XSRF-TOKEN in your header to call Setup endpoints, please read comments
in [Api.java](./geonetwork/src/main/java/au/org/aodn/geonetwork4/controller/Api.java)

| Description | Method | Endpoints | Param | Environment |
|-------------------------------|--------|---------------------------------------------|----------------------------------|-------------|
| Logfile | GET | `/geonetwork/srv/api/manage/logfile` | | Edge |
| Beans info | GET | `/geonetwork/srv/api/manage/beans` | |Edge |
| Env info | GET | `/geonetwork/srv/api/manage/env` | |Edge |
| Info | GET | `/geonetwork/srv/api/manage/info` | |Edge |
| Health check | GET | `/geonetwork/srv/api/manage/health` | |Edge |
| Read Harvester - Config | GET | `/geonetwork/srv/api/aodn/setup/harvesters` | |Edge |
| Delete All Harvester - Config | DELETE | `/geonetwork/srv/api/aodn/setup/harvesters` | |Edge |
| Delete All Category - Config | DELETE | `/geonetwork/srv/api/aodn/setup/categories` | |Edge |
| Setup from github config | POST | `/geonetwork/srv/api/aodn/setup` | source=github |Edge |

### How the Setup works?

#### source=github (default)
Run the setup endpoint trigger geonetwork load the configuration file from github [config](./geonetwork-config/config.json),
this file is the blue-print to load other configuration from github that store under the geonetwork-config folder.

By default, it load from main branch, hence during your development, you may want to use the environment variable about to
force it load from different branch.

Given the configuration is store in main branch, that means changes to configuration require a pull request.

> The POST method can carry a body with the same format as the config.json file, if this appears, then the content
> in the body will be use instead of the config.json in github. Hence, you can run individual setup one by one
## Schema folder

Expand Down
1 change: 1 addition & 0 deletions docker-gn-compose.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ services:
JAVA_OPTS: >-
-Xms256m -Xmx2g
-Dspring.profiles.active=edge
-Daodn.geonetwork4.githubBranch=${GIT_BRANCH:-main}
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000
-Dgeonetwork.data.dir=/opt/geonetwork/data
-Dgeonetwork.config.dir=/opt/geonetwork/config
Expand Down
5 changes: 5 additions & 0 deletions geonetwork-api/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,11 @@
<artifactId>json</artifactId>
<version>20230618</version>
</dependency>
<dependency>
<groupId>com.github.javadev</groupId>
<artifactId>underscore</artifactId>
<version>${underscore.version}</version>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
Expand Down
95 changes: 61 additions & 34 deletions geonetwork-api/src/main/java/au/org/aodn/geonetwork_api/openapi/api/HarvestersApiLegacy.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,16 +21,12 @@
import org.springframework.web.client.HttpServerErrorException;
import org.springframework.web.client.RestClientException;

import java.util.Collections;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.*;
import java.util.stream.Collectors;

/**
* The current api provided limited support on harvester operation, hence we implement the missing. We can demise
* the code here in the future when api get mature.
*
* We suffix it with Legacy because it used the legacy geonetwork4 api call.
*/
public class HarvestersApiLegacy extends HarvestersApi {
Expand Down Expand Up @@ -78,9 +74,17 @@ public ResponseEntity<String> checkHarvesterPropertyExistWithHttpInfo(String pro
* Delete all haverters found in the geonetwork4
*/
public void deleteAllHarvesters() {
deleteHarvester(null);
}

/**
* Delete harvesters based on name
* @param title - Contains a list of harvester name to be deleted. Null means delete all
*/
public void deleteHarvester(String title) {
ResponseEntity<String> harvesters = proxyHarvestersApiLegacy.getHarvestersWithHttpInfo();

if(harvesters.getStatusCode().is2xxSuccessful()) {
if(harvesters.getStatusCode().is2xxSuccessful() && harvesters.getBody() != null) {
JSONObject jsonObject = XML.toJSONObject(harvesters.getBody());

if (jsonObject.optJSONObject("nodes") != null && jsonObject.optJSONObject("nodes").optJSONArray("node") != null) {
Expand All @@ -89,8 +93,18 @@ public void deleteAllHarvesters() {
for (int i = 0; i < nodes.length(); i++) {
JSONObject node = nodes.getJSONObject(i);

proxyHarvestersApiLegacy.deleteHarvesters(node.getInt("id"));
logger.info("Deleted harvester - {}", node.getJSONObject("site").getString("name"));
String name = node.getJSONObject("site") != null ?
node.getJSONObject("site").optString("name")
: null;

if(title == null || title.equalsIgnoreCase(name)) {
int id = node.getInt("id");
logger.info("Delete harvester id {} - {}",
id,
node.getJSONObject("site").getString("name"));

proxyHarvestersApiLegacy.deleteHarvesters(id);
}
}
}
}
Expand All @@ -101,8 +115,8 @@ public void deleteHarvesters(Integer id) {
}
/**
* It won't check duplicate
* @param config
* @return
* @param config - List of configuration in JSON format
* @return The responses from server
*/
public List<HarvestersApiLegacyResponse> createHarvesters(List<String> config) {

Expand All @@ -119,41 +133,54 @@ public List<HarvestersApiLegacyResponse> createHarvesters(List<String> config) {
if(groupAttr.isPresent()) {
// Check if group name already exist, if yes we get the group id and set the
// group id.
String id = groupAttr.get().optString("id");
Optional<Group> group;
if(id == null) {
// Try to find group by name, there may be null issue
group = groupsHelper.findGroupByName(groupAttr.get().optString("name"));
String groupName = groupAttr.get().optString("name");
Optional<Group> group = Optional.empty();

if(!groupName.isEmpty()) {
// Name have higher prefer over id as it is more accurate across different instance
// of geonetwork
group = groupsHelper.findGroupByName(groupName);
}
else {
group = groupsHelper.findGroupById(id);

if(groupName.isEmpty()) {
// Try to find group by id, but it may fail due to different if your config
// is export form another instance
String id = groupAttr.get().optString("id");
if (id != null) {
group = groupsHelper.findGroupById(Integer.parseInt(id));
}
}

if (group.isPresent()) {
if (group.isPresent()) {
logger.info("Group found with either name or id -> {}", group.get());
// Re-parse the jsonobject due to value updated
parsed = parser.parseHarvestersConfig(
groupsHelper.updateHarvestersOwnerGroup(parsed.getJsonObject(), group.get()).toString());
}
}

Optional<JSONArray> categories = tagsHelper.getHarvestersCategories(parsed.getJsonObject());
Optional<JSONObject> categories = tagsHelper.getHarvestersCategories(parsed.getJsonObject());
if(categories.isPresent()) {
// TODO: In the legacy, there is a comment to said support first category only, Why???
// @craig may have info
// The geonetwork able to save 1 category so we use index 0 only.
// From the name we get the category id of this instance, this is more portable as the
// id is different between different geonetwork4 instance.
Optional<MetadataCategory> category = tagsHelper.findTag(
categories
.get()
.getJSONObject(0)
.getJSONObject("category")
.getString("@id"));
.getJSONObject(TagsHelper.CATEGORY)
.getString(TagsHelper.NAME_ATTRIBUTE));

if(category.isPresent()) {
parsed = parser.parseHarvestersConfig(
tagsHelper.updateHarvestersCategories(parsed.getJsonObject(), category.get()).toString());
}
}

logger.info("Adding harvestor config : {}", parsed.getJsonObject().getString("name"));
String name = parsed.getJsonObject().getString("name");
logger.info("Add harvester config : name {} {}", name, parsed.getXml());

// Delete before add to avoid duplicates
deleteHarvester(name);

ResponseEntity<Map<String, Object>> r = proxyHarvestersApiLegacy.createHarvesterWithHttpInfo(parsed);

Expand Down Expand Up @@ -204,9 +231,9 @@ public ResponseEntity<Map<String, Object>> createHarvesterWithHttpInfo(Parser.Pa

HttpHeaders localVarHeaderParams = new HttpHeaders();

MultiValueMap<String, String> localVarQueryParams = new LinkedMultiValueMap();
MultiValueMap<String, String> localVarCookieParams = new LinkedMultiValueMap();
MultiValueMap<String, Object> localVarFormParams = new LinkedMultiValueMap();
MultiValueMap<String, String> localVarQueryParams = new LinkedMultiValueMap<>();
MultiValueMap<String, String> localVarCookieParams = new LinkedMultiValueMap<>();
MultiValueMap<String, Object> localVarFormParams = new LinkedMultiValueMap<>();

String[] localVarAuthNames = new String[0];
ParameterizedTypeReference<Map<String, Object>> localReturnType = new ParameterizedTypeReference<>() {};
Expand Down Expand Up @@ -235,9 +262,9 @@ public ResponseEntity<Map<String, Object>> deleteHarvesterWithHttpInfo(Integer i

HttpHeaders localVarHeaderParams = new HttpHeaders();

MultiValueMap<String, String> localVarQueryParams = new LinkedMultiValueMap();
MultiValueMap<String, String> localVarCookieParams = new LinkedMultiValueMap();
MultiValueMap<String, Object> localVarFormParams = new LinkedMultiValueMap();
MultiValueMap<String, String> localVarQueryParams = new LinkedMultiValueMap<>();
MultiValueMap<String, String> localVarCookieParams = new LinkedMultiValueMap<>();
MultiValueMap<String, Object> localVarFormParams = new LinkedMultiValueMap<>();

String[] localVarAuthNames = new String[0];
ParameterizedTypeReference<Map<String, Object>> localReturnType = new ParameterizedTypeReference<>() {};
Expand Down Expand Up @@ -326,9 +353,9 @@ public ResponseEntity<Map<String, Object>> deleteHarvesterWithHttpInfo(Integer i
*/
public ResponseEntity<String> getHarvestersWithHttpInfo() {

MultiValueMap<String, String> localVarQueryParams = new LinkedMultiValueMap();
MultiValueMap<String, String> localVarCookieParams = new LinkedMultiValueMap();
MultiValueMap<String, Object> localVarFormParams = new LinkedMultiValueMap();
MultiValueMap<String, String> localVarQueryParams = new LinkedMultiValueMap<>();
MultiValueMap<String, String> localVarCookieParams = new LinkedMultiValueMap<>();
MultiValueMap<String, Object> localVarFormParams = new LinkedMultiValueMap<>();

String[] localVarAuthNames = new String[0];
ParameterizedTypeReference<String> localReturnType = new ParameterizedTypeReference<>() {};
Expand Down
77 changes: 9 additions & 68 deletions geonetwork-api/src/main/java/au/org/aodn/geonetwork_api/openapi/api/Parser.java
View file
Original file line number Diff line number Diff line change
@@ -1,10 +1,9 @@
package au.org.aodn.geonetwork_api.openapi.api;

import com.fasterxml.jackson.core.JsonProcessingException;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.json.JSONArray;
import org.json.JSONObject;
import com.github.underscore.U;

import java.util.Map;

/**
* This parser is used to convert configuration in json to XML, those configuration
Expand All @@ -14,7 +13,7 @@
*/
public class Parser {

public class Parsed {
public static class Parsed {
private final JSONObject jsonObject;
private final String xml;

Expand All @@ -32,77 +31,19 @@ public JSONObject getJsonObject() {
}
}

protected Logger logger = LogManager.getLogger(Parser.class);

public Parsed parseLogosConfig(String json) throws JsonProcessingException {
public Parsed parseLogosConfig(String json) {
JSONObject jsonObject = new JSONObject(json);
return new Parsed(
jsonObject,
"<?xml version=\"1.0\" encoding=\"utf-8\"?>" + this.convertJsonToXml(jsonObject, "logo")
U.jsonToXml(json, "logo")
);
}

public Parsed parseHarvestersConfig(String json) throws JsonProcessingException {
public Parsed parseHarvestersConfig(String json) {
JSONObject jsonObject = new JSONObject(json);
return new Parsed(
jsonObject,
"<?xml version=\"1.0\" encoding=\"utf-8\"?>" + this.convertJsonToXml(jsonObject.getJSONObject("harvester_data").getJSONObject("node"), "node"));
}

protected String convertJsonToXml(JSONObject jsonObject , String rootElement) throws JsonProcessingException {
StringBuilder xmlBuilder = new StringBuilder("<" + rootElement);

// Process attributes (keys starting with "@")
for (String key : jsonObject.keySet()) {
if (key.startsWith("@")) {
xmlBuilder.append(" ")
.append(key.substring(1))
.append("=\"")
.append(jsonObject.get(key))
.append("\"");
}
}
xmlBuilder.append(">");

// Process child elements
for (String key : jsonObject.keySet()) {
if (!key.startsWith("@")) {
Object value = jsonObject.get(key);

if (value instanceof JSONObject) {
// Recursive call for nested objects
xmlBuilder.append(convertJsonToXml((JSONObject) value, key));
}
else if (value instanceof JSONArray) {
JSONArray array = (JSONArray) value;
for (int i = 0; i < array.length(); i++) {
Object item = array.get(i);
if (item instanceof JSONObject) {
// Recursive call for each object in the array
xmlBuilder.append(convertJsonToXml((JSONObject) item, key));
}
else {
// Handle non-JSON object items in the array
xmlBuilder.append("<").append(key).append(">")
.append(item.toString().replace("&", "&amp;"))
.append("</").append(key).append(">");
}
}
}
else {
// Normal element
xmlBuilder.append("<")
.append(key)
.append(">")
.append(value.toString().replace("&", "&amp;"))
.append("</")
.append(key)
.append(">");
}
}
}

xmlBuilder.append("</").append(rootElement).append(">");
return xmlBuilder.toString();
U.toXml((Map) U.fromJsonMap(json).get("harvester_data"))
);
}
}
Loading

0 comments on commit 7bba833

Please sign in to comment.