resolved conflicts for #27

.github/ISSUE_TEMPLATE/feature_request.md

@@ -7,8 +7,14 @@ assignees: ''
 
 ---
 
-**Is your feature request related to a problem? Please describe.**
-A clear and concise description of what the problem is. 
+**New feature description.**
+Give a clear and concise description of the feature to be added to OLS and the potential limitation or problem it addresses. 
+
+**Use cases**
+Please detail the use cases that this new feature will serve.
+
+**User communities**
+If possible and with violating GDPR, can you refer us to user communities who are in need of this feature? 
 
 **Describe the solution you'd like**
 A clear and concise description of what you want to happen.

.github/workflows/docker.yml

@@ -3,6 +3,8 @@ name: Build & publish ols4 images
 on:
   push:
     branches: [ "dev", "stable" ]
+  pull_request:
+    branches: [ "dev" ]
 
 env:
   REGISTRY: ghcr.io
@@ -27,22 +29,20 @@ jobs:
 
       - name: Build and push ols4 dataload Docker image
         run: |
-          cd dataload
-          docker build -t ghcr.io/ebispot/ols4-dataload:${{ github.sha }} .
-          docker tag ghcr.io/ebispot/ols4-dataload:${{github.sha}} ghcr.io/ebispot/ols4-dataload:${{ github.ref_name }}
+          docker build -t ghcr.io/ebispot/ols4-dataload:${{ github.sha }} -f ./dataload/Dockerfile .
+          docker tag ghcr.io/ebispot/ols4-dataload:${{github.sha}} ghcr.io/ebispot/ols4-dataload:${{ github.head_ref || github.ref_name }}
           docker push --all-tags ghcr.io/ebispot/ols4-dataload
 
       - name: Build and push ols4 backend Docker image
         run: |
-          cd backend
-          docker build -t ghcr.io/ebispot/ols4-backend:${{ github.sha }} .
-          docker tag ghcr.io/ebispot/ols4-backend:${{github.sha}} ghcr.io/ebispot/ols4-backend:${{ github.ref_name }}
+          docker build -t ghcr.io/ebispot/ols4-backend:${{ github.sha }} -f ./backend/Dockerfile .
+          docker tag ghcr.io/ebispot/ols4-backend:${{github.sha}} ghcr.io/ebispot/ols4-backend:${{ github.head_ref || github.ref_name }}
           docker push --all-tags ghcr.io/ebispot/ols4-backend
 
       - name: Build and push ols4 frontend Docker image
         run: |
           cd frontend
           docker build -t ghcr.io/ebispot/ols4-frontend:${{ github.sha }} .
-          docker tag ghcr.io/ebispot/ols4-frontend:${{github.sha}} ghcr.io/ebispot/ols4-frontend:${{ github.ref_name }}
+          docker tag ghcr.io/ebispot/ols4-frontend:${{github.sha}} ghcr.io/ebispot/ols4-frontend:${{ github.head_ref || github.ref_name }}
           docker push --all-tags ghcr.io/ebispot/ols4-frontend
 

CONTRIBUTING.md

@@ -0,0 +1,9 @@
+Thank you for using OLS! We appreciate any feedback that can improve OLS and grow its user base. Besides being an enthusiastic user of OLS, you can contribute to OLS in the following ways:
+
+1. If you find something that does not work as expected, please open a [Github issue](https://github.com/EBISPOT/ols4/issues/new?assignees=&labels=bug&projects=&template=bug_report.md&title=).
+2. If you have an idea for a new feature for OLS, please open a [Github feature request](https://github.com/EBISPOT/ols4/issues/new?assignees=&labels=ols4-new-feature&projects=&template=feature_request.md&title=). 
+Please provide detailed information with regards to the use case(s) this feature will support.
+3. If you are hosting your own instance of OLS, please tell us about it by opening a ticket [here](https://github.com/EBISPOT/ontotools-docker/issues). 
+Please tell us about your use case and statistics like, number of ontologies hosted, number of requests per month and number of unique 
+users per month.
+4. For opening PRs against OLS, please follow the guidelines detailed [here](https://github.com/EBISPOT/ols4/wiki/Guidelines-for-opening-PR).

README.md

@@ -24,16 +24,17 @@ This repository contains three projects:
 
 # Deploying OLS4
 
-Deployment instructions will go here. OLS4 is still under heavy development, so currently we only have detailed
-instructions for developers below.
-
-However, if you just want to try it out, this should get you going:
+If you want to try OLS4 out, this should get you going:
 
     export OLS4_CONFIG=./dataload/configs/efo.json
     docker compose up
 
 You should now be able to access the OLS4 frontend at `http://localhost:8081`.
 
+If you need to set the heap size, you can do so using:
+
+    JAVA_OPTS="-Xms5G -Xmx25G"  docker compose up
+
 If you want to test it with your own ontology, copy the OWL or RDFS ontology file to the `testcases` folder (which is
 mounted in Docker). Then make a new config file for your ontology in `dataload/configs` (you can use `efo.json` as a
 template). For the `ontology_purl` property in the config, use e.g. `file:///opt/dataload/testcases/myontology.owl` if
@@ -174,6 +175,10 @@ or if you want to load all testcases, you can use
 
     ./dev-testing/teststack.sh ./testcases ./output
 
+If you need to set the Java heap size, you can set the environment the JAVA_OPTS variable as follows:
+
+     export JAVA_OPTS="-Xms5G -Xmx10G"
+
 Once Neo4J and Solr is up, to start the backend (REST API) you can run:
 
     ./dev-testing/start-backend.sh
@@ -184,7 +189,7 @@ Once the backend is up, you can start the frontend with:
 
 Once you are done testing, to stop everything:
 
-    ./stopall.sh
+    ./stopNeo4JSolr.sh
 
 ### Running the dataload locally
 
@@ -359,43 +364,74 @@ local (Dockerized) Solr and Neo4j servers:
 The frontend is a React application in `frontend`. See [frontend docs](frontend/README.md)
 for details on how to run the frontend.
 
-## Development: Updating `testcases_expected_output`
+## Development: Updating `testcases_expected_output` and `testcases_expected_output_api`
+If you make changes to the data load or API of OLS, you need to run testcases and compare it against the expected outputs 
+to ensure backward compatibility. This testing consists of 
 
-If you change something that results in the test output changing (e.g. adding new tests, changing what the output looks
-like), the CI on this repo will fail.
+1. testing the dataload outputs by comparing test outputs to expected outputs,
+2. API testing which compares API responses to expected responses, and
+3. adding the latest expected outputs to Git.
 
-To fix this, you need to replace the `testcases_expected_output` and `testcases_expected_output_api` folders with the
-new expected output. **You should do this in the same commit as your code/test changes because then we can track exactly
-what changed in the output.**
+### Testing dataload
+These tests are run locally as described in [Test testcases from dataload to UI](#test-testcases-from-dataload-to-ui).
+Ensure that the environment variables `NEO4J_HOME`, `SOLR_HOME` and `OLS4_HOME` are set up accordingly.
 
-First make sure all the JARs are up to date:
+1. First make sure all the OLS4 JARs are up to date by running :
+
+       mvn clean package
 
-    mvn clean package
+2. Generate new output files and import into Neo4J and Solr: 
+
+       ./dev-testing/teststack.sh ./testcases ./testcases_output
+
+3. Compare `/testcases_output` with `/testcases_expected_output`:
+
+       ./compare_testcase_output.sh
+
+4. The output of step 3 is written to `testcases_compare_result.log`. If no differences are found, this file will be empty. 
+Ensure that all differences in this file can be explained and that they do make sense.
+
+5. Once you are happy with the output in `testcases_output`, remove the old `testcases_expected_output` and replace with
+new expected output:
+
+       rm -rf testcases_expected_output
+       cp -r testcases_output/testcases testcases_expected_output
+
+6. Now continue with API testing.
+
+### Testing API
+Before doing API testing you must have completed the [dataload testing](#testing-dataload).
+
+7. Start the backend:
+
+        ./dev-testing/start-backend.sh
+
+8. Run API tests against backend using: 
+
+       ./test_api_fast.sh http://localhost:8080 ./testcases_output_api ./testcases_expected_output_api --deep
+
+9. The results of step 8 is written to `./apitester4.log`. Differences are written to the end of the file. When there are no
+differences, this file will end with these lines:
 
-Then run the test scripts:
+       RecursiveJsonDiff.diff() reported success
+       apitester reported success; exit code 0
 
-* `./test_dataload.sh` (~1 minute) will test the dataload locally, updating `testcases_expected_output`. All you need is
-  Java and Maven.
-* `./test_api.sh` (~15 mins) will test the entire OLS4 stack (dataload → solr/neo4j → api server) using Docker compose
-  to bring up and tear down all the services for each testcase, updating `testcases_expected_output_api`. You need to
-  have Docker and Docker compose installed.
+10. Ensure that all differences listed in `./apitester4.log` are accounted for. Once you are happy with the output, remove 
+the old `testcases_expected_output_api` and replace with new expected output: 
 
-To run both:
+        rm -rf testcases_expected_output_api
+        cp -r testcases_output testcases_expected_output_api
 
-    ./test_dataload.sh
-    ./test_api.sh
+11. Add the latest expected outputs to Git:
 
-Now remove the existing expected output:
+        git add -A testcases_expected_output
+        git add -A testcases_expected_output_api
 
-    rm -rf testcases_expected_output
-    rm -rf testcases_expected_output_api
+    **You should do this in the same commit as your code/test changes because then we can track exactly
+    what changed in the output.**
 
-Copy your new output to the respective directories:
+12. You can stop the OLS4 backend with "Ctrl-C", and Solr and Neo4J with:
 
-    cp -r testcases_output testcases_expected_output
-    cp -r testcases_output_api testcases_expected_output_api
+        ./dev-testing/stopNeo4JSolr.sh
 
-You can now add it to your commit:
 
-    git add -A testcases_expected_output
-    git add -A testcases_expected_output_api

apitester4/src/main/java/uk/ac/ebi/ols/apitester/RecursiveJsonDiff.java

@@ -76,24 +76,28 @@ public boolean diff() throws FileNotFoundException {
     }
 
     LinkedHashSet<String> getJsonFiles(String path) {
+        System.out.println(path);
+        if (path.startsWith("./"))
+            path = path.substring("./".length());
+        String finalPath = path;
         Collection<File> files = FileUtils.listFiles(new File(path), TrueFileFilter.INSTANCE, TrueFileFilter.INSTANCE);
         return new LinkedHashSet<>(
                 files.stream().map(file -> {
-                        String filePath = file.getPath();
-                        if(!filePath.startsWith(path)) {
-                            throw new RuntimeException("File " + filePath + " + did not start with prefix " + path);
-                        }
-                        return filePath.substring(path.length());
-                    }).filter(filePath -> {
-                        return filePath.endsWith(".json");
-                    })
-                .collect(Collectors.toList())
+                            String filePath = file.getPath();
+                            if(!filePath.startsWith(finalPath)) {
+                                throw new RuntimeException("File " + filePath + " + did not start with prefix " + finalPath);
+                            }
+
+                            return filePath.substring(finalPath.length());
+                        }).filter(filePath -> {
+                            return filePath.endsWith(".json");
+                        })
+                        .collect(Collectors.toList())
         );
     }
 
     JsonElement readJsonFile(String filename) throws FileNotFoundException {
         JsonReader reader = new JsonReader(new FileReader(filename));
         return JsonParser.parseReader(reader);
     }
-
-}
+}

backend/Dockerfile

@@ -1,11 +1,16 @@
 
 FROM maven:3.9.6-eclipse-temurin-17
 
-RUN mkdir /opt/ols
-COPY . /opt/ols/
-RUN cd /opt/ols && ls && mvn clean package -DskipTests
+RUN mkdir /opt/ols && mkdir /opt/ols/backend && mkdir /opt/ols/ols-shared
+COPY ./backend /opt/ols/backend
+COPY ./ols-shared /opt/ols/ols-shared
+
+RUN cd /opt/ols/ols-shared && mvn package
+
+RUN mvn install:install-file -DcreateChecksum=true -Dpackaging=jar -Dfile=/opt/ols/ols-shared/target/ols4-shared-1.0.0-SNAPSHOT.jar -DgroupId=uk.ac.ebi.spot.ols -DartifactId=ols4-shared -Dversion=1.0.0-SNAPSHOT
+RUN cd /opt/ols/backend && ls && mvn clean package -DskipTests
 
 EXPOSE 8080
-ENTRYPOINT ["java", "-jar", "/opt/ols/target/ols4-backend-4.0.0-SNAPSHOT.jar"]
+ENTRYPOINT ["java", "-jar", "/opt/ols/backend/target/ols4-backend-4.0.0-SNAPSHOT.jar"]
 
 

backend/pom.xml

@@ -21,7 +21,11 @@
     </properties>
 
     <dependencies>
-
+        <dependency>
+            <groupId>uk.ac.ebi.spot.ols</groupId>
+            <artifactId>ols4-shared</artifactId>
+            <version>1.0.0-SNAPSHOT</version>
+        </dependency>
     	<dependency>
     		<groupId>com.google.code.gson</groupId>
     		<artifactId>gson</artifactId>

backend/src/main/java/uk/ac/ebi/spot/ols/controller/api/v1/V1IndividualController.java

@@ -1,5 +1,7 @@
 package uk.ac.ebi.spot.ols.controller.api.v1;
 
+import io.swagger.v3.oas.annotations.Parameter;
+import io.swagger.v3.oas.annotations.tags.Tag;
 import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.data.domain.Page;
 import org.springframework.data.domain.Pageable;
@@ -28,6 +30,9 @@
  * @date 18/08/2015
  * Samples, Phenotypes and Ontologies Team, EMBL-EBI
  */
+
+@Tag(name = "Individual Controller", description = "NOTE: For IRI parameters, the value must be URL encoded. " +
+        "For example, the IRI http://purl.obolibrary.org/obo/IAO_0000124 should be encoded as http%3A%252F%2Fpurl.obolibrary.org%2Fobo%2FIAO_0000124.")
 @RestController
 @RequestMapping("/api/individuals")
 @ExposesResourceFor(V1Individual.class)
@@ -46,12 +51,15 @@ public RepositoryLinksResource process(RepositoryLinksResource resource) {
         return resource;
     }
 
-    @RequestMapping(path = "/{id}", produces = {MediaType.APPLICATION_JSON_VALUE, MediaTypes.HAL_JSON_VALUE}, method = RequestMethod.GET)
+    @RequestMapping(path = "/{iri}", produces = {MediaType.APPLICATION_JSON_VALUE, MediaTypes.HAL_JSON_VALUE}, method = RequestMethod.GET)
     HttpEntity<PagedModel<V1Individual>> getAllIndividuals(
-            @PathVariable("id") String termId,
+            @PathVariable("iri")
+            @Parameter(name = "iri",
+                    description = "The IRI of the individual, this value must be double URL encoded",
+                    example = "http%3A%252F%2Fpurl.obolibrary.org%2Fobo%2FIAO_0000124") String termId,
             @RequestParam(value = "lang", required = false, defaultValue = "en") String lang,
-            Pageable pageable,
-            PagedResourcesAssembler assembler) {
+            @Parameter(hidden = true) Pageable pageable,
+            @Parameter(hidden = true) PagedResourcesAssembler assembler) {
         String decoded = null;
         decoded = UriUtils.decode(termId, "UTF-8");
         return getAllIndividuals(decoded, null, null, lang, pageable, assembler);
@@ -60,12 +68,21 @@ HttpEntity<PagedModel<V1Individual>> getAllIndividuals(
 
     @RequestMapping(path = "", produces = {MediaType.APPLICATION_JSON_VALUE, MediaTypes.HAL_JSON_VALUE}, method = RequestMethod.GET)
     HttpEntity<PagedModel<V1Individual>> getAllIndividuals(
-            @RequestParam(value = "iri", required = false) String iri,
-            @RequestParam(value = "short_form", required = false) String shortForm,
-            @RequestParam(value = "obo_id", required = false) String oboId,
+            @RequestParam(value = "iri", required = false)
+            @Parameter(name = "iri",
+                    description = "The IRI of the individual, this value must be double URL encoded",
+                    example = "http%3A%252F%2Fpurl.obolibrary.org%2Fobo%2FIAO_0000124") String iri,
+            @RequestParam(value = "short_form", required = false)
+            @Parameter(name = "short_form",
+                    description = "This refers to the short form of the individual.",
+                    example = "IAO_0000124") String shortForm,
+            @RequestParam(value = "obo_id", required = false)
+            @Parameter(name = "obo_id",
+                    description = "This refers to the OBO ID of the individual.",
+                    example = "IAO:0000124") String oboId,
             @RequestParam(value = "lang", required = false, defaultValue = "en") String lang,
-            Pageable pageable,
-            PagedResourcesAssembler assembler) {
+            @Parameter(hidden = true) Pageable pageable,
+            @Parameter(hidden = true) PagedResourcesAssembler assembler) {
 
         Page<V1Individual> terms = null;
 
@@ -82,12 +99,16 @@ HttpEntity<PagedModel<V1Individual>> getAllIndividuals(
         return new ResponseEntity<>(assembler.toModel(terms, individualAssembler), HttpStatus.OK);
     }
 
-    @RequestMapping(path = "/findByIdAndIsDefiningOntology/{id}", produces = {MediaType.APPLICATION_JSON_VALUE, MediaTypes.HAL_JSON_VALUE}, method = RequestMethod.GET)
+
+    @RequestMapping(path = "/findByIdAndIsDefiningOntology/{iri}", produces = {MediaType.APPLICATION_JSON_VALUE, MediaTypes.HAL_JSON_VALUE}, method = RequestMethod.GET)
     HttpEntity<PagedModel<V1Individual>> getAllIndividualsByIdAndIsDefiningOntology(
-            @PathVariable("id") String termId,
+            @PathVariable("iri")
+            @Parameter(name = "iri",
+                    description = "The IRI of the individual, this value must be double URL encoded",
+                    example = "http%3A%252F%2Fpurl.obolibrary.org%2Fobo%2FIAO_0000124") String termId,
             @RequestParam(value = "lang", required = false, defaultValue = "en") String lang,
-            Pageable pageable,
-            PagedResourcesAssembler assembler) {
+            @Parameter(hidden = true) Pageable pageable,
+            @Parameter(hidden = true) PagedResourcesAssembler assembler) {
         String decoded = null;
         decoded = UriUtils.decode(termId, "UTF-8");
         return getAllIndividualsByIdAndIsDefiningOntology(decoded, null, null, lang, pageable, assembler);
@@ -99,12 +120,21 @@ HttpEntity<PagedModel<V1Individual>> getAllIndividualsByIdAndIsDefiningOntology(
     		produces = {MediaType.APPLICATION_JSON_VALUE, MediaTypes.HAL_JSON_VALUE},
     		method = RequestMethod.GET)
     HttpEntity<PagedModel<V1Individual>> getAllIndividualsByIdAndIsDefiningOntology(
-            @RequestParam(value = "iri", required = false) String iri,
-            @RequestParam(value = "short_form", required = false) String shortForm,
-            @RequestParam(value = "obo_id", required = false) String oboId,
+            @RequestParam(value = "iri", required = false)
+            @Parameter(name = "iri",
+                    description = "The IRI of the individual, this value must be double URL encoded",
+                    example = "http%3A%252F%2Fpurl.obolibrary.org%2Fobo%2FIAO_0000124") String iri,
+            @RequestParam(value = "short_form", required = false)
+            @Parameter(name = "short_form",
+                    description = "This refers to the short form of the individual.",
+                    example = "IAO_0000124") String shortForm,
+            @RequestParam(value = "obo_id", required = false)
+            @Parameter(name = "obo_id",
+                    description = "This refers to the OBO ID of the individual.",
+                    example = "IAO:0000124") String oboId,
             @RequestParam(value = "lang", required = false, defaultValue = "en") String lang,
-            Pageable pageable,
-            PagedResourcesAssembler assembler) {
+            @Parameter(hidden = true) Pageable pageable,
+            @Parameter(hidden = true) PagedResourcesAssembler assembler) {
 
         Page<V1Individual> terms = null;