Improve HTTP API

[jrwdunham]

Connected to #369

Summary

• Adds consistent REST (HTTP JSON) endpoints for almost every model in the Storage Service.
• Builds documentation into the API using a generated OpenAPI 3.0 YAML file and Swagger UI interface to same.
• Auto-generates Python client code which is self-documenting and has client-side validation.
• Introduces abstractions and an example of how to implement an OpenAPI-compliant custom endpoint GET locations/{pk}/browse/ using Python data structures.

This PR adds a new set of versioned API endpoints (under the beta namespace) which address #369 by being:

• consistent: same HTTP method + path regex for all endpoints or operations of a given type; all resources expose the same read/write or read-only operations.
• thoroughly documented using OpenAPI 3.0 spec—see /api/beta/—and a Swagger UI interface—see /api/beta/doc/—to same;
• able to generate API client code—see /api/beta/client/—which is generated by clientbuilder.py; and
• introduces needed endpoints, e.g., advanced search over all exposed resources, DELETE, PUT (update), etc.

How to Understand OpenAPI 3.0

OpenAPI suffers from too much documentation. I have found this one to be good:

• https://swagger.io/docs/specification/about/

How to Understand this PR

• The API endpoints introduced here are configured in storage_service/locations/api/beta/
  • __init__.py defines the resources dict that tells Remple which resources with which endpoints to expose.
  • resources/ defines the resource classes (which point to Django models to infer read schemata and Formencode schemata for generating mutate schemata).
    • resources.py — simple resources with little configuration (ie no custom endpoints)
    • locations.py — a module of its own because it has a custom endpoint: browse
  • schemata.py defines the (Formencode) schemata for viewing, creating and updating resources.
• Remple is the mini REST framework introduced here. If this PR is merged, we will probably want to move Remple into its own repo so that Archivematica can use it also.
  • resources.py provides the base classes for defining resources.
  • querybuilder.py holds the logic for transforming JSON/Python data structures into Django ORM filters.
  • routebuilder.py takes a resources config dict and can return a list of Django url() instances that can be included in urlpatterns.
  • openapi.py can generate an OpenAPI data structure given a configured Remple API and spit out an OpenAPI 3.0-conformant YAML file that can be used to generate the Swagger UI JavaScript/HTML app as well as the Python client code.
  • clientbuilder.py is a Python module that must be modified to contain an OpenAPI data structure so that it can define (at runtime) a set of Python classes that constitute a client for the SS API.

Questions

• Would it be better to use TastyPie, Django Rest Framework or APIStar (Py3 only, but has OpenAPI spec generation built-in) instead of the custom-built Remple mini REST framework Remple introduced here? At a high level, the benefits of rolling our own API mini framework allows us more control as well as the ability to be on the bleeding edge when it comes to the OpenAPI spec (cf. codegen's unavailability in OpenAPI v. 3.0). The benefits of using an existing framework are a lower maintenance burden for us and the pros of using a battle-tested code base with buy-in from disparate stakeholders.
• Can existing code generation tools be used to build the Python client/SDK instead of rolling our own in clientbuilder.py? From my research, the answer was "No, because these tools have not caught up to OpenAPI v. 3.0 yet." See Swagger Codegen. My preference would be to use clientbuilder.py and switch to a standard OpenAPI code generation tool when such tools catch up to the spec, an option that is opened up for us by virtue of using OpenAPI.

[sevein]

I think that you've accidentally checked in storage_service/.pytest_cache/v/cache/*.

[jrwdunham]

Thanks @sevein: removed those

[jambun]

Hi @jrwdunham. We have spent some time going through this and our overall impression is positive. We like the approach and were able to understand how you had done things and why.

Using clientbuilder as a stopgap makes sense. Hopefully swagger support isn’t too far away. We noticed this: https://github.com/openapitools/openapi-generator.

Our understanding is a bit abstract at this stage - we’ve only eyeballed the code. We will be able to give more detailed feedback after using it.

[jrwdunham]

This is specific to the Archivematica Storage Service so it should be removed or moved outside of Remple. Futhermore, it seems that it is not even being used.

[jrwdunham]

Document or remove the implicit usage here of the non-standard HTTP method SEARCH.

[jrwdunham]

It should be possible to remove some of the duplication here with _set_crud_paths.

[jrwdunham]

TODO: should we be able to use info in django_settings to specify a true URL here instead of just a path?

[jrwdunham]

TODO: it seems that OpenAPI's readOnly and writeOnly boolean property attributes could be used here. See https://swagger.io/docs/specification/data-models/data-types/, in particular the Read-Only and Write-Only Properties section.

[jrwdunham]

No. This comment is misleading. Update via Remple has the semantics of PUT not PATCH. A full representation of the post-update state must be sent in the update request.

[jrwdunham]

Docstring should indicate that this is done by inspecting the Django model.

[jrwdunham]

The above is only needed if we want to support semver-compliant API versions, which may not be a good idea.

[jrwdunham]

Fix function name since it returns a boolean indicating whether a field is required and whether it is null(able).

[helrond]

This is a huge step forward in making the Storage Service API consistent, predictable and documented. Thank you! I haven't had a chance to spin this up locally yet, but I'm planning to do that in the next week or so and may have more specific feedback after that.

I am not a real developer, and have a very rudimentary knowledge of the Archivematica codebase, however from my perspective I'm not sure it makes sense to implement a custom REST framework. It seems like maintenance for Archivematica is already challenging enough that alleviating some of that burden would outweigh the benefits of OpenAPI 3.0 compliance, especially since as you note the tooling isn't really there yet. But perhaps I don't fully understand the benefits of OpenAPI 2.x versus 3.0.