Schema Evolution

[wittjosiah]

Problems

1. Application updates from version 1 to version 2 which has a new data schema and it wants to migrate all of the data to the new schema (classic db migrations)
2. Application updates from version 1 to version 2 and wants to be able to read and write data of both new and old version of the schema (data lensing)
   a. Two different applications want to operate on the same data but have different ways of interpreting the same data
   b. Support for dragging data from application into another and having the destination application interpret the data in a way that makes sense to its own context
3. Application is collaborative and wants to be able to collaborate between peers without aligning on a protocol version to use (protocol/api versioning)

Prior Art

• sql database migrations
• graphql schemas
• protocol buffers definitions
• cambria data lenses

[wittjosiah]

Next Steps

While we generally need to solve all of these problems, we are looking to take an iterative approach such that we can ship a solution to a small piece of one of the problems and then grow that solution over time to support more use cases and cover more of the problem space.

The initial narrow problem which we are looking at currently is a naming problem. More specifically when generating the application graph in Composer, each object that is added to the graph needs to be given a label to describe itself when rendered, however different objects have different opinions on what that field is.

A concrete example of this is, within our codebase, it is not consistent whether schemas use name or whether they use title. The result of this is shotgun parsing code along the lines of object.name || object.title || placeholder.

[dmaretskyi]

We might want to start storing the schema in ECHO then. The peer that creates an object upserts it's schema into the DB.

This allows us to keep object - schema consistency when switching versions. For example objects created by other peers with diffrent schemas will still be introspectable despite the local peer having different/missing schema.

In the future applications might includes migration functions between schemas. Migration functions will form edges of a graph with schema versions as nodes.

[wittjosiah]

Building on this idea, I would propose the possibility that one object could have more than one schema associated with it. I shared these thoughts elsewhere already but including them here for posterity:

• currently an object just has at most a single schema
• if we wanted we could write a lens from one schema to another but then if we wanted to persist that result it would need to be in a new object
• at this point you have two objects and a problem, because in a lot of interop use-cases you're going to want to keep them in sync somehow
• but what if, instead of having to write it to a different object, one object could support multiple schemas?
• I imagine it would have some default schema for how it presents itself generally, but if you wanted to lens the data to another schema that also includes additional fields you could store that somehow on the same object.
• I think these fields for not the default schema would be hidden by default and only included if you specifically asked for the object in that schema
• This would allow someone to have a multi-faceted view of a single object

[wittjosiah]

We've begun to lay the ground work for some sort of future lensing solution with how we're handling universal drag & drop in Composer currently. #4696 introduces an abstraction that allows plugins to register data transformations between different types. It is currently very limited and primitive but we believe provides us a good abstraction for which we could layer in more comprehensive lensing later.