Support both object inlining and referencing

[Silvanoc]

According to the documentation, you can currently compose objects (also known as aggregates) from other objects by either referencing the constituent objects or embedding (inlining) them.

Why is this design so exclusive? I want the flexibility to specify that constituents can be either inlined or referenced, allowing for conversion between the two formats.

Here's an example to illustrate my goal:

• An IoT Gateway within an isolated network can discover and model connected devices, gathering information about modules, manufacturers, etc.
• The IoT Gateway lacks the resources and connectivity to separate an aggregate into its constituent parts, so it provides all information with constituents inlined.
• These models are then sent to a cloud service, which can and should split the aggregate into its constituents. This is useful for deduplication and canonicalization.
• Depending on the needs of the consumer, the models can be provided as inlined aggregates or as aggregates with references to their constituents.

In the end, if the constituent has the right range, what is the difference between inlining and referencing?

[Silvanoc]

I know I already asked this question somewhere, but I could not find it anymore. I suspect that it is a comment hidden in some issue or pull-request. IMO this is a design decision that is worth a discussion by itself. I hope it is fine for you.

[cmungall]

Why is this design so exclusive

Simply because by making the structure more predictable, we make it easier for consumers of the data. This may not make much sense from a relational model or graph database perspective, but for tree-based formalisms (JSON, YAML, Pydantic objects, ...) it's much harder for me as a user of data to defensively code against having the data be in two different possible shapes. The current design also makes the mapping to json-schema more predictable (see existing discussions about supporting Referential Integrity in json-schema)

Some of this could be obviated by using some alternative to Pydantic that takes care of automated de-referencing. In fact this is possible using a LinkML ObjectIndex. See

• Should automatic dereferencing of non-inlined objects be supported? #2176

for discussion. But this doesn't help if a consumer of the JSON/YAML I produce isn't using that.

I want the flexibility to specify that constituents can be either inlined or referenced, allowing for conversion between the two formats

This is totally reasonable. LinkML is designed for a variety of use cases. Perhaps in some scenarios the flexibility is a major win, and there are fewer concerns with making the shape of the data less predictable - it sounds like this is true for you.

In slack you provide two solutions:

• linkml-map transformation both ways
• support for "referenced OR inlined" objects, since now it is an "referenced XOR inlined" objects (this would be my favorite)

On reflection, I agree and linkml-map would be overkill for this (but there may be other scenarios that complicate the picture - e.g. having the id be optional).

Remaining questions:

• should the representation be coerced to one form, or should reads/writes be structure-preserving?
• whether to bundle the "XOR" support with general support for different object serialization forms (see Proposed metamodel enhancement: use enum to determine the different collection serialization types #1251)

[Silvanoc]

I think I understand your point better now. And I think that if I scope my use-case better, the realization might become easier:

1. My main requirement is having the possibility of validating the "inlined objects" and converting them to "referenced objects".
2. I do not need support for both inlined and referenced objects as equivalent. Inlined objects would be like an accepted exception on input where referenced objects are expected.
3. I could accept providing dummy IDs or placeholders in the inlined objects, that get replaced when converting them to referenced objects. But it would be even better not needing an ID, which gets created and assigned on conversion.

I could think of following approach:

1. New flag on linkml-validate to handle inlined where referenced was expected (something like --accept-inlined-instead-of-referenced, but shorter).
2. New flag on linkml-convert to convert inlined into referenced wherever needed (something like --disassemble-aggregate), without failing.
3. Optionally we could create a new attribute on SlotDefinition to specify that the range of the defined slot might exceptionally appear inlined and should get referenced (something like accept_inlined: boolean, defaulting to false). This attribute could help making linkml-validate and linkml-convert even more precise, accepting the exception only on marked slots.

I would not expect equivalence between both inlined and referenced in the models (which is what makes the whole thing difficult) throughout the whole life of an object, only an exceptional way to provide an object meant to be converted (into using referenced objects, wherever specified) for further handling.

Getting JSON-Schemas that can cope with it for the validation is pretty easy using anyOf.

I do not know Pydantic good enough, but I would expect that a try-error process could handle it. First try the validation without any exception, wherever Pydantic fails expecting a referenced and finding an inlined, replace with inlined and re-validate. I do not know if it would be possible to make Pydantic coerce inlined to referenced to avoid the whole try-error.

[cmungall]

I'm not sure the Pydantic automatic handling would be so easy. You might be able to auto-coerce to an ID but if you need to make sure that the referenced object is preserved elsewhere it becomes more difficult..

[sneakers-the-rat]

I think another way through is to make it so any slot can be provided by reference in the data, and the schema should only care about is the shape of the data, ie. Whether something is an array or a mapping. The current inlining system mixes both of these concerns in a way that I have had a decent bit of trouble wrapping my head around and I think is a decently confusing thing for newbies.

This would require improving the loader/dumper system rather than a metamodel, and imo I think revamping the default linkml yaml data model is one of the more pressing needs in a few ways. (Most recent e.g. at the bottom of https://github.com/orgs/linkml/discussions/2020#discussioncomment-11792951 )

whether data is inlined or referenced should be irrelevant to the schema. So if we had a linkml data model that was like linkml schema in that each element had an unambiguous identifier and class within the scope of a dataset, this:

linkml_meta:
  imports:
  - name: MySchema
    uri: example.com/schema-used-in-this-dataset

MyData:
  is_a: MySchema.MyClass
  values: [1,2,3,4,5]

Should be unambiguously equivalent to this

linkml_meta:
  imports:
  - name: MySchema
    uri: example.com/schema-used-in-this-dataset

AnotherInstance:
  is_a: MySchema.AnotherClass
  whatever: [1,2,3,4,5]

MyData:
  is_a: MySchema.MyClass
  values: @AnotherInstance.whatever

Resolving the references would happen at loading time, and whether the notion of the reference is preserved after loading time would be framework dependent. E.g. currently the pydantic models with class ranges that have identifiers have annotations that are literally str - the identifier of the class instance (and iirc aren't even a Union[ClassName, str] so you actually can't hydrate the reference?). As with creating an induced schema, the reference form is a reduced form, and the loader "induces" the dataset by hydrating the references. So something like this would happen at loading time

AnotherInstance = AnotherClass(whatever = [1,2,3,4,5])
MyData = MyData(values = AnotherInstance.whatever)

And then be passed on to the particular framework we're loading it into.

What is currently inlining should, in my opinion, be essentially removed, and instead be replaced by allowing types to be parameterized (aka generics for linkml types). This resolves the question of ambiguity in what shape a data takes by unifying it with the type system. Currently it is imo more confusing to have three possibilities for shapes for a single range (str reference, inlined array, inlined map) depending on the value of inlined and inlined_as_list. By separating the two things that inlined does, modify shape and referencedness/literalness, we make range the unambiguous declaration of shape and references are made orthogonal by allowing them anywhere.

# instead of
my_slot:
  range: MyClass
  multivalued: true
  inlined: true
  inlined_as_list: false

# declare range as a map
my_slot:
  range:
    map:
      key: str
      value: MyClass

Representation in RDBMS is a join table with a key column, representation in RDF would be a bnode with key and value predicates (not sure how maps are represented in RDF currently).

[Silvanoc]

I have somehow the feeling that @sneakers-the-rat expectations and mine are well aligned, what probably comes from an OOP background. Whereas other people in the community have a rather ontology and less software-heavy background resulting in different expectations.

I think another way through is to make it so any slot can be provided by reference in the data, and the schema should only care about is the shape of the data, ie. Whether something is an array or a mapping. The current inlining system mixes both of these concerns in a way that I have had a decent bit of trouble wrapping my head around and I think is a decently confusing thing for newbies.

That's a good point and this approach is well aligned with my expectation.

Still @cmungall is right in something: implementations caring about the shape of the data, must be able to cope with both scenarios (inlined or referenced objects). It is undeniable that it adds some complexity, but I consider it dispensable.

As of now the metamodel is specifying not only the shape of the data, but also about the representation of the data. That's the root-cause for the difficulties in the discussion. If we state in the specification that the schema is mostly focused on data structure and partially on representation, then we have paved the way to address this topic. We can say that not matter what the metamodel says, the possibility of having a reference to a separate data representation is always implicitly possible.

This would require improving the loader/dumper system rather than a metamodel, and imo I think revamping the default linkml yaml data model is one of the more pressing needs in a few ways. (Most recent e.g. at the bottom of https://github.com/orgs/linkml/discussions/2020#discussioncomment-11792951 )

I'm not sure I understand what you mean by "revamping the default linkml yaml data model". Do you mean regenerating the YAML representation of the metamodel? Apparently I has not be done for a long time and in that sense LinkML is currently not really eating its own dog food.

whether data is inlined or referenced should be irrelevant to the schema. So if we had a linkml data model that was like linkml schema in that each element had an unambiguous identifier and class within the scope of a dataset, this:

linkml_meta:
  imports:
  - name: MySchema
    uri: example.com/schema-used-in-this-dataset

MyData:
  is_a: MySchema.MyClass
  values: [1,2,3,4,5]

Should be unambiguously equivalent to this

linkml_meta:
  imports:
  - name: MySchema
    uri: example.com/schema-used-in-this-dataset

AnotherInstance:
  is_a: MySchema.AnotherClass
  whatever: [1,2,3,4,5]

MyData:
  is_a: MySchema.MyClass
  values: @AnotherInstance.whatever

I agree in considering that it should be irrelevant to the schema if data are inlined or referenced. That is in fact what I'm requesting 🙂 What I am not sure I agree with is the possibility of having a reference in a slot to another objects slot... I would rather split the discussion in two: one about full object references and another one about advanced references. I mean once to enable having a reference to an object slot, one might ask to enable something like @AnotherInstance.whatever[0].

Resolving the references would happen at loading time, and whether the notion of the reference is preserved after loading time would be framework dependent. E.g. currently the pydantic models with class ranges that have identifiers have annotations that are literally str - the identifier of the class instance (and iirc aren't even a Union[ClassName, str] so you actually can't hydrate the reference?). As with creating an induced schema, the reference form is a reduced form, and the loader "induces" the dataset by hydrating the references. So something like this would happen at loading time

AnotherInstance = AnotherClass(whatever = [1,2,3,4,5])
MyData = MyData(values = AnotherInstance.whatever)

And then be passed on to the particular framework we're loading it into.

Agree on this.

What is currently inlining should, in my opinion, be essentially removed, and instead be replaced by allowing types to be parameterized (aka generics for linkml types). This resolves the question of ambiguity in what shape a data takes by unifying it with the type system. Currently it is imo more confusing to have three possibilities for shapes for a single range (str reference, inlined array, inlined map) depending on the value of inlined and inlined_as_list. By separating the two things that inlined does, modify shape and referencedness/literalness, we make range the unambiguous declaration of shape and references are made orthogonal by allowing them anywhere.

# instead of
my_slot:
  range: MyClass
  multivalued: true
  inlined: true
  inlined_as_list: false

# declare range as a map
my_slot:
  range:
    map:
      key: str
      value: MyClass

I agree that inlined, inlined_as_list, their combinations, their defaults and simple dictionaries are really confusing/difficult topics to deal with in LinkML.

I prefer your proposal with backwards compatibility, but I am concerned about the needed effort for such a restructuring.