Data discovery

[lecoqlibre]

If we want multiple applications to exchange data in a re-usable way, we need to let one app to know where data is hosted on a second app. Example: tell that route (URL) points to the Enterprise LDP container, that other route points to the SuppliedProduct LDP container and so on.

I see some options here:

1. define a common fixed data structure for all DFC applications. Example: Enterprise will always be stored in the /agents/enterprises/ container, supplied products will always be stored in the /defined-products/supplied-products/ container and so on. The root container would be defined by a new ontology predicate like dfc-t:rootContainer: "https://example.org/user/".
2. define new ontology predicates for each kind of resources. Example: the dfc-t:routeOfEnterprise would be used to express where Enterprise can be hosted, the dfc-t:routeOfSuppliedProduct would be used to express where Supplied products can be hosted on so on.
3. Use an index like the TypeIndex used by the Solid community. This index would contain a registration for each kind of resource we have. It could look like this (turtle):

@base <https://example.pod/username/datafoodconsortium>.
@prefix solid: <http://www.w3.org/ns/solid/terms#>.
@prefix dfc-b: <https://www.datafoodconsortium.org#>.

<>
    a solid:TypeIndex;
    a solid:ListedDocument.

<#addresses> a solid:TypeRegistration;
    solid:forClass dfc-b:Address;
    solid:instanceContainer </addresses/>.
    
<#persons> a solid:TypeRegistration;
    solid:forClass dfc-b:Person;
    solid:instanceContainer </agents/person/>.
    
<#enterprises> a solid:TypeRegistration;
    solid:forClass dfc-b:Enterprise;
    solid:instanceContainer </agents/enterprises/>.
    
<#customer-categories> a solid:TypeRegistration;
    solid:forClass dfc-b:CustomerCategory;
    solid:instance </agents/customer-categories.ttl>.
    
<#catalogs> a solid:TypeRegistration;
    solid:forClass dfc-b:Catalog;
    solid:instanceContainer </catalogs/>.
    
<#catalog-items> a solid:TypeRegistration;
    solid:forClass dfc-b:CatalogItem;
    solid:instanceContainer </catalog-items/>.
    
<#functional-products> a solid:TypeRegistration;
    solid:forClass dfc-b:FunctionalProduct;
    solid:instanceContainer </defined-products/functional-products/>.
    
<#supplied-products> a solid:TypeRegistration;
    solid:forClass dfc-b:SuppliedProduct;
    solid:instanceContainer </defined-products/supplied-products/>.
    
<#technical-products> a solid:TypeRegistration;
    solid:forClass dfc-b:TechnicalProduct;
    solid:instanceContainer </defined-products/technical-products/>.
    
<#orders> a solid:TypeRegistration;
    solid:forClass dfc-b:Order;
    solid:instanceContainer </orders/>.
    
<#sale-sessions> a solid:TypeRegistration;
    solid:forClass dfc-b:SaleSession;
    solid:instanceContainer </sale-sessions/>.

4. Follow the recommendations of the Solid interoperability panel with data registrations and shape trees.

I think we should consider the investigation of the option 4 while the option 3 could be easily set up.

Currently we use a TypeIndex in our Solid specifications.

[RaggedStaff]

I think it makes sense to do as you suggest @lecoqlibre - to go with option 3 for now.

Would we require all platforms to implement the TypeIndex on a fixed URL (e.g. /.well-known/TypeIndex.rdf) ?

[lecoqlibre]

+1 for option 3.

We could use a .well-known URL but I would like to propose to use a WebID instead as it would be a more re-usable solution.

A WebID is just a URL which contain a link to a profile document in which the TypeIndex would be defined. It's the current solution used by the Solid community instead of Decentralized Identifier (DID) which is a W3C standard but far more complicated to implement. WebID is a lightweight solution that could be used prior to DID.

WebID spec: https://www.w3.org/2005/Incubator/webid/spec/identity/
WebID wiki: https://www.w3.org/wiki/WebID
WebID wikipedia: https://en.wikipedia.org/wiki/WebID
DID W3C announcement: https://www.w3.org/press-releases/2022/did-rec/
DID specs: https://www.w3.org/TR/did-core/

[RaggedStaff]

Also wondering how this ties in with datafoodconsortium/.github#5 (Platform Discovery) ?

[lecoqlibre]

This topic is focused on how to find data once we have found the platform.

[simonLouvet]

I agree with @maxime that the entry point on platforms should enable the retrieval of containers for data querying, rather than presenting a graph with the user as the root and their data as predicates in a tree structure. A method more aligned with the Solid standard would be for this entry point to return the profile linked to the user's WebID as a "Solid WebID Profile," with profile data containing links to the containers [https://solid.github.io/webid-profile/]. The containers could be represented using the VoID ontology [https://www.w3.org/TR/void/]. Another solution could be for the server to return the containers without concerning itself with the notion of a user, on an endpoint like _wellKnown/void or equivalent, as is done by semapps servers. These solutions could be complementary.

[RaggedStaff]

@simonLouvet - I get confused by the many standards out there... what's the difference between VoID & DCat (2) ?

[simonLouvet]

That's a very good question :-) to be honest, I don't know, I think it's worth exploring.

[RaggedStaff]

Ok. I agree it makes sense to use a single point of entry for a platform.

We could use a .well-known URL but I would like to propose to use a WebID instead as it would be a more re-usable solution.

Ok... which if these standards are you proposing to use: WebID OIDC; Solid OIDC; WebID-TLS ? Are these all extensions to OIDC ? Would that require a change/clarification to the standard to utilise WebID + OIDC, rather than just OIDC ?

I'm concerned that we're introducing another concept (WebID) - we currently reference foaf in the ontology, but only to provide Contributor information. Would we need to rebuild the Agent information to reference foaf concepts (arguable we should've done that in the first place 🤨 ) or can we build the WebID data as an extension of the existing ontology ?

Another solution could be for the server to return the containers without concerning itself with the notion of a user, on an endpoint like _wellKnown/void or equivalent, as is done by semapps servers.

Would this still be encapsulated in the OIDC user access layer - to provide data security ?

Also worth noting the parallel discussion around use of TypeIndexes / data structures - #3 @balessan - would love to hear your thoughts.

[balessan]

Ok... which if these standards are you proposing to use: WebID OIDC; Solid OIDC; WebID-TLS ? Are these all extensions to OIDC ? Would that require a change/clarification to the standard to utilise WebID + OIDC, rather than just OIDC ?

To my knowledge: Solid-OIDC is the renaming of WebID-OIDC, they are strictly equivalent.
WebID-TLS has been abandoned. The general consensus is to base your distributed authentication system on OIDC as it has been bulletproofed, valid for both the Solid and the more generic Data-spaces ecosystem.

[lecoqlibre]

Introducing WebID and profile

In the following proposal, any DFC user and platform would have their own WebID and profile(s). When dereferenced, a WebID would give information (profile) which can be used to find data.

The WebID could be used with Solid-OIDC for authentication and authorization instead of the email (currently used in the DFC token).

Platform WebID and profile

For instance, a public WebID of a platform (https://ofn.org/card#me) could look like the following and contain links to pieces of data:

# This is the WebID profile document
<> a foaf:PersonalProfileDocument; # (note: there is a proposal to rename it to solid:profileDocument).
  foaf:primaryTopic <#me>.
    
<#me> a foaf:Agent; # this is the WebID profile, containing useful information to discover data.
  foaf:name "Open Food Network";
  pim:preferencesFile <link/to/the/preferences/file>; # could be used to restrict access to some information (see below).
  solid:publicTypeIndex <link/to/the/public/type/index>; # we can use TypeIndex or whatever vocabulary to advertise some index documents.
  dfc-b:EnterpriseByNameIndex <link/to/the/index/document>. # or we could list the indexes directly in the WebID using custom predicates which is linked to specific indexes like this one.

If the platform want to restrict access to some information, it could move them in a private or restricted (ACL) preferences file:

<> a pim:ConfigurationFile.

<https://ofn.org/card#me> 
  solid:publicTypeIndex <link/to/the/public/type/index>; # like before we can use TypeIndex or any vocabulary of our choice.
  dfc-b:EnterpriseByNameIndex <link/to/the/index/document>. # or direct predicates.

User's WebID and profile

The following is an example of an user's WebID (https://webid.provider/user/card#me):

<> a foaf:PersonalProfileDocument;
  foaf:primaryTopic <#me>.
    
<#me> a foaf:Person;
  foaf:name "Bob";
  pim:storage <https://socleo.fr/user/dfc/> <https://ofn.org/user/dfc/>; # link to the user's data on different platforms.
  pim:preferencesFile <link/to/the/preferences/file>. # Used to restrict access to sensitive information (see below).

To restrict access to sensitive information, the user did move some information in the preferences file, protected by ACL:

<> a pim:ConfigurationFile.

# Here we are extending the profile of the user...
<https://webid.provider/user/card#me>
  dfc-b:agent # ...mapping the user to his corresponding dfc-b:Person on the different platforms he uses.
    <https://socleo.fr/user/dfc/agent/person/user> 
    <https://ofn.org/user/dfc/agent/person/user>.
		
# We also define a configuration to find the data.
<#config> a dfc-b:Configuration;
  solid:privateTypeIndex <link/to/the/private/type/index>; # TypeIndex could be used to lists containers.
  dfc-b:enterpriseContainer <link/to/the/container>; # or use DFC custom direct predicates like this one.
  void:somePredicate <...>; # or we can use void datasets or any other vocabulary of our choice.
  dfc-b:EnterpriseByNameIndex 
    <https://socleo.fr/user/dfc/enterpriseByNameIndex> 
    <https://ofn.org/user/dfc/enterpriseByNameIndex>.

[RaggedStaff]

Thanks @lecoqlibre This seems like a solid start to a proposal! 😉 Just want to clarify my understanding & have a few queries:

So this suggestion requiries each platform to publish a single WebID profile (for the platform, as a foaf:agent) to facilitate the network engaging with & discovering data on that platform ? I think that's clear & sensible.

I'm less clear about the User Profile proposal - are we requiring platforms to publish those profiles for all of their users, or will we look to publish them on a central server somewhere (maybe the OIDC server 🤨 ) ?

The first option seems messy - a user will have a profile on each platform they use: how do we ensure they're all joined up with minimum duplication ? The second option seems in theory more elegant but inherently risky: how do we determine which platforms a user is active on/ensure we're publishing up to date information for each platform storage access point ?

I'm also unsure how we would most efficiently determine all the platforms a user has profiles on, either if we're publishing the data on each platform or centrally. What are you recommendations for that @lecoqlibre ?

[simonLouvet]

In the current state, the user does not have a pod and therefore no unique location to store their profile. To my knowledge, members of the Data Food Consortium (DFC) are not ready for the user's profile to be hosted on their own platform (of the user, POD). If a user is on multiple platforms, they have a user profile on each platform. This profile could provide the URLs for accessing data from the queried platform.

I think that the most coherent approach, given the current constraints (no personal pod), is for the platforms to act as agents and respond to a yet-to-be-defined URL with their profile including the data access URLs (LDP containers). When calling the container URLs, the platforms would only return the data that the connected user (OIDC) has access to. This approach is very close to what Semapps does but moves closer to the SOLID specifications through the application's profile in the client-to-client logic.

[lecoqlibre]

So it seems we have a consensus about using a WebID + WebID profile document for platforms, that's a good news!

Regarding the users WebID, there is no problem to have multiple profiles as a user, like the Solid WebID Profile spec says:

An extended Solid profile is an additional profile document that provides supplemental description to a Solid profile to help distribute and manage information about a WebID for use in different circumstances. Extended profile documents can be used for various purposes, such as organizing information or limiting access to certain data.

Would not it be fantastic to log in on Socleo or OFN with a Solid WebID? So we can integrate both ecosystems!!!

If the platforms are "not ready" to move users data on a User's POD, one alternative would be that platforms act as PODs (only the storage part)! It could not cost so much to platforms as they could perhaps only implement the Request Flow of Solid OIDC (need to check in details).

Indeed, platforms could only serve a Solid storage and let the authentication process to a Solid dedicated server.

So users would have their POD + WebID on a Solid server of their choice. This way they will have one WebID pointing to platforms like we said before : pim:storage <https://socleo.fr/.../> <https://ofn.org/.../>;. This will allow both non-Solid users as well as regular Solid users to connect to Socleo and OFN using their WebID.

Plus, there is already a library to authenticate with Solid OIDC that platforms can already use to implement the client part.

What do you think @RaggedStaff @simonLouvet?

[simonLouvet]

As I said earlier, as far as I know, the platforms are not ready to handle solid authentication. The consensus was OIDC. If you want to convince them, you have to invite them and explain it to them.
A discussion on this subject is underway on the archipelago repo: assemblee-virtuelle/archipelago#154

[lecoqlibre]

Indeed, it is maybe the time to propose to implement Solid-OIDC as a way to tie the authentication to the identity on the Web. I would like to make a demo and a tutorial on how to implement it in platforms.

AFAIK Solid-OIDC was part of the DFC's roadmap: https://datafoodconsortium.gitbook.io/dfc-standard-documentation/technical-specifications/protocols-specifications.

[lecoqlibre]

Today during our Tech meeting with @RaggedStaff, @RachL, @simonLouvet and @lin-d-hop I presented the WebID proposal which is linked to a Solid-OIDC proposal (only the Resource server + Authorization server part of it).

Garethe and Simon said that platforms are not ready to use Solid-OIDC although we did not discuss what it is and what is required to implement it (I started to implement a Proof Of Concept which showed me that is "easily" doable). Lynne said that we should make the standard work without changing it ("rewriting" was the word I think).

It seems we want a solution right now with as few changes as possible and considering Solid-OIDC as the way to go but in a future version of the standard. Simon and Garethe was saying that Solid-OIDC has not been discussed and agreed before and we need a solution that is working with the current classic OIDC authentication. It's true but I was thinking it was maybe the time to discuss about it seriously because we gonna have to change the standard anyway.

And as an architech I prefer to take the time to design things in a long term thinking because I know the result will be stronger, easier and cheaper at the end. But I also understand real-life limitations and how short/mid term solutions could be attractive. Even if they can work we should be careful, especially when building a standard, because some changes will have a huge impact if they are not made at the right time (authentication and data storage is like the foundations of a building).

But let's go back to the meeting.

Some opinion was expressed on these subjects:

Subject	Garethe	Simon	Maxime
1. Have a platform WebID to advertise high level routes (like enterprise container is here, products are here and so on).	agree	agree	not confortable with
1b. Have a .well-known endpoint to advertise high level routes.	agree	agree	not confortable with
1c. Have a platform WebID that would be used to index the dfc-b:Enterprise of the users who allow it. Each user would have it's own space on the platform (LDP container).	disagree	disagree	agree
2. Have a fixed structure for containers like some kind of data format.	disagree	can live with it	agree
2b. The fixed structure for containers would be a recommendation for now.	agree	agree	can live with it
3. Users would have WebID.	agree	agree	agree
3b. The WebId of an user would be hosted on each platform.	agree	agree	not confortable with
3c. The WebId of an user would be hosted by the WebID provider of the user.	disagree	disagree	agree

I'm little nervous about the point 3b. This point is related to the fact that we need a way to know all the other platforms a user is using and where its data (the data of this particular user) is stored on these platforms. The question is where to save that information?

So if we choose the point 3b. we gonna have a WebID for the same user on all the platforms. We gonna have to link them together, using the owl:sameAs predicate. While a same WebID should be unique, one can agree that copies can exist.

Indeed, these WebIDs will be copies of each other, they should be identical. So we need these copies to be synchronized. And here we hit the question of synchronization with all the issues and complexity it has (network errors, authorizations, integrity, etc).

Some possible alternatives to replication that could work with the current DFC OIDC:

1. The user chooses one platform to host the WebID, and only one WebID should exist (no copies).
2. The WebID is stored at the OIDC provider. I think there is a KeyCloak API that can be used to store attributes for users.
3. We don't use WebID for users and let the platforms provide their own user_info API endpoint (which could be defined by the standard) to provide the data of the authenticated user (via the Authorization header). This is what is currently used except that the returned user data would share a better structure. The user endpoint could follow the Solid WebID profile document.

I would say the best option is 3 given the constraints. The first option seems unfair and the second is not generic and not realy secure.

Simon is also proposing to use ActivityPub to synchronize platforms. Using this mechanism the option 3b could also be used. But on my side I don't realy like this idea of synchronization. I think it will be overcomplicated to manage to get the ecosystem synced. And the Solid project and its philosophy resolve this issue, after almost ten years of research. WebID and Solid-OIDC are well designed, simple and fit the Web requirements while being secure (the Demonstrating Proof of Possession is also ten years of research in the cyber-security community).

---

Here I paste the content I wrote in the meeting notes:

Abstract: If we want multiple applications to exchange data in a re-usable way, we need to let one app to know where data is hosted on a second app. Example: tell that route (URL) points to the Enterprise LDP container, that other route points to the SuppliedProduct LDP container and so on.

We have a consensus about using a WebID + WebID profile document for platforms. When dereferenced, a WebID would give information (profile) which can be used to find data.

For describing the data that can be found, I think we should have in fact:

• An entry point for a DFC installation (like a file) (option 1 of the discussion)
• A fixed structure like a data format

Because the main high level TypeIndex should only advertise the High Level data types (and not all types like we did in option 3 of the discussion). We should instead think in term of data format. The user should not choose where he wants to store a particular RDF resource in the installation itself. He choose the installation on which he wants to work on but that’s all. The application should know the internal organisation but not the user. While the standard could use a TypeIndex to reference the internal resources of a installation this is not necessary useful.

So the WebID of a platform could reference a public TypeIndex listing all (those who agreed) the DFC (main) Enterprise the platform is hosting (or preferably all the DFC installations but we need a way to link a installation to the main enterprise(s) of the installation). This enables data discovery: other platforms would be able to know which farms are hosted by the platform by following the links found in the TypeIndex and try to dereference whatever RDF resource(s) it might need to display each farm information on its interface.

So platforms would host the users information in dedicated containers. For instance, the user “Tom Jones” could have a DFC installation at https://someplatform.org/tom.jones/. This container would host all the RDF resources of the installation including the (main) Enterprise of Tom Jones. If many installation are supported, platforms could host these installations in https://someplatform.org/tom.jones/installation1 and https://someplatform.org/tom.jones/installation2.

Having a fixed tree structure is like having some kind of file format or package format. Example: Enterprise will always be stored in the /agents/enterprises/ container, supplied products will always be stored in the /defined-products/supplied-products/ container and so on. The installation would be defined by a new ontology predicate like dfc-b:installation.

Users should have their own WebID too. This would allow him to make the link between the platforms he is using. The WebID should also be used to login on platforms. That would allow Access Control at the platform level. This requires platforms to get closer to the Solid protocol by implementing Solid-OIDC (primer) which would fix the security issues we have. Solid-OIDC is well designed and use the new Demonstrating Proof of Possession (DPoP) mechanism which has been actively seeked for more than a decade by the OAuth ecosystem. The history is showed in this presentation.

While platforms seems to not be ready to move the users data on Solid PODs yet, they could act as OAuth Resource Server and Authorisation server. Indeed, the Identity server could be delegated to a third party like DFC maybe? Indeed the DFC could propose a WebID service! So users would create their WebID on DFC and would be able to login on all the DFC applications they would use. In the same time they would also be able to use Solid applications!

At the platform level the implementation of a Resource + Authorisation server is doable. The DFC could provide some tutorials and tools to make this even easier and faster. Maxime worked several days on this topic and will show some code at https://github.com/datafoodconsortium/solid-oidc-proposal.

[RachL]

Thanks @lecoqlibre ! I was about to share my notes here and I'm really glad I didn't press enter before you did 😅

[simonLouvet]

Indeed, these WebIDs will be copies of each other, they should be identical. So we need these copies to be synchronized. And here we hit the question of synchronization with all the issues and complexity it has (network errors, authorizations, integrity, etc).

I not aggre each platfrm have divergentinformations about user. webId in a platform is the user as seen by the platform.

[RachL]

@RaggedStaff, @lecoqlibre , @simonLouvet @lin-d-hop and I have met again. On this topic we have agreed that:

• SolidOIDC is what we should use in our standard but it is a big breaking change: we should introduce it in the next major version
• In the meantime webID profile can be introduced as minor version - however platforms would not be required to do the change and can keep hardcoded URL as they do now, and only switch when the time is appropriate for them.

[RachL]

Additional info:

• A SolidOIDC platform cannot communicate with an OIDC platform, therefore we agree to tolerate one user per platform. @lecoqlibre is going to document this on gitbook
  We want in the future to only support SolidOIDC, but that's a big change for existing platform's: @lecoqlibre is going to look / demo how they could do it

[RachL]

Issue to track the documentation update: #17