Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Generic LDNodes and RO-Crate parsing rework #496

Open
wants to merge 56 commits into
base: main
Choose a base branch
from
Open

Generic LDNodes and RO-Crate parsing rework #496

wants to merge 56 commits into from
+18,091 −1,234

Conversation

HLWeil
Copy link
Member

@HLWeil HLWeil commented Feb 27, 2025 •
edited
Loading

Reworked Generic JSON-LD Objects

This is a generic JSON-LD implementation that represents those aspects of the JSON-LD specification which are relevant for handling ARC-RO-Crate:

  • Datamodel
    • LDContext (Contains mappings from compacted terms to extended IRIs)
    • LDNode (Our basic objects with @id, types and fields)
    • LDGraph (Basically a collection of nodes; Top-level object in the RO-Crate)
    • LDRef (Object containing only @id, meant for linking)
    • LDValue Expanded, self-contextualizing values
  • Algorithms
    • Flatten
    • Compact
    • WIP Unflatten
  • Generic JSON encoders and decoders

Added ARC-RO-Crate profile layer

This layer acts like a ARC-RO-Crate view on the generic JSON-LD model. All actual handled objects are either FSharp primitive types or generic JSON-LD objects. Through making use of static methods which in turn use static IRIs, a loose typing is emulated. These static IRI mostly consist of types and properties from schema.org and bioschemas.org.

  • Added a set of static classes, containing functions for

    • getting properties
    • setting properties
    • generating default @ids
    • validating given LDNode against requirements for this type
    • create functions for LDNodes matching the requiremenets for this type

  • For the following types/profiles:

    • Comment
    • DefinedTerm
    • PropertyValue
    • PostalAddress
    • Organization
    • Person
    • ScholarlyArticle
    • Sample
    • File
    • LabProtocol
    • LabProcess
    • Dataset

  • Added contexts of RO-Crate specification 1.1 and 1.2DRAFT

  • Added context for all other fields exclusive to the ARC-RO-Crate profile

ARC Scaffold <-> ARC-RO-Crate conversion layer

Added functions for Conversion between Scaffold and RO-Crate representation types.
E.g.

  • Ontology Annotation <-> DefinedTerm
  • ArcTable <-> LabProcess Sequence
  • ArcInvestigation <-> Dataset

Added top-level members to ARC Scaffold object to parse to and from an RO-Crate JSON-LD metadata string.

To-do:

  • Top level parsers
  • DOI and PubmedID as identifier
  • Bioschemas context
  • Some @id are empty (LabProtocol)
  • Top-level roundabout tests
  • Link datafiles instead of fragment selectors from datasets

To-do after this PR is merged:

  • Datamap support
  • Make use of @list for ordered process inputs and outputs
  • CI job, running the RO-Crate output for ARC(Prototype?) against RO-Crate validator

closes #326
closes #400
closes #429
closes #465
closes #494
closes #495

kMutagene and others added 30 commits February 10, 2025 15:30
@kMutagene @HLWeil
Add LDObject.tryFromDynamicObj + tests
b434ade
@kMutagene @HLWeil
LDObject: model @type/additionalType as arrays:
e5efa2f 
- Json: parse `@type`/`additionalType` from either string (-> singleton) or array field, always write array
- add tests
@kMutagene @HLWeil
Fix running single test projects
556adb3
@HLWeil
some LDObject parsing fixes
10ab9a3
@kMutagene @HLWeil
wip LDObject from dynamic obj
0eab82a
@HLWeil
update DynamicObj dependency and fix for LDObject from DynamicObj
7de9bdf
@HLWeil
fix LDObject from DynamicObj function
0ad4df6
@HLWeil
add basic ld context resolving
bc55487
@HLWeil
add ro-crate 1.1 context
015410a
@HLWeil
make ldcontext term resolution recursive
c69106a
@HLWeil
split up LDObject into LDNode, LDRef and LDValue
e7a8c6c
@HLWeil
add basic LD form algorithms
7b62486
@HLWeil
start working on basic generic LDNode access classes
4222bc3
@HLWeil
add basic LDContext tests
664b86b
@HLWeil
add context copy tests
bda9098
@HLWeil
start working on ldnode tests
bd11928
@HLWeil
add basic ldnode tests
1c43c6d
@HLWeil
some fixes for ldnodes according to tests
1ca9ac9
@HLWeil
add ldnode compact and flatten tests
27db166
@HLWeil
start on ldnode json io
477f921
@HLWeil
add various ldobject json parsing tests
030c9c3
@HLWeil
add basic ldgraph test and context parsing hotfixes fixes
c9c6020
@HLWeil
add some static types from isa-ro-crate profile
48efd4f
@HLWeil
add labprocess and labprotocol rocrate static classes
59edad3
@HLWeil
start working on scaffold RO-Crate type conversion
05d5ea6
@HLWeil
finish up preliminary version of Scaffold ROCrate Process conversion
fdb4f16
@HLWeil
add and update first bulk of RO-Crate Scaffold conversion tests
633438c
@HLWeil
some LabProcess Table conversion fixes
90647e0
@HLWeil
fix remaining LabProcess ArcTable conversion errors
e1dc2cb
@HLWeil
start working on top level RO Crate types and conversions
f18abad
@HLWeil
Copy link
Member Author

HLWeil commented Feb 27, 2025

@floWetzels could you also take a look onto whether #326 is actually fixed or not (e.g. by inspecting a written RO-Crate metadata json)

This was referenced Feb 27, 2025
Open
Open
@floWetzels
Copy link
Collaborator

I tested the metadata json file generated for the ArcPrototype with the rocrate-validator. I found the following errors that need to be fixed:

  • Studies und assays have wrong ids - the path to the isa-xlsx file. It should be the path to the directory ending with a /.
  • Studies and assays are missing the hasPart property
  • The root data entity has no license
  • The root data entity has no datePublished

Furthermore, I noticed the following when parsing the file manually:

  • Paths must be relative to the root of the ro-crate
  • Ad-hoc ids (e.g. for processes, paramters, parameter values) should begin with a # symbol
  • The id of PropertyValue objects looks weird to me. It contains the propertyID, which could be a potential source for errors. Previously, it was generated only from name and value, if I am not mistaken.

@floWetzels
Copy link
Collaborator

Looks a lot better now! The validator accepts the RO-Crate. When looking through the file, I noticed a few more things:

  • Some @id values contain spaces. This appears almost throughout all types, e.g. Organization, PropertyValue, Process, Protocol, Sample etc. This is not necessarily a problem, the json-ld playground is able to handle it for example. It is also not completely avoidable, since file paths can contain spaces. I would prefer to replace them in ad-hoc ids though. The old export did so with a simple replace statement.
  • For some types, ad-hoc ids are generated although a URI or similar identifier exists. I found two examples in the ArcPrototype, DefinedTerm and Person. A DefinedTerm should use its termCode as id, if it exists. For Person, I found an instance where an email address exists, but an ad-hoc id was assigned.
  • Some termCode entries of DefinedTerm contain multiple URIs separated by an underscore. This should not be the case, right? Here's an example: "termCode": "http://purl.obolibrary.org/obo/scoro_http://purl.org/spar/scoro/principal-investigator".

@HLWeil
Copy link
Member Author

HLWeil commented Mar 4, 2025

I incorporated your suggested fixes and included the test file in d3d786e

@HLWeil HLWeil marked this pull request as ready for review March 4, 2025 12:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
3 participants
@HLWeil @floWetzels @kMutagene