Skip to content

Commit

Permalink
Process comments
Browse files Browse the repository at this point in the history
  • Loading branch information
@ChristopherSpelt
ChristopherSpelt committed Apr 3, 2024
1 parent c04d163 commit 5ae3551
Showing 1 changed file with 64 additions and 42 deletions.
106 changes: 64 additions & 42 deletions docs/Projects/TAD/reporting_standard.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -29,37 +29,42 @@ the Hugging Face metadata.
The standard will be written in yaml. An example yaml is given in the next section. The standard contains the
following information.

1. `language` (OPTIONAL, list[string]). If relevant, the languages the model support. There can be multiple languages.
2. `license`(REQUIRED, string). Any license from the [license list](https://huggingface.co/docs/hub/repositories-licenses).
1. `algorithm-registers` (OPTIONAL, list[string]). If this algorithm is registered in an algorithm register,
this field should contain a URI to the corresponding record in the register. There can be multiple registers.
2. `language` (OPTIONAL, list[string]). If relevant, the natural languages the model supports in ISO 639.
There can be multiple languages.
3. `license`(REQUIRED, string). Any license from the [license list](https://huggingface.co/docs/hub/repositories-licenses).
If the license is NOT present in the license list, the following two fields will be REQUIRED.

1. `license_name` (string). An id for the license.
2. `license_link` (string). A link to a file of that name inside the repo, or a URL to a remote file containing the license
contents.

3. `library` (OPTIONAL, string). Any library from the [library list](https://github.com/huggingface/huggingface.js/blob/main/packages/tasks/src/model-libraries.ts).
4. `tags` (OPTIONAL, list[string]). Tags with keywords to describe the project. There can be multiple tags.
5. `owners` (list). There can be multiple owners. For each owner the following fields are present.
4. `library` (OPTIONAL, string). Any library from the [library list](https://github.com/huggingface/huggingface.js/blob/main/packages/tasks/src/model-libraries.ts).
5. `tags` (OPTIONAL, list[string]). Tags with keywords to describe the project. There can be multiple tags.
6. `owners` (list). There can be multiple owners. For each owner the following fields are present.

1. `organization` (REQUIRED, string). Name of the organization that owns the model.
2. `name` (OPTIONAL, string). Name of a contact person within the organisation.
3. `email` (OPTIONAL, string). Email address of the contact person or organization.
4. `role` (OPTIONAL, string). Role of the contact person. This field should only be set when the `name` field
2. `oin` (OPTIONAL, string). If applicable the [Organisatie-identificatienummer (OIN)](https://www.logius.nl/domeinen/toegang/organisatie-identificatienummer/wat-is-het).
3. `name` (OPTIONAL, string). Name of a contact person within the organisation.
4. `email` (OPTIONAL, string). Email address of the contact person or organization.
5. `role` (OPTIONAL, string). Role of the contact person. This field should only be set when the `name` field
is set.

### 2. `model_index`

There can be multiple models. For each model the following fields are present.

1. `name` (REQUIRED, string). The name of the model.
2. `results` (list). There can be multiple results. For each result the following fields are present.
2. `model` (REQUIRED, string). A URI pointing to a repository containing the model file.
3. `results` (list). There can be multiple results. For each result the following fields are present.

1. `task` (list).

1. `task_type` (REQUIRED, string). The task of the model, for example "object-classifcation".
2. `task_name` (OPTIONAL, string). A pretty name fo the model taks, for example "Object Classification".

2. `Dataset` (list).
2. `dataset` (list).

1. `type` (REQUIRED, string). The type of the dataset, can be a dataset id from [HuggingFace datasets](https://hf.co/datasets)
or a link to a repository containing the dataset[^1], for example "common_voice".
Expand Down Expand Up @@ -131,6 +136,8 @@ There can be multiple assessments. For each assessment the following fields are
## Example

```yaml
algorithm-registers:
- {id} # Optional. Example: https://algoritmes.overheid.nl/nl/algoritme/45991543.
language:
- {lang_0} # Optional. Example nl.
license: {licence} # Required. Example: apache-2.0 or any license identifier from https://huggingface.co/docs/hub/repositories-licenses
Expand All @@ -142,12 +149,14 @@ tags:
- {tag_1} # Optional. Example: automatic-speech-recognition
owners:
- organization: {organization_name} # Required. Example: BZK
oin: {oin} # Optional. Example: 00000001003214345000
name: {owner_name} # Optional. Example: John Doe
email: {owner_email} # Optional. Example: [email protected]
role: {owner_role} # Optional. Example: Data Scientist.

model-index:
- name: {model_id} # Required. Example: CatClassifier.
model: {model_uri} # Required. URI to a repository containing the model file.
results:
- task:
type: {task_type} # Required. Example: image-classification.
Expand Down Expand Up @@ -202,13 +211,21 @@ assessments:
```json
{
"$schema": "http://json-schema.org/draft-06/schema#",
"$schema": "http://json-schema.org/draft/2020-12/schema",
"$ref": "#/definitions/TadSchema",
"title": "TadSchema",
"description": "A schema for the TAD reporting standard",
"definitions": {
"TadSchema": {
"type": "object",
"additionalProperties": false,
"properties": {
"algorithm-registers": {
"type": "array",
"items": {
"type": "string"
}
},
"language": {
"type": "array",
"items": {
Expand Down Expand Up @@ -257,10 +274,10 @@ assessments:
"licence",
"model-index",
"owners"
],
"title": "TadSchema"
]
},
"Assessment": {
"title": "Assessment",
"type": "object",
"additionalProperties": false,
"properties": {
Expand All @@ -282,9 +299,9 @@ assessments:
"name",
"contents"
],
"title": "Assessment"
},
"Content": {
"title": "Content",
"type": "object",
"additionalProperties": false,
"properties": {
Expand All @@ -307,10 +324,10 @@ assessments:
"required": [
"question",
"answer"
],
"title": "Content"
]
},
"Authors": {
"title": "Authors",
"type": "object",
"additionalProperties": false,
"properties": {
Expand All @@ -323,16 +340,19 @@ assessments:
},
"required": [
"name"
],
"title": "Authors"
]
},
"ModelIndex": {
"title": "ModelIndex",
"type": "object",
"additionalProperties": false,
"properties": {
"name": {
"type": "string"
},
"model": {
"type": "string"
},
"results": {
"type": "array",
"items": {
Expand All @@ -343,10 +363,10 @@ assessments:
"required": [
"name",
"results"
],
"title": "ModelIndex"
]
},
"ModelIndexResult": {
"title": "ModelIndexResult",
"type": "object",
"additionalProperties": false,
"properties": {
Expand All @@ -370,10 +390,10 @@ assessments:
"dataset",
"metrics",
"task"
],
"title": "ModelIndexResult"
]
},
"Dataset": {
"title": "Dataset",
"type": "object",
"additionalProperties": false,
"properties": {
Expand All @@ -390,10 +410,10 @@ assessments:
"required": [
"type",
"name"
],
"title": "Dataset"
]
},
"Measurements": {
"title": "Measurements",
"type": "object",
"additionalProperties": false,
"properties": {
Expand All @@ -409,10 +429,10 @@ assessments:
"$ref": "#/definitions/GraphPlot"
}
}
},
"title": "Measurements"
}
},
"BarPlot": {
"title": "BarPlot",
"type": "object",
"additionalProperties": false,
"properties": {
Expand All @@ -432,10 +452,10 @@ assessments:
"required": [
"type",
"results"
],
"title": "BarPlot"
]
},
"BarPlotResult": {
"title": "BarPlotResult",
"type": "object",
"additionalProperties": false,
"properties": {
Expand All @@ -449,10 +469,10 @@ assessments:
"required": [
"name",
"value"
],
"title": "BarPlotResult"
]
},
"GraphPlot": {
"title": "GraphPlot",
"type": "object",
"additionalProperties": false,
"properties": {
Expand All @@ -472,10 +492,10 @@ assessments:
"required": [
"type",
"results"
],
"title": "GraphPlot"
]
},
"GraphPlotResult": {
"title": "GraphPlotResult",
"type": "object",
"additionalProperties": false,
"properties": {
Expand Down Expand Up @@ -508,11 +528,11 @@ assessments:
"required": [
"feature",
"data"
],
"title": "GraphPlotResult"
]
},

"Metric": {
"title": "Metric",
"type": "object",
"additionalProperties": false,
"properties": {
Expand All @@ -539,10 +559,10 @@ assessments:
"type",
"name",
"value"
],
"title": "Metric"
]
},
"Subgroup": {
"title": "Subgroup",
"type": "object",
"additionalProperties": false,
"properties": {
Expand All @@ -556,10 +576,10 @@ assessments:
"required": [
"feature",
"group"
],
"title": "Subgroup"
]
},
"Task": {
"title": "Task",
"type": "object",
"additionalProperties": false,
"properties": {
Expand All @@ -572,16 +592,19 @@ assessments:
},
"required": [
"type"
],
"title": "Task"
]
},
"Owner": {
"title": "Owner",
"type": "object",
"additionalProperties": false,
"properties": {
"organization": {
"type": "string"
},
"oin": {
"type": "string"
},
"name": {
"type": "string"
},
Expand All @@ -594,8 +617,7 @@ assessments:
},
"required": [
"organization"
],
"title": "Owner"
]
}
}
}
Expand Down

0 comments on commit 5ae3551

Please sign in to comment.