Attempt to add an affiliation for a newly added contributor results in web ui error

[yarikoptic]

index.e7a183c2.js:14 TypeError: this.value is not iterable
    at click (index.e7a183c2.js:191:39525)
    at ul (index.e7a183c2.js:10:23347)
    at o.n (index.e7a183c2.js:10:13584)
    at ul (index.e7a183c2.js:10:23347)
    at e.$emit (index.e7a183c2.js:10:28128)
    at o.click (index.e7a183c2.js:86:29785)
    at ul (index.e7a183c2.js:10:23347)
    at HTMLButtonElement.n (index.e7a183c2.js:10:13584)
    at KJ.s._wrapper (index.e7a183c2.js:10:59969)
    at HTMLButtonElement.r (index.e7a183c2.js:29:1711)

happening on https://dandiarchive.org/dandiset/000029/draft which you are welcome to torture directly - it is a test one

[waxlamp]

A newly created contributor doesn't have any affiliation value in it (see https://api.dandiarchive.org/api/dandisets/000029/versions/draft/info/; search for "Turing" to see an example, and compare to the other contributors next to that one).

When I open the Turing contributor in the meditor then click on the plus button for affiliations, I get a console error reporting that this.value is not iterable. Using the debugger reveals that this.value is null in context. I believe that is coming from the default value defined for affiliation being null.

@mvandenburgh, could you confirm my reasoning? If I'm right, then updating the schema to have [] instead of null as the default value there should fix this. A note for completeness: the jsonschema spec says that default values have no restriction but it's "RECOMMENDED" that they validate against the schema they're modifying. null definitely fails that recommendation.

[mvandenburgh]

@mvandenburgh, could you confirm my reasoning? If I'm right, then updating the schema to have [] instead of null as the default value there should fix this. A note for completeness: the jsonschema spec says that default values have no restriction but it's "RECOMMENDED" that they validate against the schema they're modifying. null definitely fails that recommendation.

This is correct. To verify, I manually modified the schema and pointed my local dev instance at it, and it fixed this issue.

[yarikoptic]

In dandischema we have Optional[list]. That's legit. Defaulting to None is legit for Optional. There is a good number of such elements of type array and default null in the jsonschema, not just "affiliation", so fixing for it alone makes little sense to me.

According to above discussion it is valid (RECOMMENDED is not MUST) as far as jsonschema concerned, and such records do pass jsonschema validation, right? If it was not - pydantic export to jsonschema had to be fixed up, but I don't think that is the case here.

So, where is the bug then really which is to be fixed in code? The vue UI library?

[waxlamp]

You're right, @yarikoptic, None is allowed for the specific type used here. But for values that are lists, what is the semantic difference between None and []? More specifically, what is the difference between those values for the list of affiliations for a Person? To me, it seems there is no difference (but that's why I ask). And if that's true, then we really should have List[Affiliation] with a default of [] for the most useful semantics.

If instead we want to keep the type as-is, we'll have to see what we can do in the client. But the error in the meditor occurs inside the framework code, so we'll essentially need to work around the issue in some way. The reason I don't really love that approach is that we would be implicitly treating the None default value as [] -- this seems to indicate that the true type really is List[Affiliation] rather than Optional[List[Affiliation]] (at least in terms of how the meditor approaches Person).

Let me know what you'd like to do here. (And in case you're unaware, there's a parallel discussion happening over at dandi/dandi-schema#282 (comment).)

[candleindark]

There may be another way around this without changing the type or the default value of affiliation in Person.

The generation of JSON schemas for Optional types in the DANDI models is not standard per Pydantic V2. I was requested to put in a customized function to generate JSON schema for Optional types to mimic JSON schema generation by Pydantic V1. If that function is removed, the affiliation property of Person in the JSON schema would be the standard generation by Pydantic V2 as the following.

        "affiliation": {
          "anyOf": [
            {
              "items": {
                "$ref": "#/$defs/Affiliation"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "An organization that this person is affiliated with.",
          "nskey": "schema",
          "title": "Affiliation"
        }

This standard generation doesn't have the default value and type disagreement issue.

Be aware that if the customized function is removed. All JSON schema generation of Optional types will be affected. I was told to put in that function, specifically for compatibility with GUI I think.

[yarikoptic]

You're right, @yarikoptic, None is allowed for the specific type used here. But for values that are lists, what is the semantic difference between None and []? More specifically, what is the difference between those values for the list of affiliations for a Person? To me, it seems there is no difference (but that's why I ask). And if that's true, then we really should have List[Affiliation] with a default of [] for the most useful semantics.

There is indeed no semantic difference and ambiguity as two different values provide the same semantic. My point is that if we were to address it as changing to [] - IMHO then we should do it for all such encounters, and likely removing Optional altogether to remove such ambiguity. Then it would be for UI facing elements to treat "empty list" as "really there is nothing". Otherwise, we would be back at similar issue either for another similar attribute (or even this one) which would be assigned None since it would remain to be allowed! And then we also need to check if changing dandischema default would be sufficient or we already have fields with 'null' in those fields and requiring metadata migration... (#565 and related)

@candleindark thanks for digging that up. In my view, it just brings us back to us detecting some shortcoming of UI and providing a workaround instead of a proper solution. So, it seems we are making yet another circle to the same problem. IMHO it would be better both short and long term to (at last) address this issue properly by consulting with upstream.

How difficult would be to construct minimal demonstration of the issue for both definitions?

Related, we pin/use

web/package.json: "@koumoul/vjsf": "2.23.3",

There was v3.0.0 in Nov, 2024 "Vjsf 3 is a complete rewrite of the library compatible with Vue 3 and Vuetify 3.". And we have fresh (thanks @waxlamp !)

• Vue 3 migration #2167

So overall question is -- may be that version already addressed it and proper solution for us is really migration to Vue 3 + vjsf 3 ? I guess should be easy to discover if we have minimal reproducer.

[waxlamp]

How difficult would be to construct minimal demonstration of the issue for both definitions?

Here's a codepen that demonstrates the VJSF 2 issue. Note that the top level value is an array, and clicking on the plus button causes the same error as we're seeing in the meditor. If you remove line 6 (or change the default value to []) then it will work. There is a similar issue for the "address" value that exists inside the value you can now add.

I don't think there's any simple workaround for this. The reason we haven't run into the problem before is that we seem to populate most instances of these list-valued metadata items with []. For example, if you create a new Dandiset and then examine its metadata, you'll see that about has a value of [] rather than null. I am not sure where that happens, but it's clearly an explicit setting of the value, since otherwise the default of null would appear there, and we'd have meditor issues with that metadata item. Unfortunately, the affiliation field of Person does not get this explicit setting of [].

So overall question is -- may be that version already addressed it and proper solution for us is really migration to Vue 3 + vjsf 3 ?

Mike and I just looked into this and it turns out that yes, VJSF 3 does solve this problem. So to solve this problem, we can just wait for the Vue 3 upgrade to happen (Mike will have some news about this at Monday's meeting).

My point is that if we were to address it as changing to [] - IMHO then we should do it for all such encounters, and likely removing Optional altogether to remove such ambiguity.

It seems we agree 😄

Upgrading to Vue3 should solve the immediate issue, but I think we should still talk about "fixing" the typing problem in our schema. Just to reiterate: list-valued items should not be Optional unless there is a specific need to distinguish between "list is missing" and "list is present but empty". In our case, I don't believe we have that need. But the nice thing here is that we have time to discuss this and come to a consensus.