Specifying Custom Attributes, Metadata etc...for Export

[dgovil]

Hi everyone,

I was wondering what people's thoughts are on how the maya-usd project plans to handle the following sets of extra data that users may want to expose to the exported USD files:

• Per Prim custom primvars
• Per Prim assetInfo
• Per Prim Kind
• USD file customLayerData

Per Prim Custom Primvars

This is mostly about being able to export attributes as primvars that the default USD export would not otherwise do.
For example extra channels that downstream processes or code could react to.

The Pixar exporter had the somewhat hidden option of using USD_UserExportedAttributesJson to specify custom attributes for export along with naming etc. There was also a basic UI to manage the options for this.

Is this the route that the Maya USD exporter will also go down? Are there any plans to use the same attribute and tools or update them to something else? If so, would this be exposable to some place easy for artists to access?

Alternatively (or additionally) would it make sense to people to allow for an alembic style approach of providing attribute prefixes to match against e.g. USD_*. I can see some pros and cons with regards to usability for both which is why I ask. Specifically the alembic style route is great for quickly specifying what you want exported without necessarily modifying all your objects, but makes it hard to capture the amount of options that the other approaches provide.

Per Prim AssetInfo

This is something I haven't seen existing tooling around. I imagine having a similar system as the USD_UserExportedAttributesJson with a JSON field would make sense?
This would be handy for exporting static metadata as part of an object.

Per Prim Kind

This one's a little more niche. Perhaps a custom string attribute? Various pipelines may have kinds that aren't exposed to Maya, but are still valid to set for their downstream pipelines to query.

USD file customLayerData

Custom Layer Data would be really useful for embedding extra data that is not prim specific but layer specific for the pipeline to use, or for general book keeping.

This one's something we use quite a bit to include metadata like copyright info etc... for .usdz files.
Right now we implement this as a custom chaser but I can see it being really useful to be exposed as an exporter argument, for general use. Perhaps as another json formatted string argument?

---

Anyway, I just wanted to start the discussion on this. The need for custom data export has come up a few times on our end, so before we dev anything internally, we want to make sure it's something representative of the general direction that Autodesk and others were planning to go down.

[wtelford]

Great discussion Dhruv,
You cover a wide range of (related) topics here. The rule of thumb I try to follow is to map USD concepts to Maya concepts when there is an analogue present. The difficulty comes in where USD introduces concepts that either don't exist or are poorly supported in Maya. In these cases Maya concepts tend to get overloaded and the implementation suffers.

That said, all functionality for transporting data in and out of Maya from the Pixar plugin remains in MayaUSD, however not all of it is exposed through the UI. We did not bring over the Pixar UI's for tagging data as we still want to investigate more native ways of defining that data.

Something that would be helpful would be production samples which would show these concepts as you would like them to exist in Maya, and then the USD as you would expect it to be exported (or vice-versa).

[dgovil]

Sorry for not replying sooner, I missed the notification. We've just been discussing this internally, and here were our thoughts so far...

Maya Side representation

With regards to how to store attributes in Maya, I cooked up this file:
custom_attributes.ma.zip

This shows what it would look like to specify primvars, assetInfo and kind on nodes in Maya.

AssetInfo and Custom PrimVars

For both assetInfo and specifying custom primvars, we were discussing the use of json versus compound attributes to represent it in Maya.

In both cases we're assuming the following attributes are important:

• The name of the attribute to export
• The name of the attribute for the export. Empty == do not rename
• The precision value of the attribute. Though per discussions on the usd-interest group, the precision discussion may become moot? I'm not sure here.

Anyway, we discussed two approaches to representing the data Maya side. We're trying to look at it from the perspective of what might be easy to pipeline, but also for non technical users to setup if needed.

Compound Attributes

Compound attributes are an option that we thought could be nice because they are exposed in an intuitive way in the Attribute Editor for less technical folk to use. We set it up as a multi plug in which case it makes it easy for a user to keep specifying new attributes directly in the Attribute Editor without any extra interfaces involved.

Another advantage of using this, is that it's easy for MEL/cmds users to modify. Whereas a JSON field makes it tricky for a MEL user to modify the attributes without adding some kind of cmd for them.

Downsides seem to be that it is less "flexible" than JSON should the number/type of the compound attribute children change.

JSON Attribute

JSON is what the Pixar setup uses, and it is a lot more compact to represent.
However it is not as easy for a non technical user to edit, and is not easy to edit from MEL/cmds should they need to do so.
That may not be a concern , but i figure I'd list it.

Another concern from our TD was that JSON parsing at scale can become expensive for large scenes.

Kind

Kind is something that I think makes sense to be a string attribute in Maya.
We'd initially discussed making this an enum, however kind is not a finite enumeration of values. New schemas and plugins can add their own kind, and indeed, we do this implicitly for our preliminary custom schemas.

Therefore, I think having a string attribute for users to fill in makes sense. If there's a need to limit this, I think additional UI tools or pipeline specific tools could be setup, but it makes most sense to us to represent usdKind as a string that a user can edit.

Custom Layer Data

Custom Layer Data is the one outlier because it can't be represented effectively in scene for exports.
My thinking here is that the mayaUsdExport command could take either of these options:

• A string flag as a json: cmds.mayaUsdExport(...., customLayerData='{"copyright":"someone"}')
• Numerical flag pairs: cmds.mayaUsdExport(...., customLayerDataName1="copyright", customLayerDataValue1="someone", customLayerDataType1="string")

I've seen both of those used in various pipelines and they both have their pros and cons. It's harder for MEL users to specify JSON, but I also don't expect MEL users to be the ones setting up pipelines like this so maybe that's moot.

Conversely numerical flag pairs are very easy for any language user to specify, but makes this pretty ugly looking.

USD Side

As for how they'd look when exported, it would be roughly similar across the board for assetInfo and customLayerData. The samples on the AR QuickLook gallery show the customLayerData and kind being used.

Custom Prims

On the USD side, it would look like any other primvar. The name of the exported primvar would be whatever the output name is specified to, otherwise it would be the name of the original maya attribute.

def Mesh "pCube1" {
      float foo = 12.0
}

Uses for this would be things like:

• At Sony we'd export certain rig attributes through into the abc/usd files to drive shader parameters in lighting. This let animators have control early in the pipe, and see it reflected in their publish renders.
• Exporting custom data to engines. e.g. exporting data for use in Unity.

AssetInfo

Asset Info on the prims would follow similar rules to the custom prims for naming, but would be represented in the dictionary.
The difference versus primvars, is that assetInfo can be explicitly used to act as metadata instead of other runtime data. It's a subtle but useful distinction.

def Mesh "pCube1" (
            assetInfo = {
                float foo = 12.0
            }
        ) {}

Kind

We use kind quite liberally to designate our items

This is from a sample asset on the AR QuickLook gallery

def Xform "tv_retro" (
    kind = "component"
)

However the preliminary schemas use this too

def Xform "Root" (
    # Because you can apply `kind` to any type of prim, that allows any prim to be a scene.
    kind = "sceneLibrary"
)

Custom Layer Data

We currently use this to store copyright and creation data in our USD files. However we think it would be useful for many content creators to be able to specify arbitrary data here:

#usda 1.0
(
    customLayerData = {
        string copyright = "Copyright (c) 2020 Foo Inc. All rights reserved."
        string creator = "USD 20.11 Maya Plugin 0.5"
    }
)

---

It would be great to hear other people's thoughts here on this.

[wtelford]

Thanks for the detailed write up Dhruv,
I like the idea of compound attributes more than the Json approach. Currently the json approach is a bit opaque.
I'm curious your thoughts on using objectSets to define and collect attributes. In that scenario names would just round trip and all data would just live as a Maya attribute.

[dgovil]

I think that could work but my one issue there would be how it applies to assets that are referenced in?

For example let's say I have a rig where I want to set it up so that every export includes some custom attributes. This rig gets referenced into each Animators scene.

Then there's other objects in the animators scene where we need to export attributes on.

Would sets be able to be referenced in and combine together effectively from the perspective of the translator?

[wtelford]

I would think it could. Maya will reference in a set and either namespace it or increment its name as needed. If the translator was set to look for a naming pattern for the set as opposed to a hard coded name, I would think it would be possible.
Sets should also be fully editable with ref-edits as well.

For transparency's sake, we have not chosen a direction, but I am interested in understanding any pitfalls with the set-based approach.Any other concerns with a set-based approach?

To summarize the methods on the table are:

1. Json-based approach (this is Pixar's original implementation)
2. Compound Attributes
3. Set-based
   One last option would be
4. Native

For #4 Maya does have the notion of metadata for nodes, its just not very fleshed out. This approach would still present the problem of "which" data to export if you did not want ALL Maya custom attributes and metadata.

[dgovil]

Would native be different than using something like MPxCustomData to define it?

As for other downsides to sets, I think the biggest would be UX locality for less technical folk. Having the set control it might be confusing for people perhaps since they'd have to look at the set node rather than the node that holds the data being exported. Though good docs and/or tools here would probably mitigate or even negate that. Just trying to think of the downsides, not necessarily saying it's a big one.

[pberto]

Compound Attributes is how this is implemented in Multiverse. @wtelford: "copying is the best form of flattery".

[kxl-adsk]

@pberto For the sake of having a constructive discussion here, can you share what other options you have considered and why the compound attributes solution was picked?

[dgovil]

I just wanted to ping this thread and ask people's thoughts on specifically USD file customLayerData.
We've had a few people internally and externally asking about this, and I've been thinking that perhaps the easiest implementation is to add three new arguments to the exporter.

1.~~ customLayerDataKeys (vector of strings): The names of the keys, matched by index to the values~~
2. customLayerDataValues (vector of strings): The values of the keys, matched by index to the keys
3. customLayerDataTypes (optional vector of strings): An optional name of the type to convert the values to. Matched by index to the keys. Default to string for all values that are unspecified.

Actually, looking at MSyntax, and as an example the chaserArgs flag, I think I can just replicate the chaserArgs argument and make it so it is a multiUse flag with three arguments: key, value, type

So then it just requires adding a single flag, and no index matching

I think this will allow people in MEL/Python to easily provide the parameters.

Do people have any thoughts on that approach? or other possible alternatives?
Based on peoples thoughts, we can implement our solutions internally to go along with those and submit a PR if people are amenable to our solution.

[wtelford]

Hey Dhruv,
This is another interesting question. I'd be interested in hearing your thoughts on what your expectations would be if the exporter were updated to output multiple files (split out material libraries, or split our payloads, etc.) in a single export. How would you expect the above flags to function? I think the flags you propose make sense for the root layer, but would there be any expectation to have custom layer data on other files?

While the flags you propose do make sense, I'm always hesitant to associate multiple arrays by index, as it can be a bit fragile. Do you have any thoughts on a different approach?

[dgovil]

Hey Will, so yes, definitely agree on not liking index based matching since it's fragile. I had forgotten/missed that you can do multi-argument multi-use. I put together a quick draft PR here: #1081 using a single attribute where you pass in the fields as triplets of (key, value, type).

As for supporting multiple layer types, I'm not sure what your chosen semantics would be for specifying those types, but I think perhaps adding a fourth argument to this flag would allow for controlling that. Something like a (regex) pattern argument where * or .* resolve to putting it on all layers, but something like mat* would put it on material layers if they had an appropriate naming convention? Just spitballing since I don't know the details of how you plan to expose multiple layer exports yet to the export command.

[cking-spire]

Has there been any resolution to this?

I am working on writing USD out of Maya to be loaded into other DCCs (Unreal, Houdini). I want to be able to tag materials with hints so I can translate those materials into something useful in those other DCCs. Is there a way to do that today?

Thanks,
Chris King
Spire Animation Studios

[bentoogood]

Can I bring this back from the dead as I have been unable to find any reference to the correct way to export arbitrary attributes to USD prim vars in any of the documentation?
Our use case is for exporting animated parameters that will be fed through to mtlx shaders. Is there a method to achieve this, or has the functionality from the Pixar plugin not made it through to the official release?

[santosd]

Hello, we have a way of tagging user attributes for export with USD and wondering if that is what you are looking for. More information on that option can be found here: https://github.com/Autodesk/maya-usd/blob/dev/lib/mayaUsd/commands/Readme.md under the Custom Attributes and Tagging for USD section. Let me know if that helps.

[bentoogood]

Thanks David - that's the section I've been looking at so far, but from what I can tell there's a line that says:

See "Specifying Arbitrary Attributes for Export" below.

however, the section 'below' that's referred to appears to be missing, having been removed from the document a little while back.
Do you have any details on how to tag attributes with the current version of the plugin?

[santosd]

Yes the docs are a little confusing I think and need to be updated, I will report this to the team and see if we can get better documentation. The way that an attribute is tagged is by using the "adaptor" helper tool:

The following script will creates the attribute subdivisionScheme and assigns it the loop value. The adaptor will tag the attribute with USD_ATTR (you can see it in the Attribute Editor under the Extra Attributes section) so on export it will set that value of subdivisionScheme to loop which is not the default. I hope that helps clear things up a little bit more. I can create a video if that would help.

import mayaUsd.lib as mayaUsdLib
import maya.api.OpenMaya as om
sl = om.MSelectionList()
sl.add("pCylinder1")
dagPath = sl.getDagPath(0)
dagPath.extendToShape()
adaptor = mayaUsdLib.Adaptor(dagPath.fullPathName())
geomSchema = adaptor.GetSchemaOrInheritedSchema("UsdGeomMesh")
subdAttrib = geomSchema.CreateAttribute("subdivisionScheme")
subdAttrib.Set("loop")

[bentoogood]

Thanks David, very helpful - but am I right in thinking that only applies to attributes defined as per of the schema? My attempts at creating arbitrary attributes just result in errors.

'Attribute 'foobar' doesn't exist on schema 'Mesh''

What I'm really looking to do is register arbitrarily named Maya attributes for export as primvar attributes in USD land.

Do you have an example of that?

thanks

[mrloafbot]

Can you set the orientation of a mesh this way? I have some CAD data that for what ever reason wants to export as lefthanded and all the rest of it is righthanded. So far my attempts at using an adapter have not worked.

Thanks for any suggestions!

[santosd]

Hi @bentoogood, I tried an arbitrary attribute on my end and I see the issue you are running into as well. I believe in this case you may need to have a custom schema that has the 'foobar' attribute on it. As you can see the adaptor uses the schema to get the attributes geomSchema = adaptor.GetSchemaOrInheritedSchema("UsdGeomMesh") here is some information on creating custom schema's:
https://graphics.pixar.com/usd/release/tut_generating_new_schema.html

[bentoogood]

Thanks David - feels like that moves it out of the realm of arbitrary if we need to implement a new schema each time :)

Would be great to revisit some of the earlier conversation where suggestions had been made for flexible methods for attribute tagging. Fingers crossed for a future release.

[santosd]

@bentoogood that's true :) I will bring this up to the team and see what we can do for future releases.

[santosd]

@mrloafbot I would imagine that should work. I tested it on my end and I am also not able to get it to work. One workaround is to bring the leftHanded mesh into Maya as a stage and then through the attribute editor change the orientation value to rightHanded and then open the Layer Editor and save your stage. That should change it. I am looking further into the issue with orientation value not working as a user defined attr.

[mrloafbot]

Thanks for the reply. I appreciate you trying it out yourself.
Keep up the good work !

[yohanndageb]

Hello!

Sorry to bring up again this post.
But I keeping don't understand how to export UsdKind as an Extra Attribute. I already add it as an Extra Attributes with the name usdKind (Make attribute Hidden | Data Type String) but when I export what I want as usd this coming always the first group as component and the rest one, don't matter if it's a group or a geometry, as empty on the drop-down menu list on the side of Kind:

Can anyone give me some help?

Thank you all!