Trying to use aot mode in unity with multiple libraries using Ceras.

[asusralis]

Hello! I currently have two libraries I import to Unity and both use Ceras. Both have classes that need formatters generated for them. So far I've been using the two libraries' Ceras, but it seems I can't do this when using Ceras.AotGenerator and Ceras.UnityAddon. Many methods say they're unacccessable. If I stop importing Ceras from the libraries, will the aot generator generate their classes as well? Is this possible?

[rikimaru0345]

Hi,
when you write code and want to access classes, properties and fields in a different assembly they have to be public. It's a requirement of the C# language.

But you can probably work around that by generating the formatters in the right assembly (or rather "in the right folder" when working with Unity).

The AotGenerator (in its current state) is not smart enough to detect .asmdef files or anything like that.
So you'll have to move the formatters into the right folder manually, and then change the code that uses them (the code that adds the formatters to your SerializerConfig).

It would be a great feature if the AotGenerator would do all of that automatically, I'll make a note of that in my todo list 👍

[rikimaru0345]

At least for debugging / testing purposes there is another workaround I can think of:

You can simply make all the relevant classes, fields and properties public.
That way the formatters will work even when in a different assembly.

[asusralis]

Oops, I didn't mean to close this. Regardless of where the class is, can the serializer access properties that are private when a formatter is generated for aot mode? Or, in aot mode, do they always have to have public get/set?

[asusralis]

The classes are all public. The access problem was from ceras specifically. Ceras itself was being imported with a .dll - unlike the guide that has you put the Ceras folder into Unity. Some of Ceras' classes from the aot generator were complaining the Ceras classes weren't accessable. However, if I understand your reply correctly, not being able to serialize private fields and properties in aot is an even bigger problem for me D:

[rikimaru0345]

Regardless of where the class is, can the serializer access properties that are private when a formatter is generated for aot mode? Or, in aot mode, do they always have to have public get/set?

However, if I understand your reply correctly, not being able to serialize private fields and properties in aot is an even bigger problem for me D:

In normal (non-aot) mode Ceras uses dynamic code generation, which has no "visibility restrictions", meaning it can read and write private fields and properties.

In AotMode formatters cannot be generated "on-the-fly" by Ceras (because the whole point is not to use dynamic code). The code generator doesn't really do anything special, it just writes plain old code that gets compiled along with your own code.
There is no way to bypass this restriction directly though.

There are a few ways to work around that though:

• don't use AotMode
• make your properties and fields public
• you could also try marking your fields and props as internal and then use an InternalsVisibleTo attribute, but I never tried using that in Unity, not sure if the compiler even understands/respects that attribute.
• (in theory) an serializer could use reflection to get/set any field and property, but those functions are notoriously slow. (like, literally 100x slower...). Ceras uses this as a fallback for readonly fields (when setting by constructor or property are not an option). But for "normal" serialization that'd be way to slow.

The access problem was from ceras specifically. Ceras itself was being imported with a .dll - unlike the guide that has you put the Ceras folder into Unity.

Which methods is it trying to use?
And yes, you should definitely use the source-code version of Ceras for Unity projects.
The source-code has quite a few compiler flags that will result in better/faster code depending on your compilation target.

[asusralis]

Oh, no! That's a shame. I would love to not use aot, but it seems it is a requirement for unity/Android as you need it for x64 builds. What to do...

Well, my Network library uses dumb dtos, so there is no problem to generate formatters there; that is where I'd need the most performance. However, the more complicated objects would only be serialized in local saves. How difficult would it be for me to use reflection in the formatters? Or did your (in theory) mean it's not possible right now?

And thanks so much for the help. I've created an issue here before and it's always so nice to interact with you :D

[rikimaru0345]

Oh, no! That's a shame. I would love to not use aot, but it seems it is a requirement for unity/Android as you need it for x64 builds. What to do...

It seems like if you use the mono backend, you don't have to use Aot for android:
https://docs.unity3d.com/Manual/ScriptingRestrictions.html
Not sure if that is an option for you, maybe it helps.

However, the more complicated objects would only be serialized in local saves. How difficult would it be for me to use reflection in the formatters? Or did your (in theory) mean it's not possible right now?

Local saves? Like for savegames or settings? I see...
In that case even a the (relatively) huge slowdown from reflection calls will most likely not be a problem.

How difficult would it be? Unfortunately there's no built-in option for that at the moment, but but depending on how familiar you are with the internals of Ceras, it shouldn't be too hard to implement.

Here's what I'm thinking:
In the v5 version of Ceras, the formatters are generated in a new way (compared to v4).
It first generates the formatter methods as if it would be generating them for the "normal mode" (non aot), and then it converts the generated expressions into C# source code.

In theory, it should be relatively easy to exchange all the "Assign()" calls (where it is either assigning a deserialized valued to a field or property, or reading an existing value in order to serialize it) with SetValue and GetValue calls.

But then again those Get/SetValue calls are called on a FieldInfo or PropertyInfo, and while it is absolutely no problem to get those inside a Deserialize/Serialize method, doing so would make everything even slower. The correct approach would be to create a list of all the required FieldInfos and PropertyInfos and then cache them in some static fields, and then use those instead of re-discovering them everytime an object should be serialized.

Hmm... it wouldn't be too hard to do, but it would definitely require a few hours or days of work to do right 😋

Or you could write the formatters manually and then replace all the gets/sets with reflection calls manually. But then again that kind of defeats the purpose of a "serialization" library (in my opinion). You'd only keep de-duplication, automatic handling of references, polymorphism, ... not sure if that's worth it.

I'd probably do the following in your case:

• ensure all members to serialize are at least "internal"
• ensure each generated formatter is actually in the right folder (so it ends up in the same assembly as the type that it serializes), that way the members only need to be internal, and not public.

And thanks so much for the help. I've created an issue here before and it's always so nice to interact with you :D

Thanks!! ❤️

[asusralis]

I'm pretty sure the Google play store requires x64, and on newer unity versions you can't build Android using mono x64 :( only x86 (?)

Is that part of v5 (new aot) already in? If so, I would definitely look into it! Does this at all affect serializing delegates? Or can you not do that too in aot?

And to have it work right now, basically replace all private with internal? That certainly sounds easier...

[asusralis]

I deleted the ceras dll and added the source files from master, but I'm running into two problems:

1): It says Color cannot be found in System.Drawling. I can just remove this.

2): It seems my libraries cannot find Ceras. "Assembly 'Assets/Plugins/netstandard2.0/Empis.dll' will not be loaded due to errors:
Unable to resolve reference 'Ceras'. Is the assembly missing or incompatible with the current platform?"

Both reference Ceras from a nuget. Is this the wrong way to do it?

[rikimaru0345]

Is that part of v5 (new aot) already in?

While v5 is not done yet, the changes to the AotGenerator are mostly complete.
Depending on what exactly you're doing, you may encounter some bugs though! I'm aware of most of them, but haven't had the time to fix them yet. They're all relatively easy to fix.

Does this at all affect serializing delegates? Or can you not do that too in aot?

Serializing delegates is possible in v4 as well, the only change in v5 (as far as I remember) is support for "MulticastDelegates" (delegates that have multiple targets, like when serializing a C# event where multiple targets can register themselves to be notified).

As for Aot and Delegates: Aot should not impose any limitations on serializing delegates whatsoever, because delegate serialization is more or less just a list of "methodName" + "targetObject" combinations. See here: https://github.com/rikimaru0345/Ceras/blob/Ceras-v5/src/Ceras/Formatters/DelegateFormatter.cs

It says Color cannot be found in System.Drawling. I can just remove this.

Yep, exactly, unless ofc you're using System.Drawing. I should add an #if guard for that...

It seems my libraries cannot find Ceras. "Assembly 'Assets/Plugins/netstandard2.0/Empis.dll' will not be loaded due to errors:
Unable to resolve reference 'Ceras'. Is the assembly missing or incompatible with the current platform?"

Maybe it is trying to load the previous version of your dlls? Delete the dlls, and let them be re-compiled from the source code as well so they reference the correct (local) version of Ceras.

Another thing to keep in mind is that you probably have to copy additional Dlls (like for example https://www.nuget.org/packages/System.Runtime.CompilerServices.Unsafe/ etc ...). But you probably already did that in order to get the original Ceras version (the precompiled one you've been using so far) working in the first place.

[asusralis]

Thanks for the info! I will first try to get the networking to work in aot and then circle back to the more problematic classes.

Hmm, I'm still confused about the Unity Ceras part. My two projects are currently referencing a Ceras dll. Which, from what I understand, has its own Assembly. When I import my two libraries to my unity project, those .dlls are going to be looking for the Ceras assembly, right? And, if I just drop Ceras into my project like the tutorial says, Ceras will share the same assembly as my project - not its own. How would my two libraries find Ceras, then? Everything is on the same Ceras version.

I had duplicates of those extra .dlls (Like the .Unsafe) but I deleted them and that now has no issues!

[asusralis]

Hey, sorry, but I still can't get the source files of Ceras to work with my libraries that reference Ceras through a .dll :( It's looking for the assembly of Ceras, right? I'm not sure how to satisfy that without using the Ceras dll (but, when I do, I cannot use the Unity addon part of Ceras).

[rikimaru0345]

So you have a pre-compiled .dll that is referencing Ceras?
I'm not sure if that will work in Unity, those libraries have to be in their source-code form as well so you can make them reference the .asmdef.

But just to be sure: Can you post the actual error you're getting?