RFC: useSerialized$/createSerialized$ + SerializeSymbol

[wmertens]

Champion

@wmertens

What's the motivation for this proposal?

Problems you are trying to solve:

• third-party libraries with non-serializable objects

---

Proposed Solution / Feature

What do you propose?

1. a Symbol prop, SerializerSymbol, you can attach to an object that will serialize the object
2. a Signal that holds the object and will lazily recreate it when needed

Code examples

class MyCustomSerializable {
  constructor(public n: number) {}
  inc() {
    this.n++;
  }
  [SerializeSymbol]() {
    return this.n;
  }
}
const Cmp = component$(() => {
  const custom = useSerialized$<MyCustomSerializable, number>(
    (prev) =>
      new MyCustomSerializable(prev instanceof MyCustomSerializable ? prev : (prev ?? 3))
  );
  return <div onClick$={() => custom.value.inc()}>{custom.value.n}</div>;
});

import {makeThing} from 'things'

const serializer = (thing) => thing.toJSON()

const Cmp = component$(() => {
  const custom = useSerialized$((data) => {
    const thing = makeThing(data)
    thing[SerializeSymbol] = serializer
    return thing
  });
  return <div onClick$={() => custom.value.doThing()}>{custom.value.stuff}</div>;
});

In detail

• you must provide [SerializeSymbol]: (obj) => serialize(obj) to any object that needs custom serialization
  • note that this can also used on regular objects to do housekeeping before serialization
  • Promise results are awaited so you can even e.g. save a reference number to a db
• useSerialized$(constructorFn) takes a function that will be executed on the server as well as the browser to create the object, and it returns a Signal
  • the Signal works exactly like ComputedSignal, except that it will always be marked to-calculate in the browser, and it passes the serialized value to the compute function
  • if you use scoped variables that change, they will cause the function to be called again and then the parameter will be the previous result

---

PRs/ Links / References

QwikDev/qwik#7223

Proposed changes

• add the serializer as an optional function to the useSerialized$(deser, [ser]) call. This would mean using a WeakMap like noSerialize() does, but it is better DX.

[GrandSchtroumpf]

I think this is a great addition to Qwik. On a devX point of view I find it a little difficult to understand because serialize & deserialize don't have the same mecanism (serialize -> SerializeSymbol / deserialize -> useSerialized$).
Maybe something like that :

const custom = useSerialized(new MyCustomClass(), {
  serialize: $((instance) => instance.toJSON()),
  deserialize: $((json) => MyCustomClass.fromJSON(json))
});

Also scoping it into a use hook limit its usage inside components, while it could be useful outside too. It won't be lazy, but as long as the dev is aware of that, I think it would be a good tool.

import { MemoryDB } from 'third-party';
export const db = serialize(new MemoryDB(), {
  serialize: (instance) => instance.getState(),
  deserialize: (state) => new MemoryDB(state)
});

// Create API
export const addFoo = $(async (foo: Foo) => {
  await db.store('foo').add(foo);
  // Do additionaly logic here
})

[wmertens]

The first interface can work (the serializer doesn't need to be qrl but can be). I think I'd still like to keep the symbol interface for cleanup before serialization etc.

For the use issue, there's also createSerialized$ which you can use anywhere, so that's not an issue.

[wmertens]

Oh but the creation needs to be a function and the deserializer does the same, so it could be like

const createSerialized$((data) => new MyCustomClass(data), (obj) => obj.toJSON())

, with the serializer optional.

EDIT: actually this would mean keeping a a WeakMap for the serialization functions, instead of the single prototype object with the serializer. So I'm not a fan.

[GrandSchtroumpf]

I'm not sure to understand how the createSerialized$ works. Could you share a compete example ?
Let's say I want to create a MemoryDB instance on the server and pick it up in the client, how would it work ?

import { MemoryDB } from 'third-party';

// Create global instance that I can access in both server & client
export const db = ... ;

// Use instance in either server or client
export const addFoo = $(async (foo: Foo) => {
  // db.add(...)
})

[wmertens]

That won't work, the db connection won't be initialized in the client.

addFoo would be a method of an object you create, and you'd use it like signal.value.addFoo, and it would create its db connection if needed.

[GrandSchtroumpf]

Let's say MemoryDB is something that doesn't require any connection, something dummy, but unserializable, like that :

class MemoryDB {
  state = {};
  add(store, value) {
    const id = randomUUID();
    this.state[store] ||= {};
    this.state[store][id] = value;
  }
  get(store, id) {
    return this.state[store][id];
  }
  ...
}

How can I use createSerialized$ to serialize it on the server and deserialize it on the client ?
(I understand I could achieve that with useStore, it's just to have some code example to better understand how it would work.)

[wmertens]

@GrandSchtroumpf something like

class MemoryDB {
  constructor(public state = {}) {}
  add(store, value) {
    const id = randomUUID();
    this.state[store] ||= {};
    this.state[store][id] = value;
  }
  get(store, id) {
    return this.state[store][id];
  }
  [SerializeSymbol]() { return this.state }
  ...
}

const dbSig = createSerialized$((data) => new MemoryDB(data))

[GrandSchtroumpf]

In a previous comment you mentioned we can use createSerialized$ to work outside a component.

For the use issue, there's also createSerialized$ which you can use anywhere, so that's not an issue.

How can I use it ? Like that ?

import { MemoryDB } from 'third-party';

// Create global instance that I can access in both server & client
export const dbSig = createSerialized$((data) => {
  const instance = new MemoryDB(data);
  instance[SerializeSymbol] = () => instance.state;
  return instance
});

// Use instance in either server or client
export const addFoo = $((foo: Foo) => {
  dbSig.value.add('foo', foo);
})

[wmertens]

Yes correct, I edited the example, I used use by accident.

[thejackshelton]

The use case of useSerialized$ is especially valuable for integrating vanilla JS libraries with Qwik.

The current challenge with libraries like TanStack Table is that they have a vanilla JS core that creates complex objects with methods. These objects often become non-serializable in Qwik's resumable architecture, leading to runtime errors when trying to pause/resume the application state. (it does not recover the right info, at least in v1)

For example, with TanStack Table, they create table instances with methods like:

const table = createTable({
  data: [],
  columns: []
})

// Methods on the table object can cause serialization issues
table.getRowModel()
table.getSortedRowModel()

The proposed useSerialized$ would let us properly serialize/deserialize these instances by:

1. Defining how to serialize the core state
2. Providing a way to reconstruct the full object with methods in the browser

const tableSignal = useSerialized$((data) => {
  const table = createTable(data)
  table[SerializeSymbol] = () => table.getState() // Serialize core state
  return table
})

This would be a huge improvement over current workarounds like:

• Not serializing at all (noSerialize around everything)
• Manually managing serialization (external to what the framework understands, similar to imperative browser changes with the DOM and the framework)
• Restructuring code to avoid serialization

[mhevery]

@wmertens

// existing propsal
const custom = useSerialized$(new MyCustomClass(), {
  serialize: $((instance) => instance.toJSON()),
  deserialize: $((json) => MyCustomClass.fromJSON(json))
});

// mhevery proposal
class MyClass {
  static serialize: ((instance) => instance.toJSON())
  static deserialize: ((json) => MyCustomClass.fromJSON(json))
}

const custom = useSerialized$(MyClass, optionalInitialSerializableValue);
// non class syntax
const custom = useSerialized$({
  serialize: ((instance) => instance.toJSON()),
  deserialize: ((json) => MyCustomClass.fromJSON(json))
}, optionalInitialSerializableValue);

I think the above syntax is cleaner and does not require that the object has to be a class.

[wmertens]

@mhevery ok, I like it, how about this:

type SerializationOptions<T, S> = {
  // No need for `serialize` in the browser
  serialize?: false | (obj: T) => S | Promise<S>,
  // If you rerun due to scope capture, the previous will be in the second argument
  deserialize: (...args: [data: S | undefined, previous: undefined] | [data: undefined, previous: T]) => T
  // put the initial data inside the segment too
  initialData?: S | () => S
})
declare const createSerialized$: (opts: SerializationOptions) => Signal<T>

const mySig = createSerialized$({
  serialize: isServer && o => o.toJson(),
  deserialize: d => new MyObj(d)
})

// This will optimize to:
const mySig = createSerializedQrl(qrl(() => import('./opts.js'), 'opts'))
// ...and ./opts.js in client:
export default opts = {
  serialize: false,
  deserialize: d => new MyObj(d)
}

We could even special-case it in the optimizer to strip the serialize in the browser.

The SerializedSignal then stores the serializer function. The advantage is that if the value gets stored outside of the signal, serialization will fail, which is a good thing.

So, 1), is this API ok?

That said, I'd still like to keep the NoSerializeSymbol and SerializerSymbol, because they allow using memory more efficiently as well as cleaning up structures before serialization.
2) Is that OK?