Handling JSON::Any

[jwoertink]

If you have any jsonb columns on your models, then you'll have to deal with JSON::Any which can be sort of a pain. Let's say that you are updating this column, and the structure is saved in a Serializer.

class DataSerializer < BaseSerializer
  def initializer(@data)
  end

  def render
    {
      who: @data.username,
      what: @data.thing_user_did,
      where: @data.location,
      when: @data.created_at.to_s("%F")
    }
  end
end

SaveUserTracker.update(
  current_user,
  metadata: .... # This field is stored as a JSON::Any
)

You have a few options here..

# this fails because you can't convert a NamedTuple to JSON::Any
# So you'd have to go back and update all of your serializers to be Hash(String, JSON::Any)
metadata: JSON::Any.new(DataSerializer.new(data).render)

# This works! However, if you benchmark it, it's about 12x slower
# than manually building a JSON::Any.. maybe that's not an issue for you?
metadata: JSON.parse(DataSerializer.new(data).render.to_json)

# This also works, and it's faster than the previous version
# but.... look at it ☹️ and imagine this gets bigger
metadata: JSON::Any.new({
  "who" => JSON::Any.new(data.username),
  "what" => JSON::Any.new(data.thing_user_did),
  "where" => JSON::Any.new(data.location),
  "when" => JSON::Any.new(data.created_at.to_s("%F"))
})

There's also a few other issues along with JSON::Any doesn't allow for UUID instances, and probably quite a few others. The error message you get is also less than optimal:

Overloads are:
 - JSON::Any.new(raw : Type)
 - JSON::Any.new(pull : JSON::PullParser)

🤔 so I guess UUID, and NamedTuple aren't a raw : Type but String is...

Lastly, just the general setup of an empty JSON::Any object gets a bit tedious.

class UserFactory < Avram::Factory
  def initialize
     metadata JSON::Any.new({} of String => JSON::Any)
  end
end

class User < BaseModel
  table do
    column metadata : JSON::Any = JSON::Any.new({} of String => JSON::Any)
  end
end

class CreateUsersTable < Migration
  def migrate
    create :users do
      add metadata : JSON::Any, default: JSON::Any.new({} of String => JSON::Any)
    end
  end
end

class SaveUser < User::SaveOperation
  needs time : Time
  before_save do
    current_metadata = metadata.value.try(&.as_h) || {} of String => JSON::Any
    new_metadata = {} of String => JSON::Any
    new_metadata["when"] = JSON::Any.new(time)
    metadata.value = JSON::Any.new(current_metadata.merge(new_metadata))
  end
end

With all this information, should Lucky handle this structure internally somehow? Like providing some wrapper? Maybe monkeypatch to include a .to_json_any type method? Or is this all complex enough to ask the Crystal language itself for something better? Or do we just say that this is the downside to using json, but it provides the most type-safe handling?

[stephendolan]

@jwoertink That last thought on .to_json_any made me think of the shard Sija maintains:
https://github.com/Sija/any_hash.cr

I know we're not looking to add third party libs as dependencies, but I wonder if there are any good nuggets in that implementation?

[jwoertink]

Oh interesting. I think @Sija is active enough in the community that it might not be bad in this case if it gives us a 10x boost in happiness 😂 without sacrificing any sort of type-safety and minimal performance at worst. At least something to look in to a bit more. I guess the end goal would still be to have something built-in, but I'm not totally opposed to stepping stones in this case.

Thanks for the suggestion!

[mdwagner]

I've always just used the second implementation, but didn't know about the performance implications. This any_hash shard seems like a better solution.

[robacarp]

Don't forget about reading from the database too.

I do wish Crystal had something better for this. Part of the reason that things are so complicated is that the crystal-pg shard is built with json::any in mind rather than json::serializable. As I recall, it would be a significant breaking change to alter this because it's not possible to opt-in to the behavior.

I like the idea of using something like sija's any_hash, but I also like the idea of implementing something which converts (both read- and write-time) between the json::any and whatever serializable object is being stuffed into the column.

[jwoertink]

Yeah, our latest Lucky app is using a ton of JSON::Any columns, and it's becoming a mess real quick. I'm just not sure how to best fix it, and part of me feels like the root fix should be outside of Lucky/Avram and more in Crystal itself, or maybe pg. I'm just not sure what that would look like 😝

[robacarp]

Yeah, me too. Here's the general idea of what I'm doing. I tried to make it generic, hopefully the code still makes some sense. Adding a couple helper methods made the any-ification a bit easier to write but it doesn't cut down on the verbosity.

module JsonHelpers
  macro any(thing)
    JSON::Any.new({{ thing }})
  end

  macro blank_any
    JSON::Any.new({} of String => JSON::Any)
  end

  def to_any : JSON::Any
    blank_any
  end
end

A model:

class Model < BaseModel
   avram_enum Type do
     JsonTypeOne
     JsonTypeTwo
  end

  table do
    column type : Type
    column config : JSON::Any
  end

  def config
    case type
    when self.class.json_type_one.to_i
      Model::JsonTypeOne.from_json(config.to_json)
    when self.class.json_type_two.to_i
      Model::JsonTypeTwo.from_json(config.to_json)
    else
      raise "cannot produce a config for #{type}"
    end
  end
end

and the config object which gets de/serialized at read/write:

class Model::JsonTypeOne
  include JSON::Serializable
  include JsonHelpers

  @[JSON::Field]
  property field1 : Bool = true

  @[JSON::Field]
  property field2 : String = ""

  @[JSON::Field]
  property field3 : Int64 = 1

  def to_any : JSON::Any
    any({
      "field1" => any(field1),
      "field2" => any(field2),
      "field3" => any(field3),
    })
  end
end