fix(store-sync): skip invalid utf-8 characters in strings before inserting into postgres

[alvrs]

Currently an invalid UTF-8 character in a string type column makes the decoded postgres indexer error with PostgresError: invalid byte sequence for encoding "UTF8": 0x00. This PR adds a cleanup step before upserting values into postgres to remove 0x00 characters.

[changeset-bot]

🦋 Changeset detected

Latest commit: 0a524cd

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 30 packages

Name	Type
@latticexyz/store-sync	Patch
@latticexyz/cli	Patch
@latticexyz/dev-tools	Patch
@latticexyz/explorer	Patch
@latticexyz/store-indexer	Patch
@latticexyz/world-module-erc20	Patch
@latticexyz/world-modules	Patch
@latticexyz/abi-ts	Patch
@latticexyz/block-logs-stream	Patch
@latticexyz/common	Patch
@latticexyz/config	Patch
create-mud	Patch
@latticexyz/entrykit	Patch
@latticexyz/faucet	Patch
@latticexyz/gas-report	Patch
@latticexyz/paymaster	Patch
@latticexyz/protocol-parser	Patch
@latticexyz/react	Patch
@latticexyz/recs	Patch
@latticexyz/schema-type	Patch
solhint-config-mud	Patch
solhint-plugin-mud	Patch
@latticexyz/stash	Patch
@latticexyz/store-consumer	Patch
@latticexyz/store	Patch
@latticexyz/utils	Patch
vite-plugin-mud	Patch
@latticexyz/world-module-callwithsignature	Patch
@latticexyz/world-module-metadata	Patch
@latticexyz/world	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[frolic]

wouldn't this corrupt the data if a \0 is in the middle of the string somewhere?

[frolic]

some interesting discussion here around this problem: ponder-sh/ponder#1456

[alvrs]

wouldn't this corrupt the data if a \0 is in the middle of the string somewhere?

Postgres doesn't support \x00 bytes in strings, so the only alternative to removing the bytes here is skipping the entire record or throwing an error, both of which are worse my opinion (especially because this is often user provided data, so a user could make the indexer break for everyone by providing a malformed string).

Seems like the discussion in ponder-sh/ponder#1456 also tends towards this behaviour as default. I don't think we need to add a flag to disable this behaviour until someone requests it.

[frolic]

I like Ponder's naming

Suggested change

	export function cleanStrings<TSchema extends ValueSchema>(
	export function removeNullCharacters<TSchema extends ValueSchema>(

[frolic]

thoughts on doing this call just before we pass into postgres, rather than here, in case we later need to use value for something?

e.g.

 await tx
    .insert(sqlTable)
    .values({
      __keyBytes: keyBytes,
      __lastUpdatedBlockNumber: blockNumber,
      ...key,
      ...removeNullCharacters(value),
    })

[frolic]

ohh I was worried that these fields were being used to re-encode the data before mutating via event, but I don't think that's a concern here because we still store the raw bytes and use that as the source of truth