How to assign unique IDs to tiddlers?

[ichorid]

I've succeeded with creating a plugin for syncing tiddlers to a web server (basically, created a cut-down version of TiddlyWeb plugin, as per advice in this discussion)

Now, to make the sync logic to work with our p2p backend, I need a unique ID on every user-created tiddler (not necessary UUID, just a random number will suffice).

How do I write a plugin that will assign each tiddler a unique ID, right when a toddler is created?

[AnthonyMuscio]

I have a wikitext and macro solution.

Just increment a global variable, and save it back. You can issue them serially, If random you need to always test for collisions. More often than not the created date would suffice, and you can increment microseconds until unique. The use of a custom field with a prefix or suffix will reduce the chance of collisions across wikis; eg use a custom wikiname in the serial number, along with a string for your solution.

Tony

[ichorid]

I have a wikitext and macro solution.

Would you kindly point me to the source code of a TW plugin that does something like that?

(I don't want to edit TW code directly, I want to produce a forward-compatible plugin that will allow me to easily upgrade TW in the future, without applying any custom patches by hand.)

Thank you! 😉

[kookma]

I think @pmario has a plugin for this! Have a look at his repos.

[Jermolene]

Hi @ichorid

I've succeeded with creating a plugin for syncing tiddlers to a web server (basically, created a cut-down version of TiddlyWeb plugin, as per advice in this discussion)

Excellent!

Now, to make the sync logic to work with our p2p backend, I need a unique ID on every user-created tiddler (not necessary UUID, just a random number will suffice).

How do I write a plugin that will assign each tiddler a unique ID, right when a toddler is created?

TW5 uses the title field as a unique ID for each tiddler. You may be able to use the title directly, but if you'd rather have a fixed length identifier you could just hash the title and take a fixed number of bits to yield the ID. The sync adaptor would probably have to maintain a mapping of ID to title.

Using the title as the ID means that renaming a tiddler has to be modelled as deleting the old tiddler and creating the new one. I believe @pmario's plugin attempts to track IDs across renames.

[ichorid]

The kind of content we're going to serve tends to rely on renaming of entities titles. E.g. "Show X ep.1-7" is going to be renamed to "Show X ep.1-8`, etc. Therefore, we need stable IDs, as they do on torrent tracker forums.

Is it possible to add an invisible field (not a title!) with a randomly generated number to a tiddler, when it is created? Creating an empty tiddler, changing even a single letter in it, everything results in sync engine sending a copy of it to the sync backend. On the backend, I got no way to distinguish if the user just created a new tiddler with copy-pasted contents from an existing one, or just renamed the original.

Adding unique internal IDs to tiddlers would solve that problem without compromising "Title is the ID" idea.

[Jermolene]

Hi @ichorid

Is it possible to add an invisible field (not a title!) with a randomly generated number to a tiddler, when it is created?

Yes, you could create an "id" field containing a GUID that you'd add to each tiddler when it is created. That should work as long as you don't have to worry about users editing tiddlers and removing/altering that field.

[ichorid]

Yes, you could create an "id" field containing a GUID that you'd add to each tiddler when it is created.

I'm perfectly fine with that!

How can I create a plugin that will extend the Tiddler class to do that? (There are no examples in the Core TW wiki for that, and the closest thing I found is for a very old TW version.)

[Jermolene]

How can I create a plugin that will extend the Tiddler class to do that? (There are no examples in the Core TW wiki for that, and the closest thing I found is for a very old TW version.)

I thought you were injecting the tiddlers from the server, in which case the server can assign the ID field. If you've got to deal with users creating new tiddlers then you'd need another approach, and I think that's where @pmario's plugin might help.

[ichorid]

@saqimtiaz pointed out that it is impossible to create a plugin that modifies the Tiddler class? Is it true?

[pmario]

I did some experiments 10 years ago with tiddler titles as UUIDs ... but that's not convenient. ... At the moment I use tiddler names in a-format-like-this and never change them. So the hardest part is finding tiddler names that actually work.

Wikitext linking is done using the "aliases" field and my uni-link plugin. So both names are human readable, which imo is the biggest advantage of my actual system. ... BUT that's not what you are after..

I personally would go for UUIDs. They are easy to create and you don't have to think about collisions. UUIDs in general means RFC 4122 IDs, which are Hex based and need 36 chars for 5.3x10^36 possible numbers.

I'd use a number which is based on 62 characters. So you get 4.4x10^37 possible "numbers" out of a string with 21 chars eg: H0k4LRUUKFns0WE6othS3

I did some other experiments which used different elements, that are "kind of" unique eg: date in miliseconds since 1970 + domain name + local miliseconds
eg:

human id  : [email protected]
machine id: 24699C90BC43DAA240EEFCD06BDB818C6C1AEA2D     <- above string hashed
UUID V4   : 24699C90-BC43-4DAA-8240-EEFCD06BDB81    <- above string modified to look like a UUID

just some ideas

[ichorid]

Thanks for the suggestion! My problem is not what to do, however, but instead how to do it specifically in TiddlyWiki, i.e. how to write a compatible plugin that will add an ID field (whatever type) to the Tiddler class instances.

FYI, in Tribler we identify pieces of information by a combination of the user's public key and a random 64-bit int. That works perfectly, because it guarantees there will be no ID collisions other than those willingly produced by the user (and we don't handle those).

[saqimtiaz]

@ichorid as far as I am aware there are no facilities for extending the Tiddler class. Plugins are loaded via Tiddlers, so it is a bit of a chicken and egg problem.

The usual way to go about this would be to look at the potential ways that your users might create new tiddlers, and ensure that they add the field that you need. E.g: the new tiddler button. How much you want to customize the TW UI, and how much flexibility to use TW features will probably play a role here.

You could try to reach into the core and patch the Tiddler class ($tw.Tiddler), but it might turn out to be a brittle approach in the long run.

[pmario]

As Saq wrote,

The usual way to go about this would be to look at the potential ways that your users might create new tiddlers, and ensure that they add the field that you need.

If your users use the UI it should be simple to add a th-saving-tiddler hook, which can add an ID field to every tiddler, that is created. .. The th-saving-tiddler hook function is executed by the navigator widget. ... So it will be UI level.

If you create tiddlers programmatically eg: $tw.wiki.addTiddler() this hook will not work, since the "wiki" function call is lower level. ...

So: How do you create new tiddlers?

[ichorid]

Typically, the user will create a tiddler by pressing a ➕ button in TW right-panel. But I guess there are other ways to do it in TW too. That's why I would prefer a lower-level solution (e.g. creating the ID in the object's constructor)

[ichorid]

Ok, so the question is: how to add a custom field programmatically, from a plugin? (e.g. in a sync plugin, on synchronization). I spent a few hours looking for a way to do it.

[pmario]

You are right. The sync mechanism could do it.

[AnthonyMuscio]

Issuing ids to all tiddlers can be done in the UI adding appropriate actions to the required buttons, one case to remember is cloning, you need to reset that field to a newly issued one. In the save mechanisium you can first trigger a check all "has changed" tiddlers for the ID AND GENERATE if not. This will catch tiddlers created from following camel case links or custom buttons not using a SET UID action. For a database the ID is only needed before saving.

Also rather than just a unique number simply allow a static prefix with a wiki name or epoch. Still increment the number but give it a seed value. The chance of duplicate ID's is reduced and can be tested for interwiki transfers.

This obviates the need to generate a UNIVERSAL UID which uses far more bytes.

Regards
Tony

[ichorid]

So, if I understand correctly, it is impossible to add a field from a plugin? If so, I'm left with two options:

1. change the GUI side to create UIDs during the creation of a tiddler
2. employ sync plugin's store to generate and store the UIDs in the plugin object (adaptorInfo property).

[saqimtiaz]

@ichorid if adding a field in the sync plugin would meet your needs, that is entirely possible. Apologies, you're getting a lot of replies that don't take into account the context of the thread, presumably out of a desire to be helpful despite the busy time of year.

If you post your plugin and point out where in the flow you want to add the field, I can assist you with that.
Otherwise the things to keep in mind are that:

• Tiddler class objects are immutable.
• The way to add a field programmatically is to essentially create a new Tiddler with the same title and add it to the wiki store.
• The usual pattern is as follows below:

this.wiki.addTiddler(new $tw.Tiddler(oldTiddler,newFields));

In your case newFields would be a simple object like { "uuid" : myid }

[saqimtiaz]

@ichorid there is no facility to to extend or hook into the Tiddler class constructor, since the Tiddler class itself is used to load plugins from tiddlers.

Technically you could try monkey patching the Tiddler class in a startup module, it can be accessed via $tw.Tiddler
However this is not an approach I would recommend as it can lead to problems down the line.

I think there is a two-fold approach that will meet your needs:

1. If the tiddler is new and being created by the sync plugin, the plugin should add the UUID.
2. You can add a hook that is triggered whenever a save happens via the UI. This is the th-saving-tiddler hook mentioned above, which can be added via a startup module. Your hook should do two things:
   • add a UUID to the tiddler being saved if it does not exist
   • ensure the UUID is unique if it does exist, if it is not unique then issue a new UUID. This will cover the situation where a user triggers the "clone" a tiddler process in the UI which would duplicate all the fields.

Here is an example of a th-saving-tiddler hook being used:
https://github.com/Jermolene/TiddlyWiki5/blob/14a28b77796461c9167898793ab9851e029e0354/plugins/tiddlywiki/savetrail/savetrail.js#L38

You can add a field and return a new Tiddler object.

Some documentation: https://tiddlywiki.com/dev/#Hook%3A%20th-saving-tiddler

Lastly you will want to hide the UUID field from being shown in the editor, by adding a tiddler via one of your plugins with the title:
$:/config/EditTemplateFields/Visibility/YOUR_UUID_FIELD_NAME
where the text field has the value:
hide

Hope this helps.

[pmario]

As Saq wrote. this.wiki.addTiddler(new $tw.Tiddler(oldTiddler,newFields)); is extreamly powerful if you do it the other way around. this.wiki.addTiddler(new $tw.Tiddler(newFields, oldTiddler)); The hashmap that is given to the new $tw.Tiddler() function will take precedence from left to right. So last one wins!

So if you define a new UID in newFields = {"uuid", createUUID()} you don't need to check if the existing tiddler has a uuid field. ... Because if it has one it will be overwritten from the tiddlers existing field.

So you don't need to add any checks, if the tiddler has an existing uuid. You only need to take care, that the existing one wins.

[ichorid]

@saqimtiaz ,sir, you just saved me a couple of days of work! 🥇

[pmario]

So the only function that needs to create the UID is a "sync write to backend". That's basically it.

[saqimtiaz]

@saqimtiaz ,sir, you just saved me a couple of days of work! 🥇

Happy to be of help.

You can just mark this post, the replies underneath it are all relevant to anyone looking for the same information.

[AnthonyMuscio]

I have spent a lot of time on this subject so please consider these comments. They are a result of considerable design and analysis and will save time in developing such a solution.

There are a range of powerful features that become possible if we have an additional field on tiddlers, that I have being calling a Tiddler Serial Number TSN. I have built this as a complete solution already, using wikitext and macros (no plugins) but of course a feature such as this would be better at a lower level plugin or core.

Whilst I support the idea of a Universal ID, I believe issuing a serial number each time one is needed, by incrementing a number and adding a configurable prefix, such as a user specified wikiname or epoch, is sufficient and requires less bytes. A utility to rename a (Serial number) prefix if necessary, provides the opportunity to adjust the unique identifier as needed.

I have also built a method to add serial numbers to existing core tiddlers without modifying them by storing the system Serial Number in a separate json file. Core tiddlers will have a "name to system serial number" table that will never change. Once a shadow is cloned or renamed they are issued a TSN.

There are many cases where only issuing a Serial number if required is also practical, such as for compound tiddlers, like we see in streams. If the parent tiddler has a serial number, each child can be issued there own serial number, add this to a field in the parent, and have the parent tiddlers serial number recorded in the children. The Parent and child tiddlers can then be freely renamed as many times as needed whilst retaining their relationships.

This same "serial number as required mechaisium" could be used for unique ID's in a database or saver backend by simply using the same on demand serial number before saving a given tiddler the first time. A database name could also be specified, and made part of the serial number used to save tiddlers. The Serial number field once issued should not be available to edit; use $:/config/EditTemplateFields/Visibility/fieldname with the value "hide".

Another trick is to create a mechaisium to increment the last serial number by one, add the current prefix and before saving convert the incremented number to a more compact base number, just as hex is more compact with its 16 characters (base 16), you can make a base 36 or base 62 using a-z A-Z and 0-9. The serial number never changes and compressing it this way consumes far fewer bytes. Starting with one byte and only growing as needed. 5 bytes can represent more than 14,700,000 tiddlers.

With an modifiable prefix, one can continue issuing new serial numbers but change the prefix, this does not affect the uniqueness but does allow a different prefix to be set using the current user name, or an epoch, such as development time, then once in "production". Thus the serial number can carry other information that is related to the issuing of serial numbers.

An Additional on demand serial number based on the existing ones, could always be generated using a registered wikiname to make such a serial number unique, and totally transferable between wikis. Not to mention the use of namespaces for tiddlers similar to the current plugin and macro tiddlers.

Regards
Tony

[tejasvi]

TiddlyMap implements a hidden tmap.id field to track tiddlers. You can start from here.

[ichorid]

Wow, thanks! That should help!