How the TW internal data structure looks like and why data-tiddlers are not optimal

[pmario]

Some TW internals.

The following description of the tiddler store is a bit simplified for explanation reasons.

0
In javascript almost everything is an Object {}, that's why the browser javascript engine is highly optimized to work with them. Access to object keys is fast! ... Really fast!

1
A simple Object can look like this: {title: "HelloThere"} ... We can say: title is a key and HelloThere is a value

2
So we have an Object that contains a key : value pair.

3
But this doesn't make a tiddler. ... We need tags, created, creator, modified, modifier and so on.

4
So the tiddler-like Object looks more like this:

{
​​​​  title: "test"
  created: Date Thu Oct 14 2021 11:24:29 GMT+0200 (Mitteleuropäische Sommerzeit)
  ​​creator: "Mario"
  tags: Array [ "a", "b" ]
  ​​text: "some text"
}

As you can see, there are some values, that are strings. text, creator, title ... AND ...
There are also some other elements: created and tags that are different. created is a Date and tags is an Array [].

5
So we have 1 "tiddler-like" object now. ... But 1 tiddler doesn't make a store. right?
We need more objects to store several tiddlers. ...

Which could looks like this: An Array[ of tiddler-Object {} ]

[
  {
    ​​​​title: "test"
    created: Date Thu Oct 14 2021 11:24:29 GMT+0200 (Mitteleuropäische Sommerzeit)
    ​​creator: "Mario"
    tags: Array [ "a", "b" ]
    ​​text: "some text"
  }, 
  {
    ​​​​title: "test-1"
    created: Date Thu Oct 14 2021 11:25:09 GMT+0200 (Mitteleuropäische Sommerzeit)
    ​​creator: "Mario"
    tags: Array [ "b", "c" ]
    ​​text: "some more text"
  }
]

6
The problem with arrays is: Accessing them will need a loop. So searching for eg: title: "test-1" will need to:

• look at the first element - "test"
• compare the title if it is "text-1" ... no
• go to the next element
• compare the title ... and so on ... and so on. ...

That's a lot of work. ... A lot of CPU cycles. ... BUT ... We want it fast. ... Really fast. ... Remember step 0?? ... We need a tiddlers object{}, where the key is the tiddler title. Like so:

tiddlers = {
"test": { tiddler Object {} },
"test-1": { tiddler Object {} }
}

That's an Object-store. ...

So the tiddler-title is the key and the tiddler Object {} is the value. .. .YES ... Objects can be nested!

To retrieve test-1 is as simple as accessing tiddlers["test-1"] now. tiddlers is not an array it's an object. ... Remember step 0 Accessing object keys is fast. ...

That's the main reason why tiddler titles have to be unique. ... Object keys have to be unique, to have the fastest access possible!

So a function similar to:

var tiddler = tiddlers["test-1"] ;

will return

tiddler =  {
    ​​​​title: "test-1"
    created: Date Thu Oct 14 2021 11:25:09 GMT+0200 (Mitteleuropäische Sommerzeit)
    ​​creator: "Mario"
    tags: Array [ "b", "c" ]
    ​​text: "some more text"
  }

All of this can be done by the browser javascript engine with max speed.

[pmario]

Why data-tiddlers are sub-optimal.

A
From the post above we know how to get the content of the test-1 tiddler

Now let's say test-1 is a data tiddler. It would look like this:

tiddler =  {
    ​​​​title: "test-1"
    created: Date Thu Oct 14 2021 11:25:09 GMT+0200 (Mitteleuropäische Sommerzeit)
    ​​creator: "Mario"
    tags: Array [ "b", "c" ]
    type: "application/x-tiddler-dictionary"
    ​​text: "
        2021.10.13 : That's some text
        2021.10.14 : That's a different text ;)
    "
  }

B
If we want to access the the 2nd index 2021.10.14 we will need to:

• read the tiddler (see post 1 step 6) that's fast.
• read the text
• read line 1
• parse it
• check if it is the right index ... no
• read line 2
• parse it ... and so on and so on.

So worst case if the index we want is the last index in the tiddler we waste a lot of CPU cycles. ... To parse data-tiddler indexes and present them in a decent way in wikitext needs definitely more code then accessing and displaying a core field.

C
From post 1 you probably know, how we could make this fast. ... right?

If it would look like this:

tiddler =  {
    ​​​​title: "test-1", 
    created: Date Thu Oct 14 2021 11:25:09 GMT+0200 (Mitteleuropäische Sommerzeit),
    ​​creator: "Mario",
    tags: Array [ "b", "c" ],
    type: "application/x-tiddler-dictionary",
    ​​text: "",
    "1" : "That's some text",
    "2" : "That's a different text ;)",
    "field with spaces": "lorem ipsum .."
  }

Access would be fast. ...

[dubiouscript]

To parse data-tiddler indexes and present them in a decent way in wikitext needs definitely more code then accessing and displaying a core field.

strange/sureal that it is posible to surgest ..., however

https://github.com/petli-full/awkjs

https://awk.js.org/index.html
https://awk.js.org/assets/libs/mawk.js ( source/build steps unknown unfortunatly coz mawk is.gd )

awk is the thing for plain text , and so (in theory) should do it well tm
😁

[pmario]

I think you did misunderstand me. I wasn't clear enough. ... I was talking about filter operators as eg: getIndex and others, as I wrote about "presenting" the info to wikitext.

[dubiouscript]

I think you did misunderstand me.

ah perhaps so.. , while reading/scanning through the above observations about js data structures
i may have drifted off ⚡ recollecting some/lots of *historical exprementing with json & methods of loading 3454+ entrys on some old crap hw

*( one page is loading /jquery-1.2.6.min,.. just checked https://archive.org/details/jquery-1.2.6.min_201511 some time around 2015 =s )

I wasn't clear enough. ... I was talking about filter operators as eg: getIndex and others, as I wrote about "presenting" the info to wikitext.

your post was perfectly clear! ..i was just day dreaming while scan/reading your post

all thought ... and i say this fuly lucidly
i was reading through the wiki parser
..thinking there is not much you cant port to awk 🤔

[yaisog]

In the solution you propose in C, data tiddlers are just regular tiddlers with a lot of fields? How would you make this work together with "regular" fields – e.g. the field chooser in draft mode could become a huge list if there are data tiddlers present?

[pmario]

e.g. the field chooser in draft mode could become a huge list if there are data tiddlers present?

Yes, but that's a UI problem and not a conceptual problem. With the right filters the lists can be shortened.

[msameen99]

Thanks a lot @pmario for this fantastic presentation.
it's very strange that I was struggling on how to work with "array of tiddlers" in JavaScript? do I use "array of objects"? or "one object" contains key/value pair for each tiddler?
and surprisingly I found this post to answer my question !!

thanks again