For easier working with data tiddler (JSON)

[kookma]

Data tiddler (especially in form of JSON) have many use cases and are very useful to have tiny databases.

However there are some issues for easy working with them

1. getting values is tedious
2. sorting and working on data based on values is tedious and using the current filter operators is very slow

While I know there is technical difficulties as @Jermolene explained here #3971 (comment) I would like to ask is it possible to at least have some improvement (not development) for better and easier working with data tiddler.

Ref

[1] Google forum https://groups.google.com/d/msgid/tiddlywiki/65d562c5-88e4-49b1-aa5f-e399ca397a81n%40googlegroups.com
[2] @pmario pull request to address some issues #3971 (comment)

[kookma]

I am aware of great https://github.com/joshuafontany/TW5-JsonMangler plugin by @joshuafontany
But here a simple solution in the core is requested including

1. easy retrieving values
2. sorting by keys and values

[pmario]

see: https://wikilabs.github.io/editions/keyvalues/
how to use my library can be seen at: https://wikilabs.github.io/

[kookma]

@pmario many thanks. Actually I use this for https://github.com/kookma/TW-Vote to create a summary table of votings/rankings and then want to sort the table. I used your solution here and extracting keyvalues.js then using the trick by @Marxsal

I came to below solution

\define show-records-by-mario()
<$vars sortby="[split[°]nth[2]]">
<$list filter="[[$:/vote/dataTiddler/votings]keyvalues:key,value:°[]] +[$(sortNeg)$sortsub:number<sortby>]">
<tr>
<td><$link to={{{ [<currentTiddler>split[°]first[]] }}}/></td>
<td><$text text={{{ [<currentTiddler>split[°]last[]] }}}/></td>
</tr>
</$list>
</$vars>
\end

The speed is fast for less than 300 entries, on 2000 entries still it is fast (like 1 or 1.2 second to refresh the tiddler) but for 10,000 entries still it is slow but works without crashing TW.

Q1. Does https://wikilabs.github.io/editions/keyvalues/ is newer or better than http://3971.tiddlyspot.com/
Q2. What do you recommend to sort the data tiddler based on the values?
Q3. Why not ask @Jermolene to add the keyvalues.js to the core?

[kookma]

@pmario many thanks. Actually I use this for https://github.com/kookma/TW-Vote to create a summary table of votings/rankings and then want to sort the table. I used your solution here and extracting keyvalues.js then using the trick by @Marxsal

I came to below solution

\define show-records-by-mario()
<$vars sortby="[split[°]nth[2]]">
<$list filter="[[$:/vote/dataTiddler/votings]keyvalues:key,value:°[]] +[$(sortNeg)$sortsub:number<sortby>]">
<tr>
<td><$link to={{{ [<currentTiddler>split[°]first[]] }}}/></td>
<td><$text text={{{ [<currentTiddler>split[°]last[]] }}}/></td>
</tr>
</$list>
</$vars>
\end

The speed is fast for less than 300 entries, on 2000 entries still it is fast (like 1 or 1.2 second to refresh the tiddler) but for 10,000 entries still it is slow but works without crashing TW.

Q1. Does https://wikilabs.github.io/editions/keyvalues/ is newer or better than http://3971.tiddlyspot.com/
Q2. What do you recommend to sort the data tiddler based on the values?
Q3. Why not ask @Jermolene to add the keyvalues.js to the core?

[pmario]

Q3: I did create the plugin, because it was refused for the core.

[pmario]

Q2. What do you recommend to sort the data tiddler based on the values?

There have been a lot of internal optimizations in the core lately. They all are "tiddler" based. Similar stuff would probably be needed for handling data tiddlers. BUT that's not what Jeremy wants. So you should have a closer look at tiddler based solutions.

[Jermolene]

Hi @kookma I think I made my position clear in the discussion in #3971. In that thread @kookma makes the point:

I think a small database inside TW can be very useful!

That remains true, and I think the best way to accomplish it would probably be to literally put a small database inside TW. I'm interested in exploring integrating a WebAssembly port of MySQL:

https://github.com/sql-js/sql.js

The interesting thing for TiddlyWiki is that the MySQL API is entirely synchronous, which means that we could use it to provide an alternate implementation of the main TiddlyWiki tiddler store.

Getting MySQL up and running seems to open up much more interesting possibilities than tweaking the data tiddler implementation.

[kookma]

Thank you @Jermolene - That is great to integrate to small sqlite or MySQL and it will opens great new opportunities.
I appreciate your quick response. I still believe the fix by @pmario worth to be considered for possible merge.