Clip as markdown

[arunnbabu81]

Currently there is an option to clip from website as html. Is it a viable option to add clip from website as markdown. I found an addon called Markdownload which download webpages in markdown format. I think many people will be interested in clipping webpages as markdown directly into tiddlywiki.

You may shift this discussion to the tiddlyclip addon repo

[buggyj]

Conversion would be done in the tiddlywiki not the addon. There is a TC macro to convert to TW2 that I wrote some years ago - @htmlToTW2(@web). I never found it useful to covert - often the conversion is not good, and with the ckeditor I found it easier to add annotations to the html.
Also I do not see how markdown would be more useful?

[arunnbabu81]

Conversion would be done in the tiddlywiki not the addon

I was asking whether the add-on can have the ability to clip the webpage content in markdown format and then send it to wiki (Similar to the MarkDownload addon I shared)

Also I do not see how markdown would be more useful?

Personally I haven't used markdown yet. But I have seen discussions in forums, that markdown is more future proof. Don't know how true is that.

There is a TC macro to convert to TW2 that I wrote some years ago - @htmlToTW2(@web). I never found it useful to covert - often the conversion is not good, and with the ckeditor I found it easier to add annotations to the html.

I have seen a macro written by Mark to convert html to wikitext in the google group. I have tried to convert some article which were clipped as html using tiddlyclip to wikitext - although the text part was converted ok, the images were couldn't be rendered after the conversion. So my experience with conversion is also not that great.

Another thing I noticed with clipped html tiddler were that - if I open them in ckeditor, there was not much lag even if the html tiddler in large. But if I tried to open them in the default TW editor, there was some noticeable lag . So I was confused whether to clip large articles as html. That's why I thought of markdown format. How is your experience with clipping of large articles as html.

and with the ckeditor I found it easier to add annotations to the html.

Mark is developing is an annotator based on the official dyna-annotate plug in. May be that can improve the annotation capabilities in TW.

[buggyj]

markdown is used in a different context to tw5 markup. tw5 is really a software development language, a bit like react. Markdown is used where some formatting of text is required but html must be avoided for security reasons.

I found annotating (in general) not to be very useful - it takes a lot of effort and I did not find the annotations very useful when re-reading (a part from when I am learning foreign languages). However with the annotations I make with ckeditor I can extract them from the text using tiddlyclip.

How big is a large clip? Ckeditor4 was the best editor for adding to webpages it was developed by a team of software engineers, i.e a lot of time and effort was put into making to work well so I am not surprised that it works better that the tw editor with large texts.

I tend to take multiple clips from different webpages on the subject I am interested in and then write some separate notes (sometimes) based upon the clips (which can be video/audio as well as text.). I do this occasionally for a client. I also create 'paper trails' of the websites I visit as part of my research.

[arunnbabu81]

However with the annotations I make with ckeditor I can extract them from the text using tiddlyclip.

How to do that?

I tend to take multiple clips from different webpages on the subject I am interested in and then write some separate notes (sometimes) based upon the clips (which can be video/audio as well as text.). I do this occasionally for a client. I also create 'paper trails' of the websites I visit as part of my research.

Can you share a sample of how you take notes in this way

How big is a large clip?

This is an example of the large clip I took

https://pubs.rsna.org/doi/10.1148/rg.2020190085

I used select feature to select the full text. I was experimenting with clipping of the entire article like this. Is that a bad strategy. Should I clip only the relevant text only

[Marxsal]

I use an extension called "copy cat" to copy selected text to markdown. (there are other extensions like this). Then I use tiddlyclip and the clipboard option to clip the contents of the clipboard. I have this set up to create the markdown field type. So it's only one extra step over directly clipping in markdown. My rule looks like this:

\rules only table wikilink prettylink

|!Title|!Body|!Tags|!Fields modifiers|!field initials|!Modes|
|((*@pageTitle*))|((*@pageRef*))\n\n((*@clip*))\n\n|Articles 2022||{"$url":"((*@pageRef*))"},{"$caption":"((*@pageTitle*))"},{"$when":"((*@dateTimeShort*))"},{"$type":"text/x-markdown"}|append add|

My feeling is that most HTML that you clip is "noisy", taking up to 3 times or more the space that the same text in markdown would take.

I do wish there was official support for WYSIWYG in TW. I really thought that TW5 was going to have a WYSIWYG editor available, at least in the plugin library.

[buggyj]

@Marxsal It is good to see people making their own methods of working with tiddlyclip(TC). I have tried to make TC as configurable as possible while still having some basic functionality out-of-the-box. tiddlyclip is not opinionated about the clips, allowing the user to decide how to created them. Personally I do not have any need for markdown conversion at this time, but when I do I will definitely consider you method.

A Webpage's html source-code can be extremely complex, so when we clip a part of the page(as html) it is not surprising that we get a lot of html markup (often broken as it is incomplete). I find that ckeditor is very good at handle these fragments of html, and it enables me to hightlight sections easily.

The first version of the ckeditor adpater plugin was release 8 years ago - I look from time to time to see if a better js html editor is available but I have not found a more suitable one.

[buggyj]

@arunnbabu81 - I try to use tiddlyclip to condense information I find interesting on the web, so I try to make the smallest clips, however sometime this is a large amount.

I use tiddlyclip with a number of other plugins that I have written - my aim is to keep clips separate from my own writings, and so I put clips inside plugin tiddlers which I call 'closedlists'.

https://github.com/buggyj/tw5-other/tree/closedlist/xplugins/bj/closedlist

[buggyj]

I need to update the plugin to work with the lastest TW releases. As you can see there are a lot of options. When I have updated it I will put a demo on tiddlyhost.com with some notes.

searchtids is a electron based desktop app that I am writing - it allows me to search all my tiddlywikis I have on my computer for tiddlers base on content, tags and title.

[arunnbabu81]

I need to update the plugin to work with the lastest TW releases. As you can see there are a lot of options. When I have updated it I will put a demo on tiddlyhost.com with some notes.

Will the closedlist plug in work with T1 5.2.1 presently. I tried it in my wiki based on 5.2.1, but i could not find the add-on context menu as seen in the demo site (as shown in the image below).

searchtids is a electron based desktop app that I am writing - it allows me to search all my tiddlywikis I have on my computer for tiddlers base on content, tags and title.

Eager to test it out once you finish the development of the searchtids

[buggyj]

you need all the plugins that are on the demo page. - BUT this is not a release version there are some issues (eg navigation in story not working), and I have to decide whether to support the new template cascade (presently it depends on the types template plugin)

[arunnbabu81]

searchtids is a electron based desktop app that I am writing - it allows me to search all my tiddlywikis I have on my computer for tiddlers base on content, tags and title.

@buggyj Is this available for testing?

Anything similar possible for mobile?

[buggyj]

there is a problem with a library that I have used - I need to find a replacement but at present I am prioritizing tiddlyclip.

I do not use tiddlywiki on my mobile, so I have not investigated this for the mobile.

[buggyj]

New tags can be added via editing the closedlist tiddler, eg here 'newtag' has just been added - it needs to appear in both places indicated. (notice the comma separator on the lower field)

The new tags then appears after saving:

clicking on a tag pill shows the clips that have that tag, the tag is remove by click on the small cross in a circle on the right side. tags are added by dragging a tag pill and dropping it on the large cross on the left side.

[Marxsal]

When I try to use "snappy", something briefly appears in the side bar making me think download is happening. But when I navigate to the docked page there is an alert that says "This page says" -- but no content. No file appears in the "media" sub directory.

Thanks!

[buggyj]

I use savemedia together with savetiddlers so myTWs are in a sub-directory of the download directory.

[Marxsal]

For these experiments I have file:

file:///home/mark/temp/tiddlywiki/bj/closed-list-demo.html

and download directory set to

/home/mark/temp/tiddlywiki/bj/

Does savemedia need savetiddlers to work?

This is chromium. There's a note in your readme, but I didn't understand what the implications were. I'm pretty sure I got savemedia to work a few weeks ago.

Thanks!

[buggyj]

savemedia is not dependent on savetiddlers. you can observe the savemedia work - go to chrome://extensions and chick on the 'background page' button and then select the console tab - then do a snap and see what appears

[Marxsal]

Ok, so the media saver wants the download directory to be 2 levels above the media directory. Or at least that's what it took for the media to save. This was different from the tiddlersaver, which was fine with having the save directory be the same as the download directory.

[Marxsal]

There's something weird going on with the tagging. I created a "test" tag and then dragged it on to an article that was untagged. Immediately it disappeared. Now I can see the item only if I click on "all tags". But then I can't tag it because there's no tagging dialog. If I click on the "test" tag, the item I tagged doesn't show up -- the list is empty!

I think the lack of transparency re the tagging would be very hard to deal with long term.

With regular plugins I can access the shadow tiddler and examine it. But with the closed list tiddler I'm kind of barred from any sort of tweaking or fixing.

[buggyj]

I guess it is pretty confusing! The interface is a template it can be improved... I have made some changes to make the tags visible and added a delete-tag drop icon (cross in a circle), I will upload it when later today.

[buggyj]

@arunnbabu81 @Marxsal I have put a demo here:

https://closedlistdemo.tiddlyhost.com/