>>7509So I've done some thinking and have some solutions and few more ideas/problems that need to be decided/figured out
So now I have:
* Some
really basic commentsJSON -> Html converter and avatar downloader written in Kotlin
* Found bug in comment downloader (it's not producing valid JSON) But I added fixer to the JSON-Html converter (I wanted to patch comment downloader but it didn't go too well)
* Figured out how the comments are linked together and it looks quite simple, but now only imageboard style is supported in converter (more on that later)
* Some more detailed info on usage included in zip
Here I have some things that I think should be decided/done/fixed
* What format to convert videos to? - youtube-dl saves it as .mkv by default and thats not friendly with browsers, so:
* Should we include more versions of video? The browser then picks what ever it can play automatically.
* if so, what format? I was of thinking of .mp4 or .webm but then there is problem with not preserving "original" or potentially taking too much space
* Should we include Meta file? (could be packaged with the archive) It could include: time of download, video name, time of release, channel name etc.
* I was thinking of JSON format because if anyone would wanted to work with these files they would need JSON library anyways
* It would be machine-readable so it could be used with potential players/viewers (I still really like the idea of having custom viewers/players one day)
* If we would need to scrap some video again for what ever reason it could be almost fully automated and repacked/sorted with little work
* Structure of the archive - What folders and what put where? I was thinking of:
*
src/ for css/js and
src/avatars/ for avatars
*
meta/ for meta file/files
*
raw/ for json comments, video, thumbnail? - I don't know if this should be in
src/ (or whatever we'll use for resources) or if there is any benefit in putting it elsewhere
* What else could we download? I noticed youtube-dl can get almost everything except comments so we could also get
* Thumbnails
* Description (Automatically archive links from description? -
may be a bit overkill actually but maybe?)
* View count
* Anything else?
* Other ways to display exported comments? Currently there is "imageboard" system, just instead of numbers you have comment IDs because it was the easiest thing to do. Some problems are:
* Currently it looks horrible (There was some attempt with CSS but I am horrible at design in general)
* It's a bit overkill as you can only reply to "OP" comment anyways
* We probably want to be closer to YouTubes look/feel?
* For the language/tooling choice. I decided to go with Kotlin for the prototypes and then when all things will be decided I would move it to either Kotlin script (so there's no need for recompilation in case of changes)(There is also Kscript for UNIX systems which could be better) or rewrite it in python as all other tools are in python and we could just use them as libraries.
None of the suggested things are currently implemented, It's just a list of ideas that crossed my mind while doing that prototype. Changes, edits and thoughts are welcomed.
Included zip contains converter/jsonFixer/avatarDownloader in one - with sources and prebuilt .jar included - JRE8 or higher required, more detailed description of contents is in README.
I can start working on it again at earliest in 1.5 months so that should plenty of time to think everything through.