/hydrus/ - Hydrus Network

Hydrus Bunker

Posting mode: Reply

Check to confirm you're not a robot
Drawing x size canvas

Remember to follow the rules

Max file size: 350.00 MB

Max files: 5

Max message length: 4096

Manage Board | Moderate Thread

Return | Catalog | Bottom

Expand All Images

Version 379 Anonymous Board owner 01/02/2020 (Thu) 03:42:33 Id: b13cd3 [Preview] No. 459
https://youtube.com/watch?v=nvy6cHhpZdQ [Embed]
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v379/Hydrus.Network.379.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v379/Hydrus.Network.379.-.Windows.-.Installer.exe
app: https://github.com/hydrusnetwork/hydrus/releases/download/v379/Hydrus.Network.379.-.macOS.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v379/Hydrus.Network.379.-.Linux.-.Executable.tar.gz
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v379.tar.gz

Happy New Year! Although I have been ill, I had a great week, mostly working on a variety of small jobs. Search is faster, there's some new UI, and m4a files are now supported.


As hoped, I have completed and extended the search optimisations from v378. Searches for tags, namespaces, wildcards, or known urls, particularly if they are mixed with other search predicates, should now be faster and less prone to spikes in complicated situations. These speed improvements are most significant on large clients with hundreds of thousands or millions of files.

Also, like how system:inbox and system:archive 'cancel' each other out, a few more kinds of search predicate will remove mutually exclusive or redundant predicates already in the search list. system:limit predicates will remove other system:limits, system:audio/no audio will nullify each other, and--I may change this--any search predicate will replace system:everything. I have a better system for how this replacement works, and in the coming weeks I expect to extend it to do proper range-checking, so a system:filesize<256KB will remove a system:filesize<1MB or system:filesize<16KB or system:filesize>512KB, but not a system:filesize>128KB.

Anonymous Board owner 01/02/2020 (Thu) 03:43:39 Id: b13cd3 [Preview] No.460 del

I have started on some quality of life for the downloader UI. Several of the clunky buttons beneath the page lists are now smaller icons, you can now 'retry ignored' files from a button or a list right-click, any file import status button lets you right-click->show all/new in a new page, and the file import status list now lets you double-click/enter a selection to show that selection in a new page.

I have rolled in a fixed derpibooru downloader into the update. It seems to all work again.

With the pixiv login script confirmed completely broken with no easy hydrus fix in sight, if you have an 'active' record with the old, now-defunct default pixiv login script, this week's update will deactivate it and provide you with a note and a recommendation to use the Hydrus Companion web browser addon in order to login.

the rest

m4a files are now supported and recognised as audio-only files. These were often recognised as mp4s before--essentially, they are just mp4s with no video stream. I have made the choice for now to recognise them as audio-only even if they have a single frame 'jpeg' video stream. I hope to add support to hydrus for 'audio+picture' files soon so I can display album art better than inside a janked single-frame video.

The 'remove' and 'select' menus on the thumbnail right-click have been improved and harmonised. Both now lay out nicely, with file service options (like 'my files' vs 'trash' when there is a mix), and both provide file counts for all options. Support for selecting and removing from collected media is also improved.

Anonymous Board owner 01/02/2020 (Thu) 03:44:49 Id: b13cd3 [Preview] No.461 del
full list

- downloaders:
- the right-click menus from gallery and watcher page lists now provide a 'remove' option
- gallery and watchers now provide buttons and menu actions for 'retry ignored'
- activating a file import status list (double-clicking or hitting enter on a selection of rows) now opens the selection in a new page
- file import status buttons now have show new/all files on their right-click menus
- on gallery and watcher pages, the highlight, clear highlight, pause files, and pause search/check buttons are now smaller bitmap buttons
- as the old default pixiv login script is completely broken, any client with this active will have it deactivated and receive an update popup explaining the situation and suggesting to use Hydrus Companion for login instead
- updated the derpibooru downloader
- .
- search:
- when search predicates are added to the active search list, they are now better able to remove existing mutually exclusive/redundant predicates:
- - system:limit, hash, and similar to predicates now remove other instances of their type
- - system:has audio now removes system:no audio and vice versa
- - any search predicate will remove system:everything (see how you feel about this)
- improved 378's db optimisation to do tag searches in large file domains faster
- namespace search predicates ('character:anything' etc...) now take advantage of the same set of temporary file domain optimisations that tag predicates do, so mixing them with other search predicates will radically improve their speed
- wildcard search predicates, which have been notoriously slow in some cases, now take full advantage of the new tag search optimisations and are radically faster when mixed with other search predicates
- simple tag, namespace, or wildcard searches that are mixed with a very large system:inbox predicate are now much faster
- a variety of searches that include simple system predicates are now faster
- integer tag searches also now use the new tag search optimisation tech, and are radically faster when mixed with other search predicates
- system:known url queries now use the same temporary file domain search optimisation, and a web-domain search optimisation. this particularly improves domain and url class searches
- fixed an issue with the new system:limit sorting where sort types with non-comprehensive data (like media views/viewtime, where files may not yet have records) were not delivering the 'missing' file results
- improved the limit/sort_by logic to only do sort when absolutely needed
- fixed the system:limit panel label to talk about the new sorted clipping
- refactored tag searching code
- refactored namespace searching code
- refactored wildcard searching code and its related subfunctions
- cleaned all mappings searching code further

Anonymous Board owner 01/02/2020 (Thu) 03:45:25 Id: b13cd3 [Preview] No.462 del
- the rest:
- m4a files (and m4b) are now supported and recognised as separate audio-only mp4 files. files with a single jpeg frame for their video stream (such as an album cover) should also be recognised as audio only m4a for hydrus purposes for now. better single-frame audio support, including functional thumbnails and display, is planned for the future. please send in any m4a or m4b files that detect incorrectly
- the remove thumbnail menu has been moved to a new, cleaner file filtering system. it now presents remove options for different file services and local/remote when available (most of the time, this will be 'my files'/'trash' appearing when there is a mix), including with counts for all options
- the select thumbnail menu is also moved to this same file filtering system. it has a neater menu, with counts for each entry. also, when there is no current focus, or it is to be deselected, the first file to be selected is now focused and scrolled to
- for thumbnail icon display and internal calculations, collections now _merge_ the locations of their members, rather than intersecting. if a collection includes any trash, or any ipfs members, it will have the appropriate icon. this also fixes some selection-by-file-service logic for collections
- import folders, export folders, and subscriptions now explicitly only start after the first session has been loaded (so as well as freeing up some boot CPU competition, a quick import folder will now not miss publishing a file or two to a long-loading session)
- the subscription manager now only waits 15s before starting first work (previously, the buffer was 60 seconds)
- rearranged migrate tags panel so action comes before destination and added another help text line to clarify how it works. the 'go' confirmation dialog now summarises tag filtering as well
- tag filter buttons now have a prefix on their labels and tooltips to better explain what they are doing
- the duplicate filter right-center hover window should now shorten its height appropriately when the pairs change
- fixed a couple of bugs that could appear when shutting down the duplicate filter
- hackily 'fixed' an issue with duplicates processing that could cause too many 'commit and continue?' dialogs to open. a better fix here will come with a pending rewrite
- dejanked a little of how migrate tags frame is launched from the manage tags dialog
- updated the backup help a little and added a note about backing up to the first-start popup
- improved shutdown time for a variety of situations and added a couple more text notifications to shutdown splash
- cleaned up some exit code
- removed the old 'service info fatten' maintenance job, which is not really needed any more
- misc code cleanup
- updated to Qt 5.14 on Windows and Linux builds, OpenCV 4.1.2 on all builds

next week

Next week is a 'medium size job' week. Now I am more comfortable with Qt, I would love to see if I can get an MPV window embedded into hydrus so we finally have legit video+audio support. I can't promise I can get anything but a rough prototype ready for 380 for all platforms, and there is a small chance it just won't work at all, but I'll give it a go.

Hydrus had a busy 2019. Starting with the jump to python 3, and then the duplicate storage and filter overhaul, the Client API, OR search, proper audio detection, the file maintenance system, multiple local tag services, tag migration, asynchronous repository processing, fast tag autocomplete, and all the smaller improvements to downloaders and UI workflow and latency and backend scheduling and optimisations for our growing databases, and then most recently with the huge Qt conversion. The wider community also had some bumps, but we survived. Now we are in 2020, I am feeling good and looking forward to another productive year. There are a couple of thousand things I still want to do, so I will keep on pushing and try to have fun along the way. I hope you have a great year too!

Anonymous 01/02/2020 (Thu) 16:39:33 Id: 15a6b6 [Preview] No.463 del
You're the best! Thanks for all the hard work and happy new year!

Anonymous 01/03/2020 (Fri) 08:22:25 Id: dd376a [Preview] No.464 del
Happy new year!

I have an issue: In the manage subscriptions dialogue, Hydrus warns that subscriptions with >200k items are a problem (that's oddly not many - why is it much more limited than the rest of Hydrus?).

But the manage subscription dialogue itself already launches suspiciously slowly, so I took the warning and started to split a small 150k item subscription. Hydrus now has been frozen for 30 minutes and still did not finish; and from the apparent inactivity in the machine's IO/process monitor I suspect it won't finish at all. Am I experiencing some bug?

Anonymous 01/03/2020 (Fri) 12:37:31 Id: ff3ce0 [Preview] No.465 del
>I would love to see if I can get an MPV window embedded into hydrus so we finally have legit video+audio support.

Anonymous Board owner 01/04/2020 (Sat) 16:49:12 Id: 5a15d0 [Preview] No.467 del
Thank you for this report. I have had a couple of other reports like this recently. I had thought I had fixed it to always work fast when subs get too big, but I will give it another look. I am not sure if I will get to it this week, but if I have time, I will.

If it takes more than ten minutes, I recommend you just kill the process. It is an inefficient CPU thing from a system that I designed for 1,000-odd files, now trying to deal with a problem a couple of orders more complicated.

The 200k soft limit is again due to legacy code. Subs are essentially URL caches bundled into one object, basically one file, that is saved to the db file. When we had one query per sub, this meant a max size of a couple hundred KB, but when you multiply that by a thousand, suddenly it hits some buffer limits with SQLite itself and a save fails. I think the actual limit is about 800,000 URLs, depending on the type of sub.

The blobby storage is why loading that panel is slow once you hit six or seven figures cached URLs--it has to load everything. Next step here is to break the monolithic sub object up and keep each query's URLs in the db until they need to be consulted. This fast access to header data will also allow the Client API to access subscriptions.

Anonymous 01/07/2020 (Tue) 08:04:56 Id: dd376a [Preview] No.468 del
Yes, I did kill the Hydrus process; it basically just wouldn't complete.

I see, it operates on BLOBs. Was it too inefficient to put URL strings (+ indexed URL hash & URL length field?) or such into SQLite? I seem to recall it wasn't too efficient at storing such data.

Anonymous 01/07/2020 (Tue) 08:10:12 Id: dd376a [Preview] No.469 del
Yes, I did kill the Hydrus process; it basically just wouldn't complete.

I see, it operates on BLOBs. Was it too inefficient to put URL strings (+ indexed URL hash & URL length field?) or such directly into a SQLite table?
I seem to recall it wasn't the database of choice for this kind of a thing. Although perhaps this is just for the trivial way of inserting the data & I figure it might still work better than the BLOB does now.

Anonymous 01/07/2020 (Tue) 16:23:05 Id: cc1628 [Preview] No.470 del
(38.59 KB 738x916 error.png)
Is the gallery downloader broken for rule34hentai? I tried entering tabs and it would error out. I have trouble accessing the site as well where it would be stuck at checking if I could access the site until I turned off my adblock and refreshed.

Anonymous Board owner 01/07/2020 (Tue) 21:15:59 Id: 5a15d0 [Preview] No.471 del
Subs are really an extension of a simple legacy system that has grown too big for its boots. Normal stuff like known URLs for files is all stored as properly indexed data in the db, but subs and gui sessions are atomic blobs atm that unpack and operate all in memory. It is technically possible to make them better, I just have to find the time.

Errors starting 5XX like that 503 usually mean a serverside error. My guess is their db crashed or the server was overloaded or something. If you try it again now, does it work?

Sorry, No Release Tomorrow! Anonymous Board owner 01/07/2020 (Tue) 21:46:10 Id: 5a15d0 [Preview] No.472 del
I had a challenging week. I was able to get a basic MPV (a proper video+audio player) player embedded into hydrus and playing video and audio great, but some IRL difficulty meant I could get nothing else done. I don't want to put out a release with nothing but a janky prototype, so I will put off v380 until next week, January 15th.

Anonymous 01/07/2020 (Tue) 22:21:43 Id: cc1628 [Preview] No.473 del
I tried it again still getting the same errors.

Anonymous 01/07/2020 (Tue) 22:49:02 Id: dd376a [Preview] No.474 del
Still based for even just trying. If it works, mpv certainly is a very nice player. Hydrus will act like a SMplayer type of UI decorating the mpv binary, right?

Hope you'll manage fine IRL!

yiff.party broken Anonymous 01/09/2020 (Thu) 08:27:32 Id: fd7013 [Preview] No.475 del
it seems like it the new server update they did broke the formulae or the parser

Anonymous 01/11/2020 (Sat) 02:22:34 Id: 231ccf [Preview] No.476 del
Hey hdev, doing a more culling to be able to put in a new hdd I got for christmas and have been dragging my ass on installing, and I come into an issue.

I have a watcher with 150~ images, I remove it, and the images all go away, this is ok but I accidently forgot to tick them all off as checked.

would there be a way to have either an option in the settings to change the behaure to not remove images from being seen? I know with opening multiple watchers it works this way.

also on that same note, would it be possible to have the 'clear highlight' button be active when any images are open in a watcher tab? also great job on the mpv implementation, ill hold off on culling further then needed on gifs till thats working since it will make it a lot easier, here's hoping I will get a scrubber that is fast acting, like when it's all the darker color on a gif and not like when it's short and loading the content in as it goes.

Anonymous 01/11/2020 (Sat) 10:34:26 Id: 44949a [Preview] No.477 del
Currently on v378 but I'm guessing this still applies for v379. In the gelboru downloader, its tagging "small breast" as creator:small breast. I know I never set it up for something likes this so I checked both parents and siblings and it had nothing for it so hydrus is somehow automatically tagging "small breast" as creator:small breast on its own.

Anonymous 01/11/2020 (Sat) 11:30:25 Id: 44949a [Preview] No.478 del
Back again, it doesn't seem to be the tag itself but certain images that would make hydrus create "creator:small breast" instead of "small breast".


and several others from this same artist. Haven't tried any other artist just yet.

Anonymous 01/14/2020 (Tue) 08:00:09 Id: 65f287 [Preview] No.479 del
SSimDownscaler is really nice with mpv and images. I forget where the white paper is unfortunately but the shader file is here

Release Tomorrow! Anonymous Board owner 01/15/2020 (Wed) 01:42:49 Id: 53bf76 [Preview] No.480 del
I had a couple of difficult weeks, but I got some good work done. A prototype for an mpv video player embed is ready for advanced users to test! This new player renders video smoothly and finally adds audio support to hydrus. I have also fixed a heap of bugs, added some new imageboard downloaders, and improved the quality of life of a variety of UI.

I still have some things to do, and plenty to test, and messages to catch up on, so the release is likely to be late tomorrow.

Anonymous Board owner 01/15/2020 (Wed) 19:18:27 Id: 53bf76 [Preview] No.481 del
I just tested again and it worked for me. Another user was telling me this week that r34H had turned on enhanced CloudFlare protection. He thought it was internet-wide, but I had the same situation with him--it worked ok on my end but he still had trouble. I assume they are doing a region, ip, or domain based block. If you are on a VPN, my best suggestion is to jump to a new location and try again.

You probably want to clear out your hydrus's r34h cookies as well, in case there is some stale CF tracker in there, under network->data->review session cookies.

Otherwise, if it works for your web browser but not hydrus, you can try Hydrus Companion to copy the cleared CF cookies from your browser to hydrus. I do not know if this will work.


Anonymous Board owner 01/15/2020 (Wed) 19:22:33 Id: 53bf76 [Preview] No.482 del
Sure--how about I make the 'yes/no' dialog that does the remove confirmation be a 'yes/yes/no' dialog when there is a highlight, and the two yeses be:

yes and clear highlight
yes but leave files in place

Yeah, letting the clear highlight button also clear files sounds sensible. I'll think about the technical stuff a bit and see if I can get it done neatly.

Anonymous Board owner 01/15/2020 (Wed) 19:30:31 Id: 53bf76 [Preview] No.483 del
Hmm, that's weird. Can you do a test for me? Hit network->downloader definitions->manage parsers->gelbooru 0.2.5 file page parser and then put that url up in the test area and run a 'test parse'. Do you get 'small breasts' as a creator tag or a regular one?

I don't know much about gelbooru. If you have your hydrus client set to log in to gelbooru, is there any chance that you have 'small breasts' tagged as a favourite or something? Something that could colour/present/star the tag differently, so could cause it to parse under a different category?

Oh, also, sorry to be pedantic, but we are talking 'small breasts' here, not 'small breast', right?

If you try to type and add 'small breasts' yourself in the manage tags dialog on some file, do you see any sibling/parent business going on, or does it present and add fine?

Anonymous Board owner 01/15/2020 (Wed) 19:32:19 Id: 53bf76 [Preview] No.484 del
Thank you. This first prototype is obviously going to be simple, but I would like to add support for custom confs and anything else that is popular.

I am new to MPV. To save me the time of looking it all up and getting it wrong five times, can you tell me how to incorporate a shader file like that into MPV? Is it a launch parameter, or something defined in a conf, or something else?

Anonymous Board owner 01/15/2020 (Wed) 19:33:48 Id: 53bf76 [Preview] No.485 del
(60.03 KB 1075x747 sb.png)
Forgot image.

Anonymous 01/15/2020 (Wed) 21:26:07 Id: 65f287 [Preview] No.486 del
Sure, there are multiple ways to do it. With the json api you can use changelist


change-list glsl-shaders append ~~/shaders/blahblah.glsl

double ~ in mpv is just the mpv home directory. they can be added via command line using --glsl-shaders-add. if you're on linux you can pipe mpv to find stuff easier, "mpv --list-options | grep glsl"

I've hacked mpv support into a ranger which is a python based file manager, for personal use, so if you have any questions regarding mpv integration I can probably answer them. Except qt specific problems.. plex-media-player does it so maybe you can use that as a reference if you run into issues there. Also yeah, MPV's documentation is like the necronomicon and if you stare into the abyss too long you can start to lose your sanity.

Some shaders I use:

Anonymous 01/15/2020 (Wed) 21:28:50 Id: 65f287 [Preview] No.487 del
Also, mpv can be made into a much better image viewer with the right config settings. And it's very extensible with both lua/javascript support baked in. If you would like inspiration for UI design you should take a look at tachiyomi, an android app. It has the best image reader I've seen hands down, although it doesn't use mpv.

Anonymous 01/15/2020 (Wed) 21:46:11 Id: 65f287 [Preview] No.488 del
Another thing, you may want to take a look at two projects that would integrate nicely (and are both python based). gallery-dl and youtube-dl. They are really nice scrapers. MPV has built in support for youtube-dl.


I'm a bit sleep deprived so my posts are all over the place, so sorry about that.

Top | Return | Catalog | Post a reply