/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.

Posting mode: Reply

Check to confirm you're not a robot
Drawing x size canvas

Remember to follow the rules

Max file size: 350.00 MB

Max files: 5

Max message length: 4096

Manage Board | Moderate Thread

Return | Catalog | Bottom

Expand All Images

Version 455 Anonymous Board owner 09/22/2021 (Wed) 22:03:37 Id: 9764e1 [Preview] No. 1138
https://youtube.com/watch?v=Gs069dndIYk [Embed]
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v455/Hydrus.Network.455.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v455/Hydrus.Network.455.-.Windows.-.Installer.exe
app: https://github.com/hydrusnetwork/hydrus/releases/download/v455/Hydrus.Network.455.-.macOS.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v455/Hydrus.Network.455.-.Linux.-.Executable.tar.gz

I had an ok couple of weeks. Some quality of life is improved, the UI should save sessions a bit quicker, and the database repairs itself more of itself.

Your client may warn you about a missing index on first boot. If it does, that's great--some new repair code is working!

session save

Some users with large sessions (typically around 2-3+ million 'weight') are experiencing severe UI lockups and general lag. This week I deal with part of that problem by greatly reducing the amount of CPU needed to save most sessions. Now pages know if they have been changed since being loaded, and if they have no changes, they can skip a bunch of session save work.

There's nothing you have to do here, but if you have been experiencing 12+ second freezes every few minutes, please let me know how you get on now. I can't promise things will be perfect, but if you have any large and inactive downloader pages, you should notice things running a bit smoother (exiting the program too, should be faster).

There is still a problem of lag and lockups when there are 200+ downloaders in the session. This is a different and unfortunately more severe problem that I have relieved in recent weeks but needs some other large work to completely fix. Best solution for now is to clear out finished downloaders if you can.


Many of the simple system predicates, like height and filesize, now support the '≠' (not equal) operator! This is still in the early stages, but I'd like to add some options of the thumbnail menu so you can say 'find files that (do not) have this file's precise resolution/duration/size' etc...

You can now copy thread subjects to clipboard in the watcher downloader page list!

The advanced file deletion dialog now remembers the last reason you chose!

The network job widget (which shows download progress, you see it on any downloader) now shows the current URL it is working on under its cog menu. You can click it to copy to clipboard. May help to debug some odd network errors etc...

The way data sizes over 1KB are rendered has been overhauled. It should now generally be to three significant figures, so 3.11MB instead of 3.1MB, 12.3GB instead of 12GB. See if you like the change, and if not, try tweaking it to 2sf under the new EXPERIMENTAL option in options->gui.

Anonymous Board owner 09/22/2021 (Wed) 22:04:44 Id: 9764e1 [Preview] No.1139 del
database repair

tl;dr: Boot checks are better, you don't have to do anything.

I have spent most of my 'cleanup' time this year breaking the client database into smaller modules. These modules are now responsible for their own repair (which happens on boot), and they do much of it in a neater automatic way, using the same set of definitions it uses to initialise on first start to check whether an existing database is missing anything now. Most of my old hardcoded 'oh, you are missing this table' code is now simplified and modularised, works more reliably, and will be much easier to maintain in future.

You may get a popup when you boot saying you are missing one or more indices. There are several legacy reasons why this might be the case, but most often it will be because you once recovered from former hard drive damage and a clone/recover was unable to rescue an index. Whatever the case, these can now be regenerated automatically (they store duplicated data, so can be regenerated 100%), so please let it run. You might even notice PTR processing working faster afterwards.

Many tables can also be regenerated, and some can be completely repopulated. It is now possible (though not a great idea!) to start a client without a client.caches.db or client.mappings.db file and have it recover to a bootable state. Error presentation around this is improved, with instructions on what to do next in the case of critical problems.

I will continue modularising more of the database and fleshing out this repair code to cover more of the schema!

full list

- misc:
- many of the simple system predicates (width, size, duration, etc..) now support the '≠' (not equal) operator! searches only support one of these at once for now (e.g. you can't say height != 640 AND height != 1080--it'll pick one of these pseudorandomly)
- the watcher page list right-click menu now has 'copy subjects', which copies the selected watchers' 'subject' texts to clipboard
- the advanced file deletion panel now remembers which reason you last used. it remembers if you selected the 'default' value up top versus specific reason text, and if you enter custom text, it remembers that for next time too
- the network job widget now shows the current URL as tooltip over the progress gauge, and under the cog menu, where clicking it copies it to clipboard
- the various menu labels across the program should now show ampersand (&) correctly (e.g. in URLs)
- the way byte sizes (like 21.7KB or 1.24GB) above 1KB are rendered to strings has been overhauled. they now generally show three significant figures. a new EXPERIMENTAL option in 'gui' options panel lets you change this, but only 2 or 3 are really helpful
- if a repository clears the message on your account, you no longer get a popup telling you 'hey, new message from server x: ...'
- the new ≠ system preds should be parseable (but be careful, likely buggy) using the client api's new system predicate parser, with '≠', '!=', 'is not', or 'isn't'
- cleaned up some old data presentation methods and improved how client specific options are patched in to base hydrus string conversion code

Anonymous Board owner 09/22/2021 (Wed) 22:08:42 Id: 9764e1 [Preview] No.1140 del
- ui freezes:
- session pages can now detect if they have had no saveable changes since a certain time. they use this ability to skip redundant session save CPU time for pages with no changes since the last session save
- for now, since the smallest atom of the session system is a whole page, gallery and watcher pages can only save time if _every_ downloader in the page has had no changes, so in their case this optimisation mostly only applies to completely finished/paused pages. it is still better to have several medium size downloader pages than one gigantic one
- a new database maintenance task ensures that optimisation cannot accidentally lose a page (from something like an unfortunate timing of a session save after many manual session deletes)
- the existing optimisation that skips 'last session' save on no changes now initialises its data as the 'last session' is first loaded (rather than on first save), meaning that if there are no changes while the client is open, no new 'last session's will be saved at all
- misc session save code cleanup
- .
- database repair, mostly boring:
- a client can now boot with client.caches.db missing and will rebuild that file. almost all of its tables are now able to automatically repopulate (issue #975)
- all the new modules I have been working on are now responsible for their own repair. this includes missing indices, missing tables, and table repopulation where possible. modules now know which of their tables are critical to a boot, what version each table and index was added, and now manage both initial and later-created service tables and indices
- essentially, all newer database repair code is now modularised rather than hardcoded. the repair and creation code actually now share the same table and index definitions. the code is more reliable, checkpoints its good work in case of later failure, and will be much easier to maintain and expand in future
- lots of module repair and initialisation code is refactored and generally given a full pass
- the core mappings cache regeneration routine now takes transaction checkpoints throughout its job to save progress and reduce journal size
- master definition critical error detection code is no longer hardcoded!
- mapping storage repair code is no longer hardcoded!
- similar files repair code is no longer hardcoded!
- parent or sibling cache repair repopulation is no longer hardcoded!
- the local hashes cache module can now repopulate itself during repair
- the notes fast search table can now repopulate itself during repair
- the similar files search tree table can now rebuild itself during repair

next week

I'm not really happy with my productivity over the past two weeks. This repair and session save worked bogged me down a bit, but I also didn't get as many hours in as I had planned. In any case, I will try to prioritise smaller bug fixes next week. Keep on pushing.

Release Tomorrow! Anonymous Board owner 09/29/2021 (Wed) 02:57:53 Id: bc88c5 [Preview] No.1141 del
I had an ok week. I eliminated more UI lag, added downloader support for a special kind of URL, and fixed some more bugs.

Anonymous 09/29/2021 (Wed) 11:36:19 Id: ce2e29 [Preview] No.1142 del
(47.12 KB 1356x621 457474.jpg)
Not sure if this is related to version 455 since I haven't used the site in awhile but I get this error when trying to add links from rule34.xxx about a missing login cookie and I don't see this type of cookie in my browser.

Anonymous Board owner 09/29/2021 (Wed) 23:00:02 Id: 866d21 [Preview] No.1143 del
Hey, thank you for this report. It may be that the default login script is now out of date. I will investigate this, and if it is an easy fix, I will roll out an update to that login. You may want to deactivate that login for now and maybe try Hydrus Companion to copy your browser login cookies over directly using the Client API:


Top | Return | Catalog | Post a reply