/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.

Posting mode: Reply

Check to confirm you're not a robot
Name
Email
Subject
Comment
Password
Drawing x size canvas
File(s)

Board Rules

Max file size: 350.00 MB

Max files: 5

Max message length: 4096

Manage Board | Moderate Thread

Return | Magrathea | Catalog | Bottom

Expand All Images


Version 425 Anonymous Board owner 01/13/2021 (Wed) 22:39:14 Id: eb79da [Preview] No. 897
https://youtube.com/watch?v=r1nn-tp26KE [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v425/Hydrus.Network.425.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v425/Hydrus.Network.425.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v425/Hydrus.Network.425.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v425/Hydrus.Network.425.-.Linux.-.Executable.tar.gz

I had a good week. I optimised and fixed several core systems.

faster

I messed up last week with one autocomplete query, and as a result, when searching the PTR in 'all known files', which typically happens in the 'manage tags' dialog, all queries had 2-6 seconds lag! I figured out what went wrong, and now autocomplete should be working fast everywhere. My test situation went from 2.5 seconds to 58ms! Sorry for the trouble here, this was driving me nuts as well.

I also worked on tag processing. Thank you to the users who have sent in profiles and other info since the display cache came in. A great deal of overhead and inefficient is reduced, so tag processing should be faster for almost all situations.

The 'system:number of tags' query now has much better cancelability. It still wasn't great last week, so I gave it another go. If you do a bare 'system:num tags > 4' or something and it is taking ages, stopping or changing the search should now just take a couple seconds. It also won't blat your memory as much, if you go really big.

And lastly, the 'session' and 'bandwidth' objects in the network engine, formerly monolithic and sometimes laggy objects, are now broken into smaller pieces. When you get new cookies or some bandwidth is used, only the small piece that is changed now needs to be synced to the database. This is basically the same as the subscription breakup last year, but behind the scenes. It reduces some db activity and UI lag on older and network-heavy clients.

better

I have fixed more instances of 'ghost' tags, where committing certain pending tags, usually in combination with others that shared a sibling/parent implication, could still leave a 'pending' tag behind. This reasons behind it were quite complicated, but I managed to replicate the bug and fixed every instance I could find. Please let me know if you find any more instances of this behaviour.

While the display cache is working ok now, and with decent speed, some larger and more active clients will still have some ghost tags and inaccurate autocomplete counts hanging around. You won't notice or care about a count of 1,234,567 vs 1,234,588, but in some cases these will be very annoying. The only simple fixes available at the moment are the nuclear 'regen' jobs under the 'database' menu, which isn't good enough. I have planned maintenance routines for regenerating just for particular files and tags, and I want these to be easy to fire off, just from right-click menu, so if you have something wrong staring at you on some favourite files or tags, please hang in there, fixes will come.


Anonymous Board owner 01/13/2021 (Wed) 22:40:48 Id: eb79da [Preview] No.898 del
full list

- optimisations:
- I fixed the new tag cache's slow tag autocomplete when in 'all known files' domain (which is usually in the manage tags dialog). what was taking about 2.5 seconds in 424 should now take about 58ms!!! for technical details, I was foolishly performing the pre-search exact match lookup (where exactly what you type appears before the full results fetch) on the new quick-text search tables, but it turns out this is unoptimised and was wasting a ton of CPU once the table got big. sorry for the trouble here--this was driving me nuts IRL. I have now fleshed out my dev machine's test client with many more millions of tag mappings so I can test these scales better in future before they go live
- internal autocomplete count fetches for single tags now have less overhead, which should add up for various rapid small checks across the program, mostly for tag processing, where the client frequently consults current counts on single tags for pre-processing analysis
- autocomplete count fetch requests for zero tags (lol) are also dealt with more efficiently
- thanks to the new tag definition cache, the 'num tags' service info cache is now updated and regenerated more efficiently. this speeds up all tag processing a couple percent
- tag update now quickly filters out redundant data before the main processing job. it is now significantly faster to process tag mappings that already exist--e.g. when a downloaded file pends tags that already exist, or repo processing gives you tags you already have, or you are filling in content gaps in reprocessing
- tag processing is now more efficient when checking against membership in the display cache, which greatly speeds up processing on services with many siblings and parents. thank you to the users who have contributed profiles and other feedback regarding slower processing speeds since the display cache was added
- various tag filtering and display membership tests are now shunted to the top of the mappings update routine, reducing much other overhead, especially when the mappings being added are redundant
- .
- tag logic fixes:
- I explored the 'ghost tag' issue, where sometimes committing a pending tag still leaves a pending record. this has been happening in the new display system when two pending tags that imply the same tag through siblings or parents are committed at the same time. I fixed a previous instance of this, but more remained. I replicated the problem through a unit test, rewrote several update loops to remain in sync when needed, and have fixed potential ghost tag instances in the specific and 'all known files' domains, for 'add', 'pend', 'delete', and 'rescind pend' actions
- also tested and fixed are possible instances where both a tag and its implication tag are pend-committed at the same time, not just two that imply a shared other
- furthermore, in a complex counting issue, storage autocomplete count updates are no longer deferred when updating mappings--they are 'interleaved' into mappings updates so counts are always synchronised to tables. this unfortunately adds some processing overhead back in, but as a number of newer cache calculations rely on autocomplete numbers, this change improves counting and pre-processing logic
- fixed a 'commit pending to current' counting bug in the new autocomplete update routine for 'all known files' domain
- while display tag logic is working increasingly ok and fast, most clients will have some miscounts and ghost tags here and there. I have yet to write efficient correction maintenance routines for particular files or tags, but this is planned and will come. at the moment, you just have the nuclear 'regen' maintenance calls, which are no good for little problems


Anonymous Board owner 01/13/2021 (Wed) 22:43:02 Id: eb79da [Preview] No.899 del
- network object breakup:
- the network session and bandwidth managers, which store your cookies and bandwidth history for all the different network contexts, are no longer monolithic objects. on updates to individual network contexts (which happens all the time during network activity), only the particular updated session or bandwidth tracker now needs to be saved to the database. this reduces CPU and UI lag on heavy clients. basically the same thing as the subscriptions breakup last year, but all behind the scenes
- your existing managers will be converted on update. all existing login and bandwidth log data should be preserved
- sessions will now keep delayed cookie changes that occured in the final network request before client exit
- we won't go too crazy yet, but session and bandwidth data is now synced to the database every 5 minutes, instead of 10, so if the client crashes, you only lose 5 mins of login/bandwidth data
- some session clearing logic is improved
- the bandwidth manager no longer considers future bandwidth in tests. if your computer clock goes haywire and your client records bandwidth in the future, it shouldn't bosh you _so much_ now
- .
- the rest:
- the 'system:number of tags' query now has greatly improved cancelability, even on gigantic result domains
- fixed a bad example in the client api help that mislabeled 'request_new_permissions' as 'request_access_permissions' (issue #780)
- the 'check and repair db' boot routine now runs _after_ version checks, so if you accidentally install a version behind, you now get the 'weird version m8' warning before the db goes bananas about missing tables or similar
- added some methods and optimised some access in Hydrus Tag Archives
- if you delete all the rules from a default bandwidth ruleset, it no longer disappears momentarily in the edit UI
- updated the python mpv bindings to 0.5.2 on windows, although the underlying dll is the same. this seems to fix at least one set of dll load problems. also updated is macOS, but not Linux (yet), because it broke there, hooray
- updated cloudscraper to 1.2.52 for all platforms

next week

Even if this week had good work, I got thick into logic and efficiency and couldn't find the time to do anything else. I'll catch up on regular work and finally get into my planned network updates.

It looks like 8kun is going off clearnet, or at least expecting to. The site has been dying for a long time now, and it is well past time I moved. I simply procrastinated. I also regret being behind the 8kun bug report and question threads for several weeks now. In the coming week I will clear out appropriate responses and lock the board in prep for deletion. I will update 426's hydrus links so Endchan is our primary board for the time being.


Anonymous 01/15/2021 (Fri) 07:16:15 Id: 389199 [Preview] No.902 del
> It looks like 8kun is going off clearnet, or at least expecting to. The site has been dying for a long time now, and it is well past time I moved. I simply procrastinated. I also regret being behind the 8kun bug report and question threads for several weeks now.
Unfortunate.

Obviously Hydrus isn't the reason for any of this. What's the place if Endchan goes down? Some other imageboard? Tunguska/Misskey?


Also I think I get a Hydrus problem, one of the last messages before a crash was a nitter rate limit HTML dump which essentially (real thing is detected as spam, can't post) says "Instance has been rate limited. Use https://github.com/zedeus/nitter/wiki/Instances another instance"

I wonder if this was the direct reason, going to check a few more times.


Anonymous 01/15/2021 (Fri) 07:18:18 Id: 389199 [Preview] No.903 del
> It looks like 8kun is going off clearnet, or at least expecting to. The site has been dying for a long time now, and it is well past time I moved. I simply procrastinated. I also regret being behind the 8kun bug report and question threads for several weeks now.
Unfortunate.

Obviously Hydrus isn't the reason for any of this. What's the next place if Endchan also goes down? Some other imageboard? Misskey/Pleroma?

Also I think I get a Hydrus problem, one of the last messages before a crash was a nitter rate limit HTML dump which essentially (real thing is detected as spam, can't post) says "Instance has been rate limited. Use https://github.com/zedeus/nitter/wiki/Instances another instance"

I wonder if this was the direct reason, going to check a few more times.


Anonymous Board owner 01/16/2021 (Sat) 19:36:37 Id: 562180 [Preview] No.906 del
>>902
>>903
If Endchan goes down, or even more boards across the community, my more normie places like discord or twitter will be the places to check for the new home. Or you can just keep updating, and I'll make sure the help that is bundled with the client and the links in the help menu will link to whatever board I can figure out.

I'd like to have an imageboard somewhere. Somewhere with anon image posting, so lads can talk somewhat privately and make bug reports on nsfw pictures and so on.

I'll stick on just endchan for now. I am not a comfortable board owner, and I have never been great at checking boards regularly for spam and so on, so I think when it feels like time to expand again, maybe figure out a board on the larger webring or similar, I may try to just be a janny on a /hydrus/ run by others. The hydrus discord works like that, and it works well for me. But I also don't want to make others do hotpocket work for free unless they are keen, so I'll think about it a bit.

Thank you for this information about nitter. I hadn't heard about rate limiting for them before--I guess if they are directing to different nitter mirrors, that means our connection to nitter, as opposed to nitter's connection to the twitter api? Sounds like the hydrus downloader could do with some default bandwidth rules? If you discover more, especially something like 'no more than 500 requests an hour pls' kind of statement, please let me know and I'll fold it in to a future release.


Anonymous 01/17/2021 (Sun) 05:27:54 Id: 96dee4 [Preview] No.919 del
>>906
>>906
>I'd like to have an imageboard somewhere. Somewhere with anon image posting, so lads can talk somewhat privately and make bug reports on nsfw pictures and so on.
An idea. Perhaps the board /1ntr/ (internets) of /mlpol/ would be suitable. The fags are friendly and the mods are legit anons and refugees from 4cuck.


Anonymous 01/19/2021 (Tue) 17:05:20 Id: c130d4 [Preview] No.927 del
Sorry for being blunt but how long are you going to leave the manage tags window unusable? Some of us do actually use it, you know.

I just spent 5 minutes trying to find which tag parented in the gender:male tag on an image with no males in it, and I wasn't successful.

Please don't leave important features half-finished.


Anonymous Board owner 01/20/2021 (Wed) 05:04:19 Id: d36528 [Preview] No.935 del
>>906
>>919 Thanks m8, we'll see how this goes!

The plan to move to Endchan only has changed!

Sorry for the sudden change, this all came together this week!

Codexx on 8chan.moe has kindly offered to host me on /t/ over there. As I have never been a comfortable BO, this is a much better solution for me as the primary place to talk Anon about hydrus.

I will be maintaining a Hydrus Network General, the first instance is here: https://8chan.moe/t/res/2219.html

The links in the release tomorrow will reflect this. Endchan will remain as a bunker.


Release Tomorrow! Anonymous Board owner 01/20/2021 (Wed) 05:08:35 Id: d36528 [Preview] No.936 del
I had a great week. I fixed up some issues with new systems; one where tags were sometimes being forgotten by the new tag search cache, and another where late-created network sessions were sometimes causing harmless but annoying errors on client shutdown. There are also some fixes for the gelbooru downloaders, improved handling for when you try to import a donwloader your client cannot yet support, and some help and code cleanup.

The release should be as normal tomorrow.

>>927
I am sorry for the trouble. It has been tricky to fit this in, as it involves a complicated new expansion to the taglist control. Since the network object breakup and the new tag cache seem to be doing ok, my next 'medium size' job week is free to do anything. I think that will be for 428. I agree this is currently bad, so how about I aim to make it nicer for then? If I can't get nice ghost tags, I think I can get something good going.


Anonymous 01/22/2021 (Fri) 00:39:33 Id: c130d4 [Preview] No.949 del
>>936
>I think that will be for 428. I agree this is currently bad, so how about I aim to make it nicer for then? If I can't get nice ghost tags, I think I can get something good going.

Thank you. Sorry if I was rude, I was just frustrated. Even if you can't get the ghost tags you have planned functioning a less elegant temporary solution would be good just to make it usable for now. I need to be able to see the parents and siblings added on the manage tags window, or else it's just a huge slow down on my work flow.



Top | Catalog | Post a reply | Magrathea | Return