/hydrus/ - Hydrus Network

Hydrus Bunker

Posting mode: Reply

Check to confirm you're not a robot
Drawing x size canvas

Remember to follow the rules

Max file size: 350.00 MB

Max files: 5

Max message length: 4096

Manage Board | Moderate Thread

Return | Catalog | Bottom

Expand All Images

Version 376 Anonymous Board owner 12/05/2019 (Thu) 06:17:35 Id: 00e85b [Preview] No. 400
https://youtube.com/watch?v=J4wKjGHvs_4 [Embed]
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v376/Hydrus.Network.376.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v376/Hydrus.Network.376.-.Windows.-.Installer.exe
app: https://github.com/hydrusnetwork/hydrus/releases/download/v376/Hydrus.Network.376.-.macOS.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v376/Hydrus.Network.376.-.Linux.-.Executable.tar.gz
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v376.tar.gz

I had an ok to good week. Subscriptions run better and there are more UI fixes and basic theming for Qt.


Subscriptions have always been resource heavy. As last year's downloader overhaul extended their capabilities, we saw increased CPU lag with many subscription operations. I had hoped to completely fix this this week, even permitting subs with >250,000 items cached, but the surgery needed to figure this out was more complicated than I expected. While I otherwise prepped for this next step, I also tightened up subscription timing code and wrote a new scheduler that reduces overall subscription resource needs.

Subscriptions will now try to load when they are due, and only then. So, if a query in your sub is due to check in 45 minutes, or if it has file to download but has run out bandwidth for 45 minutes, it should now run, typically, in 45-60 minutes. They will also respond better to cancelling and pause events, and if only 30 of 100 queries are due for a check in this run, or have outstanding files to download, you will now see x/30 on the popup progress text and gauge, just as better indication of how much work remains.

Previously, subs checked every four hours. It meant a bunch of wasted CPU for subs that were not due, late checks for subs that were due just moments after the check time, and also a delayed 'clumping up' of work. Today's change is subtle but significant, so I am interested in feedback from all users of subs--if you notice anything odd, bad, or great, please let me know. Ideally, subs will be quieter for you, running more often when you are not looking.

Although the data overhaul was too much work to do in one week, I now have a good plan and am in better position to get it done. I really want subs to load and save their metadata in milliseconds so the manage subs dialog can launch equally quickly and the client api can start talking to them. The year is fading, so I am not sure when this will get done, but hopefully this week's scheduling will at least relieve some of the work involved.

Anonymous Board owner 12/05/2019 (Thu) 06:20:41 Id: 00e85b [Preview] No.401 del

Hydrus now has very simple support for Qt styles. Please check the new options->style page. All users are encouraged to try out the 'style' option, which should have one or more flavours for your OS and also 'Fusion'. These will change how buttons and other widgets are drawn across the program.

Advanced users may wish to play with 'stylesheet' support, which uses css/qss files to change colours and layouts using portable files. This is still prototype, just rough importing of stylesheets from a barebones folder in the install dir, and is likely to be replaced with something more user-friendly and technically sensible in the coming weeks. If you play with this, I am interested in areas where hydrus's old colour system conflicts foolishly with a qss stylesheet (such as some panel backgrounds and the green/red yes/no buttons on many dialogs), and also recommendations for stylesheets you would like bundled as future hydrus defaults.

The macOS build can now do page tab drag and drop again! Media drag and drops between tabs should also work. Everything should line up right and work like the Windows and Linux builds, but please let me know if it does not. A tip here, if you still have trouble with a centered page tab bar, is just to set the Fusion style as above, which will make hydrus look more like a standard PC application. You may prefer the macOS look, but if not, you now have the option.

Beyond a bunch of general bugfixes, I also improved the stability of some review bandwidth panels and some bad 'should I close/exit?' dialogs that were either not respecting 'no' answers or causing unusual behaviour as 'yes' closes were going on.

the rest

Pixiv tag search and twitter username lookup get updated default downloaders this week. Pixiv should be fixed, and twitter should work more efficiently, only getting 'media' tweets.

Client shutdown should now be faster and snappier for most users.

File export filename phrases now support a new '{file_id}' term that substitutes a unique permanent numerical identifier. If you need to create guaranteed-unique filenames, this is easier than using the '{hash}' term.

Anonymous Board owner 12/05/2019 (Thu) 06:23:44 Id: 00e85b [Preview] No.402 del
full list

- subscriptions:
- wrote a new subscription manager to better look after subscription scheduling
- rather than checking every four hours or after manage subs dialog close, subscriptions now record an indication of when they are next due for work, whether that is the estimated next check time or when bandwidth is free on remaining file downloads, and launch in a fifteen-minute window around that time. delays due to previous errors or user cancels are also taken into account. this reduces background cpu and i/o greatly for clients with large subs
- if a sub is paused, or all its queries are paused, it will now never be reloaded after first load until a change via the manage subs dialog
- furthermore, if a single sub takes a very long time to work, the whole sublist can re-cycle if they come up due for more work before it is finished
- if a sub query is DEAD but still has outstanding files to download, it will no longer automatically pause
- subs now clean up more tidily if they are running on a program exit
- the subscription popup now shows check/file progress based on the number of queries that appear to have pending work. instead of 'query 300/450' with 420 that aren't due, you'll get 'query 12/30'. if a query becomes due during a round of checking, another round of checking will run
- if a subscription fails to load from the db, the error is handled better and no more subs will run in that boot
- improved subscription startup checking logic, tightening up various paused/dead/cansync tests
- improved subscription interrupt checking logic, tightening checks on global network pause and various shutdown scenarios
- cleaned up some more subscription code in prep for data storage breakup
- .
- qt:
- added experimental Qt style settings to the new options->style page! all users should now be able to set Fusion style, and perhaps some alternate OS styles. advanced users are invited to play around with QSS stylesheets (although be warned that some of hydrus's custom colour system overrides QSS, so more work is needed here), which will be extended and made user-friendly in coming weeks
- fixed tab position calculations for all tab/media drag and drops for tab bars that are centered or otherwise positioned far off top-left alignment
- fixed tab drag and drop event object handling for macOS. tab and media DnD is now enabled for macOS
- the popup toaster can now unhide if an on-top-of-parent non-modal frame (like review services) is focused (so hitting 'process now' should show you the work)
- fixed a variety of old hacky wx close-window veto tests. the 'close client?' confirmation dialog will now reliably veto a close requent on 'no'/cancel, dialog close events that are vetoed (such as closing the manage tags dialog with pending tags) will now veto more than just the first time, and several bad media viewer archive/dupe filtering cancel and end-of-window events should now work more cleanly and correctly. users who had crashes at the end of filtering may find they are stable again
- as a quick patch against some multiline notes and statuses, list controls now force single-line text in all cells
- list controls now tooltip all cells
- fixed the shutdown splash not updating after the daemons shut down (lmao)
- 'modal' message dialogs, which are created by blocking maintenance tasks such as vacuum, will no longer raise the program to the foreground on creation
- should have fixed the taglist vertical positioning jank that could occur in the row after a tag with a tall emoji unicode character (and also sometimes kanji/hangul)
- fixed a typo bug that was throwing an error for the upnp port widget in the local client server management panel when 'allow non local connections' was checked
- improved stability of bandwidth review panel bandwidth rules refresh
- improved stability of review services bandwidth rules refresh
- improved some dialog cleanup code
- reverted a bad environment-setting change put in last week that was causing some running-from-source users trouble
- misc qt code cleanup

Anonymous Board owner 12/05/2019 (Thu) 06:25:21 Id: 00e85b [Preview] No.403 del
- the rest:
- updated the default pixiv tag search downloader to one submitted by a user. it now uses their api
- updated the default twitter username lookup to a downloader submitted by a user. it fetches just the media tweet feed, making it more efficient. also added (but not linked by default) is a new tweet parser that can fetch most videos using a third-party site, advanced users may wish to play with this
- added a {file_id} term for file export phrases that substitutes a unique and permanent numerical file identifier
- fixed an issue where idle maintenance jobs could sometimes sneak in a few milliseconds of work during certain long shut down pauses, such as while waiting for a 'should I do shutdown work?' dialog to return. program shutdown should be snappier for many users as forced startup delays in these calls will no longer trigger
- added a date 'encode' string transformation rule, which takes an integer timestamp and converts it to a pretty date string. the date rules are now renamed to the clearer 'datestring to timestamp' and vice versa
- fixed page parser edit panel's 'test parse' when string transformations perform pre-parsing conversion. the handling and passing of test data for all the panels here is cleaned up throughout
- system:limit predicate edit panel now has a small label describing its sampling behaviour
- updated the various 8chan links in the client and help to 8kun, let me know if I missed any, and added Endchan bunker link to help menu
- improved some misc status text handling across the program
- refactored cache and manager code into different, simpler files
- updated sqlite on windows build to 3.30.1

next week

It is already December! For me, and I know for many of you, 2019 has just disappeared. Christmas is exactly three weeks away, so I am afraid some of my hopes for significant tag overhaul like tag siblings or processing filters may have to be delayed a little. I have a bit more Qt work to do, but also plenty of regular small jobs and bugfixes, so for next week I will take a simple bit of misc work, nothing huge, just to catch up on some things. I also have messages to catch up on.

If you were an 8chan user and missed the news, the /hydrus/ board is back on 8kun.top. The site is still unstable, and reception from the community is mixed, but against all odds they are back up and improving. I will keep posting release posts on Endchan and 8kun simultaneously for the time being, so please use whichever you feel better about, or just hit me on discord/tumblr/email if you prefer.

Anonymous 12/05/2019 (Thu) 16:55:29 Id: 960f45 [Preview] No.404 del
Thank you based dev

Anonymous 12/07/2019 (Sat) 14:12:43 Id: f0eeb8 [Preview] No.407 del
welp, I have a session weight of 1430000~ and this session would have used around 6-8gb with no real use on wx, on qt it was using about 6-7 till I did shit and would bloat to hell and back, being around 14-16gb

current build, after culling files is hovering 4.8-5.2gb...

good job on the memory aspects
and this is not mentioning the far fewer hiccups with 10~ second long freezing has been addressed, though session saves still do cause hitches, its nothing I cant anticipate.

the only new issue, and not even 100% on the cause, is in media viewer when I click to drag things they don't move at speeds I expect them to, I have my mouse set to 2000dpi, but this feels like it drags the mouse to 400~ dpi

Anonymous Board owner 12/07/2019 (Sat) 21:22:13 Id: da7731 [Preview] No.410 del
Thanks. I am glad it is doing better. I hope to have a little more memory stuff done this coming week, maybe a few hundred megs.

When you drag media around, can you check Task Manager and see if your client.exe CPU spikes? There's some garbage old manual layout stuff going on in the media viewer, and I think Qt doesn't like it, so when I do manual pans, it has to dump a bunch of extra CPU into redrawing everything from scratch rather than, say, re-blitting the existing bitmap. If you do get a CPU spike, is it making out for one core (e.g. 20-25% if you are on a 4-core processor)? And what is the CPU like when you are just moving your mouse across the media viewer, without a drag?

Anonymous 12/08/2019 (Sun) 06:23:23 Id: 29b830 [Preview] No.413 del
sorry if you are already aware of this but:

when I click on move database I get this

'DirDialog' object has no attribute 'SetPath'
File "include\ClientGUICommon.py", line 199, in EventButton
self._func( *self._args, **self._kwargs )
File "include\ClientGUIScrolledPanelsReview.py", line 551, in _MigrateDatabase
dlg.SetPath( source )

Anonymous 12/08/2019 (Sun) 11:29:19 Id: f0eeb8 [Preview] No.414 del
on an 8 core 16 thread, so the program is using 1-2% idle, moving the file manually pegged it to 6% and looking at the graph it looks like about 80-95% use on that single core, panning the way I do with my g13 thumbstick doesn't even really register as use.

yep, that seems to be the issue.

Anonymous Board owner 12/09/2019 (Mon) 22:17:00 Id: c02d5e [Preview] No.418 del
Thank you for this report. I am sorry for the trouble. It is a typo from the wx->Qt migration and will be fixed for 377.

Thanks. Yeah, looks like that wx layout hack just isn't playing around well in Qt. I have a job queued up to give it a proper rework. I am not sure if that will make this work smooth as butter, but I think I can improve it.

Anonymous 12/10/2019 (Tue) 08:40:45 Id: f0eeb8 [Preview] No.421 del
found a rather fun image
The tag that this one has that is a bit suspect is the guro one.
long story short, doing a parse on tags that I don't like most images from, guro is one where I don't really like dismemberment or death as part of my fapping catalogue, but cant get rid of it all because some of them are really well done images that I like or in the case of skull girls where one of the characters can cannonocily dismember themselves I cant really just remove that either.

so back on topic, the guro tag.
I have no idea why that's there, but it is.
I also have no idea how to remove the tag, but this brings up a problem I will have in the future, and its just easy tag management.

There was a program a long time ago for tagging images, and it was great for adding tags, but also fucking horrible in the way it managed images to the point it was unusable because one directory change and everything was gone and no getting it back. I think someone posted an image of it, where it has every tag right there for you to use, and with check boxes, and at the bottom had an area for you to add new tags. now, the only way I know to do anything with tags is right click>manage>tags, now I cant do shit with the public tag repository, which looking at this images, there are quite a few tags I would just remove for either being redundant, never going to be important, or in the case of guro, just wrong.

not 100% on me doing something wrong, or the system to do tags is just currently not great

If you have the time, get comicrack if you don't, and go into the info for any comic you add to the library, the tag section here is amazing, you can make your own tags, and with the star thing you get 3 ways to list your tags, list/checkbox/text which gives you what you would expect. now for hydrus, automatically adding everything would be... impossible/unfeasible and would make it a fucking mess, but imagine this being below the tag section on the main window, a text box to add a tag, and a drop down of commonly used tags/favorites

granted, for tagging, this doesn't affect me right now, as i'm doing mass culls and sorting with ratings, but once this is done (if it ever gets done) a better tagging system would be appreciated.

(the three images are showing genre, which has the same system as tags, but is not personalized to me yet)

Anonymous 12/10/2019 (Tue) 20:20:29 Id: 29b830 [Preview] No.423 del
Oh, I was messing around and noticed another thing:
going into options and then gui and trying to change the main gui title results in no changes
anyone else confirm?

Top | Return | Catalog | Post a reply