2 / I:
0 / P:
Qt5 zip: https://github.com/hydrusnetwork/hydrus/releases/download/v495/Hydrus.Network.495.-.Windows.Qt5.-.Extract.only.zip
Qt6 zip: https://github.com/hydrusnetwork/hydrus/releases/download/v495/Hydrus.Network.495.-.Windows.Qt6.-.Extract.only.zip
Qt5 exe: https://github.com/hydrusnetwork/hydrus/releases/download/v495/Hydrus.Network.495.-.Windows.Qt5.-.Installer.exemacOS
Qt5 app: https://github.com/hydrusnetwork/hydrus/releases/download/v495/Hydrus.Network.495.-.macOS.Qt5.-.App.dmg
Qt6 app: https://github.com/hydrusnetwork/hydrus/releases/download/v495/Hydrus.Network.495.-.macOS.Qt6.-.App.dmglinux
Qt5 tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v495/Hydrus.Network.495.-.Linux.Qt5.-.Executable.tar.gz
Qt6 tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v495/Hydrus.Network.495.-.Linux.Qt6.-.Executable.tar.gz
I had a great week working on more Qt6 support and getting Note Import Options ready for advanced users to try out.Qt6
The Qt6 launch last week went generally well. There were a couple of little typo bugs as expected, but most users reported nothing drastic.
I have fixed several issues and also improved graphical quality at >100% UI scale. Qt6 handles UI scale tech much better, but that also exposed all the better where my custom UI was failing. Thumbnails at 125% were looking pretty ugly, with nearest-neighbour scaling, so I knuckled down and did my homework on how all this is supposed
to work, and I think I have it fixed. Thumbnails should look ok at any UI scale in Qt6, and their banner text too. My fixes apply to Qt5 too, but as far as I can tell that only really works comprehensively for 100%/200% scale. I will try to tackle the media viewer next week.
If you are an experienced user with a backup, please feel free to try Qt6 out on your real install. If both Qt5 and Qt6 are available, the client will now default to Qt6, so you shouldn't need to do a 'clean install' like last week. My test of this went fine, but if there is some odd dll conflict when you try to boot, check here on how to clear things out and either revert to Qt5-only or try Qt6-only:https://hydrusnetwork.github.io/hydrus/getting_started_installing.html#clean_installs
Any reports on failures here would be useful so I can write in any needed 'delete these old files' rules to the Qt6 Win installer.
As a reminder, afaik Windows 7 cannot run Qt6, so don't try it out if that's you. I will switch over to Qt6 exclusively in a few weeks, at which point I'll update the help and talk more about your future options, which will be: stop updating the client; move to a newer OS; run in Win7/Qt5 from source.Note Import Options
Unfortunately I couldn't fit all this in again, but I've done work I'm happy with and have parts of it ready for advanced users to play with. Fingers crossed, the first simple version of this will be completely integrated next week.
The File Import Options update last week went ok. I messed something up with the Presentation Options, so highlighting a gallery or watcher with a default FIO was always showing the same thing (new or all files) instead of what the default actually said. This is fixed.
I have pushed FIO and TIO further together this week. File Import Options and Tag Import Options are now merged into one button and one dialog across the program. The dialog is tabbed, so you edit both sets of options at the same time, and in future, any new Import Options will work through the same interface.
3 / I:
0 / P:
Qt5 zip: https://github.com/hydrusnetwork/hydrus/releases/download/v494/Hydrus.Network.494.-.Windows.Qt5.-.Extract.only.zip
Qt6 zip: https://github.com/hydrusnetwork/hydrus/releases/download/v494/Hydrus.Network.494.-.Windows.Qt6.-.Extract.only.zip
Qt5 exe: https://github.com/hydrusnetwork/hydrus/releases/download/v494/Hydrus.Network.494.-.Windows.Qt5.-.Installer.exemacOS
Qt5 app: https://github.com/hydrusnetwork/hydrus/releases/download/v494/Hydrus.Network.494.-.macOS.Qt5.-.App.dmg
Qt6 app: https://github.com/hydrusnetwork/hydrus/releases/download/v494/Hydrus.Network.494.-.macOS.Qt6.-.App.dmglinux
Qt5 tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v494/Hydrus.Network.494.-.Linux.Qt5.-.Executable.tar.gz
Qt6 tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v494/Hydrus.Network.494.-.Linux.Qt6.-.Executable.tar.gz
I had a great week with two big changes. First, there is an optional 'Qt6' version of the program for advanced users to try out, and second, File Import Options has some important updates.Qt5 and Qt6If you are a regular user, stick with the Qt5 versions of the release this week. It is the same as before.
Hydrus is moving up to a new version of its UI library, Qt. The new version has a ton of bug fixes and generally better support for newer OS concepts like UI scaling. I am going to be putting out releases for both 5 and 6 for a month or two, testing it with advanced and then normal users, and then will switch to 6 exclusively. Everything seems to be going well, and you don't have to do anything.
If you are an advanced user though, please try out the Qt6 builds. They work exactly the same as the old ones. Just to be careful, I recommend you not try them on your real database first off, and doubly so if you do not have a great backup. I am not worried about database damage, but you never know, and if there are problems, I don't want to give you inconvenience on your main install. Try a fresh extract on your desktop first to make sure it boots ok, and then delete that extract. Then, if you want to try it on your real database, try doing a 'clean install' as here: https://hydrusnetwork.github.io/hydrus/getting_started_installing.html#clean_installs
If you don't do a clean install, you'll still have Qt5 dlls in your install folder and the client will default to the older version (for now). If you are a macOS user, you don't have the concept of 'clean install', so just run the App as normal, but make sure you have a backup of your database first
. There's also no Windows Qt6 installer yet. You can check help->about
to see what version of Qt is currently running.
So far, the update has been remarkably smooth for me, with very few bugs. A user has been watching the situation for me and kindly provided a patch to deal with the most important syntax changes, so moving over has not been a massive pain in the neck. I've been using it IRL for a few days now and I think things are just that bit smoother and less flickery.
I am particularly interested in Linux and macOS users' feedback. So far, the main limitation I know about is that Windows 7 can't run Qt6 (it is just too old), but there may be other issues in other platforms. Let me know, and we'll see if we can iron them out.
I am going to keep hydrus Qt5 compatible, so anyone who needs to stay on it but wants to keep updating will have the option of running from source.
204 / I:
16 / P:
5 / I:
1 / P:
I had a great week working on some fixes and prototype EXIF support.EXIF
EXIF are metadata labels embedded in some media files, usually JPEG. They might say the make of camera that took a picture, or its aperture/ISO, the GPS coordinates it was taken at, how an image is rotated, or the DPI of a logo for printing purposes. There are many different possible fields. I have wanted to add support to view and even search this data for a long time, and this week we start with something simple but not user-friendly.
The media viewer now has a 'cog' icon on the top hover. On JPEGs, it lets you check for EXIF data. Most files don't have it, but if it does (usually this is photos or professional art exported from Photoshop etc...), it now throws up a little window to see every field.
The duplicate filter now actively scans for EXIF data and says if one or both files have it, just like the recent addition for ICC Profiles. Many websites strip EXIF data on upload, so if you have two exact dupes, the one with EXIF data is probably closer to the 'original' version.
Now I have this framework, I would like to extend it. Beyond general polish like replacing the cog icon with something nicer, and only enable it when I know there is some EXIF to show, I want to cache 'has exif data yes/no' in the database and allow you to search by that. I expect I'll also add the actual EXIF data itself to the database one day, so you'll be able to search all your pictures for iPhone 6 photos or whatever.
So, if you are interested in EXIF, please give this a go and let me know what you think. This feature was taking so long to happen that I decided to just spam out a rough v1.0 that I can keep improving.full list
- in the first step of 'official' EXIF support, the media viewer now has a 'cog' button on the top hover, enabled when looking at a jpeg, that will check the file for EXIF data. if found, it will throw it up on a simple new window that shows EXIF id, label, and value. this is a hacked-together prototype, not super user-friendly, but it works. let me know what you think, and please send me any files that have weird EXIF that doesn't parse right but you think should. I already discovered a file with a null character that wouldn't display in UI, that sort of thing
- GPS EXIF values are also parsed and extracted
- made it so you can double-click a row in this new window to copy an EXIF value to clipboard
- in the duplicate filter, if one or both files have exif data, this is now noted in the comparison statements, just like ICC profile! (issue #469)
- obvious future extensions here will be storing 'has exif' in the database and allowing its presence to be searchable and enabling the cog button (or a nicer 'exif' button) only when there is known data to see. a subsequent step would be actually caching the data in the database for full EXIF search
- as a side thing, we're now set up on the hydrus end to pull TIFF EXIF, but PIL doesn't seem to offer it, so we'll have to wait for a different solution there
1 / I:
0 / P:
Is there a way to change the upper limit on gifs from 32?
I'm trying to save a 52Mb sized gif and I got prompted with this message.
26 / I:
3 / P:
1 / I:
0 / P:
I had a good week mostly cleaning code and adding some things for advanced users.highlights
If you are in advanced mode, the file sort and collect controls now have a 'tags' button. This lets you determine which tag service that particular sort/collect applies to. If you are this tag-clever, let me know how it works for you. This tag button is the same thing that autocomplete dropdowns use, and I expect it to soon get a 'multiple location' makeover like the file button did with multiple file services.
The archived file delete lock (under options->files and trash
) gets a pass this week. If you try to delete files that are currently locked, it now makes a popup with a button to see those specific files, so you can decide what to do. The duplicate filter also handles the different situations (like 'archive both files' + 'delete the worse file') better.
The duplicate filter also now shows if one or both files have an ICC profile.
The shortcut actions to 'move page selection home/left/right/end' now try
to stay on the same level if you hit them several times. If you use these actions, try them out through a mix of page of pages and you'll see how it works now. It remembers the current level within three seconds of the last move event. This was requested by several users, and there isn't a nice way to do it so I hacked an answer, so let me know what you think.full list
- sort and collect updates:
- for big brain users, the collect control now has a tag domain button. it only shows if you are in advanced mode (issue #572)
- the sort control also has a tag domain button hidden behind advanced mode. it applies to system:num tags and namespace sorting
- the collect control now appears on all import pages
- archived file delete lock:
- the duplicate processing action code now no longer archives files that are due for deletion right before that deletion. this was hitting the archive delete lock
- if archive delete lock is on and the 'other' file in the duplicate filter is archived, the option to 'this is better, delete the other' is now disabled
- if you attempt to delete a delete-locked file during normal browsing, or if an automatic system like export folders wants to but fails on some, a popup is now made with a button to show the files that were filtered out so you can review the situation and fix it if you want
- I am considering adding a dialog to say 'hey, this is locked, want to send back to inbox?' to fix these situations in a nice way, but I think this is probably a bad idea in terms of workflow, design, and my sanity given all the edge cases and potential future expansions of lock rules. maybe I'll add a simple 'delete and override lock checks' option, but a lock is a lock tbh. for now, I will focus on this better UI feedback of currently delete-locked files and make it simpler for humans to remove any locks
3 / I:
0 / P:
I had a good couple of weeks doing a behind the scenes overhaul and a variety of quality of life work. I was ill last week and put off the release, but I am feeling great now.metadata import/export
As is often the case, an important overhaul makes few actual changes front end. I've been trying to do this for a while, but now the 'export/import tags in a neighbouring .txt file' routine now works on completely new tech. Rather than hardcoded tags into .txt files, it now uses a modular system that I will be able to expand in future to support filtering and string processing, more metadata types like ratings and URLs, JSON and XML as the file formats, and will even allow funky migrations like converting tags to URLs.
As a side benefit, Export Folders now support tags .txt export.
For now, all the UI front-end looks the same, but please expect this to change in future. I'll be writing some nice unified panels and dialogs to handle the new objects as I write them.advanced user highlights
The 'OR*' advanced tag input now supports system predicates! It uses the same system predicate parser as the Client API, so you can now type or copy/paste most
system predicate text and get something both useful and complicated. The text that parses isn't always exactly the same as the predicate label, so check out the big list of example parseable system predicates here: https://hydrusnetwork.github.io/hydrus/developer_api.html#get_files_search_files
Also, if you are a parser creator, String Processing now has a Tag Filter processing step. Let's say you can grab all the tags from somewhere but you need to filter out a handful of non-tag text like '+' and '?', or you are able to create hydrus namespaced tags and want to filter by namespace, just insert this into your string processing and it should be much easier than messing around with long regexes.
0 / I:
0 / P:
Moving Hydrus library between computers
I'm running Hydrus on a Mac, and tried to move the .app of my library (23GB) onto another Mac, but found it didn't work. Is there an easy way to do this?
3 / I:
0 / P:
I had a good week working on a variety of small jobs.highlights
While working with a user, I discovered I had recently messed up the initialisation of the image caches, causing them to use default values and be too small for many power users. This is fixed, so I hope you will get some smoother media viewing, especially when it comes to very large images.
I tidied up some weird file service bugs and annoyances that came from the multiple local file services transition. Fixed some weird 'delete from x' entries on the thumbnail menu, stopped spamming 'all my files' in places it wasn't helpful, figured out better service ordering, just some simple workflow stuff.
And I think
I have fixed another PTR processing bug some users had. If you have had 'this update file was missing' errors that wouldn't fix themselves, please try again and let the automatic maintenance run one more time--it should
repair your records this time.
And just a fun thing--Mr Bones now has more numbers, and more neatly laid out.full list
- fixed a stupid bug that meant the image caches were initialising with default values (as under _speed and memory_) until you opened and OKed the options dialog (or did some other options-refresh events). sorry for the trouble, please enjoy some smoother image browsing.
- mr bones now shows more numbers, and in a neater table. it should be clearer what the percentages are for now, too
- the _manage->regenerate_ thumbnail menu has additional quick maintenance commands for presence and integrity checks and regenerating data in the similar files system
- wrote a new 'special duplicate' button for the edit shortcut set dialog. the list on this dialog doesn't allow duplicates (which meant the old 'duplicate' button was doing nothing), so this duplicates the current actions with 'incremented' shortcut keys. 'a' becomes 'b', 'ctrl+5' becomes 'ctrl+6', and so on. it doesn't always work, but if you want to make ten shortcuts for setting rating 1-10, this should help
- fixed an issue where the thumbnail banner text and the media viewer background text was not changing size or font according to QSS stylesheet rules (issue #1173)
- SIGTERM should now cause a clean program exit (previously it killed the GUI App but left some daemon threads alive for thirty seconds or more). unlike SIGINT, it will not ask you if you are sure you want to exit or if you would like to do shutdown maintenance--it just closes the client promptly
- fixed a bug in last week's importer page status improvements--the hard drive import page wasn't showing all the updates it should have
- brushed up some backup help
- file services:
- fixed a bug where advanced users could set 'all known files'/'all known tags' on a search dropdown. this search domain is not supported
- in the archive/delete filter, if the current location is 'all my files' and the files being deleted are only in one local file domain, the surplus 'all my files' will no longer appear at the top of the filter's commit dialog
- the file services in the thumbnail select/remove menu are now sorted in the same order as the file domain button in search dropdowns
- the thumbnail select/remove menus now exclude 'all my files' and 'all local files' if those choices are redundant (e.g. if you only have files in 'my files', 'all my files' will be hidden)
- fixed some incorrect 'delete from x' actions appearing in thumbnail right-click menus
2 / I:
0 / P:
I had a good week getting back into the swing of things. I fixed some important bugs and improved some UI.highlights
All the downloader pages--gallery, watcher, urls, and simple--have a revamped status system. All the text that shows how file or gallery downloads are going is now generated in a better way, with more error states (e.g. it will tell you when your gallery stopped because it hit the file limit, or when one of the emergency pause states under the network
menu has kicked in), and logic in edge cases is improved. Everything is unified now, so the texts are the same across all pages. Also, if a gallery query or watched thread is 'pending', its text now reports that it is waiting for a work slot, rather than staying blank. There _shouldn't_ be any situations now where a downloader is unpaused with work to do but has blank status.
If you use the multiple local file services system, the archive/delete filter now presents more options when you are done. If the files are in more than one local file service, you can choose where you delete them from, including all applicable. This was confusing and opaque before, so I hope this makes it more clear what is happening and gives you more choice.
I _believe_ I have fixed an important bug some users were having with PTR processing. There was an annoying issue about a 'definitions' file being seen as a 'content' file, or vice versa, that the automatic maintenance could not fix. I finally managed to reproduce the issue and fixed it. I schedule a fix in the update this week, so if you have been hit by this, please wait for one more round of file maintenance 'metadata' scans, and then unpause the PTR one more time. Essentially, I think I fixed the automatic maintenance. Let me know how you get on!full list
- downloader pages:
- greatly improved the status reporting for downloader pages. the way the little text updates on your file and gallery progress are generated and presented is overhauled, and tests are unified across the different downloader pages. you now get specific texts on all possible reasons the queue cannot currently process, such as the emergency pause states under the _network_ menu or specific info like hitting the file limit, and all the code involved here is much cleaner
- the 'working/pending' status, when you have a whole bunch of galleries or watchers wanting to run at the same time, is now calculated more reliably, and the UI will report 'waiting for a work slot' on pending jobs. no more blank pending!
- when you pause mid-job, the 'pausing - status' text is generated is a little neater too
- with luck, we'll also have fewer examples of 64KB of 503 error html spamming the UI
- any critical unhandled errors during importing proper now stop that queue until a client restart and make an appropriate status text and popup (in some situations, they previously could spam every thirty seconds)
- the simple downloader and urls downloader now support the 'delay work until later' error system. actual UI for status reporting on these downloaders remains limited, however
- a bunch of misc downloader page cleanup
1 / I:
0 / P:
I had a good simple week making a clean release before my vacation.
Everything is misc this week, nothing earth-shattering, just a bunch of cleanup and little stuff. If you have any wavpack files, try importing them!full list
- the client now supports 'wavpack' files. these are basically a kind of compressed wav. mpv seems to play them fine too!
- added a new file maintenance action, 'if file is missing, note it in log', which records the metadata about missing files to the database directory but makes no other action
- the 'file is missing/incorrect' file maintenance jobs now also export the files' tags to the database directory, to further help identify them
- simplified the logic behind the 'remove files if they are trashed' option. it should fire off more reliably now, even if you have a weird multiple-domain location for the current page, and still not fire if you are actually looking at the trash
- if you paste an URL into the normal 'urls' downloader page, and it already has that URL and the URL has status 'failed', that existing URL will now be tried again. let's see how this works IRL, maybe it needs an option, maybe this feels natural when it comes up
- the default bandwidth rules are boosted. the client is more efficient these days and doesn't need so many forced breaks on big import lists, and the internet has generally moved on. thanks to the users who helped talk out what the new limits should aim at. if you are an existing user, you can change your current defaults under _network->data->review bandwidth usage and edit rules_--there's even a button to revert your defaults 'back' to these new rules
- now like all its neighbours, the cog icon on the duplicate right-side hover no longer annoyingly steals keyboard focus on a click.
- did some code and logic cleanup around 'delete files', particularly to improve repository update deletes now we have multiple local file services, and in planning for future maintenance in this area
- all the 'yes yes no' dialogs--the ones with multiple yes options--are moved to the newer panel system and will render their size and layout a bit more uniformly
- may have fixed an issue with a very slow to boot client trying to politely wait on the thumbnail cache before it instantiates
- misc UI text rewording and layout flag fixes
- fixed some jank formatting on database migration helpnext week
I am now off for a week. I think I need it! I'm going to play a ton of vidya, shitpost the big streams that are happening, fit some Wagner in, and get on top of outstanding IRL stuff. I'll be back to catch up my messages on Saturday the 18th.
2 / I:
0 / P:
I had an ok week. I was a unexpectedly short on time, so I couldn't get everything I wanted done, but I cleared out some small work.highlights
The big update last week, which I recommended only for advanced users, went well. There don't seem to be any obvious problems with the logic of the new search cache, so I now recommend it for everyone. You will be presented with a popup just before the update runs, giving you an estimate of how long it thinks it will take. Most users should take 5-10 minutes, but if you have millions of files, it will be longer. Just let it run and some things will run a bit faster and neater in the background. If you have played with 'multiple local file services', then check out the new 'all my files' domain you will see--this is basically an efficient umbrella of all your local file services. It works super fast for things like the duplicates system.
I also put some time into the duplicate filter this week. The logic of the queue is improved again, so some rare errors when reaching the end of a batch should be fixed. I also integrated manual file deletes into the queue processing: now, when you manually delete a file, or both, the deletes will not happen until you commit--just like the other decisions you are making--and they are undoable if you select 'forget' or go back a pair. You also won't see a file you manually deleted again in a batch (it'll auto-skip if that file comes up again).
Also, the duplicate filter now has a little 'send pair to page' button, which publishes the current pair to the duplicates page that made the filter, just in case you want to save them for some extra processing after you are done filtering. You can do this with multiple pairs and they'll just stack up in the page.
A couple other neat things happened in last week's advanced-user-only release, which I will repeat here:
The 'media viewers' shortcut set has three new zoom actions: 'switch between 100% and max', 'switch between canvas and max', and 'zoom to max'.
When you enter pairs in the tag sibling dialog, it shouldn't complain about loops anymore, but instead automatically break them, just like how it will auto-petition an A->B, A->C conflict.
3 / I:
0 / P:
I had a great week working on a variety of smaller issues and some important database updates.The release this week is only recommended for advanced users. I make an important change, and I want to make sure the update works quickly and without problems before I roll it out to everyone. If you are not an advanced user, please check back in next week!The update will also take a few minutes this week.all my files
So, I have made a new virtual service, 'all my files', which covers the union of all your local file services. This service is very similar to 'all local files', but it does not include trash or repository files. It provides a bunch of tools across the program for quick and precise searching of all the files that have value and are worth looking at.
When you update, this new service will be created and populated. It will take a few minutes, longer if you have millions of files and tags. My 2.8-million-file ptr-syncing client took 32 minutes.
There are progress updates on the splash window.
Once you are booted, you will see 'all my files' in review services and the file domain selector if you have more than one local file domain. Feel free to play around with it--it will run a lot faster than previously going 'multiple locations' and unioning all your local file services.
The code is working really well on my end, and I am not afraid of anything being damaged, but if something goes wrong, it may require some clever/slow regeneration to fix. The main things I would like to know are:
1) Did your update take significantly longer than ~100k files/minute? Did it get held up on anything?
2) After some use, have you noticed any file/tag miscounting with 'all my files'?
As always, make a backup before you update.other highlights
The 'media viewers' shortcut set has three new zoom actions: 'switch between 100% and max', 'switch between canvas and max', and 'zoom to max'.
When you enter pairs in the tag sibling dialog, it shouldn't complain about loops anymore, but instead automatically break them, just like how it will auto-petition an A->B, A->C conflict.
The database now cleans up after itself more thoroughly. Some users have been having trouble with very large 'WAL' files, some getting to be multiple GB, and perhaps seeing bloated memory use along with it. A set of new maintenance routines now force write-flushing at regular intervals. In my testing, there is no lag related to this, but I will be interested to hear if anyone gets new commit hang-ups during very heavy work. If you have had a huge WAL, let me know if this helps!
10 / I:
1 / P:
I had an ok week. I fixed some things, improved some quality of life, and made internal file migration a bit easier.highlights
Last week's debut of multiple local file services went well! As far as I know, no one who tried it out had any big problems, and my main concerns--mostly that it needs some better migration tools and workflows and 'this file is in here' UI--proved true. So, I know what I have to do and will keep working. Multiple local file services remains for advanced users for now, but I hope to launch it properly for everyone, with nice help, next week.
However, while doing this work, I did accidentally break the simple version of the 'delete files' dialog when files were in the trash--rather than say 'delete these permanently?', it just wouldn't appear. This was due to a logical oversight where it wasn't testing and counting up 'trash' status correctly. It is fixed now.
Also, there was a problem with the new duplicate filter queue for users who have done a good bit of processing. A certain function that in complicated situations automatically skips some pairs was failing whenever it hit the end of a batch. This is also fixed now, thank you for the great reports on this.
For multiple local file services, I updated the UI code, fixing some little bugs and improving the workflow when you have complicated situations, and I integrated the shortcuts system and the media viewer. You can now create 'add/move to service x' actions in the 'media' shortcut set, and the media viewer has the same add/move menu on right-clicks.
The media viewer has several other improvements: I think I fixed that annoying bug where a fullscreen borderless view of media that exactly fits the screen would sometimes not resize when you went back to normal window mode! Also, scrolling the mouse over the taglist hover window should no longer ever cause a 'previous/next media' event. And I have implemented a 'short and simple' version of the video/audio scanbar to show (instead of completely hiding it) when your mouse is away--just a few pixels to show things 'at a glance'. Even though it covers a few pixels of video at the bottom, I liked this so much that I set it as the default for all users. If you don't like it, you can hide it again with the new setting under options->media
1 / I:
0 / P:
4 / I:
0 / P:
I had a good couple of weeks doing some regular work and getting 'multiple local file services' ready for testing.multiple local file servicesThis is not ready for everyone yet! Advanced users only for now please.
I turned multiple local file services on in debug mode last week, just to see how things were looking, and it turned out suprisingly great, no big problems. For several months now I have been doing prep work for it, and that seems to have paid off. I decided to finish the last important things and get a v1.0 out.
So, it is now possible to have multiple 'my files' services in your client, and to search, import to, and migrate files between them. These services are completely blind to each other, so searching for autocomplete tags in one will not return suggestions from another. The hope is this will allow fairly good sfw/nsfw-style separations in clients and open up interesting new contained workflows.
I am recommending this only for advanced users for now, and moreso only those who have been following this feature. I have not yet written up nice help for this, and some of the UI/workflow is still not user friendly, so what I would like is for people who are enthusiastic to try it out and let me know what they think. I really haven't run into any massive errors, but I won't encourage you to go crazy on a real client yet. Go nuts on a new empty test client, or experiment carefully on a real client, just in case something goes wrong, and I will keep polishing the experience.
The basics are: you can now make a new 'local file domain' in manage services
. file import options
now lets you import to different or multiple local file domains, and thumbnail right-click lets you copy or move files between them too. The normal search page dropdown lets you jump between local services just like searching trash, and of course it now supports multiple domains if you want to do a union. The delete and undelete commands are similarly a little more powerful when you start adding new services. Check out the changelog for more specific details.
Next step I think is to make it more obvious when thumbnails/files are in certain services, since at the moment you have to scan the text on the status bar, top media hover, or thumbnail menu. Maybe custom icon rules (e.g. 'when the file is in "sfw" domain, give it a flower icon'). Then general polish like shortcut integration, maybe some more search tech, and then I really want to write a nice help document for it all to introduce normal experienced users to the idea, and some 'merge these clients' tech would be great, so users who have been using two or more clients for years can finally combine them into one.
3 / I:
0 / P:
I had a great week doing cleanup and other simple work.highlights
I fixed the problem where clicks on a scrolled taglist were going to the wrong location. I was cleaning up some ancient wx->Qt code hacks and it seems I rarely scroll and click when working, so I never noticed the problem. I have a new test to make sure this does not happen again. Sorry for the trouble!
The URLs in the top-right hover menu are now styled better. No longer underlined, and now colourable by QSS. I have updated all the default stylesheets that come with the client (you can set these under options->style
) to have some decent colours. If you have your own custom QSS, check my default to see how to set it yourself.
You can now set duplicate action options to 'always archive both files', if you want to play with making the duplicate filter do some of the work of the archive/delete filter.
Also, the duplicate filter now has improved image prefetch. There should be less flickering when you switch from A to B the first time and when you action a pair and move to the next. Please note that if you still get flicker for 4k images, try boosting the image cache size under options->speed and memory
(I boosted the default up to 384MB this week, so you might like to give it some more too).full list
- fixed the stupid taglist scrolled-click position problem--sorry! I have a new specific weekly test for this, so it shouldn't happen again (issue #1120)
- I made it so middle-clicking on a tag list does a select event again
- the duplicate action options now let you say to archive both files regardless of their current archive status (issue #472)
- the duplicate filter is now hooked into the media prefetch system. as soon as 'A' is displayed, the 'B' file will now be queued to be loaded, so with luck you will see very little flicker on the first transition from A->B.
- I updated the duplicate filter's queue to store more information and added the next pair to the new prefetch queue, so when you action a pair, the A of the next pair should also load up quickly
- boosted the default sizes of the thumbnail and image caches up to 32MB and 384MB (from 25/150) and gave them nicer 'bytes quantity' widgets in the options panel
- when popup windows show network jobs, they now have delayed hide. with luck, this will make subscriptions more stable in height, less flickering as jobs are loaded and unloaded
- reduced the extremes of the new auto-throttled pending upload. it will now change speed slower, on less strict of a schedule, and won't go as fast or slow max
- the text colour of hyperlinks across the program, most significantly in the top-right media hover window, can now be customised in QSS. I have set some ok defaults for all the QSS styles that come with the client, if you have a custom QSS, check out my default to see what you need to do. also hyperlinks are no longer underlined and you can't 'select' their text with the mouse any more (this was a weird rich-text flag)
- the client api and local booru now have a checkbox in their manage services panel for 'normie-friendly welcome page', which switches the default ascii art for an alternate
- fixed an issue with the hydrus server not explicitly saying it is utf-8 when rendering html
- may have fixed some issues with autocomplete dropdowns getting hung up in the wrong position and not fixing themselves until parent resize event or similar
3 / I:
0 / P:
I had a great week. Lots of different small jobs done.notes and hover windows
I'm happy with last week's work making notes show in media viewers, but I introduced some little bugs while rewriting hover windows. I have now fixed the bad text colour behind the top hover, the problem where clicking on tags or greyspace was propagating up to the archive/delete and duplicate filters, the bad hover panel colour on non-default stylesheets, and some note window position and size issues.
Also, for notes, you can now right-click them to collapse them in the hover window. Right-click again on the name to expand again. This is a test, really, just to see if it helps navigating files with many long notes. Double-clicking on the note tab in the edit dialog lets you rename, and a checkbox under the new options->notes
now lets you choose whether the text caret starts at the beginning or end of the document when editing.
Furthermore, I have updated all the icon buttons in all the hovers to no longer take focus when you click on them. They were previously stealing arrow key and space after a click (to do button-to-button form navigation), which meant you couldn't click on, say, a duplicate filter action button and then go back to arrow keys to navigate. Now you should be able to mix clicks and arrow keys without trickery. If this affects you, let me know how it goes!other highlights
If you didn't like the recent 'ctrl- and shift-clicks no longer show files in the preview viewer' change, check out the new checkboxes under options->gui pages
. You can make either click type focus for all files again or just files with no duration--if you don't want noisy videos being annoying while you ctrl-click.
The 'advanced mode' autocomplete dropdown now has two 'OR' buttons. The left one opens a new empty OR edit dialog, the right one opens the advanced text parsing input as before.full list
- fixes and improvements after last week's hover and note work:
- fixed the text colour behind the top middle hover window
- stopped clicks on the taglist and hover greyspace being duplicated up to the main canvas (this affected the archive/delete and duplicate filter shortcuts)
- fixed the background colour of the hover windows when using non-default stylesheets
- fixed the notes hover window--after having shown some notes--could then lurk in the top-left corner when it should have been hidden completely
2 / I:
0 / P:
I had a good week. Notes now display on the media viewer.notes
Notes have always been a slightly hidden system, a bit like ratings were. Today is a step forward to exposing them. Any file that has notes (you can start adding notes to a file by hitting manage->notes
on their right-click menu) will now show them in the media viewer, just below the top-right hover window. They get their own hover window too, if you mouse over them. If you click on a particular note, the 'edit notes' dialog opens on it.
This is a first version, and a little ugly, but I'm happy we now have something I can iterate on in future. If you are a big notes person, please let me know how it works best and worst for you. If you have unusual font style, size, or colour, let me know if it goes crazy or sizes too short or tall.
While working on this, I rewrote the media viewer's hover windows to be more sensible, something I have been planning for a long time. They are now 'embedded' into the parent canvas, which should reduce a variety of jank behaviour--particularly, if you now click a hover, the main media viewer window no longer loses focus. There is still some hackery in the system to clean up, but I hope it'll work better overall for you.
Unfortunately, I just did not get to note merge in the duplicates system or note parsing. That'll have to be for the future.the rest
A user is working on a neat 'gallery share' system that plugs into the Client API, here: https://github.com/floogulinc/hyshare
. It looks like a great replacement for my old 'local booru', so if you are interested in sharing groups of files straight from your client with friends over an attractive booru-like interface, check it out!
I copied the 'file log' and 'search log' button menus, where you can do en masse
actions like 'retry all failed' and 'export all to clipboard', to both the log review panels and the downloader/watcher list right-click menus. It is now possible to big actions on logs without highlighting anything.
Just a small thing, but when you select a gallery in the gallery downloader page, the focus moves straight to the query text input, so you can start typing immediately.
3 / I:
0 / P:
I had an ok week doing a mix of work.highlights
I made it so when you shift-select some thumbnails, you can now move 'back' to deselect what you just selected. This also remembers what was previously selected before the shift-select started, so it works basically like an undo. I like how this works, but as part of it I had to make every thumbnail 'hit' focus in the preview viewer, which is not how all selects worked before. I already find this annoying, so I think I am going to make the system more clever and add some options around this behaviour.
I think I improved the duplicate filter's zoom locking, particularly when one of a pair is portrait and the other is landscape. It should generally be more 'stable' now, but let me know if you still have any trouble.
WebPs should now show transparency correctly!
The new file history chart has another round of better math, an 'archived files' line, and you can hide the deleted files line if it is too big.
I reworked a little help to make some 'ok, I know the basics, what next?' things clearer to find. If you missed learning about the autocomplete dropdown, tag wildcards, or OR searching, please check here: https://hydrusnetwork.github.io/hydrus/getting_started_searching.htmlfull list
- when shift-selecting some thumbnails, you can now reverse the direction of the select and what you just selected will be deselected, basically a full undo (issue #1105)
- when ctrl-selecting thumbnails, if you add to the selection, the file you click is now focused and always previewed (previously this only happened if there was no focused file already). this is related to the shift-select logic above, but it may be annoying when making a big ctrl-selection of videos etc.. so let me know and I can make this more clever if needed
- added file sort 'file->hash', which sorts pseudorandomly but repeatably. it sounds not super clever, but it will be useful for certain comparison operations across clients
0 / I:
0 / P:
Does Nijie work for anyone else anymore? I get 503 errors every time, no matter the user-agent or cookies I use.
2 / I:
0 / P:
I had a good week mostly fixing some bugs and cleaning things up behind the scenes. There's nothing super big to highlight, but I did improve the new file history chart (help->view file history
). The axes are a bit nicer, and I fixed a small counting logic bug in the 'inbox' line.full list
- if a file note text is crazy and can't be displayed, this is now handled and the best visual approximation is displayed (and saved back on ok) instead
- fixed an error in the cloudflare problem detection calls for the newer versions of cloudscraper (>=1.2.60) while maintaining support for the older versions. fingers crossed, we also shouldn't repeat this specific error if they refactor again
- file history chart updates:
- fixed the 'inbox' line in file history, which has to be calculated in an odd way and was not counting on file imports adding to the inbox
- the file history chart now expands its y axis range to show all data even if deleted_files is huge. we'll see how nice this actually is IRL
- bumped the file history resolution up from 1,000 to 2,000 steps
- the y axis _should_ now show localised numbers, 5,000 instead of 5000, but the method by which this occurs involves fox tongues and the breath of a slighted widow, so it may just not work for some machines
- cleanup, mostly file location stuff:
- I believe I have replaced all the remaining surplus static 'my files' references with code compatible with multiple local file services. when I add the capability to create new local file services, there now won't be a problem trying to display thumbnails or generate menu actions etc... if they aren't in 'my files'
- pulled the autocomplete dropdown file domain button code out to its own class and refactored it and the multiple location context panel to their own file
- added a 'default file location' option to 'files and trash' page, and a bunch of dialogs (e.g. the search panel when you make a new export folder) and similar now pull it to initialise. for most users this will stay 'my files' forever, but when we hit multiple local file services, it may want to change
- the file domain override options in 'manage tag display and search' now work on the new location system and support multple file services
- in downloaders, when highlighting, a database job that does the 'show files' filter (e.g. to include those in trash or not) now works on the new location context system and will handle files that will be imported to places other than my files
- refactored client api file service parsing
- refactored client api hashes parsing
- cleaned a whole heap of misc location code
- cleaned misc basic code across hydrus and client constant files
- gave 'you don't want the server' help page a very quick pass
- client api:
- in prep for multiple local file services, delete_files now takes an optional file_service_key or file_service_name. by default, it now deletes from all appropriate local services, so behaviour is unchanged from before without the parameter if you just want to delete m8
- undelete files is the same. when we have multiple local file services, an undelete without a file service will undelete to all locations that have a delete record
- delete_files also now takes an optional 'reason' parameter
3 / I:
1 / P:
I had a good week. There is a mix of small work, an expansion to the Client API, and a bit of fun.misc
The network engine now pulls source time directly from file downloads if the server provides a date. This means a whole bunch of sites that haven't provided a good source time until now suddenly do, which improves the new aggregate modified time and also subscription and watcher check timings.
With our new apng parsing tech, I fixed up apng duration parsing, which was until now relying on a fallback default of 24 fps if ffmpeg couldn't figure it out. All apngs are scheduled for another scan.
I fixed an important precedence bug in the network engine that matches URLs to URL Classes. If you have been making a downloader and had Gallery URLs matching as Post URLs, please give it another go. Sorry for the trouble!Client API
A user has written a cool expansion to the Client API, which I appreciate. You can now fetch, set, and delete file notes! If you are an API dev, check out the documentation for the new calls (fetching notes is now a parameter on file_metadata).
He also made technical improvements. The Client API now supports far longer GET requests, up to 2MB of URL if needed, and the whole API has tentative and experimental support for CBOR instead of JSON if you wish.file history chart
Another user has for some time been playing around with drawing charts of a client's file history in matplotlib using raw database data. You may have run his script yourself. We have been talking for a while about integrating this into hydrus, and this week I finally got around to implementing it in QtCharts.
Please hit 'view file history' on the help menu to see the new chart. This is a simple, first attempt on my end, but it should show you a cool history of how many files you have had. If you have been using the client for any time, the lines for deleted files and inbox will be very incomplete, but this data will fill out in time.
This was fun to do, and I learned a bit more about QtCharts. I fixed a couple of ugly things in the bandwidth bar chart I made before, and I think I'll do some more here too. I have a thought to start drawing some of our other data, let's say file size or number of file views, and seeing if pareto or normal distributions pop out. Anyway, let me know what you think, and feel free to share your file history chart!
3 / I:
0 / P:
I had a good week integrating two new features: autocomplete tag search in the client api, and saved 'post times' from downloaders.post times
'Modified time' is neat, but it isn't super useful for downloaded files--since the file was only just added to your hard drive, it'll always be the same as import time. This week I integrate the 'source time' we parse from various websites to improve the modified time for downloaded files. The objective is to make 'modified' a fairly decent 'this file was completed around this time' number for searching and sorting purposes.
I'm being careful not to overwrite anything. The client now saves its best 'source time' for every different site it downloads from and then the earliest of those + modified date is used as the aggregate modified date. You don't have to do anything, but with luck you will see your new watcher and gallery files start to get some nicer modified times in the media viewer and thumbnail right-click menu.
There are many potential future expansions here. I can grab better post times from sites, show and edit every stored timestamp in UI, allow clever search and sort of those specifically (e.g. 'sort all these files by their danbooru post time), and most importantly make some sort of maintenance system to retroactively fetch a good post time for all the files we downloaded before post times were saved. This is just a first step.
I integrated the new 'archive time' too during this work. This now shows in the media viewer and thumbnail right-click menu similarly and can be sorted by. Search will come soon. I also want to think about optionally filling in some estimate dummy data here for all the files we archived before timestamps were tracked.client api autocomplete search
A user has helpfully written Client API routines for autocomplete tag search, which is something I have had trouble fitting in. I appreciate the work. This should let the various tools that use the Client API do more tag browsing in future.
The documentation is here: https://hydrusnetwork.github.io/hydrus/developer_api.html#add_tags_search_tags
There are several ways to expand this too, so if you are an API dev interested in it, let me know how it goes.
3 / I:
0 / P:
I had a good couple of weeks. There's a long changelog of small items and some new help.new help
A user has converted all my old handcoded help html to template markup and now the help is automatically built with MkDocs. It now looks nicer for more situations, has automatically generated tables of contents, a darkmode, and even in-built search.
It has been live for a week here:https://hydrusnetwork.github.io/hydrus/
And is with v475 rolled into the builds too, so you'll have it on your hard disk. Users who run from source will need to build it themselves if they want the local copy, but it is real easy, just one line you can fold into an update script:https://hydrusnetwork.github.io/hydrus/about_docs.html
I am happy with how this turned out and am very thankful to the user who put the work in to make the migration. It all converted to this new format without any big problems.misc highlights
I queue up some more files for metadata rescans, hopefully fixing some more apngs and figuring out some audio-only mp4s.
System:hash now supports 'is not', so if you want to paste a ton of hashes you can now say 'but not any of these specific files'.
Searches with lots of -negated tags should be a good bit faster now.
I fixed a bug that was stopping duplicate pages from saving changes to their search.pycharm
I moved to a new IDE (the software that you use to program with) this week, moving from a jank old WingIDE environment to new PyCharm. It took a bit of time to get familiar with it, so the first week was mostly me doing simple code cleanup to learn the shortcuts and so on, but I am overall very happy with it. It is very powerful and customisable, and it can handle a variety of new tech better.
It might be another few weeks before I am 100% productivity with it, but I am now more ready to move to python 3.9 and Qt 6 later in the year.full list
- new help docs:
- the hydrus help is now built from markup using MkDocs! it now looks nicer and has search and automatically generated tables of contents and so on. please check it out. a user converted _all_ my old handwritten html to markup and figured out a migration process. thank you very much to this user.
- the help has pretty much the same structure, but online it has moved up a directory from https://hydrusnetwork.github.io/hydrus/help
all the old links should redirect in any case, so it isn't a big deal, but I have updated the various places in the program and my social media that have direct links. let me know if you have any trouble
- if you run from source and want a local copy of the help, you can build your own as here: https://hydrusnetwork.github.io/hydrus/about_docs.html
. it is super simple, it just takes one extra step. Or just download and extract one of the archive builds
- if you run from source, hit _help->open help_, and don't have help built, the client now gives you a dialog to open the online help or see the guide to build your help
- the help got another round of updates in the second week, some fixed URLs and things and the start of the integration of the 'simple help' written by a user
- I added a screenshot and a bit more text to the 'backing up' help to show how to set up FreeFileSync for a good simple backup
6 / I:
0 / P:
I had a great week working on small fixes and improvements. There's nothing earth-shattering here to highlight, but just a mix of little work.full list
- command palette:
- the guy who put the command pallete together has fixed a 'show palette' bug some people encountered (issue #1060)
- he also added mouse support!
- he added support to show checkable menu items too, and I integrated this for the menubar (lightning bolt icon) items
- I added a line to the default QSS that I think fixes the odd icon/text background colours some users saw in the command palette
- file archive times are now recorded in the background. there's no load/search/sort yet, but this will be added in future
- under 'manage shortcuts', there is a new checkbox to rename left- and right-click to primary- and secondary- in the shortcuts UI. if you have a flipped mouse or any other odd situation, try it out
- if a file storage location does not have enough free disk space for a file, or if it just has <100MB generally, the client now throws up a popup to say what happened specifically with instructions to shut down and fix now and automatically pauses subscriptions, paged file import queues, and import folders. this test occurs before the attempt to copy the file into place. free space isn't actually checked over and over, it is cached for up to an hour depending on the last free space amount
- this 'paused all regular imports' mode is also now fired any time any simple file-add action fails to copy. at this stage, we are talking 'device disconnected' and 'device failed' style errors, so might as well pause everything just to be careful
- when the downloader hits a post url that spawns several subsidiary downloads (for instance on pixiv and artstation when you have a multi-file post), the status of that parent post is now 'completed', a new status to represent 'good, but not direct file'. new download queues will then present '3N' and '3 successful' summary counts that actually correspond to number of files rather than number of successful items
- pages now give a concise 'summary name' of 'name - num_files - import progress' (it also eli...des for longer names) for menus and the new command palette, which unlike the older status-bar-based strings are always available and will stop clients with many pages becoming multi-wide-column-menu-hell
- improved apng parsing. hydrus can now detect that pngs are actually apngs for (hopefully) all types of valid apng. it turns out some weird apngs have some additional header data, but I wrote a new chunk parser that should figure it all out
- with luck, users who have window focus issues when closing a child window (e.g. close review services, the main gui does not get focus back), should now see that happen (issue #1063). this may need some more work, so let me know
- the session weight count in the 'pages' menu now updates on any add thumbs, remove thumbs, or thumbnail panel swap. this _should_ be fast all the time, and buffer nicely if it is ever overwhelmed, but let me know if you have a madlad session and get significant new lag when you watch a downloader bring in new files
7 / I:
0 / P:
I had a mixed week. Unfortunately some IRL reduced my work time. There's a neat new widget to play with though!command palette
A user has written a cool widget that helps you navigate the program by keyboard. I have integrated the first version and am interested in feedback. If you have nothing on Ctrl+P for your 'main window' shortcut set, you should get it mapped to that on update.
So, hit Ctrl+P and you'll get a palette where you can type and press up/down and enter to quickly navigate to any of the pages currently open. If you are an advanced mode user, you will also search all of the menubar actions and also the current thumbnail selection menu. This latter part is unfiltered at the moment--you'll just see everything--so be careful. The system needs more polish, including filtering out these more advanced database routines, and proper display for checkbox items 'check' status, and so on.
I can do a lot more with this widget, so give it a go and let me know what you think. I think some of the labels can probably be improved, and I am sure some would like to customise it a little. If you don't like Ctrl+P, just hit up file->shortcuts->the main window
and re-map it!full list
- fixed the recent problem with drag and dropping thumbnails to a level below the top row of pages. sorry for the trouble!
- fixed a bug where the client would not load results sorting by 'import time' when the search file domain was a single deleted file domain
- fixed a list display bug in the edit page parser dialog when a subsidiary page parser has two complicated string-match based content parsers
- collections now sort by modified time, using the largest known modified time in their collection
- added sqlite3.exe console back into the windows build--sorry, it was missing since the github build changeover!
- added a note to the help about backing up when tight on space, which I will repeat here: the sqlite database files are very compressible (70GB->17GB on default 7zip settings!), so if you need more space on your backup drive, this is a good way to reclaim it
- command palette:
- a user has written a cool 'command palette' for the program! it brings up a type-and-search interface to navigate to pages or menu entries.
- I have integrated his first version and set the default shortcut to Ctrl+P. users who update will get this shortcut if they have nothing else on Ctrl+P on 'main window' set. if you prefer Ctrl+K or anything else, you can change it under _file->shortcuts->the main window_
- regular users will get a page list they can search and select, advanced users will also get the (potentially dangerous) full scan of the menubar and current thumbnail right-click menu. I will be polishing this latter feature in future to filter out big maintenance jobs and show checkbox status and similar, so if you are advanced, please be careful for now
- try it out, and let me know how it goes. the underlying widget is neat, and I can change its behaviour and extend it significantly
9 / I:
2 / P:
I had a good week. There are some neat new UI features and several bug fixes.new stuff
Although it isn't super useful yet, searching in multiple file domains is now complete. If you click the file domain button on a search page's tag autocomplete dropdown, there is now a 'multiple locations' entry. This lets you select a union of several domains. New users will probably only see 'my files' and 'trash', but advanced users will see more, and deleted domains too. Please feel free to play with this. Autocomplete counts should have accurate ranges and the union file searches, limited though they currently are, do work. We are a step closer to fully supporting multiple 'my files' services.
, there is now a dropdown for 'thumbnail scaling'. The default remains 'scale down only', but you can now 'scale to fit' (which scales small things up) or 'scale to fill', which crops and scales up so the thumbnail fills the whole thumbnail space. Give it a go--the animation as one format changes to another is accidentally one of the coolest things I have done.
If you right-click on the page tab bar, there is now a 'pages' submenu, listing all the pages at or below that level, that lets you quickly navigate to them. This is a prototype, basically a copy of the same thing you'll see in a web browser, so let me know what you think--I expect to put more time into it.bug fixes
I fixed a stupid issue where search pages were refreshing on session load. I regret this slipped through (again!), so I made an explicit test to catch it in future. Sorry for the trouble!
I also fixed an issue with 'do work on shutdown' cancelling repository processing and a couple of other jobs as soon as they started. The 'fast shutdown' logic was working a little too well!
Images with an alpha channel are now tested on load--if the alpha channel is completely opaque, it is stripped. This saves a little memory and CPU, and it means they will be correctly detected as pixel duplicates of their non-alpha-channel'd equivalents.
3 / I:
0 / P:
I had a great week. The program can now track the last time you viewed a file.last viewed time
By default, file viewing statistics are on. You can control how they work and turn them off under options->file viewing statistics
and if you like clear all your records under the database
If you have them on, hydrus now records the last time you saw a file in the media viewer and preview viewer. A 'last viewed' here counts with the same rules as recording view time (typically you need to look at the file a few seconds), so if you just scroll through ten files in a second, those won't be recorded as views.
You can see the last viewed time in the thumbnail right-click menu with the other view time numbers. You can also search and sort it, like with import or modified time.
These three 'time' sorts are now grouped into a new 'time' entry in the sort menu, and for searching under a new 'system:time' system predicate.
Let me know how this works for you! The search and sort uses the media viewer number for now, but if you are a preview or combined person and this does not work for you, let me know how you would like your situation supported.other highlights
I added some new ways to import the downloader-pngs you can drop onto network->downloaders->import downloaders
and other places. The program is now ok with different versions of this data, so you can now drag and drop an image from your browser onto Lain and it should still work. You can also do 'copy image' and click the new paste button. Should just be a bit more convenient when you only need to use the file once. I hope to add more export methods too (e.g. export to clipboard bitmap, which you can then paste into a site/discord), so it should be possible soon for downloader makers to share their work direct to a person without either ever touching a file.
For advanced users: The colours of the scanbar I put below videos is now editable in QSS! This is a simple test for now, as this was more complicated than you'd think. You should now be able to set the base, border, and nub colour. Check out the example in default_hydrus.qss and give it a go in your own custom QSS files. I expect to move all the other hardcoded colours except for probably tag namespaces and maybe some custom thumbnail borders to QSS in future.
0 / I:
0 / P:
Parser Sharing Thread
Can't see one here, maybe I'm blind, I'll start.
3 / I:
0 / P:
I had a great week. The big rewrites are finally doing something interesting, and I got some other stuff done as well.cleverer search
First of all, I finished my file domain rewrites. File search, tag search, and all the UI that displays it now works on n file domains at once rather than just one. Also, any of those n can be the deleted files of a service now. You can't make new local file services yet, obviously, but all search is now ready for it.
Since adding deleted files search was easy, I am adding it this week. Users in advanced mode will now see a list of deleted file domains on the tag autocomplete file domain button menu. While it is rare you ever want to do this, it was never actually possible to search these domains completely before, and definitely not quickly. Give it a go and let me know if you run into any trouble!
The next step here will be to write a new widget, probably some sort of checkbox list, that lets you select multiples of this new list. Then in future, if you had, say, a 'sfw' file service and a 'nsfw' service, you would be able to seach either or both at once easily. Then, I'll have a handful more things to do: an expansion to file import options to determine where imports are going, cleverer trash that supports n locations, and migration tools so you can move/copy between services, and then I _think_ I will be able to just let you add new local file services in manage services
I spent a little time with weird files this week. I added support for audio-only mkvs/webms and improved the test that checks whether a video with an audio track actually just has a silent one. I know I get annoyed when a video seems to have audio but actually doesn't, so this week will queue up all your videos for a recheck and hopefully fix a bunch. I also fixed the colours of some weird LAB TIFF files. If you have some jank test images in other unusual colourspaces, please send them in, it was fun figuring this one out.
The program should also shut down a little quicker now!
8 / I:
0 / P:
I had a good first week back. A mix of bug fixes and little improvements.couple of highlights, otherwise all misc this week
Thanks for the quick issue reports over the holiday. The scanbar had an annoying thing where the new single pixel border was making it awkward to drag when in borderless fullscreen--you'd move your mouse to the bottom of the screen, but then you'd click the border and not the scanbar, and the whole video would move--this I have now fixed. Also an issue with new imports' 'pixel duplicates' data not being saved correctly--I have fixed the problem and scheduled all affected files to regen their pixel duplicate data.
I also put some time into multiple local file services. It was more background work, this time to do with system predicate fetching, so there isn't much neat stuff to show off unless you are an advanced user. Almost all the background work is done now, though, so I hope to start spinning up more complex search UI and actually adding new file services pretty soon. One neat thing, now that I have new file filtering tools, I was able to expand 'system:file service' to search for deleted and petitioned files.full list
- fixed an issue where the one pixel border on the new 'control bar' on the media viewer was annoyingly catching mouse events at the bottom of the screen when in borderless fullscreen mode (and hence dragging the video, not scanning video position). the animation scanbar now owns its own border and processes mouse events on it properly
- fixed a typo bug in the new pixel hash system that meant new imports were not being added to the system correctly. on update, all files affected will be fixed of bad data and scheduled for a pixel hash regen. sorry for the trouble, and thank you for the quick reports here
- added a 'fixed font size example' qss file to the install. I have passed this file to others as an example of a quick way to make the font (and essentially ui scale) larger. it has some help comments inside and is now always available. the default example font size is lmao
- fixed another type checking problem for (mostly Arch/AUR) PyQt5 users (issue #1033)
- wrote a new display mappings maintenance routine for the database menu that repopulates the display mappings cache for missing rows. this task will be radically faster than a full regen for some problems, but it only solves those problems
- on boot, the program now explicitly checks if any of the database files are set as read-only and if so will dump out with an appropriate error
- rewrote my various 'file size problem' exception hierarchy to clearly split 'the file import options disallow this big gif' vs 'this file is zero size/this file is malformed'. we've had several problems here recently, but file import options rule-breaking should set 'ignore' again, and import objects should be better about ignore vs fail state from now on
- added more error handling for broken image files. some will report cleaner errors, some will now import
- the new parsing system that discards source urls if they share a domain with a primary import url is now stricter--now discarding only if they share the same url class. the domain check was messing up saving post urls when they were parsed from an api url (issue #1036)
- the network engine no longer sends a Range header if it is expecting to pull html/json, just files. this fixes fetching pages from nijie.info (and several other server engines, it seems), which has some unusual access rules regarding Range and Accept-Encoding
2 / I:
0 / P:
I had a good week cleaning some things up for end of year.couple of scanbar things
I polished the new autohiding video scanbar. A bunch of the layout and coordinate detection code of the scanbar and the media canvas behind it is less jank. If you had some flicker or weird mouse-popup behaviour last week, I hope it is better now!
If you are a macOS user however, I held back one of the changes. There's a background-erasing hack I put in a couple years ago because without it macOS media viewers went to 100% CPU. I hope this is no longer true, but I'm not certain, so you still have the flag on by default. Please hit _help->debug->gui actions->macos anti-flicker test_ and then try browsing some images and video in a new viewer. Does your client lock up, or is it ok now? If everything is good, I'll remove the flag and you can get some nicer anti-flicker tech too.full list
- new scanbar cleanup:
- the media container's scanbar and volume control are now combined on the same widget, meaning they now show/hide in sync and faster. their layout calculation is also more sensible. the new controls bar also has a thin border to make it pop better against a background video
- improved the way some auto-hide anti-flicker tech on the scanbar now works. it all hides a frame faster sometimes
- figured out some new anti-flicker tech to reduce/eliminate a frame of stretch when flicking from a static image to an mpv video, particularly for the first or second time in a session
- fixed a bug where clicking the global mute/unmute on an mpv player meant that certain shortcut keys (usually those with arrow keys) would not work on that player again. (it was a focus issue on the button, which then captured some form navigation keys but they had nowhere to go)
- brushed up some mouse coordinate testing logic across the program. some linux clients had trouble with the new animation scanbar popping up over mpv, I think I improved it!
- fixed another type problem with newer python/PyQt5 on Arch, also in scanbar coordinate testing
- fixed some dodgy colours in the scanbar initialisation and volume control border
- macOS users: I undid a long-time paint hack on the media container and the static image canvas. Qt is responsible for clearing the background again, which allows me to remove some jank anti-flicker tech. HOWEVER, the original reason for this hack was because without it, old macOS went to 100% CPU whenever the media viewer was showing something. therefore, to be safe, this option is still on for macOS users for now. you'll get a little flicker when browsing. please try hitting _help->debug->gui actions->macOS anti-flicker test_ and do some mixed video/image browsing. does your whole damn client lock up?
0 / I:
0 / P:
Malformed Drive -- Help
I've got multiple drives I've tested Hydrus on, and can't figure out why I'm having issues. All my drives are healthy and run without problems. At first I thought Window's built in drive compression might be fucking with Hydrus, but I decompressed my whole SSD, and I'm still getting errors. Can someone suggest a possible reason for my issue? Will provide logs if needed, Thanks in advance.
4 / I:
0 / P:
I had a great week. The duplicate filter now supports filtering by pixel duplicates, and videos now fit better in the media viewer.better duplicate search
Now we have pixel duplicate data stored in the database, I can search it. The duplicate filter page now has a dropdown that lets you select 'must be pixel dupes', 'can be pixel dupes', and 'must not be pixel dupes'. It does exactly what you think, and with luck it should be pretty fast no matter what you select.
Note that you may not have all your old pixel dupe data calculated yet. I started it for everyone last week, but even if it is doing 10,000 files a day, if you have a big client it will take a little longer. It may also still be working on the ICC profile queue. Hit up database->file maintenance->manage scheduled jobs
to see how it is doing. You can rush it there if you like.
As I was working in this system, I discovered that at some point I had started recording at what search distance potential duplicates were matched. This is to say if you were set up to search at distance 4 and two files were matched as potential duplicates with a distance of 2, it would save that 2 into the database. I am not totally sure how retroactively accurate this data is, but I've added a control for it to the filtering panel too. You can now tell the filter to only present files that are 'exact match' at distance 0 and it should
work mostly ok.
Some pretty complicated database work went into this. The most complicated search joins seven different things together. I know it is fast on my test machine, but if you have a really large client with a lot of searched dupes, some search types may be unbearably slow. Please let me know how you get on, and I'll optimise what I can.scanbar
The audio/video scanbar in the media viewer is now embedded inside the media frame. It shows and autohides when your mouse moves closer and away. It now means you can now go borderless fullscreen on a 16:9 display and finally have a 16:9 video fit perfectly!
I have wanted to do this for a really long time, but some of the layout code here is really awkward, and getting widgets to pop on top of each other can be tricky. Thankfully it turns out Qt has a nice way to do it, so I've now hacked this together. There is still a small amount of jank--the scanbar and volume control are currently separate objects, so sometimes they'll show/hide in separate frames, and you might also see the scanbar nub pop in a frame late, but I can work on these issues in future. I will also add some options so you can change the size of the show/hide activation area around the scanbar. But for now, I am pretty happy with this.
If you are a keyboard user, please check out the new shortcut in the 'global' set that flips on/off a 'force the animation scanbar to show' mode. I don't really want to bring back the old always-on hanging-below scanbar, so I hope this will be a good enough substitute. But let me know, and if you really hate this new scanbar, we'll see what we can do.
4 / I:
1 / P:
I had a good week. I focused on background improvements to the database. If you have a big client, it will take several minutes to update the database. My 2.4 million file PTR-syncing IRL client took 8 minutes.misc
First off, some quick fixes: I fixed unnamespaced wildcard tag searches (e.g. 'sam*s'), which were recently not producing namespaced results. I also improved handling in the new ICC profile system when the ICC profile with an image being loaded was completely borked.
Also, it seems the login-required 'gelbooru favourites by user id' downloader recently(?) broke severely--as well as pulling favourite links, it was also parsing and visiting the 'remove from favourites' link and deleting them! I fixed the gelbooru gallery parser to never pull a delete link again, but if you used this parser to grab all your favourites, please check your favourites list, and if you can, dig that downloader back out of hydrus and do a mass 'open sources' on the file log or the thumbnails so you can re-favourite any files that were dropped. Thanks to the users who noticed what was going on here and figured out what needed to be fixed.
I added some new options. The 'default tag service in manage tags' choice is reset this week, and it now starts off working different: it now remembers the last used service the next time you open a tag service dialog. Let's see if this works out, but if you don't like it, you can go back to a fixed default under options->tags
. There's also a new checkbox in options->search
that lets you default new file search pages to 'searching paused' rather than 'searching immediately'.ipfs and deleted files searchThis is mostly database prep for future multiple local file service expansions.
If you have an IPFS service, you can now search it in a normal search page. Just switch 'my files' on the autocomplete dropdown to your IPFS service and you should be able to search it with tag counts and everything. IPFS works a little differently to a normal file service in hydrus, so this will need some more work to get those workflows integrated. Also, while an IPFS service in hydrus only knows about your pins atm, in future I would like hydrus to harvest more info from external sources so this search space could potentially populate with remote files that you could then command the client to download.
In a related but quieter move, I did the same thing here for a new 'deleted files' umbrella domain. It'll take a few minutes to calculate this search cache on update. This will be of use in the near future when I let advanced users start searching deleted files.
3 / I:
0 / P:
I had a good week. Some images get better colour, and the software now deletes files more neatly.ICC profilestl;dr: Some image colours are better now.
Some images come with an 'ICC profile'. A scan of one ICC-heavy client showed that 10% of all images had them. ICCs are basically colour correction data like 'actually, the blue should be this much more intensely blue than is stored' in a variety of sometimes complicated ways. Some nice cameras attach a fast ICC based on lighting conditions or known defects in the camera, and some image editing software does similar. ICCs are also used for broader and more complicated colour conversion tasks like in the migration from sRGB to HDR formats if you have a new tv/monitor.
Hydrus now has basic ICC tech. It recognises when an image has an ICC and will apply it, converting the rendered colour to 'normal'. This happens right on image load, so it should apply in every media viewer and all new thumbnails. Should work for the server, too. With luck, this will be one of those things where you don't really notice any difference, but you should just have some nicer pictures, mostly photos, in your client. Let me know if you have any files that you know are still not rendering right!
This work took a long time to get done, but a user helped me out with some great examples and more background info on how to handle it all properly. The next step will be to add more indication of when an image has an ICC and add the ability to flip it on and off. Another topic is 'display ICCs', where some artists and other enthusiasts will have an ICC calibrated for their particular monitor or for different display conditions (e.g. 'how would this look like under this lighting?' or '...from this printer?'). Since we have the tech now, I think I should be able to support this sort of ICC conversion too.
3 / I:
0 / P:
I had a great week. I was able to catch up on the work I wanted and do some neat stuff besides.import presentation
I completely overhauled the importer 'presentation' options, which you can see under all 'file import options'. These have always been a bit advanced and confusing, so I rewrote it to specifically say a more human-friendly summary like 'show new files' and added text and tooltips to better explain what was going on. It also works a bit faster and cleaner.
You can also now say 'show inbox files', regardless of 'successful' or 'already in db', which I think may be neat as the default for a watcher page that you process and revisit several times--every time you were to re-highlight a watcher, it would filter out anything you previously archived.
And I updated the right-click menu on downloader and watcher pages' lists, where it said "show importers' new files", to use the new options. It now tucks it into a submenu, adds the 'inbox only' option, and if only one importer is selected it says what the default actually is and removes any duplicate action.other highlights
The client now recognises ogg files with video as ogv! Previously, they were always considered audio, but now it will make thumbnails and parse resolution and duration and let you zoom properly with the video player and so on. All existing ogg files will be scheduled for a rescan when you update, so your ogvs should just pop into place.
I think I fixed the deviant art tag search downloader. They changed their tag search routine recently, so I had to change some things behind the scenes. Everyone will get it on update, and the existing 'deviant art tag search' downloader should just work again.
The manage ratings dialog now has copy/paste buttons for quick copying of multiple ratings across sets of files. This is a prototype, so give it a go and let me know what you think. It copies 'null' ratings at the moment, but maybe we'll want a second option too that only copies set ratings.
2 / I:
0 / P:
I had a good week with a variety of fixes and optimisations. New users also get a couple of user-friendly services to start with.highlights
A recent update to the mpv seekbar accidentally caused bad errors during audio playback. I have now fixed it. I regret the regression, and I have added tests to make sure this does not happen again.
All 'file log' import queues now search and count themselves faster. Any heavy import session, like a very large downloader or many medium size downloaders working at once, should use less CPU just to do normal work.
I sped up some file searches that were running super slow. The general problem file search was typically a mix of 'has multiple search predicates' and 'includes a short unnamespaced tag or namespace:anything'. This combination was taking two minutes or more rather than a handful of seconds due to an overly complicated database query that wasn't running well. Should be fixed now, but let me know if you have any more of this sort of trouble!new user services
To help new users, new clients now start with two extra services:
First is a local tag service called 'downloader tags', which the initial default tag import options is now set to get tags for. I hope this will reduce some of the confusion around default tag import options, merged downloader/personal tags in 'my tags', and having to retroactively fill in missed tags for files downloaded early on.
Second is a like/dislike rating service called 'favourites'. Many users are unaware of the existence of ratings services at all, so now everyone will start with a simple clickable star. When you update, if you currently have no rating services, you will be asked if you want to get this new default. You don't have to, but if you haven't tried ratings out yet, give it a go!full list
- fixed a recent serious regression that could cause a crash when playing audio in mpv (issue #1007)
- the main importer file log now does 'get next/all/count imports with status y' calls significantly faster, particularly on very large lists. these calls happen all the time for different status text changes and general 'which import to try next?' work. all busy downloader situations should see CPU gains to regular and background work
- fixed a problem where importing with the min/max file resolution options set would give a typo error when the size was violated rather than 'ignored'
- I think I have fixed an issue with subscriptions not wanting to run a query if by random accident that query has an invalid URL selected as the query's 'example url' for various pre-work login and bandwidth tests
- hydrus can now capture duration/fps of videos that specify two very close fps, e.g. 60 and 59.99. previously, these would default to the 24 fallback and could cause some weirdness in mpv
- replaced the default pixiv artist page api parser with one that fetches the newer url format, matching the tag search. existing users will see no automatic change but will receive the new parser, so if you are a big pixiv user, you might like to switch 'pixiv artist gallery page api' to the 'new urls' parser variant under _network->downloader components->manage url class links_. note that if you do this, your pixiv artist subscriptions will do a mini-resync (which involves some wasted time/bandwidth) as their urls change!
3 / I:
0 / P:
I had a great week doing a lot of background cleanup work, and I also fixed some things and improved some quality of life.highlights
I brushed up last week's .clip file metadata parsing. Some clip files were getting too-small resolutions and thumbnails due to a 'canvas unit' issue, basically some clips store resolution in mm or inches etc... rather than pixels. Thanks to a user who also provided some great examples, we managed to pin down a solution to get actual pixel resolution, and I also figured out duration and number of frames for animated clips. All clips will reparse again on update and generate nicer size thumbs if they need it.
Also, if you ever got a webm that seemed normal except it somehow had a crazy 7 hour duration, I think I figured out the problem here too. All superlong-but-actually-small files will be reparsed on update and should get a more accurate duration.
When you right-click on tags in the taglist, the 'search' submenu now provides actions to require/exclude 'namespace:anything' if your selection all shares the same namespace.
Also, as a little test, I am trying out a new quiet shortcut on the file selection taglist. If you now hold down control when double-clicking or hitting enter on tag(s) here, they now go up to the search negated. Double-click = 'add tag to search', ctrl+double-click = 'add -tag to search'. Of course ctrl+click is awkward since it can cause a deselection, so I think this could do with some more work, and eventual integration into the shortcuts system so you can change it as you like, but let me know what you think. A user recommended adding the +/- buttons as you see on some boorus, so this is my interim step towards that.
When you paste query texts into the edit subscription dialog, if any of the texts you paste are already in the subscription and currently DEAD
, the dialog now revives those subs (basically the same as 'check now'). I have personally wanted this for ages and kept forgetting to add it. The sub will just do another quick check on what you paste, just in case that query started getting things again recently, rather than swallowing your input as forevermore DEAD.
I moved some autocomplete options from 'gui pages' options panel to 'search'. If you are a Linux user who has trouble with the floating autocomplete dropdown window, check it out and see if 'embedding' the dropdown works better for you.
2 / I:
0 / P:
I had an ok but light week of various work.misc
When I improved the 'apng' file parsing the other week, there was a mistake in how I hooked it up and not all the file metadata was being propagated to the database. A couple of users got some weird '1 frame' apngs as a result. I have fixed this problem, and improved the parsing to deal with more sorts of apngs, and scheduled a complete rescan of all your apngs to get some nicer 'number of frames' counts.
In a related thing, a user sent in a way to get some cool metadata from 'clip' files. These now get resolution and a rough thumbnail. I have scheduled all your clip files, if you have any, to update too.
The Client API now supports a better way to refer to tag services when editing tags. If you are a Client API developer, please check out the updated help--you can now user service keys instead of service names. I am in no rush to delete the 'names' parameters, but I expect I will eventually. 'names' was always an early hack that has flaws as an identifier, so when it is convenient, please try to migrate to /get_services and service keys.
A new help file in install_dir/db directory, 'help my mpv crashes with WASAPI or ASIO audio', now has a neat couple lines to add to your mpv.conf if you have this problem. Many thanks to the user who figured this out!full list
- client api:
- /add_tags/add_tags now supports 'service_keys_to_tags' and 'service_keys_to_actions_to_tags'.
- /add_urls/add_url now supports 'service_keys_to_additional_tags'
- /get_files/file_metadata now returns with duplicates of the tag structures using 'service_keys_to_statuses_to_(display_)tags'
- added unit tests for the above
- updated the client api help for the above
- I recommend you move from 'service_names_to...' to 'service_keys_to...' when convenient. 'names' was an ugly old hack, and while I am not in a rush to delete it from the client api, I think I will eventually
- client api version is now 21
- apngs and clips:
- fixed a problem where the new apng metadata parsing was not completely hooked up, so num_frames was not being parsed correct for the final metadata row even when 'apng' filetype was, leading to some odd '1 frame apng' situations
- apng parsing now recognises more kinds of apngs--if one has an unusual scaling chunk in the header, this is recognised and the correct animation chunk searched for
- clip files now get resolution on import and a thumbnail! big thanks to the user who dug up how to extract this--it actually comes from a sqlite file embedded in the clip! (issue #996)
- on update, all apngs and clips will be scheduled for metadata rescan, and all clips will be scheduled for thumbnail generation
3 / I:
0 / P:
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v459.2/Hydrus.Network.459.2.-.Linux.-.Executable.tar.gzHey, the release I put out earlier had a problem! The links above point to the fixed version. If you got the old one within an hour of me putting this post up, please download and install/extract again and you'll be able to boot. Sorry for the trouble!
I had a great week mostly clearing out bugs.highlights
I gave the new duplicate filter zoom locking calculation another pass. Thank you to the users who submitted examples of my last attempt not working right. It is now more careful and targeted towards fixing the 'a watermark banner spilled over my monitor edge' problem, and it deals with different combinations of portrait and landscape images and viewers better. Let me know if you still have any borked zooms!
The client's downloader system can now deal with some 'partial/interrupted download' situations and automatically resume from where it left off. If you ever saw a network job widget error out with 'was expecting 21.3MB but got 458KB', this will try to fix that!
The program should
be better about saving the work of subscriptions that are running when you exit the program. The way their jobs ran was a little hacky, previously, so I moved them to the main worker pool everything else runs on and brushed up their shutdown signals. Let me know if you encounter any situations of them not shutting down nice and fast with client shutdown.
To help with debugging and other screenshotted reports, the UI now puts the program version number in the title of windows and dialogs. Also, if you previously set a custom 'main gui title' under options->gui
, I have reset this this week and renamed it to 'application display name'--it now shows on all windows.
Big Sur macoS clients have had a problem for a while where menus spawned from dialogs can't be clicked. It seems to be a weird Qt bug. Until there is a better fix, I hacked together a new window to display the menu items instead. If you are a Big Sur user, let me know how it works for you. You can turn it off if you need to under options->gui
I wrote a new help document, 'help my media files are broke.txt', in the install_dir/db directory. This collects the different repair routines I have developed with users who have run into trouble with broken drives or incomplete backup restores. If you are missing many files--or you know you have many surplus files--and were not sure how to fix the situation, please check it out!
I fixed the new 'advanced file deletion' checkboxes under options->files and trash
! Sorry, they weren't hooked up right in 458!
4 / I:
0 / P:
I had a good week. I fixed some more bugs and improved some quality of life.highlights
I fixed an html parsing problem that started in the past few weeks in the release build. I understand it broke Newgrounds searching, but it should
be fixed now, so if this affected you please give it another go.
You can now set the client to always boot with network traffic paused under network->pause
. Useful if you have an unusual network/vpn situation that is often not connected!
I have improved the duplicate filter zoom lock calculation. Previously, it forced the width of the two images to be the same--now it chooses height if it thinks that fits things better. This should stop 'B' images with a watermark banner on the bottom accidentally spilling that watermark difference below the screen.advanced user highlights
I improved some repository update error handling and auto-healing. If you have had an issue processing the PTR related to 'this file was the wrong type' that did not seem to fix by itself, please unpause processing and let it try one more time.
If you use the advanced file deletion dialog, you can now choose to remember the last used reason and
action under options->files and trash
. I also fixed a bug where some number of files were forgetting their given deletion reason once they were permanently deleted from the trash. I am sorry for the trouble with this bug, and I will make sure to add unit tests so it cannot happen again.
File import objects now track 'primary' URLs (that directly lead to a file download) separately from parsed 'source' URLs. They are now split under the new 'file log' URLs menu, which also now shows the referral URL. Furthermore, file import options
now lets you choose whether primary URLs are associated with imports (which you might want to turn off if you are doing some clever one-time proxy download etc...).
3 / I:
0 / P:
I had a good week. Some UI jank should be much better!highlights
I reworked how the menubar menus (file, network, pending etc...) update this week. They update labels and swap items in and out more efficiently. Most importantly, the 'pending' menu should
now update with no flicker at all, so if you have a bunch of downloaders/subscriptions running and continually pending new tags, you should be able to commit and upload with no trouble! Let me know how it works for you.
I also removed some ancient and now obsolete safety checks that stop the client from updating certain UI while the program is minimised. With luck, when you restore the client--and more significantly, when you restore it from minimised to system tray--there should be less judder, and hopefully no freezing, since the client no longer has to catch up with hundreds or thousands of updates. If you minimise to system tray and have had hangs, let me know if the situation is better now. Also, if you suddenly get popup message sizing problems or similar UI jank after minimising or putting the client on a virtual desktop or similar, please let me know.
The client now uses much less CPU to parse and import png files!
I updated the mid-tier file searching help here: https://hydrusnetwork.github.io/hydrus/help/getting_started_more_files.html
2 / I:
0 / P:
I had an ok week. With luck, the client should have less UI lag, and I also fixed a bunch of stuff and improved some quality of life.basic highlights
I removed some very hacky memory management code this week. It was eating far more CPU than it was worth, particularly for large clients. If you have a very heavy client, particularly if it has lots of heavy subscriptions, please let me know if you A) have fewer UI lockups, and B) see any crazy memory spikes while running subs. As bad as the old routine was, it was aggressive and effective at what it did, so I may have to revisit this.
All multi-column lists across the program now catch enter/return keystrokes and trigger an 'activate' call, as if you had double-clicked. Should be easy to navigate and highlight a downloader page list just with the keyboard now.advanced highlights
The 'file log' window now lets you see and copy an import item's hashes, tags, and urls from its right-click menu. I hope this will help debug some weirder downloader problems and generally inform on how the downloaders work.
The Edit URL Class dialog has had a layout makeover. Also, URL Classes now support matching and normalising 'single value' parameters (this is where you have a token/keyword parameter rather than the traditional key=value pair).
The hydrus server now remembers custom update and anonymisation periods! Previously, it was resetting to defaults on a restart! Thank you for the reports here--I apologise for the inconvenience and delay.
I added an 'mpv report mode' to the debug menu. If mpv loads for you but you have silent audio or similar (and perhaps some crashes, but we'll see if this catches useful info in time), this'll dump a huge amount of mpv debug information to the log.
5 / I:
1 / P:
I had an ok couple of weeks. Some quality of life is improved, the UI should save sessions a bit quicker, and the database repairs itself more of itself.
Your client may warn you about a missing index on first boot. If it does, that's great--some new repair code is working!session save
Some users with large sessions (typically around 2-3+ million 'weight') are experiencing severe UI lockups and general lag. This week I deal with part of that problem by greatly reducing the amount of CPU needed to save most sessions. Now pages know if they have been changed since being loaded, and if they have no changes, they can skip a bunch of session save work.
There's nothing you have to do here, but if you have been experiencing 12+ second freezes every few minutes, please let me know how you get on now. I can't promise things will be perfect, but if you have any large and inactive downloader pages, you should notice things running a bit smoother (exiting the program too, should be faster).
There is still a problem of lag and lockups when there are 200+ downloaders in the session. This is a different and unfortunately more severe problem that I have relieved in recent weeks but needs some other large work to completely fix. Best solution for now is to clear out finished downloaders if you can.misc
Many of the simple system predicates, like height and filesize, now support the '≠' (not equal) operator! This is still in the early stages, but I'd like to add some options of the thumbnail menu so you can say 'find files that (do not) have this file's precise resolution/duration/size' etc...
You can now copy thread subjects to clipboard in the watcher downloader page list!
The advanced file deletion dialog now remembers the last reason you chose!
The network job widget (which shows download progress, you see it on any downloader) now shows the current URL it is working on under its cog menu. You can click it to copy to clipboard. May help to debug some odd network errors etc...
The way data sizes over 1KB are rendered has been overhauled. It should now generally
be to three significant figures, so 3.11MB instead of 3.1MB, 12.3GB instead of 12GB. See if you like the change, and if not, try tweaking it to 2sf under the new EXPERIMENTAL option in options->gui
3 / I:
0 / P:
I had an ok week mostly working on making images look better.warped tiles
tl;dr: Images look better now.
I put a lot of time into the tiled renderer this week, mostly fixing 'stretch-warped' tiles. These are most obvious in the duplicate viewer when you are flipping between two very similar images that have slightly different resolution--you would see one part of one of the images stretched one pixel horizontally or vertically compared to the other.
This turned out to be complicated to fix, and moreso for unusual zooms (like a 124.65% style canvas fit zoom), but I think I have fixed almost all warping, and tile join artifacts, for all zooms below about ~6000%, with the exception of some bottommost and rightmost tiles.
I also added an option to options->speed and memory
to govern how large the tiles should be. This is 768-pixel squares by default, but if you still have trouble with warping in the duplicate viewer or other high zoom situations, please try boosting it a bit. Larger numbers will be less memory and CPU efficient, so I don't recommend 4096 to start off, but let's see what we find in real situations.
I like the tiled renderer, so I don't want to abandon or completely replace it. I thought seriously of ditching it this week, when I could not figure out a solution to warping, but I slept on it and figured it out. If I can keep patching remaining problems, I'll be happy, but I do have a backup plan as well. Let me know how you get on!the rest
I think I fixed the issue where, after some serious network errors, the downloaders could get in a 'traffic jam' and all stay on 'pending', with nothing going to 'working'.
Finishing a big archive/delete filter should result in less UI lag. If you are a user who hits F5 right after committing archive/delete, let me know how this works for you!
The green/blue/red texts in the duplicate filter's right-hand hover window are now moved to the style system, so QSS stylesheet files (as under options->style
) can finally change their colour. I set up some basic dark/light blues for all the default QSS files, so it should be a bit easier to read the blue text in dark mode styles now. These were the last hardcoded text colours in the program, so we are now one step closer to moving everything to QSS.
3 / I:
0 / P:
I had a great week. I ended up mostly fixing bugs. If you have a large client, the update may take a few minutes this week.bug fixes
I discovered a semi-important tag processing bug last week--for some users, files that were imported before
they added the PTR were not getting all the correct tag info and would not appear in some tag searches. I fixed the underlying file-tracking bug and have added a retroactive 'gap-filler' to this week's update. If you used the client for a long time but added/reset the PTR recently, you'll have more gaps so your update will take longer.
Another persistent issue has been the 'tiling artifacts' with my new tile-based image renderer, where at certain zooms an image would have one or more horizontal or vertical blurred/janked line of pixels. You'd notice it on cleaner vector images where a smooth curved black line would suddenly jag at one zoom. I had trouble reproducing this at first, but some users got it quite often, and with their help I was able to figure it out. I have rewritten the part of my renderer that was failing, and I think
I now have the artifacts completely fixed for all 'normal' zooms. Let me know how you get on!misc
Network job widgets have a little user-friendly update this week. I brushed up some of the status texts, made the text lay out better, and when the widget counts down waiting for something (like free bandwidth), if it doesn't get it because another downloader did, it says so before resetting the countdown clock.
A helpful user wrote a new darkmode style called OledBlack. Check it out under options->style
I added OR predicates to Client API file search!
If you are an advanced user, you might like to read this idea I had for setting up multi-tag processing workflows: https://hydrusnetwork.github.io/hydrus/help/advanced_parents.html#parent_favourites
I suddenly thought of it a couple weeks ago and it worked so well for me that I decided to write it up. It basically involves using local parent tags to group PTR tags into a pseudo-OR search that you can easily edit.
2 / I:
0 / P:
I had an ok week. I ended up doing a lot of boring behind the scenes prep, but there is also some nice new quality of life. If you sync with the PTR, update will take a few minutes this week.all misc this week
After a very long delay, I finally have shortcuts for seeking video back/forwards in the media viewer. New users will get ctrl+left/right to seek back 2.5 seconds or forwards 5 seconds. Existing users need to add their own--please check it under the 'media viewers - all' shortcut set. You can set whatever seek distance you like, and even set multiple with different distance jumps. Sorry for how long it took to get this in--I had to update my shortcut system first!
I did some behind the scenes tag work this week. The database can now handle larger 'tag as number' searches. Update will take a few minutes to load these bigger numbers into the fast search cache.
If you have a subscription that is not meant to completely sync (e.g. it pulls a sample from a gallery with unusual sort order, or you have the 'normal file limit' intentionally small), there is a new checkbox that suppresses the 'hey, there was a gap, click here to fill it in' popup windows.full list
- my 'simple' shortcut commands can now store additional variables. to start things off, I have finally added 'seek video' shortcut commands that have back/forwards and second+millisecond values. these should work on the native video viewer and mpv, audio and video. existing users will have to add their own (do it to the 'media viewers - all' set), but new users will get ctrl+left/right for 2.5s back and 5s forwards as the new defaults. let me know if you have any trouble!
- the maximum number tracked by 'tag as number' system predicate is expanded from -99999999->99999999 to -2^63 -> 2^63 - 1. tag caches will be regenerated on update to store these, it will take a few minutes. the input ui for the system predicate is temporarily limited to -/+2^31, but I'll expand it
- subscriptions now have a checkbox for 'do not worry about subscription gaps'. if you have a subscription that gets files randomly, or gets an intentionally small sample, this will disable the 'hey, there was a gap, click here to fill it in' popup messages
- you can now set negative values for the duplicate score weights in options->duplicates
- also added a weight for the 'nice ratio' duplicate comparison
3 / I:
0 / P:
I had a great week cleaning code and fixing bugs. If you have a big database, it will take a minute to update this week.all misc this week, most bug fixes
I fixed a critical bug in tag siblings. It was causing some tag siblings to be forgotten, particularly on local tag services, and particularly when a tag had a sibling removed and added (e.g. 'delete A->B, then add A->C') in the same transaction. I will keep working here and will trigger a new sibling reprocess in the future just as 450 did so we can fix more PTR issues.
The new content-based processing tracking had a couple more issues. Some Linux users, in particular, it just broke for, due to a SQLite version issue. I have fixed that, and I have fixed some issues it caused for IPFS. There are new unit tests to make sure this won't happen again.
I fixed an issue with sessions recently not saving thumbnail order correctly!
I fixed issues with some of the new Client API file search parameters not working right!
Big video files should import a bit faster, and will show more status updates as they do their work.
We have had more anti-virus problems in recent weeks. We'd hoped building on github would eliminate them, but it hasn't completely. A user helped me trace one issue to the Windows installer. We have 'corrected' it (believe it or not, it was removing the 'open help' checkbox on the final page of the install wizard, and yes the reasons for why this was causing problems are ridiculous), so with luck there will be fewer problems. Thank you for the reports, and let me know if you have more trouble!full list
- stupid anti-virus thing:
- we have had several more anti-virus false positives just recently. we discovered that at least one testbed used by these companies was testing the 'open html help' checkbox in the installer, which then launched Edge on the testbed, and then launched the Windows Update process for Edge and Skype, which was somehow interacting with UAC and thus considered suspicious activity owned by the hydrus installer process, lmao. thereafter, it seems the installer exe's DNS requests were somehow being cross-connected with the client.exe scan as that was identified as connected with the installer. taking that checkbox out as a test produced a much cleaner scan. there is a limit to how much of this nonsense I will accomodate, but this week we are trying a release without that 'open help' link in the installer, let's see how it goes
- semi-related, I brushed up the install path message in the installer and clarified help->help will open the help in the first-start welcome popup message
69 / I:
13 / P:
Please feel free to submit any bug reports here.
3 / I:
0 / P:
🎉🎉 MERRY 450! 🎉🎉
I had an ok week. Last week's experimental release went well, so I have polished that, and I otherwise caught up with a variety of small work.
If you sync with the PTR, update will take a couple of minutes.all misc this week
So, the update storage change went ok! There were a couple of little sync bugs to clear up, but overall it works--tag repositories now track their processing progress by mappings, siblings and parents separately. You don't have to do anything, and this doesn't matter much for day to day work, but it allows for individualised reset and reprocessing. All 448 users will have their siblings and parents reset and reprocessed, which will take a couple of minutes to do on update, and about fifteen minutes on your next processing job to fill back in, and which should eliminate some bad siblings and parents due to years-old processing bugs that long term users have been dealing with (leaving only current bugs, which I am also working on). The reset will not delete any pending siblings or parents you have, so no worries if you have a bunch waiting to be uploaded.
Advanced Users: The PTR sibling and parent reset will however remove any siblings and parents you uploaded that were then denied by jannies (which your client would have added to itself anyway). Everyone is reset to a 'clean' sync with this change, so if you know you have a ton of surplus denied siblings you rely on, perhaps from years ago that we agreed I would deny on the PTR to help you hack in an overwrite in the old system, you might like to hold off updating and first figure out a PTR sibling/parent backup to a local service using tag migration.
I fixed some things with Mr Bones. His numbers are accurate to your 'my files' again, and he now talks about your total deleted files and also your earliest file import time. I divided the ugly growing stack of numbers into tabs, which I am sort of happy with, sort of not. People like to take screens of Mr Bones, but they have different preferences on what to show, so I may replace this with expand/collapse buttons or similar, so you can show everything if you want.
If you use the export files window to get a lot of files out, it now makes a progress popup. You can close the window while it is exporting and still see and cancel the job.
The Client API file search now supports file and tag domain selection (like the 'my files' and 'all known tags' buttons on a normal search page), and also file sort for searches. I know the Client API guys have been waiting on this, so with luck we should see some neat new search options in the Client API programs in the near future.
6 / I:
0 / P:
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v448/Hydrus.Network.448.-.Linux.-.Executable.tar.gzHey, I am sorry, Endchan has been down or not allowed login/posting the past couple weeks when I needed to get release out, so there was a gap.
I had an ok couple of weeks. I was pretty ill in the middle, but I got some good work done overall. .wav files are now supported, PSD files get thumbnails, vacuum returns, and the Client API allows much cleverer search.client api
I have added some features to the Client API. It was more complicated than I expected, so I couldn't get everything I wanted done, but I think this is a decent step forward.
First off, the main 'search for files' routine now supports many system predicates. This is thanks to a user who wrote a great system predicate text parser a long time ago. I regret I am only catching up with his work now, since it works great. I expect to roll it into normal autocomplete typing as well--letting you type 'system:width<500' and actually getting the full predicate object in the results list to select.
If you are working with the Client API, please check out the extended help here:https://hydrusnetwork.github.io/hydrus/help/client_api.html#get_files_search_files
There's a giant list of the current supported inputs. You'll just be submitting system predicates as text, and it handles the rest. Please note that this is a complicated system, and while I have plenty of unit tests and so on, if you discover predicates that should parse but are giving errors or any other jank behaviour, please let me know!
Next step here is to add file sort and file/tag domain.
Next there's a routine that lets you add files to arbitrary pages, just like a thumbnail drag and drop:https://hydrusnetwork.github.io/hydrus/help/client_api.html#manage_pages_add_files
This is limited to currently open pages for now, but I will add a command to create an empty file page so you can implement an external file importer page.misc
.wav files are now supported! They should work fine in mpv as well.
Simple PSD files now get thumbnails! It turns out FFMPEG can figure this out as long as the PSD isn't too complicated, so I've done it like for .swf files--if it works, the PSD gets a nice thumbnail, and if it doesn't it gets the file default icon stretched to the correct ratio. When you update, all existing PSDs will be queued for a thumbnail regen, so they should sort themselves out in the background.
Thanks to profiles users sent in, I optimised some database code. Repository processing and large file deletes should be a little
faster. I had a good look at some slow session save profiles--having hundreds of thousands of URLs in downloader pages currently eats a ton of CPU during session autosave--but the solution will require two rounds of significant work.
Database vacuum returns as a manual job. I disabled this a month or so ago as it was always a rude sledgehammer that never actually achieved all that much. Now there is some UI under database->db maintenance->review vacuum data
that shows each database file separately with their current free space (i.e. what a vacuum will recover), whether it looks like you have enough space to vacuum, an estimate of vacuum time, and then the option to vacuum on a per file basis. If you recently deleted the PTR, please check it out, as you may be able to recover a whole ton of disk space.
I fixed Mr Bones! I knew I'd typo somewhere with the file service rewrite two weeks ago, and he got it. I hadn't realised how popular he was, so I've added him to my weekly test suite--it shouldn't happen again.
7 / I:
0 / P:
I had an ok week. The client does not have a huge changelog this week, but the server has a neat privacy improvement.
The PTR is going to be doing a heap of maintenance this week. It will be 'busy' a lot, approximately one hour of busy and then three hours free. Please bear with it, and if you have a million mappings to upload, I recommend you just give it a break and come back later. I am not totally sure how long it will take. Best case it is a day, worst case it might take four or five.null accounttl;dr: The hydrus server is now even more anon. You don't have to do anything.
As the PTR has moved to multiple accounts, we've had several good discussions about privacy. Separate accounts, despite being anon, could potentially leave a fingerprint of preferences on the server. If the server were to fall into bad hands many years from now, someone could mine those records--maybe mixed with one time you casually said 'yeah, I added that sibling'--and perhaps derive something from it.
There is no technical need to remember which account uploaded what long term, so now all hydrus servers completely anonymise their content after a certain duration, default 90 days. A new non-useable 'null' account takes possession of files, tag mappings, siblings, or parents after the delay, letting the original uploaders be forgotten. Janitors will still have time to work on recent account-based problems, but the historical record works just like the old shared public account: all merged together.
I have updated the privacy document in the help to talk a little about it. As long as you do not tag your own face in pictures or something, I think we are pretty great now, especially if you use a VPN.https://hydrusnetwork.github.io/hydrus/help/privacy.html#account_history
It will take some time to anonymise the PTR or any other big server, as it has to go through its whole historical record to catch up. Please bear with it.
3 / I:
0 / P:
I had a great week mostly working on optimisations and cleanup. A big busy client running a lot of importers should
be a little snappier today.optimisations
Several users have had bad UI hangs recently, sometimes for several seconds. It is correlated with running many downloaders at once, so with their help I gathered some profiles of what was going on and trimmed and rearranged some of the ways downloaders and file imports work this week. There is now less stress on the database when lots of things are going on at once, and all the code here is a little more sensible for future improvements. I do not think I have fixed the hangs, but they may be less bad overall, or the hang may have been pushed to a specific trigger like file loads or similar.
So there is still more to do. The main problem I believe is that I designed the latest version of the downloader engine before we even had multiple downloaders per page. An assumed max of about twenty download queues is baked into the system, whereas many users may have a hundred or more sitting around, sometimes finished/paused, but in the current system each still taking up a little overhead CPU on certain update calls. A complete overhaul of this system is long overdue but will be a large job, so I'm going to focus on chipping away at the worst offenders in the meantime.
As a result, I have improved some of the profiling code. The 'callto' profile mode now records the UI-side of background jobs (when they publish their results, usually), and the 'review threads' debug dialog now shows detailed information on the newer job scheduler system, which I believe is being overwhelmed by micro downloader jobs in heavy clients. I hope these will help as I continue working with the users who have had trouble, so please let me know how you get on this week and we'll give it another round.the rest
I fixed some crazy add/delete logic bugs in the filename tagging dialog and its 'tags just for selected files' list. Tag removes will stick better and work more precisely on the current selection.
If you upload tags to the PTR and notice some lag after it finishes, this should
be fixed now. A safety routine that verifies everything is uploaded and counted correct was not working efficiently.
I fixed viewing extremely small images (like 1x1) in the media viewer. The new tiled renderer had a problem with zooms greater than 76800%, ha ha ha.
A bunch of sites with weird encodings (mostly old or japanese) should now work in the downloader system.
Added a link, https://github.com/GoAwayNow/Iwara-Hydrus,
to Iwara-Hydrus, a userscript to simplify sending Iwara videos to Hydrus Network, to the Client API help.
If you are a Windows user, you should be able to run the client if it is installed on a network location again. This broke around v439, when we moved to the new github build. It was a build issue with some new libraries.
4 / I:
0 / P:
https://youtube.com/watch?v=0OmOQs0ziSU [Embed]This whole vid is great. Watching a ton of this stuff was a decent whack of my vacation.windows
I had an ok week getting back to speed. There's some new help regarding the new PTR accounts and I fixed some bugs.accounts and help
If you use the PTR, you may have seen a note appear on the public account that it is now 'read only' and can now only download updates. If you would like to upload tags, siblings, or parents, please generate your own account either by clicking the popup button you get when trying to upload, or just hit up the PTR on services->manage services
and click the easy button there. This pretty much completes the long project me and the guys running the PTR have been working on: moving all uploading users to their own still-anon-but-seperate accounts so the janitors can work through groups of petitions and undo future mistakes without it all being merged together in one account. I have given both of these pages a complete pass to represent the changes and updated screenshots etc... to be more modern:https://hydrusnetwork.github.io/hydrus/help/access_keys.htmlhttps://hydrusnetwork.github.io/hydrus/help/privacy.html
The privacy page now compiles the various conversations I have had with users over time and most recently about this account change, and my current understanding of exactly how private a hydrus repo is (really good), and what you can do to make it almost perfect, even against esoteric problems or an untrustworthy administrator (basically using a VPN is always a good idea when you are doing anything fun online, and don't tag your private photos on the PTR with your real name lol). I have tried to be comprehensive and even tried to imagine some advanced future problems, so it is a long read, but I'll try to keep updating all my thoughts to there so it can be a good reference point in future. If you are interested in this stuff, please read it and let me know what I have missed or if you have any other ideas!
Also, if you are a Linux user and get OOM (out of memory) crashes, please check the new thorough user-written guide linked in this new 'running' section here:https://hydrusnetwork.github.io/hydrus/help/getting_started_installing.html#runninghttps://hydrusnetwork.github.io/hydrus/help/Fixing_Hydrus_Random_Crashes_Under_Linux.mdall misc otherwise this week
I rewrote a heap of thread-UI object interactions that should improve stability on some Linux flavours and also get rid of the 'QtDeadWindow' errors some users have seen (usually in manage tags).
The new popup message button that fills in subscription gaps would sometimes not assign the correct downloader (usually when the button was opening a new page). This should be fixed!
A variety of error reporting is more reliable and presents better text.
An odd error when loading the downloader .png files is fixed, so if you were unable to import new downloaders in the past few weeks, please try again and let me know how you get on!
3 / I:
0 / P:
I had a great week doing nice cleanup and quality of life work.Hey, we had a problem getting the macOS release to build this week. The macOS link above goes to a build using a simpler and faster method. It should work fine, but please let me know if you have any trouble. As always, back up before you update!highlights
Popup messages can now launch complex jobs from a button. The first I've added is when a subscription hits its 'periodic' file limit. The situation itself is now better explained, and a button on the popup will create a new downloader page with the specific query set up with an appropriate file limit to fill in the gap. The second is if you try to upload some content to a repository that your account does not have permission for (this is affecting sibling- and parent-uploading PTR users as the shared public account is changing), the popup message that talks about the issue now has a button that takes you straight to the manage services
panel for the service and starts automatic account creation.
Subs should now be more careful about determining when they have 'caught up' to a previous sync. Small initial file limits are respected more, and the 'caught up' check is now more precise with sites that can give multiple files per URL or very large gallery pages.
I gave options->speed and memory
a full pass. The layout is less crushed and has more explanation, the options all apply without needing a client restart, and the new, previously hardcoded cache/prefetch thresholds are now exposed and explained. There's a neat thing that gives an example resolution of what will be cached or prefetched, like 'about a 7,245x4,075 image', that changes as you fiddle with the controls.
The client has recently had worse UI lag. After working with some users, the biggest problems seemed to come in a session with lots of downloaders. I traced the cause of the lag and believe I have eliminated it. If you have had lag recently, a second or two every now and then, please let me know how things are now.
If you use the Client API a lot while the client is minimised, you can now have it explicitly prohibit 'idle mode' while it is working under options->maintenance and processing
5 / I:
0 / P:
I had a great week. An important part of GUI Sessions is overhauled, which should save a lot of hard drive time for larger clients.gui sessionsI always encourage a backup before you update, but this week it matters more than normal. If you have a client with large sessions with many important things set up, make sure you have a backup done before you update! I feel good about the code, and I try to save data on various failures, but if your situation gives errors for an unforeseen reason, having the backup ready reduces headaches all around!
Like the subscriptions and network objects breakups I've done in the past year, I 'broke up' the monolithic GUI Session object this week. Now, when your session has changes, only those pages that have changed will be saved, saving a ton of CPU and HDD write I/O. Furthermore, sessions that share duplicate pages (this happens all the time with session backups), can now share that stored page, saving a bunch of hard drive space too. Like with subscriptions, some users are pushing multiple gigabytes of session storage total, so there is a good amount of work to save here.
You don't have to do anything here. Everything works the same on the front end, and all your existing sessions will be converted on update. Your client should be a little less laggy at times, and client shutdown should be a bit faster.
If any of your old sessions fail to load or convert, a backup will be made so we can check it out later. Let me know if you have any trouble!Advanced stuff:
Another benefit is the old limit of 'sessions fail to save at about 500k session weight' now applies to pages individually. Please don't immediately try to nuke your sessions with five million new things, but if you do end up with a big session, let me know how other performance works out for you. Now this bottleneck is gone, we'll start hitting new ones. I believe the next biggest vulnerability is thread starvation with many simultaneous downloaders, so again please don't paste-spam a hundred now queries (for now).
If you have been tracking session weight (under the pages menu), I am rebalancing the weights. Before, the weight was file = 1, URL = 1, but after all our research into this, I am setting it to file = 1, URL = 20. In general, I think a page will fail to save at the new weight of about 10 million. If you are in advanced mode, you can now see each page's weight on page tab right-clicks. Let's get a new feeling for IRL distribution here, and we can aim for the next optimisation (I suspect it'll eventually be a downloader-page breakup, storing every query or watcher as a separate object). Since URLs seem to be the real killer, too, see if you can spread bigger downloads across multiple download pages and try to clear out larger completed queries when you can.
2 / I:
0 / P:
I had an ok week. Not as much as I wanted, but there are some nice Client API improvements.all misc this week
The test builds from last week seem to work ok, so they are now master. The built clients now use Python 3.8, and the security libraries (like OpenSSL) are all much newer--and will reliably stay up to date in future--so a whole bunch of things across the client should have slightly better performance. There are no special install instructions, they seem to work on an existing install just as normal. Let me know if you do run into any problems!
I fixed some more bad tiles calculations for the new tiled image renderer. Some files that seem to have little black lines on an edge at some zooms, or previews that just turn up black, should be fixed! Error reporting is also nicer.
The Client API can now do a couple more things. Particularly, it can now set your client's global User-Agent, which should
help fix some difficult CDN and login problems in future. Please watch this space.
For advanced users, if you have help->advanced mode
on, then setting a namespace file sort now allows you to choose which 'tag context' the sort works on. If you hide certain tags in single or multiple media view (as set in tags->manage tag display and search
), then those hidden tags will not count for the sort. This is obviously advanced, so if you hadn't thought of it, you can just set 'display tags' to keep 'normal' behaviour.full list
- after successful testing, all the master builds are now made on github rather than my home dev situation. the clients now work off python 3.8, and several security libraries (e.g. OpenSSL) are now always going to be latest, so there should be several quiet performance and reliability improvements across the program. there are no special install instructions--normal update seems to go fine--but let me know if you do have any trouble. big thanks to the user who did the leg work on developing the workflow build scripts here
- if you are in advanced mode, namespace file sorting now allows you to set the 'tag display context' on which it will sort. this appears as a new menu button or a button list selection dialog wherever you edit namespace file sorts. if you are not in advanced mode, the default is the 'display tags' I switched to last week (i.e. before any tags are hidden by your tag display options)
- namespace sort has some related code cleanup. the 'defaults' object is updated and moved to the newer options object
- the new tiled renderer now checks for rounding errors in zoom calc, which in some cases was giving a single extra (non-existing) native pixel row or column on rightmost or bottommost tile samples
- the new tiled renderer now double-checks clip regions for validity before attempting to crop
- improved the reported error information when a tile fails to render
- when pasting an uneven number of tags into manage siblings/parents, the error is now a nicer popup dialog. I'm pursuing a related error here--if you get this a bunch, please let me know what more info you discover
- when repositories fail to fetch the update hashes to process, they now force a metadata resync. any processing error should force a metadata resync now
- added a default url class for the new pixiv _artist_ page format
- fixed a recent typo bug with ipfs pinning
1 / I:
0 / P:
unable to keep my tags visible and overall finicky
I've been fiddling with hydrus for a bit but cannot figure out how to keep my tags visible and usable at all times. I have two tags so far, but expect to use dozens, if not hundreds in total in the future.
The whole thing isn't easy for me introductory tutorial that I've gone over so far has basically been "select image, press f3, do ur tag stuff".
Can I see all my own created tags without selecting an image and pressing f3 and be able to edit/delete them? Also, how do I keep tags in the "selection tags" section at all times?
Thanks in advance.
3 / I:
0 / P:
I had an unfortunately short week, but I did some good work. The tiled renderer has nice fixes.tiled renderer
I regret the tiled renderer, while good most of the time, had crashes when it did go wrong. To stop this with any new errors that might pop up, the whole block now has an additional layer of error catching around it. If a tile fails to render for any reason, you now get a black square, and if some coordinate space cannot be calculated or a portion of the image is truncated, the system will now print some errors but otherwise ignore it.
A particular problem several users encountered was legacy images that have EXIF rotation metadata but were imported years ago when the client did not understand this. Therefore, hydrus thought some old image was (600x900) when it then loaded (900x600). In the old system, you would have had a weird stretch, maybe a borked rotation, but in my new tiled system it would try to draw tiles that didn't exist, causing our errors-then-crashes. The client now recognises this situation, gives you a popup, and automatically schedules metadata regeneration maintenance for the file.some misc
You can now set a custom 'namespace' file sort (the 'series-creator-volume-chapter-page' stuff) right on a page. Just click the new 'custom' menu entry and you can type whatever you like. It should save through your session and be easy to edit again. This is prep for some better edit UI here and increased sort/collect control, so if you do a lot of namespace sorting, let me know how you get on!
I prototyped a new 'delete lock' mode, which prohibits deletion of files if they match a criteria. I am starting with if a file is archived. You can turn this mode on under options->files and trash
. It mostly just ignores deletes at the moment, but in future I will improve feedback, and maybe have a padlock icon or something, and eventually attach my planned 'metadata conditional' object to it so you'll be able to delete-lock all pngs, or all files with more than four tags, or whatever you can think of.
3 / I:
0 / P:
I had an ok week. The new tiled renderer is improved.tiled renderer
The new image drawing system generally worked well! There were a couple of bugs, and it still has some limitations, but in general it really improved zoom and precache performance.
For the bugs, first of all, there was a rare crash, I think triggered by loading a very unlucky coincidence of tile and image size. Then clipboard bitmap copy threw an error, tiny images could not deal with extremely small zoom, and clients under heavy load could sometimes have trouble initialising the viewer. I have fixed them all, but let me know if you have any more trouble!
There was also a problem with PyQt5, an alternative version of the Qt UI library that some 'running from source' users use. It was an object handling difference between PyQt5 and PySide2 that broke the tile caching system. I think I have fixed it, so if you are running PyQt5, please give this version a go.
Beyond bugs, there were tiling artifacts visible at higher zooms. Essentially, where the tiles lined up, there were small disagreements in resize math, resulting in little lines of mismatching colour gradients along tile borders. I worked on the tiling algorithm and have significantly mitigated the problem--I mostly only see artifacts at extreme zooms now, about 2000%.
Since people are suddenly zooming more, users who have mouse-centered zooming were having more images accidentally flying off screen too. I've hacked in off-screen rescue after a zoom, sliding it back to the nearest border, so the image should always stay in view. If people like it, I may patch this in for all media for dragging events too. There's not much need for non-visible media, and when it does happen it can sometimes be a pain dragging around to find where it went.
I hope this basically makes the tile render a complete '1.0' now. In the future, I would like to rejigger some of the virtual geometry, since at the moment a limit in Qt means I cannot zoom higher than a 'virtual' 32,768x32,768 canvas (e.g. 4k at about 800% zoom). I'll also replicate the tiling for my native Animation widget, which displays gifs and video when mpv is not available.
4 / I:
0 / P:
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v438/Hydrus.Network.438.-.Linux.-.Executable.tar.gzHey, this causes errors if you are running from source and using PyQt5 (PySide2 is fine)! All the releases above are PySide2, so they are ok! I will fix this for next week, so if you are source+QtPy5, please hold off for now.
I had a great week overhauling the media viewer's image rendering. Zooming and navigation should be a lot smoother now!image tilestl;dr: the media viewer now zooms and navigates with less lag and flicker
Zooming in a long way, particularly for large images, has been pretty hellish for as long as the program has existed. Historically, the client drew the whole image in memory at the zoom you desired so it could quickly show you the bit it needed on screen. Of course this meant zooming in to 400% on anything above 4k was suddenly taking a very long time to set up and eating a lot of memory to do it. As images have naturally grown over time, the problem has occurred more often and was starting to affect other systems.
My plan to fix this has been to break the image into tiles that then render on demand. The parts of the image off-screen are never drawn, saving CPU and memory and allowing arbitrary zoom. This is a significantly more complicated idea, and rewriting the whole rendering pipeline was always expected to be a multi-week 'big job'. I originally planned to just optimise and tweak the secondary systems and add in some sanity brakes this week, but I ran a couple of small tiling tests and realised if I went bonkers it would be possible to hack in a prototype. So I did!
In the media viewer, images now draw in tiles. It works a little like a browseable satellite map, where when you zoom in and pan about you see squares of data fading in (except in hydrus they appear instantly). You should now be able to zoom in as far as you like on an image pretty quick and you won't have any sudden memory needs.
Furthermore, I have written a cache for these image tiles. This saves CPU when revisiting different images or zooms, so when you flick back and forth between two normal things, it should
now be instant! It still takes 20-200ms to view or zoom most images the first time, but going back to that view or zoom within a minute or so should be really smooth. The cache starts at a healthy 256MB this week. I think that will cover most users very well (in screen real estate, it works out to about 35 x 1080p worth of tiles), but you can alter it under the settings at options->speed and memory
And I did some misc work improving the rendering pre-fetch logic when you browse in the media viewer. Huge files won't stomp all over the image renderer cache any more, which should make browsing through a series of giant images far less jank. If you are feeling advanced, you can now edit the prefetch timing and distance settings too, also under options->speed and memory
3 / I:
0 / P:
I had a good week mostly fixing bugs and optimising. It will take a couple of seconds to update this week.all misc this week
I reduced a heap of UI lag on clients that have pages with a lot of collected (like 'collect by creator') media. If you often have five or ten thousand files collected and you noticed your client was getting choppy just when running some downloaders, I hope this improves things. Let me know how you get on!
I started real work on multiple local file services. Most of it is boring behind the scenes stuff, but as part of it, I overhauled the trash system this week. A heap of logic is improved, it is ready for more than one 'my files' service, and now hydrus remembers when files are deleted. Delete timestamps have never been recorded clientside, and unfortunately we cannot recover old information retroactively, but it is stored for all deletes from now on. Whenever the client wants to say 'this file was deleted', it should now have 'at an unknown time' or a nicer '3 days ago' suffix.
A second neat thing with the improved deleted files storage is I hope in the nearish future to let you search
deleted files. This is a rare, clever query, like 'all known files', but there will be some kind of button you can press to flip your 'my files' or 'all local files' search to go through what has ever been removed from them. 'system:file service' will get similar improvements.
To reduce confusion, I renamed some hydrus network concepts across the program. The 'access key' (secret password for account), 'account key' (account identifier), and 'registration key' (one time token to create an account) are now known as 'access key' (i.e. no change), 'account id', and 'registration token'. There is more work to do here, particularly improving server setup and account management workflows to suit the user (rather than my technical ease), so I will keep at it.
In a related topic, the PTR is updating its accounts. The public account is moving more towards a 'read-only' account, and accounts that can upload siblings and parents (and perhaps tag petitions, eventually) will be individual to you and freely auto-creatable in manage services
. This is mostly an attempt to make janitorial decisions easier and more accurate, since at the moment everything on that side is merged due to the shared account. Permissions have not been used much in hydrus network yet, and the workflows and user notifications here are bad and buggy. Please bear with me as I iron out the problems and make it all nicer to use.
3 / I:
0 / P:
I had a great few days mostly cleaning and fixing things. If you sync with the PTR, update will take a minute this week.macos release polish
I cleaned up the new macOS release. It seems to have launched and otherwise generally worked last week, but there was a bug in finding the specific database location macOS users are used to. Without the '--d' launch parameter, it was creating an empty new db inside the app, in the 'db' dir hydrus would normally use (and the really
old App used to use, if you remember that), and hence would say 'hey, this looks like the first time you are running the program...' on boot. I have fixed the 'I am running in an app' detection and the ~/Library/Hydrus database path calculation routine, so everything should
be back to normal.
It also has the old readme and Applications shortcut in the dmg, and the filename should be fixed too. I expect this to be the only macOS release I put out from now on. Let me know if you have any more trouble!miscount fix
Last week, I made the number in the 'pending (1,234)' menu title add up in a more efficient way. Rather than counting raw mapping rows every time, it uses a table of pre-computed numbers, the same used for autocomplete results. It turns out there were some legacy (from a long time ago) miscount bugs in there for some users. This resulted in a 'sticky' number that would not go away even after committing. A maintenance routine exists to fix this, but it is a sledgehammer when we need a scalpel.
So, I have written a maintenance routine to regen this pending data efficiently and correct these old bugs. It is basically the same as I did a few months ago with the 'display' caches during the siblings and parents work, but for a deeper level of tags. It will be run on update, along with a new thing that forces the menu's count to regen, both of which can now be accessed from database->regenerate
menu in case we need them again in future. If you sync with the PTR, it may take a minute or so to finish.
I hope this will fix the issue completely, but if you still have a bad count, or if your count drifts off zero again over time, please let me know!underscores
After discussion with some users, I have added an experimental setting to options->tag presentation
that replaces all underscore characters in tags with space characters, as long as you are in 'front-facing' UI like regular search pages or the media viewer. It works on the same system as the 'hide namespace' option--and siblings--in that you still see the raw truth in manage tags
and other edit locations.
This setting is experimental since it will add a bit of CPU lag to tag presentation and may result in some seemingly duplicate rows. I have long planned to fix the underscore issue with a really nice system, but I was convinced that adding a hacky system in the meantime would be a good thing to play with. If you care about this issue, give it a go and let me know if you run into any problems.
3 / I:
0 / P:
NEW app: https://github.com/hydrusnetwork/hydrus/releases/download/v435/HydrusNetwork-v435.dmg
OLD app: https://github.com/hydrusnetwork/hydrus/releases/download/v435/Hydrus.Network.435.-.macOS.old.-.App.dmglinux
I had a good week. I continued last week's server work and have a macOS Big Sur build available for testing.new macOS build
This was delayed far longer than I wanted, but I think I have it figured out now. Thanks to work from Suika and ReAnzu on github figuring out the script, the macOS build is now put together on github itself, on Catalina. This new build thus works on Big Sur, and should generally have less compatibility jank on any newer macOS machine. I have included my old 10.12-built release as well today, just in case I messed something up, but if you are a macOS person, please give the new one a try and let me know how you get on.
Assuming this works out ok for the majority of people, I'll be moving to it as the official macOS build. Anyone who is still on an old macOS will have to run from source to stay updated with hydrus. The new release is just out of testing, so I'll clean up the filename and add the readme.rtf and so on so it looks prettier. I will also explore reducing the size (300MB vs 137MB wew), but I am told this may be unavoidable due to some new macOS thing that bundles both old and new versions of libraries in the App.editable repository update periodsJust in case, it is probably a good idea to back up your server before you try this!
Hydrus repositories now have an editable update period! This was an oft-requested feature for users running a server on a LAN or between just a few friends, so if this is you, please give it a go. There is a new 'edit options' permission for accounts, which any account with 'manage account types' (admins basically) will inherit on update. If you have this, the 'admin services' menu will have a new 'edit update period' entry, which launches a simple time delta widget where you can change it.
Changes occur immediately. The server will create any updates that are now due, and your client will do a resync to keep up to date with the new numbers, so if you have 'review services' open when you make the change, you should see everything update in realtime. Other clients that sync with the server will catch up with the new check times when they next do their normal sync, unless they too force a resync with the new 'review services' button I added last week.
You can change the update several times, up or down, and it shouldn't be a big deal--the client no longer cares if one set of updates were short or long--so please feel free to play around with this a bit, just don't go crazy with super short or long times. The minimum is 10 minutes, but I recommend you only go down to about 60 minutes if you really want to be that fast, at least as we experiment with this. The system was originally built with 100,000 seconds in mind, so I think super short periods will add overhead bloat over time (not to mention, you only see content after processing, which atm only happens in idle time anyway).
2 / I:
0 / P:
I had a great three weeks working on long overdue network updates. Most of the changes this week restore old account management UI for server admins, but there are a couple of fixes and improvements for regular users as well.
The network version goes up to 20 today. Clients and servers can only talk to each other if they are on the same version, so if you want to talk to the PTR, you will want to update at some point. No rush, but if you try to talk to the server on an older client, you will get a polite error message.regular user stuff
On review services
, you can now pause repository account sync, update downloading, and update processing separately. If you don't want to process right now but still want the client to be grabbing update files for later, or you want to pause all service network traffice for a while but still process your backlog, it should now be easy to set up.
I fixed an annoying bug where going from 'random' file sort to one with asc/desc sort order by loading a favourite search would always default the asc/desc to the top value. Guess what hydev's default sort is.
Server administrators can now set messages to accounts. If you get a message, your client should get the update within a day or two, at which point it will appear in a popup box and be viewable on review services
Server administrators can now set some account types to be user-creatable--no need to mess around with registration keys or access keys yourself, the client can fetch them for you from the server. A new button on the manage services
edit panel checks this and can do it for you. I understand the PTR is likely to move to this in future for accounts with sibling and parent petition permissions, since grouping these by account will be useful for janny petition workflow. Having separate accounts for these complicated uploads will also allow better account management (including the above rudimentary messaging) for users who make common errors with sibling or parent submissions.
I believe I fixed a bug in repository processing that caused some newer users to get stuck with siblings and other data that was later deleted. This should not happen again, and I am now planning ways to efficiently fix the problem retroactively. (If you were hit by this and want the non-efficient fix, hit 'reset processing->reprocess content' for the repo under review services
Some users recently got some scary errors about an invalid repository update file. This was the result of the client becoming confused about which update files it should have for a repository. I am still investigating why this happened, but to fix the immediate issue I have added a maintenance routine (under the new review services->reset downloading
) that triggers a full resync of update metadata and delete surplus records. This function will fire automatically in future if the service encounters borked update files of any kind. If you got these errors, please try firing it and let me know how you get on.
5 / I:
0 / P:
I had a good week doing simple cleanup and fixes. This is intended to be a 'clean' release. There will be no release for about three weeks.all misc this week
Thanks to the work of a user (who was also very patient as I slowly added the features he needed), we now have a new native twitter downloader! It gets video now! (Including gif, which on twitter is just a small mp4!) It is called 'twitter syndication' and uses a new access method. Its main limitation seems to be that its search can only reach about 500 tweets into the past. Please check it out, and if it works well for you, please move your subscriptions over from nitter to save them some bandwidth. This is the first version, so it may well have future updates.
I have written a new 'scrollable' menu button, which works like the file sort selector--instead of a dropdown, you get a menu, and you can scroll through it in either direction with your mouse. I have converted several old small choice dropdowns to this new button: file sort ascending/descending, collect-by 'collect/leave unmatched', and all the new tag sort dropdowns I added last week. Try putting your mouse over and scrolling them! This is a little thing, but I hope it smooths out the workflow here. I'll likely use it on more things.
Some more mouse changes: Middle-clicking on the collect-by dropdown now clears it. Middle-clicking or right-clicking the 'page tab bar' to open the 'new page' chooser or the page right-click menu should now only work on the tab bar's empty greyspace (previously, this was confusingly happening on a click on greyspace in any child panel, like some random downloader page panel).
Namespace file sort (the 'series-creator-title...' stuff) now support a-z/z-a ordering. Files that have no match are still counted as 'less than a' in a-z terms, but I can change this stuff easier these days. I don't use this myself much, so if you do, let me know what you'd like in future.
Network job controls (the little panel that shows your downloads) now let you edit their active job's bandwidth rules from the cog menu! It will show all the network contexts and which are currently waiting on bandwidth. If the job is using the 'default' rules, it will say so and allow you to set specific ones just for that domain.
3 / I:
0 / P:
^ After years of seeing all sorts of smaller edits using this footage, I finally discovered this source vid this week. If you like fine art, please watch it in full, it is eight minutes of unbroken kino. It gets a bit heavy at the end, not appropriate for kids. If you don't like old art, no worries, I was just chuffed this week to have finally found this thing.windows
I had a great week. There are a bunch of different little fixes and improvements this week, and the tag sort dropdown is now easier to use.tag sort
Rather than a big list, the tag sort dropdown is now a dynamic control that splits the different variables (sort type, asc/desc, group by) into different mini-dropdowns that show and hide when available. Confusing labels like 'lexicographic' and 'ignore namespace' are replaced with clean 'sort by tag/subtag/count'. I hope it is a bit easier to work with now. Give it a go and let me know what you think.
If you don't like your default tag sort, you can set it under options->tags
. Your existing default should
be converted to the new format on update.
I also added the option to sort by sibling text or the actual tag text when you have the manage tags dialog open.
There is more work to do. One thing I don't like is if, say, on the asc/desc dropdown, you have 'a-z' selected, you can hover your mouse over and scroll down to get 'z-a', but you can't scroll up to get it. I prefer to have scroll work like 'flip' when there are only two values, where either scroll direction will make a change, so I'll figure that out, and in the file sort/collect controls, so there is even less to think about when you edit. I'll also see about saving the tag sort in the session.
I also updated the back-end of tag sorting here, so it'll be easier to update in future. Some users have asked for custom namespace sorts (e.g. creator, character, series, then the rest (a-z), then unnamespaced), so I'd like to start thinking about that properly.misc
As some users have not realised they were getting a big session that was lagging them, the client now makes a once-per-boot gentle warning popup if your session exceeds 500k weight. If you cannot conveniently reduce your session size, you can turn this warning off under options->gui pages
Subscriptions now work more efficiently with a very small 'first run' file limit and a larger 'normal' file limit. Previously, the second run was often charging right through the handful of files caught on first sync and getting more than it should, but now it should better recognise it has 'caught up' and stop at the right position.
3 / I:
0 / P:
I had a challenging week, but I got some decent fixes in.all misc this week
Pretty much everything was little fixes this week. Nothing hugely significant, but check the changelog if you have been waiting on something.
The e621 parser now pulls 'lore' tags, which seems to be some new 'canonical' tag. I have put them in 'lore' namespace, so you can exclude them easily if you want. One user suggested they are useful for certain blacklists.
For advanced users, I added a 'sorter' to string processing. It has a neat thing where you can sort by a regex'd substring of the main string. I was happy about this, but then I realised this morning that it is only half of the job I wanted to do. I will add a String Slicer next week so you can just sample the first item or the last three items or whatever you like, once you have your sort figured out.full list
- when parents are hidden in the edit/write taglists (e.g. in manage tags), there is now a '(n parents)' suffix
- thread watchers that are DEAD or 404 but still have files downloading now report 'working' status until that is done
- search terms with ';' like 'steins;gate' should now work in downloaders. sorry for the trouble!
- fixed an issue where un-ideal tags were sometimes becoming non-searchable when they were entirely replaced in manage tags with their 'ideal' siblings (i.e. their autocomplete count went to 0). this was due to overzealous deletion in the new tag definitions cache not filtering out sibling/parent chain members. a small routine will run on update to resynchronise affected tags
- fixed an issue when loading up files in the main 'import files' dialog where a critical error (as opposed to a nice 'couldn't figure it out, sorry') in mime detection would cause the whole job to hang
- that main 'import files' dialog now counts 'missing' files separately in the error count
- fixed tags not updating on the filename tagging dialog when double-clicking to remove from the simple taglists
- fixed the sort on the manage tags dialog's suggestion taglists--they now preserve their original sort, rather than alphabetising once sibling/parent data is populated
0 / I:
0 / P:
tried to login after changing password and I get this message about remember cookie missing in post login step
it says in logged in when i look at the manage logins
3 / I:
0 / P:
I had an ok week. I mostly fixed and improved things. If you sync with the PTR, it will take a minute or two to update.all misc this week
When I did the autocomplete lookup speedup recently, we unfortunately lost the ability to match unusual characters with short tag inputs. Entering 'a' would not return a match of '/a/', and in fact there was no way to look that tag up without some slow cleverness like wildcards. I have fixed this this week. If you sync with the PTR, it will take a minute or two to update while a small new cache is constructed.
You can now hide/show the 'hanging' parents in taglists (in manage tags dialogs etc...) from the list right-click menu. You can also set in options->tags whether they should start shown/hidden for normal taglists and autocomplete dropdown results.
Taglists that show '(displays as xxx)' sibling info now sort by that sibling for lexicographic purposes. I had thought to make this optional, but when I tried it out, it felt great to me, so I left it as default. Let me know what you think.
I fixed an issue that meant some texts across the program were staying blank for a few seconds when they should have been updating. This mostly affected the 'highlighted downloader' panel in gallery download pages. This was personally driving me nuts IRL.
I fixed the tumblr downloader. Sorry for the problems here, I misunderstood initial reports. This was due to a logical change in hydrus that just happened to hit tumblr's URL format. If you have tumblr subscriptions, please go into them and see if any recent URLs have the status 'Found 0 new URLs.'. Right click them and say to 'try again' and they should fix up.
6 / I:
1 / P:
I had a good simple week just working on smaller jobs and cleanup. If you have a large client, it may take a minute for you to update this week.all misc this week
The new taglists seem to have gone generally well. There was one display/update error that hit some users, which I have fixed, along with some little menu logic cleanup. I will work on more display and sort options for them in future.
Since I added 'pending' status for thread watchers last week, you may have seen some apparently stuck on that status. This was a harmless legacy issue, just old check jobs that were never closed out correctly, which I have fixed. I also cut out the 'checking in' prefix from the waiting-to-check status text, let's see if that fits better.
Dialogs with multi-column lists should be less of a pain this week. I think I fixed the issue where the last column could sometimes shrink to a few pixels on the next dialog launch, and I think I also fixed the issue where columns would sometimes get a few pixels thinner every launch. Just set them once more to the size you want, and they should
'.clip' files, those for Clip Studio Paint, are now recognised and importable. Too complicated for me to support thumbs, but they have their own icon. I cleaned up the behaviour of the increasingly tall filetype selection widget, so I am happy to start adding simple new filetypes for a bit. If you would like a new media filetype added to hydrus, please send an example of that file to me with a bit of info about it. Nothing too obscure, please, and I can't promise I can add everything quick, but many are actually fairly simple. I'd like to see about .sai and .wav, and maybe start on an xml scanner so I can at recognise things like .svg.
I took a bit of time to check out animated webp this week. Unfortunately, gathering duration and frame info about webps seems to be much more difficult than actually making
webps, so I am afraid we will have to wait for an update to FFMPEG or one of the other libraries we rely on to get animated webps going.
A new cache speeds up a bunch of file database lookups this week. If you have a large client, it may take a minute to update as this is populated.
11 / I:
0 / P:
I had a good couple weeks working on the taglist code and some other jobs.
If you are on Windows and use the 'extract' release, you may want to do a 'clean' install this week. Extra notes below.taglists
So, I took some time to make taglists work a lot cleaner behind the scenes and support more types of data. A heap of code is cleaner, and various small logical problems related to menus are fixed. The tag right-click menu is also more compact, quicker to see and do what you want.
The main benefits though are in the manage tags dialog. Now, the '(will display as xxx)' sibling suffix colours in the correct namespace for the sibling, and parents 'hang' underneath all tags in all the lists. It is now much easier to see why a parent or sibling is appearing for a file.
This is a first attempt. I really like how these basically work, but it can get a bit busy with many tags. With the cleaner code, it will be much easier to expand in future. I expect to add 'expand/collapse parents' settings and more sorts, and maybe shade parents a bit transparent, in the coming weeks. Please let me know how it works for you IRL and I'll keep working.the rest
The main nitter site seems to be overloaded. They have a bunch of mirrors listed here: https://github.com/zedeus/nitter/wiki/Instances
I picked two roughly at random and added new downloaders for them. If you have Nitter subs, please move their 'sources' over, and they should start working again (they might need to do a bit of 'resync' and will complain about file limits being hit since the URLs are different, but give them time). If you would rather use another mirror, feel free to duplicate your own downloaders as well. Thanks to a user who helped here with some fixed-up parsers.
I gave the recently borked grouped 'status' sort in thread watchers and downloader pages another go, and I improved the reporting there overall. The 'working' status shouldn't flicker on and off as much, there is a new 'pending' status for downloaders waiting for a work slot, and the 'file status' icon column in galleries now shows the 'stop' symbol when files are all done.
The menu entry to 'open similar-looking files' is now further up on thumbnails' 'open' submenus.
The duplicate filter has its navigation buttons on the right-hand hover window rearranged a bit. It is silly to have both 'previous' and 'next' when there are only two files, so I merged them. You can also set 'view next' as a separate shortcut for the duplicate filter, if you want to map 'flip file' to something else just for the filter.windows clean installIf you use the Windows installer, do not worry, these issues are fixed automatically for you from now on.
I updated to a new dev machine this week. Some libraries were updated, and there is now a dll conflict, where a dll from an older version is interfering with a new one. As it happens, the library that fails to load is one I made optional this week, so it doesn't seem
to actually stop you from booting the client, but it will stop you from running the Client API in https if you never did it before (the library does ssl certificate generation).
It is good to be clean, so if you extract the Windows release, you may want to follow this guide this week: https://hydrusnetwork.github.io/hydrus/help/getting_started_installing.html#clean_installs
Remember: Don't delete your db folder.
1 / I:
0 / P:
e621 downloader doesn't grab "lore" tags
they added a new "lore" tag section, for tags that are all on my blacklist. incest, tranny, """"""nonbinary"""""", etc.
since they're apparently separate from the other tags, my blacklist on hydrus isn't catching them.
this is a bug report, i guess. i can't use github bc they don't accept cock.li emails, can't use discord bc they require a phone number. hope you see this
4 / I:
0 / P:
I had an ok week. I mostly fixed and cleaned things.more ghosts
There was another 'ghost' pending tag issue. Sometimes, on committing pending tags with siblings up to the PTR, the pending record and its count was not clearing properly. I reproduced the problem and fixed it, and a new test ensures it will not happen again. Thank you to the reports about this, they helped me to figure it out.
Since we have had some trouble with pending tags in particular, I have written a new maintenance routine to recalculate pending tags and their counts efficiently without having to wait for siblings/parents re-sync. This will run on update, so with luck, your tags and autocompletes should be fixed up (until the next problem!).the rest
There is a new shortcut set under file->shortcuts
to handle tag autocomplete shortcuts. Any 'switch searching immediately' that was previously on 'main gui' will be migrated over to this on update, and I have also moved several previously hardcoded and undocumented shortcuts to the new shortcut system, so you can now edit them! The new actions (with their defaults) are: forcing tag autocomplete lookup now (ctrl+space), flipping an IME-friendly input mode on/off (insert), navigating autocomplete results tabs left/right when input is empty (left/right arrow), navigating the service pages in manage tags dialog left/right when input is empty (up/down arrow), and navigating a parent media viewer previous/next when input is empty (page up/down).
I fixed a bad auto-setup link in the gelbooru downloader last week that messed up some users. Also rolled in is a new URL definition that should fix 420chan thread watching for their newer url format.full list
- ghost pending tags:
- fixed another ghost pending tags bug. this may have been new or there since the display cache started, I am not sure, but it shouldn't happen again. it was occuring when a pending tag was being committed to 'current' and another tag in its sibling group already existed as a current tag for that file. the pending tag and its count would not clear for non-'all known files' domains, causing ghosts to appear in search pages but not typically manage tags. you may have noticed ghost tags hanging around after a pending commit--this was it. the true problem here was in the 'rescind pending' action that occurs just before an add/commit. a new unit test tests for this situation both for two non-ideal tags being pend-merged, and a non-ideal tag being pend-merged into the existing ideal
- wrote a routine to regenerate _pending_ tag storage and autocomplete counts from scratch for the combined and specific display tag caches. this job is special in that it regens tags instantly without having to reset sibling/parent sync. you can run the job from the database->regen menu. this is the start of 'fix just this tag' maintenance ability
- the pending regen routines will occur on update. it shouldn't take long at all, unless you have five million tags pending, where it could be a couple minutes
1 / I:
0 / P:
Is it me or is this UI text ant-sized? I've been using hydrus on ubuntu and got used to bigger windows but now im on w10 and this is some small print, so small I dont know how I first got used to it and almost think I've done something wrong. I'm just on 1920x1080. Can I do something about this?
4 / I:
0 / P:
I had a great week. I mostly fixed and cleaned things.8kun board deleted
The 8kun /hydrus/ board is now deleted. I was planning last week to move to Endchan as the new primary, but Codexx of 8chan.moe kindly offered to host me on 8chan.moe /t/, in a new Hydrus Network General. The thread is here https://8chan.moe/t/res/2219.html
. The old /hydrus/ board is also migrated to 8chan.moe /hydrus/, if you would like to search it or recover something important to you. All the links across the program and help are updated to reflect the changes. I appreciate the help from Codexx in getting this going. I have never been a comfortable or great BO, so this arrangement is a better fit for my situation for the primary place to talk Anon about Hydrus. Endchan will remain as a bunker, and I will keep making release posts here. Please feel free to use whichever you are most comfortable with.downloaders from the future
It seems gelbooru changed their markup recently, and our default downloader stopped pulling tags, at least for some. A user helpfully created an update--which is rolled into today's release if you missed it--but unfortunately users who were running older clients ran into an unusual versioning bug. The updated downloader would only work correctly on above v422 or so, leading older clients to get a variety of annoying errors about 'md5', 'hex' or 'additional_info' when either booting or trying to inspect the broken object.
I have fixed the problem here. Anyone who recently imported a 'downloader from the future' should have it fixed today. Furthermore, the various ways you import downloaders now try to check against versions your current client can support, and if they are too advanced, you now get a nice non-spammy popup message and the too-complex objects are discarded from the import. If the advanced object is buried deeper inside the downloader, it may not be able to discard neatly yet, and you will just get a 'soft' popup message about it, but I hope to tighten this up in future.
This version checking is also applied more softly across the whole program. It is less likely that one of your internal objects will be from the 'future', but the client will now make a popup if this is so. Please let me know if you get a whole bunch of these. With luck, this is ultimately a rare problem and I can toughen up this error and actually stop clients from running if their objects ever turn invalid.
I also fixed up error recovery in the downloader system UI and multi-column lists across the program. When some borked object enters them, they now try to show an appropriate 'cannot render' style text, rather than raising an error.
10 / I:
0 / P:
I had a good week. I optimised and fixed several core systems.faster
I messed up last week with one autocomplete query, and as a result, when searching the PTR in 'all known files', which typically happens in the 'manage tags' dialog, all queries had 2-6 seconds lag! I figured out what went wrong, and now autocomplete should be working fast everywhere. My test situation went from 2.5 seconds to 58ms! Sorry for the trouble here, this was driving me nuts as well.
I also worked on tag processing. Thank you to the users who have sent in profiles and other info since the display cache came in. A great deal of overhead and inefficient is reduced, so tag processing should be faster for almost all situations.
The 'system:number of tags' query now has much better cancelability. It still wasn't great last week, so I gave it another go. If you do a bare 'system:num tags > 4' or something and it is taking ages, stopping or changing the search should now just take a couple seconds. It also won't blat your memory as much, if you go really big.
And lastly, the 'session' and 'bandwidth' objects in the network engine, formerly monolithic and sometimes laggy objects, are now broken into smaller pieces. When you get new cookies or some bandwidth is used, only the small piece that is changed now needs to be synced to the database. This is basically the same as the subscription breakup last year, but behind the scenes. It reduces some db activity and UI lag on older and network-heavy clients.better
I have fixed more instances of 'ghost' tags, where committing certain pending tags, usually in combination with others that shared a sibling/parent implication, could still leave a 'pending' tag behind. This reasons behind it were quite complicated, but I managed to replicate the bug and fixed every instance I could find. Please let me know if you find any more instances of this behaviour.
While the display cache is working ok now, and with decent
speed, some larger and more active clients will still have some ghost tags and inaccurate autocomplete counts hanging around. You won't notice or care about a count of 1,234,567 vs 1,234,588, but in some cases these will be very annoying. The only simple fixes available at the moment are the nuclear 'regen' jobs under the 'database' menu, which isn't good enough. I have planned maintenance routines for regenerating just for particular files and tags, and I want these to be easy to fire off, just from right-click menu, so if you have something wrong staring at you on some favourite files or tags, please hang in there, fixes will come.
1 / I:
0 / P:
No discord hydrus shenanigans for now.
Locked out of discord because of their shitty mobile verification, so doing my part tagging paywalled furry art ¯\_(ツ)_/¯> https://youtube.com/watch?v=SMTz9nIUkGc [Embed]
18 / I:
1 / P:
I had a great week. The client can now save and load searches.favourite searches
Every tag autocomplete input text box that searches for files--the most obvious being the one on normal search pages--now has a star icon button beside it. Click this, and you get a menu to save your current search, manage your saved searches, or load up one that is saved!
Currently, the saved information is the list of search terms (the tags and system predicates), the current file and tag domains (e.g. my files/all known tags), whether the system is 'searching immediately' or waiting, and optionally the current sort and collect.
Each saved search has a name and optionally a folder, for easy grouping. I have started all clients with an example inbox processing search, just as an example.
If you, like me, keep ten or fifteen empty search pages open (for me it is mostly different creator+inbox searches), I hope this system lets you collapse it down, making for a lighter and simpler session. It should also help for more unusual workflows like duplicate filtering and even file maintenance jobs.
There may be a couple of bugs in the system, something like a collect-by not being saved or updated correctly. Let me know how you get on!
6 / I:
1 / P:
I had a good week. There are some quality of life improvements and faster tag search across the board.
The update will take some time this week to update a cache. If you do not sync to the PTR, it will be just a few seconds. If you sync to the PTR, expect about 5-15 minutes.faster tag search
In the second half of 2020, I tried several times to tune the database for different sorts of wildcard tag search, which is used in all autocomplete lookups and many file searches. I was sometimes able to get small clients always running well, or complicated large systems running, well, but I failed to get it good for all situations with code alone--the structure of the database tag lookup cache made the tuning difficult.
So, I have updated how that cache works. Rather than always searching one big master table, the client can now 'zoom' in on the appropriate search context based on the type of search page or manage tags dialog or whatever.
Pretty much anything related to autocomplete and tag-based file searches is faster. Most importantly, the worst-case time for these searches is greatly improved. Complicated searches, like a 'namespace:*anything*' file search, should no longer have sudden gigantic lag spikes. These searches may still take ten seconds or more when searching millions of tags and files, but they won't accidentally lag out for two minutes on some tiny 'my tags' search with only 60 results.
The only exception in my testing is 'number of tags' searches still have bad cancelability. It is better, but not great. I'll keep working here.
The cache replaces an existing one. It will take some time to build it on update. If you do not sync with the PTR, it should just be a few seconds. If you sync with the PTR on an SSD, it should be 5-15 minutes (on my heavy client with a nice SSD, it was 7 minutes). If you sync with the PTR on an HDD, it will take significantly longer, so please plan for it. If you sync with the PTR, you will see some numbers count up as it builds the different parts of the cache. There will be some deletion work to start, then counting up to perhaps a million, and then up to 16 million or so, at about 30,000 a second.
I have more plans here, and more work to do to optimise the tag display system, but I will let this new cache breathe for a bit before going back in here with a machete.
4 / I:
1 / P:
5 / I:
0 / P:
5 / I:
0 / P:
I had a good week. I fixed some important bugs, and cleaned up some core access and maintenance code. This week's changelog gets pretty technical, which you can safely ignore. Database works better™.highlights
I fixed a problem importing files with 'only add tags that already exist' checked in tag import options
. Sorry, this was a stupid typo. I added a unit test here to ensure it doesn't happen again.
When you search for potential duplicates from the duplicates page, this now happens in a non-interrupting popup in the bottom-right. You can keep searching and browsing while it works. All duplicate pages sync with each other better, too, and share more CPU work.
Thanks to a user's contribution and help, the 'discord drag and drop' BUGFIX mode, which is really a 'export files to a temp location before starting the drag and drop' mode, now supports the same 'filename pattern' you see in the export files dialog. Check it under options->gui
. This is experimental, but the first step to automatic export with nice filenames. If you care about export filenames, let me know how it works for you, and what sorts of filenames you want to make in future. I really want to update the old 'filename pattern' system to something with conditional 'if' statements.
If you are a new user and had trouble syncing to the PTR in the past week with an error about an invalid tag, you got caught by my new 'invalid tag catching' code being overzealous. This is now fixed, so if your PTR automatically paused due to the error, please hit services->review services->PTR
and then unpause it. When a repository gives an invalid tag (we think there are about 17 really broken tags in the PTR atm, from older days), the client now recovers.
4 / I:
0 / P:
I had a great week fixing a whole bunch of bugs.bugs
I fixed taglist drag-select, which was not moving the to-be selected indices down with the scroll. Sorry for the trouble here. You can now also ctrl+drag-select to deselect.
There was a bug in the new virtual siblings and parents lookup system that meant some grandparents and siblings were not appearing. For instance, for parents, 'samus aran' might have 'metroid', and 'metroid' would have 'nintendo', but 'samus aran' would not have 'nintendo'. Thanks to help from users, I was able to reproduce it and fix the problem. When you update, the client will spend a few seconds regenerating the lookups and finding the missing links. It will queue up a bit more work for the background display sync to do later on. In my test situation, the PTR went from 189,000 sync rows to 192,000.
Autocomplete results in manage tags
now list parents beneath every tag they apply to. Previously, parents could only exist in the list once, so they were accidentally de-duping and only ending up beneath the last applicable tag. Now you get plenty.
Also, these 'write' autocomplete results now show sibling and parent data for the tag that matches your input text even if that tag has no count. When that tag has sibling data, all the siblings are loaded as well. This sounds obscure, but you'll notice when you next start entering tags on a service with a lot of siblings. It should make it easier to quick-type complicated tags in manage tags
8 / I:
0 / P:
2 / I:
0 / P:
I was ill this week, so I didn't have as much work time as I wanted. I did get some basic system predicate editing done though, if you want to try that out!editing system predicatesThis works, but it is basic, not super polished.
If you shift+double-click on a system predicate that is in your active search (like 'system:filesize>200KB'), it should now launch a dialog to edit it in place. You can also edit several at once. This should make doing several 'time imported < 7 days ago' ... '6 days ago' ... '5 days ago' searches in a row to chase something down a bit easier!
If a predicate is too simple for a panel but has an 'invert' version (e.g. archive/inbox, has audio/no audio), then it flips to that invert on a shift+double-click. This also works for regular tags, flipping between 'tag' and '-tag'.
As I was short on time, I did not finish everything and could not add too many bells and whistles. The stacked edit panel can look a little ugly, there is no menu command to edit, selection disappears after an edit, I have no edit support for 'system:rating' or 'OR' predicates, and I was unable to reintroduce optional default values for system predicates. I'll go at this work next week.
In any case, give it a go and let me know what works well and what gives you trouble. I am not super enthused about shift+double-click as the edit shortcut, but it is a good compromise. I'll likely end up adding it to the shortcut system so you can edit it.
BTW: A couple of clever system pred options that were accidentally hidden are reintroduced to the new options->search
3 / I:
0 / P:
I had a great week working on a variety of small jobs.
The hydrus network version updates this week, so any clients that currently sync with the PTR will need to update to continue syncing. There is no rush for this.all misc this week
First of all, the hydrus network version increments from 18 to 19 this week. Clients and servers can only talk to each other if they are running on the same version, and the PTR will update today, so if you want to keep syncing with the PTR, you will have to update. This version update was to ensure that all clients syncing with tag repositories are now on the new virtual sibling and parents systems. There is no rush to get this done, and no penalty if you delay--you will just get a polite popup saying there is a version mismatch, and your PTR service will be paused.
If you get a popup about the network version mismatch in an older version, after you update to 417, go to services->review services
and make sure the PTR service panel is unpaused. You might also want to hit 'refresh account'.
I added some shortcut actions to the 'main window' shortcut set--one for 'run all export folders now', and several for navigating the current page selection. You can navigate left, right, and to the leftmost or rightmost tab of the current row. Unlike (shift+)ctrl+tab behaviour, the left/right actions jump through page of pages when you are at a boundary, letting you step through all of your tabs in a depth-first manner.
3 / I:
0 / P:
I had a great week catching up on a mix of smaller work.highlights
The manage tags
dialog is now more efficient at making changes for very large numbers of files. My test application of 6 tags to 10,000 files went from 52 seconds to 4.8! A variety of other tag presentation updates should also benefit from the optimisations here.
I reworked my hacky multi-list sizing code. The various instances of the final column sizing weird should be much better. The horizontal scrollbar is enabled again.
A new checkbox under options->files and trash
now lets you prefix any copied hash with the hash type, like "md5:2496dabcbd69e3c56a5d8caabb7acde5". If you often paste hashes into boorus or other search engines that take this format, this should make it a bit quicker!
I am rolling in default thread watchers for warosu, prolikewoah, and smuglo.li. Warosu seems to have some CloudFlare stuff going on, so your mileage may vary.full list
- the new siblings and parents taglist menus now copy just the actual tag when you click, excluding the 'ideal/child/parent:' prefixes
- added a checkbox to _options->files and trash_ that allows you to automatically prefix hashes copied to clipboard with their hash type in a booru-lookup friendly manner, such as "md5:2496dabcbd69e3c56a5d8caabb7acde5"
- the media viewer now remembers if it was previously maximised when you set it to un-fullscreen (before, it would always restore-window-ise)
- fixed the 'test address' button in _manage services_ for hydrus administration services
- improved the 'add upnp mapping' error handling to better catch 'already mapped' error, with separate errors for redundant, already-on-but-wrong-port, and already-on-another-computer
- improved error handling when saving objects to the database, particularly for encoding or giganto-size-session errors
- rewrote my tag sibling lookup unit tests to deal with more situations
- wrote similar fairly comprehensive tag parent lookup unit tests
- new downloaders:
- rolling in a user-created thread watcher for warosu. it may be CloudFlare hampered depending on your situation
- rolling in a prolikewoah thread watcher
- rolling in a smuglo.li thread watcher
2 / I:
0 / P:
I had a mixed week. I greatly reduced the lag that 414 could get after large sibling/parent changes, but unfortunately I could not get much else done.all misc this week
I had wanted to do a bunch of small work this week, but my schedule fell apart a bit, and then the 'one last thing' for the new parents cache, which was rejiggering the background maintenance code to do smaller jobs at once, ended up being very difficult. I worked hard on figuring out a method, and then when I was ready to move on, I realised there was a design flaw and had to re-do a large part of it! In any case, it now works much better--even if you have thousands of tags with one common parent, they can now calculate one at a time, nice and quietly in the background. Some million-strong siblings and parents might still be a bit too laggy for an HDD to calculate in normal time, but an SSD should be generally ok. If it still lags for you, you can make it only work in idle time under tags->siblings/parents sync
In that vein, the new tag display dialogs now have a nice green/red text label to describe the current maintenance options, so you better know when to expect any changes to be calculated.
I fixed and improved a couple small things, but other than that, the main highlight is options->gui pages
now lets you put the main window's page tabs on top/left/right/bottom (previously it could only do top/left). This now also updates as soon as the options dialog is applied.
5 / I:
0 / P:
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v414.tar.gzThis is the first release of a complicated new system. If you are a cautious user, feel free to wait a week for me to iron out any surprise bugs.
I had a good four weeks converting tag parents to the new 'virtual' database cache.
The release does not need to do a big update this week. It will ask you if you have an SSD or an HDD (EDIT: with a special caveat for PTR users, see below), but that is it. All the mandatory heavy work that came in with 407 now happens in the background.parentstl;dr: You don't have to do anything. If you haven't heard of a tag parent before, no worries. The database should work better now.
I did the same to parents as I recently did with siblings. They are now virtual
, which means when the database needs to 'fill in' a missing tag because of a parent relationship, that new tag now does not actually exist in storage, only in a special computed database cache. If parents change, the implications in this cache can be recalculated with no permenant changes made. Parents are now undoable.
Parents are also comprehensive. Previously, due to different logical holes, not every parent was filled in correctly. There were several retroactive holes. Parents now always apply to all their children, and they adjust to siblings and sibling changes.
Also, you can now apply parents across services. This matters less than siblings, but if you have particular parent needs, you can now set your 'my tags' parents to work on the PTR, or remove all parents entirely, under the new tags->manage where tag siblings and parents apply
This took a bunch of work. I am happy with all of it except how parents now display in manage tags
. It was complicated to insert 'ghost' tags to represent parent implications and I just ran out of time, so for now, tags with parents just get a '(2 parents)' suffix after their tag label. You can now right-click on tags in most places to see all their sibling and parent data.
6 / I:
0 / P:
I had an ok week. I mostly fixed bugs to make a 'clean' release before I go for the tag parents database cache.
It is all misc work this week, no highlights.full list
- added 'sort by number of files in collection' file sort type. it obviously only does anything interesting if you are collecting by something
- when you enter a tag from a manage tags suggested tags column with a double-click, the tag input box is now immediately focused. entering it with a keyboard action does not move the focus
- wrote a new routine for the 'check and repair' database menu that scans for and fixes invalid tags. this might be some system:tag that snuck in, superfluous unicode whitespace, or some weird website encoding that results in null characters, or any other old tag that has since become invalid. tag translations are written to the log
- added an experimental 'post_index' CONTEXT VARIABLE to subsidiary page parsers--whenever a non-vetoed post has pursuable URLs, this value is incremented by one. this is an attempt to generate a # 0,1,2,3 series. feedback on this would be appreciated, so I can formalise and document it
- added 'no_proxy' option for the options->connection page. it uses comma-separated host/domains, just like for curl or the NO_PROXY environment variable. it defaults to 127.0.0.1. in future, options will be added to auto-inherit proxy info from environment variables
- fixed an error when subscriptions try to publish to a page name when a 'page of pages' already has that name
- activated some old 'clean url' parsing tech I wrote but never plugged in that helps parsing urls from source fields on sites that start with non-url gubbins
- fixed the v411->v412 update step to account for a tags table that has duplicate entries (this shouldn't ever happen, but it seems some legacy bug or storage conversion indicent may have caused this for some users). if a unique constraint error is raised, the update step now gives a little message box and does dedupe work
- fixed an issue where the 'will display as' tag was rendering without namespace when 'hide namespace in normal views' was on
- fixed a recent character encoding routine that was supposed to filter out null characters
- fixed some UPnP error reporting
- _may_ have fixed an odd and seemingly rare 'paintevent' issue when expanding the popup toaster from collapsed state--it may also have been a qt bug, and fixed in the new qt:
- updated qt to 5.15.1 for windows and linux builds. it fixes a couple of odd issues like 'unclicking' to select a menu item (issue #296)
- added session save to holistic ui test suite
- misc code cleanup
4 / I:
0 / P:
I had a great week catching up on smaller jobs, improving search speeds, and adding a 'lite' 407->408 update mode for HDD users who sync with the PTR. There are also a couple of new applications for the Client API.Update this week will take a few seconds to a few minutes as new database indices are created.sibling and search speeds
Thanks to feedback from some PTR-syncing HDD users, the new siblings update code, most importantly in step 407->408, takes way too long for them - perhaps more than 24 hours. I have written a little yes/no dialog popup into the update step that talks about this and optionally activates a 'lite' mode that does not apply siblings. This still requires some basic cache copying work, but it is significantly less. If you are still on 407 or before and have been waiting to update, please give this a go and let me know how it works for you.
The 'manage where tag siblings apply' dialog now has some red text to warn about the high CPU/HDD of applying many siblings to a large number of tags. I am still not happy with the 'monolithic' way this db work goes, so when I get stuck into the parents cache, I will write an asynchronous system that does this work in the background, pause/resumable, without interrupting browsing and so on, much like I did with repository processing.
Some things were working slow since siblings (e.g. in a search, mixing wildcard tags with regular tags), but I went through every instance of my new optimisation code, fixing bugs, testing it at large scale, and smoothing spikes out further. Tag, namespace, wildcard, tag presence/count, known url, and file note searches should all be more reasonable. A neat new tag search pre-optimisation routine that checks autocomplete counts for expected result size before deciding how to search now works for more sorts of tags and also kicks in for namespace and wildcard searches, which now break their work into smaller and simpler pieces. I also added and reshaped some database indices, which will ensure that more unusual search types and general operations can still run efficiently. The update will take a few seconds to a few minutes as tag indices are regenerated.
I have learned a bunch about speeding up multi-predicate searches recently - how to get it wrong and how to get it right. I have a plan to speed up rating and known url results, which are still generally not able to speed up with multiple predicates on large clients.
3 / I:
0 / P:
I had a good week. The siblings work is complete.siblings
The last areas that used the old siblings systems are now migrated to the new fast cache. The 'loading tag siblings' step of client boot, which for PTR users could be several seconds of work, is no longer needed!
These last parts are all user-facing UI code. Mostly tag autocomplete behaviour and the 'will display as' labels you see in tag edit contexts. Everything now goes through the database, and all in a nice asynchronous way, so (fingers crossed) it will not add noticeable lag.
As I unified code here and moved to nicer logic, I was able to clean up some behaviour. Matching siblings based on what you type (for instance, if you have 'lotr' siblinged to 'series:lord of the rings', text-matching the latter even if you type the former) should be cleaner and more reliable. Also the 'will display as' labels should appear in all the tag suggestion columns, and some other 'edit tag' places.
There is still more I could do here. Using 'manage tags' on thousands of files at once can be a bit laggy, both in working and once you hit apply, and I could make '(will display as xxx)' labels colour according to the xxx namespace to make them 'pop' a bit, and I would still like to ultimately replace the individual 'will display as' labels with something neater for workflow. But I have been focusing on siblings for more than a month. Now the main virtualisation and boot lag elimination objectives are done, I should move on to other work.
If you have been waiting to update from 407, this is a fine week to do it. There will still be a whack of update CPU if you sync to the PTR, but on an SSD it shouldn't be more than a one-time shot of 10-30 minutes.
3 / I:
0 / P:
I had an ok week. I wasn't as productive as I hoped, but I am happy with the mostly optimisation work.optimisations
After some more profiling in IRL situations, and with more helpful info from users, I have done another round of profiling for the new sibling cache, and more besides. A database technique I use for many purposes is now more reliable (fewer lag spikes), and has less CPU overhead. If you found some systems (like the 'related tags' suggestions in manage tags dialog) sometimes took a few seconds to work in the past couple of weeks, they should now be fast again. And you should find many types of file search, particularly those with multiple search predicates, and general tag processing, should be faster than before.
My dev machine went from about 3-8k rows/s processing speed in a test environment up to 8-20k rows/s, which is faster than it was before the siblings cache was added.
3 / I:
0 / P:
I had a great week fixing some bugs and optimising the new tag siblings cache. The new code works much faster now.siblings
I am very happy that there do not seem to have been any obvious errors with the new sibling database cache. Unfortunately, a couple of areas were working inefficiently, which IRL testing helped to diagnose. I put a lot of time into this this week and was very successful - some sections take 10% less time, some 90%, and one critical query now takes 99% less time. It depends on many factors, but many things are faster overall. In particular, tag processing speed, which took a real hit, is back up to good speed, and setting new tag sibling application rules now only needs to regenerate for changed
siblings, so if you add (or remove) your own five 'my tags' siblings onto the PTR, the client now only has to do two seconds of work, not ten minutes.
I made some progress on the final awkward things to migrate. Most autocomplete results you see are now able to give themselves the 'will display as xxx' label when needed and match against sibling input (e.g. having an input of 'lotr' match 'series:lord of the rings' due to siblings) on their own, which should save some CPU time when typing. There is still more to do, so I'll keep hammering at it for the next two weeks and see if I can get rid of 'loading tag siblings' on boot before I start on parents db cache.
If you have been waiting for faster code before you update, you might want to wait another week. I just did a test re-do of the 407->408 update step in IRL conditions, and it was not as fast as I wanted it. I'll keep pushing at this.
I am increasingly looking forward to doing that parents db cache, which will extend this work in a new dimension. That will be v412, which I am very confident will be another two-week release. This is some of the most intricate work I have done.
6 / I:
0 / P:
I had a great couple of weeks moving tag siblings forward. The update this week will take some time as a new cache is generated. On an SSD with the PTR synced, about 10 to 20 minutes, depending on how many files you have. Some of the new tag sibling application rules may have 5-10 minute delays on edit as well. If you have a large/PTR client and are on an HDD, or you would rather just wait for the kinks to be ironed out, you might like to wait a week or two for me to further optimise this new code.siblingstl;dr: Siblings are faster and better now, you don't have to do anything. Some parents will not appear with new downloads - don't worry about it, they will all fill back in nicely soon.
I have finished the first version of the database 'display tags' cache. This is a long-planned system that keeps track of how tags are 'collapsed' by siblings on an ongoing basis. Until now, siblings were always calculated on the fly at the UI level whenever media was loaded, which lead to slow and inaccurate sibling-based search results and a variety of logical problems. Search results generally load much faster now, particularly on subsequent loads.
On the front end, tags appear exactly they did before, but when you do tag-based searches or tag autocomplete, the files and counts that come back should be more accurate. For instance, if you have a file tagged 'metroid' that appears as 'series:metroid', and you search 'series:anything', that file now shows. Siblings logic is also tighter, and changes apply faster.
And now, if you go to services->manage where tag siblings apply
, you can now decide which services' siblings apply where. If you do not like the PTR's siblings, or you have a couple of different preferences you want to add on top, you can say things like 'apply siblings from "my tags" then "PTR"', or you can turn them off entirely. This powerful service-based system replaces the old 'apply all siblings to all services?' checkbox that used to be in the options.
This is some complicated code affecting core systems. I have tested it extensively, but IRL situations, particularly with large sessions or storage, will be more complicated and useful than what I have here. Please let me know where this new system is laggy or miscounts anything. And of course, if anything outright breaks, let me know!
The help on siblings is going to be increasingly outdated as this work continues. If you don't know what siblings and parents are, please wait until I am done with this multi-month job, and I'll have some nice new help written.
6 / I:
0 / P:
I had an ok week. I did some prep work for siblings, and fixed some bugs.misc
The right-click menu off a normal taglist now provides a whole bunch of different copy options, depending on whether you want all or selected, tags or subtags, and with or without counts.
I fixed a bunch of weird hover window issues in the media viewer. On a thin media viewer, the top-right window now squeezes into the smaller space better, and if it can't fit, it bumps down so you can still use it. The center-right duplicates window also positions correct on a thin window. And now moving your mouse off a clicked hover window will immediately restore focus to the main media window, which should eliminate some annoying click focus issues (usually when you click quickly after moving your mouse off, when the click can be swallowed).
A user has written a simpler getting started guide here: https://github.com/Zweibach/text/blob/master/Hydrus/Hydrus%20Help%20Docs/00_tableOfContents.md
This is now linked at the top of my help as well. If you have had trouble with the denser official help, or you know someone else interested in hydrus who is similar, please give that a go.
2 / I:
0 / P:
I had a good week. It is now easy to deduplicate queries in subscriptions.subscriptions
Managing multiple subscriptions can get complicated. Figuring out which queries are in which sub, whether and where you have duplicates, is tedious. A recent bug with query pasting also could have introduced some duplicates within the same sub. This week should make it simple to manage.
The main manage subscriptions
dialog now has a 'deduplicate' button. It lights up any time your subs of a particular downloader have the same query, whether that includes dupes within the same sub or across different subs. Click it, and it launches a thorough process to determine what you want to do:
- First, it asks if you want to deduplicate by case or not--e.g. whether "Samus_Aran" and "samus_aran" should be considered dupes, or only exact matches.
- Then it summarises which downloaders appear to have dupes, letting you choose which to work on.
- Then you have the option of selecting which of the queries you wish to deduplicate--e.g. if you want to do two runs to dedupe, say, artists to one sub and characters to another.
- Then you select which subscription(s) will retain the deduplicated queries.
Every step explains itself. Mostly it is simple, but there are some odd situations. When a sub dedupes queries within itself, it keeps the one(s) with the most files. I hope I have accounted for everything reasonable.
Adding to caseless matching, also added is a 'lowercase' button, which will coerce all the queries in the selected subscriptions to lowercase, just to help keep things neat. And lastly, when you paste queries into a sub, the 'already in the sub' test is now caseless.
3 / I:
0 / P:
I had a good week. 'system:number of tags' now supports namespace filtering.number of tags
The 'system:number of tags' predicate now lets you attach a namespace, so you can search for 'files with two character tags' or 'files with more than one creator', or any other combination you can think of. Also, all 'number of tags' queries have been optimised, and are now typically much faster, particularly when mixed with other tags. They are also quickly cancellable, so if you do enter a very slow one, it should respond fairly quickly if you hit the 'stop search' button that appears after a delay.
I updated a bunch of the database code here to support new types of search. With the namespace searching, you can enter a blank namespace to search for 'unnamespaced' tags, and if you have a very specific need, you can now enter a wildcard
namespace, like 'crea*', perhaps to help search for typos.
Unfortunately, I am removing 'system:number of tags' from the 'default system predicates' options panel. EDIT: I am hiding the whole panel now.
This horrid stack of UI runs on ancient and difficult-to-update code, so rather than wrestle with it more, I plan to retire it and replace it with a clean system that also allows for editing search predicates in place (e.g. right-clicking an active search predicate and changing the >2 to >3 or similar).
I am really happy with how namespace counting turned out, and I have wanted it myself for a long time. Let me know how it works for you.duplicate subscription queries
I believe the subscription data storage rewrite introduced a bug where when you paste queries into a sub and it says 'x y z are already in the sub and will not be added', they were being added anyway!
I fixed it. If you paste into subs a lot and get this message regularly, please check your subs for dupes! Now we have faster and more flexible subs, some users have also asked for broader de-duping tech that spans across multiple subs or does upper/lower case deduping, so I also expect to write buttons to do this in the near future, so you may also just want to wait for that.
3 / I:
0 / P:
I had a good week working almost entirely on the client's multi-column lists. They now remember their column widths and sort.multi-column lists
All the tables across the program are now plugged into a unified options structure. This remembers the lists' columns' widths, so if you resize the manage subscriptions list a little tighter, then the next time you load that dialog, it should come back the same. The lists across the program that automatically start and maintain a sort (e.g. manage subs, or the query list in a downloader page), also remember if you change the sort.
This sounds simple, but it involved rewriting critical sections for every list, and there are about 66 different ones! I am happy with the change, as it means a load of duct-tape mess is unified and easier to maintain and extend in future.
I built the system with several near-term extensions in mind: having all lists sort cleanly, quickly, and by default, and show sort up/down arrows; allowing columns to be rearranged; and allowing columns to be hidden, which will also allow me to add complicated columns (e.g. statistically expensive subscription quality info) that start hidden by default.
Although the system is now easier to check for errors, the sheer number of lists and amount of code I changed this week means there is probably a typo somewhere in a column name or initial size. I have checked the main lists closely, but if you encounter anything odd, or even a list that does not boot at all, please let me know. There are still a couple of sizing calculation peculiarities I need to fix.
6 / I:
0 / P:
12 / I:
2 / P:
I had a great week just doing cleanup and other small work.all misc this week
Most of the pages in the client now have special status bar text for when they have no files. If your search results in no files, instead of the old '0 files', you'll get 'no results for this search', and 'search cancelled!' if the search was cancelled, and so on. It just adds a bit of quick feedback to some actions--like highlighting a gallery query with no new files--where the new blank page may just be replacing another blank page.
I added 'copy_xxx_hash' shortcuts to the media shortcut set, for 'md5', 'sha1', and 'sha512'. All the hash-copying code is neated up as well, so if hashes can't be found for some reason, you'll still get what could be found and get nice info on how many were missing and why.
The client's shutdown code is untangled. A variety of unusual shutdown situations should
be smoother, less buggy, maybe a little faster, and when errors do occur, they should be caught and reported more gracefully.
Gallery and watcher pages' list right-click menus now have direct links to the selected query or watcher's file and gallery logs. It is just an easier shortcut than highlighting and finding the button.
Forced file maintenance jobs now list a correct total x/y progress on their popups. Previously, they would show x/256 in the batches of 256, with no total summary.
Most of the lists with clipboard/png import/export buttons now also support json files, which is just a straight copy of what would go to your clipboard, but to a file. You can drag and drop compatible json files on the lists as well, just like with pngs.
I have updated the Windows and Linux client to Qt 5.15, up from 5.13. We had a variety of problems with 5.14, but this version seems to be working well. Let me know if you have any trouble, or if any problems are magically fixed.
4 / I:
0 / P:
I had an ok week, although I was a little short on work time. This is mostly a bugfix release that tweaks how the new subscriptions work and fixes some downloader update issues. There is also an important new way to make feature suggestions for hydrus.mostly misc work this week
Now that subscriptions can run in small efficient pieces, I removed several old limiters that stopped the old system running too often. Subscriptions should boot with less delay when needed (e.g. after the client loads or when you ok the dialog after changes), and an old system that batched queries together is gone, so if a query is due, it should run generally as soon as it can.
I also re-added the import/export/duplicate buttons to the manage subscriptions dialog. It also supports importing (and converting to the new format) the old subscription format, so if you have an old .png sub, or you want to reimport the text that was backed up to your database folder last week, this is now possible. Exporting or duplicating a very large subscription may now need a few seconds to do some database work, but the dialog will warn you beforehand.
As users have made and shared more custom downloaders, I have had reports here and there about new default downloaders sometimes messing up the custom links, unintentionally resetting back to those defaults or other unusual link remapping. I have tightened up all this code. Please let me know if you have any more trouble.
And speaking of new defaults, I am rolling out new objects for nitter, derpibooru, and e621 again this week to improve object linking and URL matching. If you had a subscription complain last week about an e621 gallery URL not matching, this should now be fixed.
1 / I:
0 / P:
It looks like Hydrusdev enabled "issues" on the github repo a few hours ago. Was there any update or notification given by him about this? He seemed pretty insistent on using the boards and his own personal to-do list for keeping track of development and bugs, so I'm curious if he explained somewhere why he changed his mind.
4 / I:
0 / P:
🎉🎉🎉 MERRY v400! 🎉🎉🎉
I had a great week of vacation, and then a great week finally getting the subscription data overhaul done.subscriptions
When I first wrote subscriptions, they could only hold one simple query each. Queries have become much more complicated since then, and subscriptions can of course hold many queries at once, sometimes hundreds. The old monolithic method of storing and loading subs was creaking at the seams. This week fixes it, and subscriptions should now load and operate quickly for all normal operations.
Subscriptions are now broken into pieces. Essentially, instead of one thing holding everything, they now store each query as a separate object and load and save each from your database as they are needed. The 'top' of a subscription is now always in memory and allows the manage subscriptions dialog to start instantly. Subscriptions can also boot quickly, and will cause less lag as they finish up. It all saves time and database read/write. The old '200,000 files' limit for subscriptions is gone. I wouldn't advise you make a sub with 10,000 queries just yet, but you do not have to worry about the size of any one sub too much any more.
There are no significant changes to how subscriptions look or are edited. All your existing subscriptions will be converted to the new format on update. However this is a big change behind the scenes, and if you have big subs, it may take a minute or two to update your databases. Your old subscription objects will also be backed up to a new subdirectory in your db directory, just in case anything goes wrong now or in the near future.
Unfortunately, as subscriptions are now more complicated, I did not have time to write new import/export system for them. The duplicate/import/export buttons on the manage subscriptions
dialog are hidden for now.
This took a lot of planning, prep, and work. I hope you find your subscriptions work nicer, and if you have any trouble, please let me know.new downloaders
Twitter retired their old API on the 1st of June, which broke our downloader. There is unfortunately no good hydrus solution for their new API, but thanks to a user's efforts, I am rolling in a parser for nitter, a twitter wrapper, this week. It has three downloaders--one for media posts, one for retweets, and one that does both--so please play with it and then move your twitter subscriptions over to it.
Also fixed should be derpibooru search and the md5 hash parsing of the danbooru downloader (which speeds up some downloading).
10 / I:
1 / P:
I had a great week tidying up smaller issues before my vacation.all small items this week
You can now clear a file's 'viewing stats' back to zero from their right-click menus. I expect to add an edit panel here in future. Also, I fixed an issue where duplicate filters were still counting viewing time even when set in the options not to.
When I plugged the new shortcuts system's mouse code into the media viewer last week, it accidentally worked too well--even clicks were being propagated from the hover windows to the media viewer! This meant that simple hover window clicks were triggering filter actions. It is fixed, and now only keyboard shortcuts will propagate. There are also some mouse wheel propagation fixes here, so if you wheel over the taglist, it shouldn't send a wheel (i.e. previous/next media) event up once you hit the end of the list, but if you wheel over some hover window greyspace, it should.
File delete and undelete are now completely plugged into the shortcut system, with the formerly hardcoded delete key and shift+delete key moved to the 'media' shortcut set by default. Same for the media viewer's zoom_in and zoom_out and ctrl+mouse wheel, under the 'media viewer - all' set. Feel free to remap them.
The new tag autocomplete options under services->tag display and search
now allow you to also search namespaces with a flat 'namespace:', no asterisk. The logic here is improved as well, with the 'ser'->'series:metroid' search type automatically assuming the 'namespace:' and 'namespace:*' options, with the checkboxes updating each other.
I fixed an issue created by the recent page layout improvements where the first page of a session load would have a preview window about twenty pixels too tall, which for some users' workflows was leading to slowly growing preview windows as they normally used and restarted the program. A related issue with pages nested inside 'page of pages' having too-short preview windows is also fixed. This issue may happen once more, but after one more restart, the client will fix the relevant option here.
If you have had some normal-looking files fail to import, with 'malformed' as the reason, but turning off the decompression bomb check allowed them, this issue is now fixed. The decomp bomb test was itself throwing an error in this case, which is now caught and ignored. I have also made the decomp bomb test more lax, and default off for new users--this thing has always caught more false positives than true, so I am now making it more an option for users who need it due to memory limitations than a safeguard for all.
11 / I:
2 / P:
I had a good work week. Tag autocomplete gets some new search options, and advanced users who make downloaders get some new text processing tools.tag autocomplete
When I recently overhauled the tag autocomplete pipeline, I eliminated some unusual logical hoops where you could accidentally fire off expensive searches that would fetch all tags. Now the code is clean, I am adding them back in as real options.
The main thing here is that services->tag display
is now services->tag display and search
. It has several new options to change search based on what the autocomplete's current 'tag domain' is (i.e. what the button on the dropdown says, "all known tags" or "my tags" or whatever else). The options are available for every specific tag domain and the "all known tags" domain, and only apply there.
There are three new search options: You can have full namespace lookup, so an input of 'ser' also finds 'series:metroid' and all other series tags; you can have an explicit input of 'series:*' show all 'series' tags; and you can have '*' show all tags. These queries are extremely expensive for a large service like the public tag repository (they could take minutes to complete, and eat a ton of memory and CPU), but they may be appropriate for a smaller domain like "my tags". Please feel free to play with them.
There are also a couple of clever options setting how 'write' autocompletes (the ones that add tags, like in the manage tags
dialog) start up, based on the tag service of the page they are on. You can set them to start with a different file or tag domain. Most users will be happy with the defaults, which is to stick with the current tag domain and "all known files", but if you want to change that (e.g. some users like to get suggestions for "my tags" from the PTR, or they don't want tag counts from files not in "my files"), you now can. The old option under options->tags
that did the "all known files" replacement for all write autocompletes is now removed.
I have optimised the database autocomplete search code to work better with '*' 'get everything' queries. In the right situation, these searches can be very fast. This logic is new, the first time I have supported it properly, so let me know if you discover any bugs.
9 / I:
2 / P:
I had an ok week. I mostly did small fixes and improvements.highlights
I added 'system:has/has no note with name xxx' to search for files with specific note names. This is bundled into the same dialog as last week's 'number of notes', for the new generic 'system:notes' entry in the normal system predicates list.
When you have many page tabs in a row, they will now middle...elide their name text to try to fit better. You can turn this off if you like under options->gui pages
When files in the duplicate filter are pixel duplicates, the file size and import time comparison statements now have no 'score' and colour blue. Pixel duplicates are often a different judgment call, so until I can get more flexible scoring options here, I am hardcoding it to be less visually green/red.
If you need to run a gallery download with a search term that includes '#', this should now work.
I updated the IPFS code to work with the new IPFS 5.0 daemon. I think I have things mostly correct, but the new daemon has some new CORS restrictions that may need configuration on the IPFS side, depending on your situation, to give hydrus permission to talk to it.
The left page panel's position should set itself more reliably for users who start the program minimised to system tray. If you have had trouble here, please let me know how you get on. I believe
the changes here will also reduce some UI lag for many other page adding and moving operations, particularly for large sessions.
14 / I:
2 / P:
I had a good week. The client now has better file 'notes' support.
There are several database updates this week. Users with large databases can expect the first boot to take a few minutes.notes
A long time ago, I added prototype 'notes' to files. You can access it under a media right-click->manage->notes
. It was a bit hacky, functioning just like a plain .txt attached to files, but some users really picked it up. This week it becomes more mature.
Firstly, files can now have multiple notes, each with their own name. Existing notes will get the simple default name of 'notes'. Also, files that have notes now show a notes icon on their thumbnails and in the media viewer top-right hover window. Clicking the hover window icon quickly opens the edit notes dialog.
And to find files with notes, there is now 'system:number of notes'. Its panel has a couple of quick buttons for finding files with any notes or no notes. Notes are stored in a faster and more space-efficient way, so this should work fairly quickly for most situations.
The next step here is to add note parsing to the downloader system. The main thought is to get artist post text and similar imported with nice 'blah gallery artist comment' style names as you download files. I had hoped to figure something out for this this week, but I didn't have time to do it properly, so I focused on polishing the above. I would like to fold parsing in in upcoming normal weekly work.
The new storage system for notes also builds a fast search cache for their text. It will not be extremely difficult to add some sort of 'system:note has text xxxxx' in future as well, if you want to search for parsed mega links or whatever else. Beyond that, I'd love some note 'preview' on the media viewer and comprehensive import/export/Client API support for notes.
10 / I:
0 / P:
I had a good work week. I fixed a variety of bugs, improved the feel of the suggested tags columns, and made zooming in the media viewer nicer.suggested tags quality of life
I recently did some work on the different 'suggested tags' panels the manage tags
dialog can support. This week I worked on some of the feedback from that pass, hopefully making it a bit easier to get what you want done. As a reminder, you can set these columns up under options->tag suggestions
The columns are now 'synced' with the current media. They all refresh/repopulate reliably when you move to another media in the media viewer. Also, all columns now hide tags that all the selected media already have, and they update this ~as you edit~, so if both 'favourites' and 'related' suggest 'character:samus aran', and you add that through any means, both lists will remove that tag! Furthermore, if you remove a tag that one of those lists would show, they now re-show that tag instantly, without having to do any new lookup.
Furthermore, the tag lists now try to retain the selection location when you add a tag through the keyboard enter key, so even though a tag you hit enter on will disappear, the focus should stay on the tag above or below. I hope this makes it easier to work with these columns with just the keyboard. As a second remind to keyboard users, you can currently set up some slightly mickey-mouse shortcuts to move focus around here under the 'main_gui' shortcuts, the actions starting 'show_and_focus_manage_tags_...'.zoom center
When you zoom a file in or out, there is a 'centerpoint' about which the zoom happens. This position 'stays still', while everything else grows or shrinks around it. Until now, this has been hardcoded on the media center, which was sometimes unhelpful when the media had been panned. This centerpoint is now customisable under options->media
. It now defaults to the media window
centerpoint, which means the image will always grow and shrink around the center of what you are looking at. This makes it much nicer to pan and zoom in the duplicate filter. You can change it back to the media centerpoint, the media top-left corner, or even the current mouse cursor position, for RTS-vidya-like zooming.
4 / I:
0 / P:
14 / I:
7 / P:
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v393/Hydrus.Network.393.-.Linux.-.Executable.tar.gzEDIT: This release had a hotfix two hours after initial release. If you got it early from github or elsewhere, please redownload. The links are the same, the build is new.
I had a good week catching up on messages and small jobs. There is also a (hopefully) neat prototype solution for some cloudflare issues.cloudflare and network
CloudFlare hosts content for many sites online. They have a variety of anti-DDoS tech, normally a variation on a 'is this a web browser?' test, that sometimes stops hydrus from downloading. If you have seen unexplainable 503 errors on a site that works ok in your browser, this may have been it.
9 / I:
4 / P:
I had a mixed week, but I am happy with my work. Some large database jobs are faster and tag autocomplete results have better sibling support.tag sibling database cache
I was not able to get as much done here as I had hoped, but I was able to finish the first step and fix some sibling bugs. The client now keeps a cache for fast sibling lookups, which will be a great tool going forward.
The tag autocomplete system now uses this new cache. When you type in the start of a tag, possible matches' siblings are added to the results list, even if they do not strictly match what you typed. These siblings used to be computed on the fly, but now it is looked up using fast pre-generated data. I have also improved the accuracy of the search and added 'pending' siblings in.
As I was working here, I also fixed some sibling presentation and search bugs. If you discovered that a 'series:blah' search was delivering some unexpected 'cosplay:blah' results, this should now be fixed. There is still more to do here, which I hope to naturally fix as I replace more old systems with nice new caches.
There are two next steps here: first to write a 'what file tags should look like with siblings collapsed' cache, which will speed up tag search and file load and also eliminate many of the uncertain counts you get with autocomplete results like 'character:samus aran (500-700)'. Second is to write a full-featured 'which siblings apply to this service?' options system, so you can say 'ignore the ptr's siblings, use mine instead' and other clever combinations, rather than the current 'always apply all to all, or one to one'.the rest
e621 and sankaku get revised default downloaders, fixed for getting 'rating' and 'genre' tags respectively.
The system tray icon got some bug fixes. It is also more stable under non-Windows, but doesn't seem to hide reliably yet.
The hydrus server and the client's API and booru now try to host their services on IPv6 as well as IPv4. This is new stuff to me, so let me know if you come across any problems!
7 / I:
6 / P:
I had a great week. The client now has 'minimise to system tray' support, you can search by framerate and 'number of frames', and a bunch of bugs are fixed.system trayThis first version is buggy/crashy under non-Windows, so I have disabled it for non-advanced mode non-Windows users for now. The icon menu doesn't work well for non-Windows, but double-clicking seems ok. I will keep working on this.
The hydrus client can now make a system tray icon for those OSes that support it. You can customise it under the new options->system tray
panel. You can set the client to minimise to and/or close to system tray, and you can set the icon to always display if you like. There is also a new 'hide_to_system_tray' action under the 'globals' shortcut set.
Clicking on the system tray icon will raise a 'shown' client to the front or show a hidden client. Double-clicking or middle-clicking will do hide/show. The icon also has a menu to show/hide the client, pause/unpause subscriptions or network traffic, and exit the program.
You can also set the client to boot minimised into the system tray, but there may be layout issues once you do show the client--please let me know if you get any!frames
You can now search for file framerate or 'number of frames' under an expanded 'system:duration' search predicate. It is difficult with the data hydrus currently knows to get very precise framerate, so system:framerate searches +/-5% of what you set. You can't differentiate between 30fps and 29.97fps yet.
You can now sort by number of frames.
Video framerate is now shown in the 'file info summary' text that displays on the status bar, thumbnail menu, and media viewer.
11 / I:
2 / P:
6 / I:
2 / P:
I had a great week. I fixed many small bugs, added some quality of life, and am rolling in updated downloaders for e621 and Deviant Art.downloaders
Unfortunately, last week's e621 downloader was not getting md5 hashes reliably. This coupled with the URL format change often meant an increased bandwidth load for the subscriptions that were trying to re-find their place. Thankfully, another user has provided a more accurate one that rolls into today's update. It also pulls rating tags. If you paused your e621 subs, please resume them again.
Deviant Art have also been through a layout change recently. The existing file page parser was getting a little shaky, so I wrote a new one this week that uses their API. The default hydrus DA downloader is now wholly API-driven, so it should all be more resistant to any further changes on their frontend. If you would like to search for nsfw DA content, the default DA login still seems to work, but Hydrus Companion is also an excellent option.For advanced users:
Thanks to user contribution, there are two newer default file lookup scripts, an updated iqdb-based danbooru lookup and one for direct danbooru lookup using md5 hash.For advanced users:
The parser String Transformation system now has a 'url percent encoding' option for the encode/decode types. This lets you convert complicated text to a URL-parameter-friendly percent-encoded string, or vice versa
, if you need to do something like a clever API URL transformation.
3 / I:
0 / P:
I had a great week mostly fixing things and adding and improving small features.all misc this week
The 'sort files by' dropdown on all pages is now a button. It launches a menu that groups the different sort types, cutting the long list down into easier to navigate groups. Mouse wheel still works on it!
Also, 'sort by framerate' is added. It just does a simple num_frames / duration calculation for now. Fps isn't surfaced in the UI atm, so I expect in the near future to add it to normal file labels and also to add a system search predicate for it.
got its overhaul. Managing namespace sorts is much more sane, support for namespaces with hyphens (like 'creator-id:') is added, and you can edit the order in which they appear in the sort by menu.
Some logic behind tag autocomplete lookup is improved this week. The way special characters like braces and paretheses are handled is better (for instance if you want to search for '[intensifies]'), and now hyphens and underscores are included in these special rules. Typing in 'blue_eyes', 'blue-eyes', 'blue eyes', or 'eyes' will match all of 'blue_eyes'. 'blue-eyes', and 'blue eyes'! Although the results may still be separate without tag siblings, you should never have to worry about searching an underscore version of a tag again. It will take up to five minutes to update your client
to reflect the new rules. This is a simple change to a complicated system, so let me know if it fails anywhere, particularly in namespace or wildcard searches!
5 / I:
0 / P:
I had a great week. I mostly cleaned code, moving old wx definitions and calculations up to Qt format, fixing bugs and colours along the way. There is not much significantly new this week, but I am happy to have cleared out some behind-the-scenes mess.gif and mpv.conf
Some gifs have metadata that says 'play this once' or 'play this five times' rather than looping infinitely. Hydrus now parses this information, and if you tell it to under options->media
, it will obey it.
MPV is extremely configurable. It uses a file called 'mpv.conf' to load its settings. Hydrus has a very bare conf file at the moment, but I expect to expand it as we experiment more with settings that are useful for hydrus. Now you can set the path of the mpv.conf file you want to use, again under options->media
. I have added three test mpv.conf files to the hydrus static directory--one for high quality, and two for different kinds of audio normalisation--if you are interested in this subject, or if you have your own mpv.conf files you would like to experiment with, please give them a go and let me know what you think. Once we know what works well for different situations, I will make some more well-rounded profiles for users to choose from.misc code cleanup notes
You can now edit how fast the mouse cursor autohides on the media viewer, including turning autohide off completely. Yet again, under options->media
. The default is 700ms.
The window positioning and sizing system has had several calculation improvements, and it will also now be better at rescuing off-screen windows (for instance if you have a multi-monitor setup that changes between client boots), all with better error handling. Furthermore, some windows like the splash screen--which finally has fixed colours--will now appear on the monitor your mouse is on. If you have had problems with window positioning or sizing, particularly if you have an unusual desktop or multi-monitor setup, let me know if this improves or worsens your situation.
If you are interested in editing your own hydrus QSS stylesheet, please check the install_dir/static/qss directory again this week. I have moved many of the previously hardcoded colours in the client (like green/red dialog button labels) to a new 'default_hydrus.qss' file. This file is loaded by hydrus, and is prepended to any custom stylesheet. You can add these hydrus-specific qss class definitions to your own qss to overwrite them. I expect to continue migrating hardcoded colours to this file in future.
12 / I:
12 / P:
I had an ok week. There are more improvements to mpv and the shortcuts system, and a few misc small jobs done as well.mpv
Some more general improvements this week:
MPV windows now work in slideshows, and all audios/videos are smoother to move to the next file when slideshows are running.
When you scrub the seekbar below a native or mpv video player, the video should now more reliably pause itself while you drag. Also, more kinds of one-frame music videos (where it is just an image webm with a song in the background) should now appear with their seekbars now, and the seekbars for all these unusual videos and audio files should work better.
A zoom bug where video could navigate down a few pixels on zoom in/out cycles is fixed, and the blank audio mpv window now has some hardcoded zoom capability and should scale itself to fit in the small preview window, always letting you interact with its seek bar.
I attempted to get mpv working for macOS but ran into some technical walls, I think because I am limited to 10.12 Sierra. I do not think there are excellent solutions for this, at least for my macOS expertise, so mpv support on macOS is unfortunately on hold until I learn more.
5 / I:
1 / P:
I had a good but light week. MPV should be working better, and some of the recent UI animation flicker/jitter should be gone.mpv
Introducing MPV added a weird bug--after the first time an mpv window was opened, the client would then in some cases fail to animate thumbnail fade or process certain intra-program events unless another event, such as a mouse move over the program, restarted the event loop. Some media viewer transitions have also had some single-frame flicker. I have banged my head at this problem for a couple of weeks now, thankfully improving some behind-the-scenes code along the way, but I could not find the full solution. As it happens, rolling Qt back to a slightly older version (5.14.0 to 5.13.0, for users running from source) seems to have completely fixed it. It could simply be a 'native window' bug in the new version of Qt, but at the moment I still suspect my mostly borked wx-code-trying-to-talk-to-Qt-code. Until I can clean that code up, I am happy to have this easier solution.
I fixed a recent issue with the preview viewer not clearing out media correctly on page changes (which was particularly significant now we have audio) and also fixed an mpv crash for Linux/macOS users who happened to already have MPV available through their OS. Also, MPV is now configured to have a 'lighter touch' on filesystems, so if you have a client with a laggy file storage system (e.g. a very large client stored over a network share) and have experienced extreme latency when trying to open MPV, this situation should be much better. MPV should be fairly stable now, so my big remaining tasks are to get nicer volume controls and proper build support for Linux and macOS.
MPV as a program relies on '.conf' files for most of its configuration. Hydrus now follows this format. If you have your own conf or otherwise would like to edit this, please check out the mpv.conf under install_dir/static/mpv-conf. If you have settings you think should be in the default for all users, let me know. I'll likely let you optionally override where hydrus looks for its conf (where it can't be accidentally overwritten on update) in future.
11 / I:
3 / P:
I had a good week with a couple of challenges. MPV is now ready for all windows users and is turned on by default.MPV
Thank you to the advanced users who tested and gave feedback on MPV. I have eliminated the crashes, tightened up the jank, and am now rolling it out to all Windows users by default for video, audio, and gif/apng. All media view settings under options->media
will be reset this week.
MPV is a good free media viewer. One of the core benefits of moving hydrus to Qt was being able to plug it into our media player, enabling hardware-accelerated video playback and audio. It looks just like the native player, with the seek bar beneath, but it works much faster, able to play 1080p or 4k videos at 60fps at full or unusual zooms. And of course, it makes noise!
This is early days. I have only just started plugging into MPV, so many features are basic or not yet available. The global volume and mute controls are currently some very ugly controls in the top media hover panel. Slideshows will not move forward on an MPV window (hydrus doesn't know when an MPV player has 'played once through' yet), and some processes like the archive/delete filter will need some extra workflow options now that more users will be playing videos at high res (left-click on the player pauses the video, so to set 'keep' on archive/delete, you'll have to click on some whitespace, of which there is so much less when the video is so big). Please let me know what your top priorities for improvement are, and of course, if you encounter errors or crashes, let me know. I'll keep working.
Even though it was a lot of difficult work, I am overall really pleased with how this has gone. The only big remaining bug that I need to nail down is an unusual thing where after multiple mpv viewings in a preview window, that page will stutter some query/thumbnail loading unless the mouse is moving. This is a slightly frustrating bug, but the benefits of MPV are enough that I am happy to live with it. I will also get it going for Linux and macOS, which I did not have time for this week.
15 / I:
1 / P:
1 / I:
1 / P:
Invitation to Endchan's Fourth Christmas Special!
28 / I:
4 / P:
Happy New Year! Although I have been ill, I had a great week, mostly working on a variety of small jobs. Search is faster, there's some new UI, and m4a files are now supported.search
As hoped, I have completed and extended the search optimisations from v378. Searches for tags, namespaces, wildcards, or known urls, particularly if they are mixed with other search predicates, should now be faster and less prone to spikes in complicated situations. These speed improvements are most significant on large clients with hundreds of thousands or millions of files.
Also, like how system:inbox and system:archive 'cancel' each other out, a few more kinds of search predicate will remove mutually exclusive or redundant predicates already in the search list. system:limit predicates will remove other system:limits, system:audio/no audio will nullify each other, and--I may change this--any search predicate will replace system:everything. I have a better system for how this replacement works, and in the coming weeks I expect to extend it to do proper range-checking, so a system:filesize<256KB will remove a system:filesize<1MB or system:filesize<16KB or system:filesize>512KB, but not a system:filesize>128KB.
11 / I:
3 / P:
I had a great, simple week. Searches are less likely to be very slow, and system:limit searches now sort.all misc this week
I identified a database access routine that was sometimes not taking an optimal route. Normally it was fine, but with certain sizes or types of query, it could take a very long time to complete. This mostly affected multi-predicate searches that included certain tags or system:duration and system:known urls, but the routine was used in about 60 different places across the program, including tag and duplicate files processing. I have rewritten this access routine to work in a more 'flat' way that will ensure it is not so 'spiky'.
Also in searching, I managed to push all the 'simple' file sorts down to file searches that have 'system:limit'. If you search with system:limit=256 and are sorting by 'largest files first', you will now see the 256 largest files in the search! Previously, it would give a random sample. All the simple sorts are supported: import time, filesize, duration, width, height, resolution ratio, media views, media viewtime, num pixels, approx bitrate, and modified time. If you want something fun, do a search for just 'system:limit=64' (and maybe system:filetype) and try some different sorts with F5--you can now see the oldest, smallest, longest, widest, whateverest files in your collection much easier.
There are also some fixes: if you had sessions not appearing in the 'pages' menu, they should be back; if you have had trouble with ipfs directory downloads, I think I have the file-selection UI working again; 'remove files when trashed' should work more reliably in downloader pages; and several tag and selection lists should size themselves a bit better.
9 / I:
2 / P:
I had a good week. I mostly caught up with my smaller jobs queue, just pushing on work that had piled up.Qt
Thank you for the continuing Qt bug reports. I fixed a variety of typo-broken buttons this week, mostly buried in less-used UI, and if you have set a browser launch path override in the options, hyperlinks across the program should obey that again. I also believe I fixed the annoying issue where media viewer hover windows that needed to shrink (because of switching to a new media that had fewer info lines or known urls) would linger too tall for one frame. EDIT: I'm still having slight hover window resize flicker in my IRL client when I keep my mouse over the top-right hover, I'll give it another go next week.
Furthermore, all non-menubar menus across the program now open on right-click release instead of press. This sounds trivial, but there was a mix of down/up click handling here, now unified, and some things like thumbnail and tag selection are handled more naturally in the initial click down. This also fixes an issue where users who had menu styles with no significant border were sometimes accidentally activating the first menu item on a slow click (the click down would show the menu, the click up would activate it).
Last week's support for styles and stylesheets for Qt seems to have gone well. There is more to do here, but this week brings better error handling for unusual situations, better 'we already have this set, no need to waste CPU re-applying it' detection, and fewer hacky colour overrides, so custom stylesheets should be able to colour regular panels without lots of unintended grey or white boxes.the rest
The shortcut system should now be able to support any text key. This includes extended characters like ø or æ. If you need to apply a modifier to get these keys (e.g. ctrl+alt+e), these modifiers are likely to show up spuriously in the shortcut's label (e.g. ctrl+alt+æ), but everything should nonetheless line up in the program correctly. Let me know if you discover any new bugs here!
There are also some new shortcut commands this week, for the 'media viewer' set. They are 'pan_top/bottom/left/right_edge' and 'pan_horizonal/vertical_center'. They move the whole image around to edges, or center it again, and should be helpful if you often look at tall or wide comics at 100% zoom. There will be more to come here in coming weeks.
The work on making the UI smoother is also continuing. Session saving and a variety of 'media refresh' events that occur on file import or maintenance are now significantly faster and do the bulk of that work in the background. Users with very large sessions should experience less juddery behaviour during both idle and heavy work periods.
0 / I:
0 / P:
I had a good week. I focused on catching up on smaller work. More bugs are fixed, there's some neat new shortcuts to pan media around, the UI should be smoother when importing files and saving sessions, and the system:known urls predicates now run much faster.
The release should be as tomorrow.
11 / I:
5 / P:
I had an ok to good week. Subscriptions run better and there are more UI fixes and basic theming for Qt.subscriptions
Subscriptions have always been resource heavy. As last year's downloader overhaul extended their capabilities, we saw increased CPU lag with many subscription operations. I had hoped to completely fix this this week, even permitting subs with >250,000 items cached, but the surgery needed to figure this out was more complicated than I expected. While I otherwise prepped for this next step, I also tightened up subscription timing code and wrote a new scheduler that reduces overall subscription resource needs.
Subscriptions will now try to load when they are due, and only then. So, if a query in your sub is due to check in 45 minutes, or if it has file to download but has run out bandwidth for 45 minutes, it should now run, typically, in 45-60 minutes. They will also respond better to cancelling and pause events, and if only 30 of 100 queries are due for a check in this run, or have outstanding files to download, you will now see x/30 on the popup progress text and gauge, just as better indication of how much work remains.
Previously, subs checked every four hours. It meant a bunch of wasted CPU for subs that were not due, late checks for subs that were due just moments after the check time, and also a delayed 'clumping up' of work. Today's change is subtle but significant, so I am interested in feedback from all users of subs--if you notice anything odd, bad, or great, please let me know. Ideally, subs will be quieter for you, running more often when you are not looking.
Although the data overhaul was too much work to do in one week, I now have a good plan and am in better position to get it done. I really want subs to load and save their metadata in milliseconds so the manage subs dialog can launch equally quickly and the client api can start talking to them. The year is fading, so I am not sure when this will get done, but hopefully this week's scheduling will at least relieve some of the work involved.
3 / I:
0 / P:
QSS Stylesheet Testing
Was asked to forward this here.
Hydrus using PySide2, QSS & CSS Stylesheets currently 'mostly' work as intended. However, all Sub-Panels including Docked Panels (media browser and the like with a centered title), Hover Panels (media viewer panels for example) ignore styling in QSS or CSS formats.
Here's a handful of linked Stylesheets to try during testing: https://wiki.qt.io/Gallery_of_Qt_CSS_Based_Styles
Qt Dark Orange was used in the image.
0 / I:
0 / P:
I had an ok week. Subscriptions now operate more efficiently, using less resources and choosing their sync times more precisely. Some simple Qt theming support is added for experimentation by advanced users, and a number of bugs are fixed, including page tab drag and drop for macOS users and anyone else with a center-aligned tab bar.
I have a busy day, so the release will be quite late tomorrow.
7 / I:
0 / P:
I had a great week. There are a bunch more Qt fixes, and a few other things as well.Qt
I have fixed a bunch more bugs in the Qt code. We are getting to the end now--this is mostly smaller stuff like an unusual dialog button not working, but I have fixed another important memory leak that was causing some backend not to be deleted correctly when a media viewer closed on a video. This should radically reduce memory use for some heavily used clients.
Some windows that were large, or could expand to be, like the options dialog on some pages, were sizing off the edge of the screen. This should be fixed now, and a variety of child-window initial size calculations should be a bit more accurate--Qt manages window size a little differently than wx, and the additional buffer represented by the window title frame and border was not being taken into account.
The menubar menus should work a little snappier this week. Things like the rapid 'pending (xxx)' menu updates when a downloader is importing files with tags. The whole UI should get a little latency benefit from this during these high-traffic times.
I also gave the layout and scaling another go on clients that use UI scaling (when you tell your OS to display UI at >100% on a monitor) on high dpi displays. The experiment from last week did not go how I wanted, with pixelly scaled-up thumbnails, so I have reworked that and fixed the taglist, which was crunching tags together on these displays.
0 / I:
0 / P:
trying to move my database and I get this error when I click "move entire database and all paths":
'DirDialog' object has no attribute 'SetPath'
File "include\ClientGUICommon.py", line 199, in EventButton
self._func( *self._args, **self._kwargs )
File "include\ClientGUIScrolledPanelsReview.py", line 551, in _MigrateDatabase
dlg.SetPath( source )
18 / I:
4 / P:
0 / I:
0 / P:
I had a great week. I have fixed some more Qt problems, including some bad window positioning, thin popup message widths, a video memory leak, and another round of improvements for high-dpi displays. Also are some improvements to the file maintenance system for recovering (including auto-redownloading) missing files, some better network retry timing code, and new downloaders for some broken sites.
The release should be as normal tomorrow. /hydrus/ is back up on 8kun, and I will be making a release post there as well as here.
56 / I:
13 / P:
Version 373 (Qt)
The Qt update is ready for Windows and Linux!This week's release is for all users, but please bear in mind it has some small layout and positioning bugs, such as subscription popups sizing a little strangely. If you would rather wait a week or two for these last issues to be cleared (and any others that pop up as more people play with this), that is totally fine.Qt background
Since hydrus began as an application, it has used wxWidgets to draw all the windows and buttons on screen. wx has served us well, but hydrus has grown to be a complicated program with hundreds of different custom things going on, and it was starting to show. Lots of windows were flickery, modern tech like 4k screens were not excellently supported, and operating systems and window managers were unstable. If I could have moved to a more flexible and more frequently updated UI library by snapping my fingers, I would have, but the total UI code is almost three megabytes, far too much to reasonably convert as I kept at my normal weekly schedule.
A user contacted me I think about a year ago talking about Qt and possibly making some scripts to automatically convert hydrus's wx code to Qt. I said it sounded like a good idea, and he worked in the background trying to figure it out and add manual tweaks. He was very successful, ultimately getting an essentially functional build going a couple of months ago. He passed the code to me four weeks ago, and I have since crash-learned Qt and fixed the great majority of the bugs that slipped through the automatic conversion process.
I am extremely grateful for this user's work--this would not have happened otherwise--and I am very happy with the result. Qt is a nicer library than wx for our purposes, runs faster, has much less flicker and related jank, and provides many new options for future extensions and customisation. I also enjoy working with Qt--the library is good.
0 / I:
0 / P:
I had a great week mostly working on a variety of bugs in the new Qt build. The majority of the jank and annoyances are fixed, performance and memory use are improved, and I have a macOS release ready.
The release should be as normal tomorrow.
2 / I:
2 / P:
Test Build of 373
THIS IS FOR ADVANCED USERS ONLY. IF YOU ARE JUST A REGULAR USER LOOKING TO UPDATE, THE PROPER RELEASE WILL BE OUT ON THE 13th.
Hey, here are test builds for Windows and Linux:https://mega.nz/#F!ojwgFa6C!ycjHGcP7FLDYe6hXNO16GQDO NOT USE THESE ON A REAL CLIENT. THERE ARE BUGS, AND YOU CANNOT ROLL BACK TO 372!
If you are interested, please feel free to extract one and try out a fresh client for five minutes. I have already received plenty of feedback on smaller layout issues and some broken controls, so I won’t waste your time asking for smaller reports, but if you are unable to run the client or import a handful of typical files, I would be interested to know what you get and what OS you are on.
Please forgive me for being late to replying to messages for the next week. There is still a good bit to do, but I feel good about having a decent release ready for the 13th.
3 / I:
0 / P:
wx->Qt Status Update 1
I had a good week. I continued to work on the Qt version of hydrus, fixing bugs and cleaning code. There is still a good bit to do. I now have working builds for Windows and Linux. I have a working OS X environment but need to put more work in to get a running App.
I still feel good about the schedule. I hope to have 373 ready for the 13th November. I will put out a preview build for advanced users to give feedback for on the 6th.
0 / I:
0 / P:
Qt Release Tomorrow!
I had a great and very busy four weeks. I appreciate everyone's patience.
The hydrus client is now switched over to a more stable and feature-rich Ui library, Qt. This is a big and important change behind the scenes, but the client looks and works essentially the same as before, though smoother and less flickery. The update process should be as normal--no special instructions. I also folded in a little normal work, including an important fix to the file maintenance system and some quality of life gui improvements.
The release is likely to be late tomorrow, as I want to do some more testing on my own system and with advanced users. I have a working macOS build, but it has some sizing/layout problems still to fix, so if I cannot nail them down, the macOS release will be delayed a week for 374.
1 / I:
0 / P:
wx->Qt Status Update 3
Things are going well with the update. I am coming to the end of my ‘must do before a release’ todo list. I feel good about having a decent release ready for the 13th, and hopefully with a couple of new bells and whistles thrown in.
I would still like to do a test release for advanced users, mostly to test that my builds will at least boot on various systems and there are no very unpleasant surprises. I had thought to do this later today, the 6th, but there are still some important things I want to do, and I do not want to rush it out just for the sake of it. I will now aim for Saturday the 9th.
18 / I:
12 / P:
4 / I:
0 / P:
wx->Qt Status Update 1
I had a good week. I received the Qt code from the user who put the conversion work in and have familiarised myself with the very basic differences between wx and Qt. The client boots and runs well. I also managed to create a working build for Windows today, so there do not seem to be any huge technical hurdles to getting this done.
There are still things for me to learn, bugs to deal with, and miscellaneous work (fixing layout issues and so on) until the release will be ready. I feel very good about things and believe I am on schedule to put out 373 on the 13th of November. I will put out a preview build for advanced users to give feedback for on the 6th.
11 / I:
1 / P:
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v371/Hydrus.Network.371.-.Windows.-.Installer.exeos x
I had a good week. There are several fixes, and I have added 'favourites' management for tag filters.favourite tag filters
The Tag Filter is the newish whitelist/blacklist panel that manages which tags will be parsed, blacklisted, migrated, or, more recently, hidden on the new tag display system. It is being used in more and more places, so I have added the ability to save and load certain filters for quicker re-access.
So, now every tag filter edit panel has save/load/delete buttons in the top-right. They all share the same list of favourites. I also put in some example defaults. Let me know if you run into any trouble with it!the rest
Thumbnails now sort by 'number of tags' and namespace sorts using the 'single' tag display context. So, if you have your thumbs set to hide page tags from the PTR, these page tags will no longer be included in a 'creator-series-volume-chapter-page' sort, for instance.
I fixed several smaller issues--a kind of video that would not import correctly, an unusual 'options' saving error after setting a number of ratings on several thumbnails, seeing duplicate thumbnails on some downloader highlights, gallery downloads that stopped early due to malformed post URLs, download pages sometimes secretly performing a collect and sort when a default collection is set, the media viewer being unable to show certain malformed files, and remote thumbnails not refreshing correctly when their respective file is imported.
Advanced users only: URL Classes now let you set whether requests for URLs matching the class will send a 'referer' http header. Normally, hydrus will send any 'previous' URL as the 'referer' (typically the gallery page when visiting a post page, or the post page when downloading a file), just as if you were visiting it in your web browser, but you can set it to never send any referrer, always use an auto-generated referrer, or use the auto-generated one when no natural referrer is available (which can happen on a user drag-and-drop URL import). The auto-generated referrer is a standard StringConverter operation on the URL being visited itself--if you can figure out a way to infer it from the original URL, or just replace it with a static value, I think some more perculiar downloaders should be possible.
1 / I:
0 / P:
I had an ok week. I mostly fixed bugs to make a nice 'clean' final release in wx.
The release should be as normal tomorrow. It will be the last release for about four weeks, as the next job will be finalising an large and long-planned wx->Qt (UI library) conversion.
22 / I:
2 / P:
0 / I:
0 / P:
I had a good week. I mostly fixed things and cleaned code, and I added 'favourite' tag filters to make it easier to manage and reload common filters.
The release should be as normal tomorrow.
25 / I:
5 / P:
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v369/Hydrus.Network.369.-.Windows.-.Installer.exeos x
I had a good week. The file maintenance system works better, and files now know their 'modified' dates.file maintenance
The new file maintenance system now works smarter and more productively. Previously, it did rare bursts of hard work in idle time, and now it does small amounts of work all the time. You do not have to do anything--it will now keep up with its behind-the-scenes queues more quickly and quietly, hopefully without giving you any hassle. The only significant difference is file maintenance no longer occurs on shutdown. If you are not interested in any details, you can skip the following completely.
Its settings under options->maintenance and processing
have changed a little--rather than the old '200 files per day' default, it now has faster '1 work unit per 2 seconds/20 seconds' for idle/normal time. These values should not tax your system or interfere with other programs or interrupt your browsing. If you have a faster or slower computer, feel free to change them. If the work does make your browsing juddery, please do turn off work in normal time, which you can do in the options or now more quickly at database->maintain->file maintenance
. The 'work unit' is part of a new weighting system that lets fast jobs--like checking for file presence on the hard drive--count for less work in the throttle and process faster than bigger jobs like regenerating thumbnails
When I added the 'go through all video files and scan for has_audio property' job some weeks ago, it added 100,000+ jobs for many users. The 200/day default was not cutting it, so this new 'lots of light work all the time' mode should get these big jobs done in reasonable time without going nuts. It also updates thumbnails in real-time, so there is the smallest chance you'll see one of the new 'has audio' icons fade into one of your thumbnails right as you look at it! The new processing also helps for another large retroactive job being added this week--figuring out file modified dates:
0 / I:
0 / P:
I had a great week. As well as a variety of fixes and little improvements like new 'duration' icons on thumbnail and system:modified date, I finished the first version of the new tag display/filtering system. You can now hide specific tags and namespaces from the 'tag selection' and media viewer lists separately, and a variety of tag tasks like media selection in a big thumbnail view are a bit faster. There is also an important fix to the new asynchronous tag repository processing.
The release should be as normal tomorrow.
1 / I:
0 / P:
1 / I:
0 / P:
Co-development generals thread
13 / I:
2 / P:
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v368/Hydrus.Network.368.-.Windows.-.Installer.exeos x
I had a great week. The PTR has moved successfully, and I got multiple local tag services working.PTR has moved
The Public Tag Repository has changed management. I am no longer running it or involved in working as a janitor. There is more information here:>>62
As a result, the PTR no longer has bandwidth limits! The user now running it is also putting together a janitorial team to catch up on petitions as well. About ten million delete mapping petitions and six thousand add sibling petitions had piled up! If you would like to talk to the new management, they are available on the discord. The current plan is to keep running the PTR with the same loose rules as I did--the main concern was overcoming my bandwidth limits.
If you currently sync with the PTR, you will be given a yes/no dialog when you update asking if you would like to keep using the PTR at the new location. If you select yes, the client will automatically update your service (the only credentials difference is that instead of it being at hydrus.no-ip.org, it is now at ptr.hydrus.network), and you will keep syncing and be able to continue uploading without skipping a beat. If you select no, your PTR stay as-is but will pause. If you still sync with my read-only test file repository, this will be paused automatically.
While I am very thankful for the 650 million submissions I have had to the PTR over the past seven years, it is a load off my mind to no longer be responsible for it. I much prefer being a developer than an administrator and hope to use the extra time and upcoming feedback to work on improving the admin side of hydrus repositories, which, due to me being the primary user, have always had debug-tier UI and half-broken features.
The various areas in the help have been updated to reflect that the PTR is no longer mine, and the quick setup under the help menu is now just help->add the public tag repository
0 / I:
0 / P:
I had a good week. As well as some smaller fixes and improvements, I added file 'modified date' metadata to the client, letting you see and sort by it, and improved the file maintenance system (which will retroactively generate modified date info and catch up on the existing 'has audio' detection queue) to work quietly all the time in the background, so its gets much more work done with much less hassle.
The release should be as normal tomorrow.
13 / I:
0 / P:
6 / I:
1 / P:
The PTR will undergo a change of management in two weeks
I have run the PTR since the start of hydrus, seven or so years now. I remember hitting the first 1,000 tag mappings, then being stunned at 1,000,000, and now we are at 650,000,000. It was always meant to be a test prototype, but I kept it going because it was so neat seeing the tech work, getting positive feedback, and dealing with all the interesting problems millions and then hundreds of millions of tags presented.
Unfortunately, as it has grown, it has become increasingly unwieldy to manage by myself. I have fallen behind on dealing with petitions, and I have increasing bandwidth problems you have no doubt noticed. I have never been an excellent server administrator, and I don't like being a single source of failure, either. A user who has more bandwidth has kindly offered to run it in my stead, so therefore I will be shutting it down on my end, likely on September 16th, in time for version 368.
I will sanitize the database so it has no more information than a syncing client currently knows (there have never been IP logs or anything, but I will do some additional work such as deleting the petition 'reasons' users have submitted over the years, some of which have personal joking messages to me) and put the frozen database up on a Mega account with scripts to generate new admin access keys and so on. Any user who wants to download it and continue their own tag repository with its data, whether that is private or public, will be able to. I will also create some Hydrus Tag Archives for the tags and parents and siblings for users who are separately interested in that data.
0 / I:
0 / P:
I had a great week. The PTR has moved to its new higher bandwidth location and management successfully. If you sync with the PTR, tomorrow's client will offer to automatically move your settings to the new location and you will be able to continue syncing and uploading tags without skipping a beat. Also, I was able to get multiple local tag services working completely!
The release should be as normal tomorrow.
1 / I:
0 / P:
nope, running windows 7 and from installer, no idea why it wont import for you, but it was already in my db and not really sure where I would find it again to dick around.
I got somewhat done with the image thing I was doinghttps://mega.nz/#!Dl4ABYwL!t53V4rjD7nqPYrPq-PYDT4ug-Uo67B0OfvnCOIpXdqc
its I think 9 images, with source files where I ran them through waifux2 at 1.0 magnification trying to see where they started to fall apart in quality, some of them go fairly low before really shitting the bed, some of them aren't able to handle the conversion to jpeg at all.
one of the images I went x2 on source because I have seen a lot of jpegs where they are twice the resolution, about the same file size but far higher quality then the one that should have been higher quality.
the main reason I did this was a thought. now, i'm not sure about you but I have a fuckload of images that I could run though this and get a reasonable file out of, but the only way to do this in bulk is to copy every image from the archive out, convert them 1 by 1 and zoom in on them which is honestly a shit work flow.
if this was capable in hydrus, you could have the source image convert the image, then save both images to a duplicate detector, hell if you wanted to get anal about it, you could have them go
20 - 40 - 60 - 80 - 100 and you pick where the image falls apart, then from there it goes into more fine grain. like lets say 40 was where it fell apart, then it would explode 40-60 with 40-45-50-55-60 and you figure out where the image goes to shit. most images seem to be good with about 1/5 to 1/10 their file size.
This would also put waifux2 in the program, so images that are fucked that you like could potentially be fixed inside.
each one of these going through a 'dup detector' assigned to conversions, so you can manually tell it 'yea, this worked' as this workflow is far less of a dick pain as trying to compare source to conversion out of program, you could also, in the case of the overwatch image, likely shrink it in waifux2, haven't tried, shit took up way to much ram to work with that monstrosity.
18 / I:
2 / P: