Catalog of /hydrus/

Bottom

Mode: Thread

Max file size: 350.00 MB

Max files: 5

Remember to follow the rules

Max message length: 4096

Open

R: 0 / I: 0 / P: 1

Version 566

https://youtube.com/watch?v=PlvK2pabBqI [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v566/Hydrus.Network.566.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v566/Hydrus.Network.566.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v566/Hydrus.Network.566.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v566/Hydrus.Network.566.-.Linux.-.Executable.tar.zst

I had a good week. The long-awaited incremental tagger is ready, and the program supports some more document types. You will get a yes/no on update, but it isn't a big deal.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

incremental tagger

When you open the manage tags dialog on several thumbnails, it now has a '±' button that lets you tag the files 'page:1', 'page:2', 'page:3' and so on. You can set the namespace (or no namespace), the starting point (so you can start at 'page:18' if you need to), the 'step' (so you can count by +2, or even -1 to decrement), and even prefix/suffix for the number if you need to decorate with 'page:x (wip)' or something.

I'm quite happy with how this worked out. There's some live text that gives you a preview of what's about to happen, so the best way to get to grips with it is to play with it. Just click and poke around, and let me know how you get on. Namespace will be remembered between opens, and if the first file in your selection has a number tag for that namespace, it'll set the 'start' position to that value. If you are setting page tags to a bunch of chapters, or gaps in a larger body, a bit of prep/overlap may help things here.

We've wanted this for years and years, and while I'm not expecting the program to handle paged content beautifully now, this is a decent step forward.

docx and friends

The document types .docx, .xlsx, and .pptx, which are the newer Microsoft Office 'Open' formats, are now recognised by the client. These are secretly zips, so there's a chance you have some in your client already. On update, you'll be asked if you want to scan your client's existing zips to see if they were really one of these. You probably don't, so it isn't a big deal, but if you think you do, hit yes and they should appear.

next week

Cloudflare are rolling out some annoying new cache-optimising tech that is causing file dupes, so I need to work on URL handling so we can patch the most-affected sites.
Open

R: 1 / I: 0 / P: 1

Version 565

https://youtube.com/watch?v=lgpD6OsHCKU [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v565a/Hydrus.Network.565a.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v565a/Hydrus.Network.565a.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v565a/Hydrus.Network.565a.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v565a/Hydrus.Network.565a.-.Linux.-.Executable.tar.zst

I had a simple week. Lots of small changes today. The update step may take a couple of minutes.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

You are going to get a couple yes/no dialogs on update this week talking about deleting some mis-parsed URLs. If you do not manually store weird data in your 'known urls' store, just click yes. If you have lots of URLs, the work will take a couple of minutes.

options->sort/collect now has four different default tag sort widgets! You can set the default tag sort for search pages, the media viewer, and the manage tags dialogs launched off them.

There's a new 'media' shortcut action 'copy file known urls', that copies all the known urls of your current media selection.

Sidecars set to import to file notes now have an optional 'forced name' field, so if you ave a .txt file with only note text, no name, you can now force it. Some of the UI is less jank here too.

The tag filter UI also got a little polish. There's less logic jank, better labels and tooltips, and you can now copy listed namespace entries to the clipboard and get something you can paste back in elsewhere.

next week

I did not get to the manage tags dialog incremental tagging thing this week, so I'll try again.
Open

R: 1 / I: 0 / P: 1

Version 564

https://youtube.com/watch?v=Oo0o84-TJTU [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v564/Hydrus.Network.564.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v564/Hydrus.Network.564.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v564/Hydrus.Network.564.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v564/Hydrus.Network.564.-.Linux.-.Executable.tar.zst

I had a good week, several system predicates have better range-based searching.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

The system predicates for width, height, num_notes, num_words, num_urls, num_frames, duration, and framerate now support two different kinds of approximate equals (≈): absolute (±x), and percentage (±x%). Previously, the ≈ secretly just did ±15% in all cases, but now you set how and how far they go.

Also, any 'system:framerate' predicate that was '=' will now be converted to ±5%, which is what it was secretly doing before, and any 'system:duration' '=' predicate will also be converted to ±5%, which is what it really should have been doing before. 'system:duration' also allows hours/minutes input, for longer videos.

This predicate overhaul was an important cleanup job, replacing a ton of hacky ancient code with something that is easier to update and maintain. I've collapsed all these preds down to a lot of shared UI and logic, so let me know if there are any display/search quirks, but once we have it nailed down, I hope to replicate this work for the more complicated system predicates.

I reworked what 'Space' does in the media viewer by default. I am updating existing users too, so you'll probably get a little popup about it when you update. Essentially, if you are still on the default shortcuts, Space will now only send 'pause/play media'. It no longer does 'yes' on the archive/delete filter or 'this is better' on the duplicate filter. If you want to go back to how it was, sorry for the trouble--hit up file->shortcuts to set it back.

Thanks to a user, space also does a new 'Quick Look' for macOS users on thumbnails. Try it out! If you are a 'running from source' macOS user, make sure to rebuild your venv this week, or it won't work!

next week

I'd like to figure out incremental tagging on the manage tags dialog, so you can select 20 files and tag them page:7 through to page:26 in one step. Let's see how it goes.

R: 5 / I: 4 / P: 1

html sidecar attempt

Hey,

Since my last message, I recently had time to work with Hydrus, and I developed a semi-functional parser for my HTML files. However, I don't want to include the description string in the tags, so I need to remove it. However, I'm not sure how to do this. I assume that it has something to do with the "string selector/slicer" component, but I'm not sure how that component works. I managed to do an incredibly crude filter by regex filtering by ^[^<]+$, which blocks the < in html tags, but this will presumably fail for any description that doesn't include html tags, for whatever reason. I would really appreciate your advice on this. If you need to see my (bad) code, I have attached the code to copy as a text file.

Thanks!

Ps: I can't import sidecar parsers through pngs, as the file selection dialogue doesn't show png files, just folders. Is there something I'm doing wrong?
Open

R: 1 / I: 0 / P: 1

Version 563

https://youtube.com/watch?v=XXlzWhBvYwg [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v563/Hydrus.Network.563.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v563/Hydrus.Network.563.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v563/Hydrus.Network.563.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v563/Hydrus.Network.563.-.Linux.-.Executable.tar.zst

I had a good week. There's a mix of small fixes and improvements.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

macOS

A user has improved the macOS release in many ways, mostly in brushing up the App to normal macOS standards. The menubar should now plug into native global bar, with some standardised command labels. The program icon is better, some colours should be improved, dialog menus are no longer a crazy hack by default, and the system tray icon is turned on. In a side thing, I also added Cmd+W to close pretty much any dialog or non-main-gui window in macOS, just like hitting Escape.

I think we can expect more work here, so let us know how you get on!

all misc

In more shortcut news, I've added Alt+Home/End/Left/Right to the 'thumbnails' defaults to perform the new thumbnail rearranging. Existing users will also get these! (assuming it doesn't conflict with something you already have set)

Also, the shortcut system now, by default, converts the 'numpad' variants of non-number key presses (think 'numpad Home' or 'numpad Return') into the non-numpad 'normal' ones. You now only need to map 'Left' or 'Delete' once when setting a shortcut, and, no matter how crazy your keyboard's internal mapping is, it should just work. If you do need to differentiate between the numpad and normal variants of these keys, you can turn this new behaviour off under options->shortcuts.

The menu on a 'file log' button now lets you delete all the remaining 'unknown' items or set them all to skipped.

I fixed another damn problem with copy/pasting timestamps into the manage timestamps dialog. When you paste a timestamp, it should 'stick' better now!

If you have had problems with mpv sometimes going silent on 'every other video' or had your windows being rescued from 'off screen' even though they were supposed to appear just on some monitor, check the changelog for some special new BUGFIX options.

next week

I want to put some work into system predicates, specifically startng with 'system:duration'. The aim is to eventually get rid of all the +/-15% '~=' stuff and replace it with actual customisable values, along with better behind the scenes storage of that data. We'll see how it goes!
Open

R: 1 / I: 0 / P: 1

Version 562

https://youtube.com/watch?v=XnwHtzBf-c8 [Embed]
(go to 4:44:00 for a wild fight)
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v562/Hydrus.Network.562.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v562/Hydrus.Network.562.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v562/Hydrus.Network.562.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v562/Hydrus.Network.562.-.Linux.-.Executable.tar.zst

I had a tepid week, but there's some decent fixes and quality of life improvements.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

all misc this week

I fixed a stupid typo error in the manage times dialog when you go into a single time and try to copy/paste the timestamp. The buttons also add millisecond data now.

Fingers crossed, drag and drops of thumbnails and page tabs will feel snappier this week.

You might see some heavy 'analyze' database maintenance coming down the pipe. Let me know if it proves annoying, or if you even notice it at all--my hope is to iron out some super slow PTR-based tag updates that hit some users, but we'll see how it goes.

If you are an AUR user or otherwise updated to a very new Qt version (6.6.1) and suddenly the column widths of multi-column lists all went ~100px wide, I think I've fixed it! If you were affected by this, I can't recover your old settings, but recall that you can right-click any list header to reset the widths to the default.

next week

I was short on work time this week, so I'll try to hack away at github bug reports again.

R: 2 / I: 0 / P: 1

html sidecars - possible?

Hey,

I was recently trying to import some files into hydrus that I had downloaded using gallery-dl. Thanks to a config-file mistake, all the tag files were saved as html pages. Is there a way to use regex to extract the tags? I attached one of the html files, if that helps.

Thanks!
Open

R: 1 / I: 0 / P: 1

Version 561

https://youtube.com/watch?v=GuVnZZ1sFIc [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v561/Hydrus.Network.561.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v561/Hydrus.Network.561.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v561/Hydrus.Network.561.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v561/Hydrus.Network.561.-.Linux.-.Executable.tar.zst

I had a mixed week. Thumbnail rearranging is added, and some bugs and quality of life issues are cleared.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

thumbnail rearranging

So, if you right-click any thumbnail, you'll now see a 'move' menu. This makes it simple to rearrange your thumbnails. You can move your current selection to the start, the end, left one, right one, or 'to here' if you have multiple selected. If your multiple selection is non-contiguous, it will be made so on move, with the move focusing around the position of the first selected item.

You can also map these commands to keyboard shortcuts under the 'thumbnails' shortcuts set. I haven't added any default shortcuts for this yet, but let me know if and what you would prefer--I've been playing around with ctrl+numpad numbers on my dev machine, and it feels nice to me.

In future I'll try and figure out mouse drag-and-drop rearranging. It might have to wait for a larger pending rewrite of the thumbnail grid though--we'll see.

other highlights

Unfortunately, there were a couple of stupid typos in the content processing changes last week. One in the duplicate filter (which the v560a hotfix addressed), and another fixed today that was causing 'already in db' results to not get metadata updates correctly. Sorry for the trouble, and thank you to the users who reported these.

Ctrl+C/Ctrl+Insert is now hardcoded to copy tags from the taglist.

The thumbnail and media viewer menus should now be much thinner. I hate how wide they can get, how annoying it is to hit their many nested submenus when they get like that, so let me know if they still go crazy in some situations.

There was a bitrot issue in v559, the millisecond timestamps conversion, that made it impossible/bad to update from a much older version. This has always been a tricky technical issue to talk about and communicate to the user, so I've now written a better in-client error reporting process that stops the user before the update is even attempted. The upshot this week is that if your client is v551 or older and you try to update to v561 or later, you will be told to update to v558 first.

In lieu of a proper rewrite, I've made some semi-hacky newline processing improvements to the parsing system. If you make downloaders and hate having to deal with extra whitespace in multiline content, notes or otherwise, check out the full changelog and let me know how you get on with it all.

next week

I want to work on github bug reports, which I haven't put proper time into for too long!
Open

R: 3 / I: 0 / P: 1

Version 560

https://youtube.com/watch?v=UBYThAP_C5A [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v560/Hydrus.Network.560.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v560/Hydrus.Network.560.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v560/Hydrus.Network.560.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v560/Hydrus.Network.560.-.Linux.-.Executable.tar.zst

I was suddenly without internet for a while, so there was no release last week, but everything is back to normal now, and I had a great couple weeks' work. The 'edit times' dialog can now handle multiple files.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

editing times for multiple files at once

If you launch 'edit times' on a single file, everything is the same. If you open it on a larger selection, it will now show, in a summarised way, the range of each particular time type, and how many files in the selection have or do not have a time set there. The controls all work as before, but in general, when you set a time, all files are now locked to that new time. Give it a play and you'll see how it all works.

SInce we don't always want to set exactly the same time to a set of files, but rather ideally a similar, staggered time, there is also a new 'cascade step' system in the multiple file version of the dialog. Every time-edit you open up has the ability to enter a little step, say 100ms, which will cause the dialog to set each successive file in the selection to be that much later (or earlier, negative values are allowed!) than the last. This way, if you have a bunch of files of something contiguous, like a comic chapter, that all have different, merged, or otherwise awkward import times, you can now manually sort them in the UI using tags or hackery and apply a new import time based on the first file plus a few milliseconds each time. Thereafter, any time you sort those files by import time, they'll also be nicely in page order time, just as if you had imported them all together from one directory.

I am quite happy with the new dialog, and I plan to copy some of the new techniques I figured out to other dialogs for similar better multi-file handling. I'd also really like, as we have discussed before, to tackle a 'cascading' tagger in 'manage tags', so we can quickly set 'page:n' tags to many files at once.
Open

R: 3 / I: 0 / P: 1

Version 559

https://youtube.com/watch?v=zBuJD-ugT_g [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v559/Hydrus.Network.559.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v559/Hydrus.Network.559.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v559/Hydrus.Network.559.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v559/Hydrus.Network.559.-.Linux.-.Executable.tar.zst

I had a great week working mostly on one thing: converting the timestamps in the program from seconds to milliseconds.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

milliseconds

tl;dr: time numbers better, you don't have to do anything

Since the program started, it has tracked the various file 'times', like import and archive time, using a system that stores time with second precision. It knows that file x was imported at 2023-12-02 14:04:51, but nothing finer. This is common in many computing applications, but sometimes you want more. Particularly, in hydrus, whenever you have a fast importer that brings in two files within the same second, the program does not know, later, which was actually imported first (e.g. when you sort by 'imported time'). This can be annoying when you import a whole chapter of a comic and don't have good page tags set up yet--random pairs of pages will be flipped whenever you next try to sort those files by 'import time'.

So, this week I converted the database to use a time system that has millisecond resolution. Any files imported from now on, or deleted, archived, modified, or 'last viewed', will now get a millisecond timestamp. You don't have to do anything, and nothing outside of the 'edit file timestamps' dialog will appear any different, but it may take a minute to update.

This work was quite simple, but there was a ton of it, years and years of system build-up to go through. Rather than make an already messy system more complicated with a rough injection, I decided to invest the time and clean everything as I moved to the new format. I've tested it back and forth, but given the number of changes, I wouldn't be surprised there is a typo in there somewhere--if you see a file saying it was last viewed '54 years ago' or similar, let me know!

Also, the Client API can now edit timestamps, including with the new millisecond precision. If you are an API dev, check out the new call (and 'edit file times' permission) here: https://hydrusnetwork.github.io/hydrus/developer_api.html#edit_times_set_time

misc

Just a side note, I have removed the sankaku downloader defaults from the program, so new users won't see them. None of them were working well/at all, especially in recent weeks. If you want to grab from complicated sites, and there isn't a clean solution on the shared downloader repo here, https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts , I recommend going with a more robust solution like gallery-dl/hydownloader or just finding the content elsewhere.

If you ever run something like 'system:known url: regex = blah.com/(regex stuff)' and are annoyed at how slow it is, try adding a 'system:known url: domain = blah.com' pred in addition. This combination will now be explicitly much faster than just the bare regex (previously, it could be faster, by accident).

next week

Now we have ms timestamps and nicer code, I'd like to try tackling an 'edit timestamps' dialog that works for multiple files, including a 'cascade' command for force some clever sorts.
Open

R: 5 / I: 0 / P: 2

Version 558

https://youtube.com/watch?v=crJst7Yfzj4 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v558/Hydrus.Network.558.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v558/Hydrus.Network.558.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v558/Hydrus.Network.558.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v558/Hydrus.Network.558.-.Linux.-.Executable.tar.zst

I had an ok week. I figured out 'system:number of urls', and you can now import rtf files!

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

So, you can now search for 'num URLs'. You will find it under a new stub, 'system:urls', at the bottom of the normal system predicate list (where 'system:known urls' has also moved). This first version is simple--it counts all URLs, regardless of how important, but I can see a future version having the ability to scan by 'post URLs' or specific URL classes. Give it a go and let me know if you run into any trouble!

Also, thanks to a user, you can now import rtf files! There's no word count yet, but I think it should be doable in future.

If you select multiple tags, you can now hide or favourite them all at once. There's a bit of workflow and presentation improvement here, too.

The old 'tag suggestions' box called 'favourites' is now called 'most used'. It was stupid to have two tag systems called 'favourites', so this is now fixed. You still edit them under options->tag suggestions; they just have a more precise and less confusing name now.

If you are an advanced macOS user (i.e. you know how to build something with xcode), you might like to check out Hydra Vista, a new user-made macOS App that presents your client (via the Client API, e.g., perhaps on another computer) in a booru-like wrapper. https://github.com/konkrotte/hydravista

next week

I would like to add timestamp editing to the Client API.
Open

R: 6 / I: 0 / P: 2

Version 557

https://youtube.com/watch?v=xNKXyr6YN8Y [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v557/Hydrus.Network.557.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v557/Hydrus.Network.557.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v557/Hydrus.Network.557.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v557/Hydrus.Network.557.-.Linux.-.Executable.tar.zst

I had a good week back after the holiday. There are some bug fixes and improvements to system:hash parsing.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

Tag filters (which operate the various tag whitelists/blacklists across the program) now edit much much faster when they are full up with stuff. You can paste 5,000 items into an empty tag filter in less than a second now, and removing one or a handful of items from a filter with thousands of items already in is now instant. Previously, these things could take many seconds or even minutes due to inefficient overhead.

CBZs that have four or fewer pages should now be recognised correctly.

When you copy 'system:hash' and the 'system:similar to' predicates to clipboard, you now get a much longer string that includes all the hashes in the predicate. These copied strings are all parsable, meaning you can paste them into the same client elsewhere or a different client entirely and it should all just work in a complete loop. The human labels are updated to give you more information, too.

There may be other predicates that are currently parsable if you type carefully but do not themselves copy a parsable string, but I am not sure which they are, so if you run into one, let me know and I'll see what I can do.

If you run from source on Windows--or you'd like to--but you haven't installed the extremely convenient but bulky-to-install 'Git for Windows', I had to go through the installer again to set up my new dev machine, and I wrote out a guide for the full 12-page wizard here: https://hydrusnetwork.github.io/hydrus/running_from_source.html#core It is actually easy to do, and almost everything can be left as default and things will be fine. In any case, Git is great and lets you update in about three seconds, so if you run from source on Windows, give it a go!

next week

My old dev machine died just before Christmas, but I bought a new one and am back to normal work. For next week, I'd like to get some sort of system:num_urls figured out.
Open

R: 3 / I: 0 / P: 2

Version 556

https://youtube.com/watch?v=e3mkeK6pRaY [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v556/Hydrus.Network.556.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v556/Hydrus.Network.556.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v556/Hydrus.Network.556.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v556/Hydrus.Network.556.-.Linux.-.Executable.tar.zst

I had an ok week. I fixed some bugs and added a system to force-set filetypes.

You will be asked on update if you want to regenerate some animation thumbnails. The popup explains the decision, I recommend 'yes'.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

forced filetype

The difference between a zip and an Ugoira and a cbz is not perfectly clear cut. I am happy with the current filetype scanner--and there are a couple more improvements this week--but I'm sure there will always be some fuzziness in the difficult cases. This also applies to some clever other situations, like files that secretly store a zip concatenated after a jpeg. You might want that file to be considered something other than hydrus thinks it technically is.

So, on any file selection, you can now hit right-click->manage->force filetype. You can set any file to be seen as any other file. The changes take affect immediately, are reflected in presentation and system:filetype searches, and the files themselves will be renamed on disk (to aid 'open externally'). The original filetype is remembered, and everything is easily undoable through the same dialog.

Also added is 'system:has forced filetype', under the 'system:file properties' entry, if you'd like to find what you have set one way or the other.

This is experimental, and I don't recommend it for the most casual users, but if you are comfortable, have a play with it. I still need to write better error handling for complete nonsense cases (e.g. calling a webm a krita is probably going to raise an error somewhere), but let me know how you get on.

other highlights

I fixed some dodgy numbers in Mr. Bones (deleted file count) and the file history chart (inbox/archive count). If you have had some whack results here, let me know if things are any better! If they aren't, does changing the search to something more specific than 'all my files'/'system:everything' improve things?

Some new boot errors, mostly related to missing database components, are now handled with nicer dialog prompts, and even interactive console prompts serverside.

I _may_ have fixed/relieved the 'program is hung when restored from minimise to system tray' issue, but I am not confident. If you still have this, let me know how things are now. If you still get a hang, more info on what your client was doing during the minimise would help--I just cannot reproduce this problem reliably.

Thanks to a user who figured out all the build script stuff, the Docker package is now Alpine 3.19. The Docker package should have newer libraries and broader file support.
Open

R: 1 / I: 0 / P: 2

Version 555

https://youtube.com/watch?v=BE4ptZRUJZo [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v555/Hydrus.Network.555.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v555/Hydrus.Network.555.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v555/Hydrus.Network.555.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v555/Hydrus.Network.555.-.Linux.-.Executable.tar.zst

GET

I had a good week. There's some cbz/Ugoira follow-up, nicer system:time parsing, and much better boot error-handling.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

The CBZ/Ugoira stuff last week went ok! We had a few too many false positive Ugoiras, so that test is tightened up, and those files should soon become CBZs. For false positive CBZs (i.e. just a zip of CG images or similar), we don't have a good automatic solution, but I still plan to roll out 'force this file to be this filetype' tech in future so we can, amongst other things, assign these fixes manually.

The various 'system:archived since xxx' predicates' parsing is now plugged into the same excellent date parser we are using in the downloader system. If you type them in, they'll now take all sorts of date phrasing. Try 'since 01/05/2016' or 'before june 2022' into the autocomplete--you may even be able to use your own language!

There is still a little work to do with the 'since/before x time units ago' variants, though. In some unclear cases (including foreign languages), the before/since may be flipped to what you type. Also, I have decided to soon migrate these predicates to just store days and hours (no more year/month). You can still enter '1 year ago' with this new parser, but on my end, trying to calculate leap years and weird month durations has caused too many problems, so I am going to simply pull back over the near future and let you put 365 or 30 in yourself! In any case, give this stuff a go and let me know how you get on.

When the hydrus client fails to boot really early on, before the main UI system is live, it should now nonetheless pop up a dialog saying what happened! The only way this will fail is if the problem with the boot is the Qt UI library, lol. In either case, the 'hydrus_crash.log' file is still made on your desktop.

After talking it out with users, I have decided to move towards dropping the image library OpenCV from the program. It has served us well, but it is often difficult to install and a bloat, and our flexible alternative, Pillow, works extremely well these days. I'm not ready to flick the switch yet, but we have done work here and there, and if you would like to help me test this out, please hit the 'IN TESTING: Load images with PIL' checkbox under options->media and let me know if you have any images that suddenly load incorrectly.

The String Splitter and Joiner objects in the parsing system now accept \n and \t for newline and tab. If you need to split or join by \, use \\. To not break any existing parsers, existing objects that have a \ have been updated to have \\.

next week

I only have two more weeks in the year, so I'll just do some cleanup and little jobs.
Open

R: 1 / I: 0 / P: 2

Version 554

https://youtube.com/watch?v=HUVtoWIeyp8 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v554/Hydrus.Network.554.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v554/Hydrus.Network.554.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v554/Hydrus.Network.554.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v554/Hydrus.Network.554.-.Linux.-.Executable.tar.zst

I had an excellent week. Some important bugs are fixed, and we have some basic support for CBZ and Ugoira files.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

bugs

I screwed up two important calls last week, in fixed-period checker timers and job statuses. If you had watchers not start, subscriptions with incorrect check times, and/or popup messages that threw display errors or wouldn't auto-dismiss, I am sorry! I didn't plan the changes here properly, and several things slipped through my tests.

All the affected systems have been given proper rework this week and have some unit tests to make sure this doesn't happen again. Please let me know if you still have any problems.

animations

I fixed up more of the 'while checking for transparency, this file produced x error!' issues. Checking GIFs for transparency should be a bit faster and more fault tolerant too, now.

Also, the native GIF renderer has much improved transparency support. The multilayered 'noclip' artifacts should be gone, and damaged GIFs will recover better. GIFs also now get thumbnails that are x% in with proper transparency.

Also, APNGs now get transparency: when rendering in the native renderer; in their thumbnails (which are also now x% in); and for 'has transparency' checks.

CBZ and Ugoira

I am adding basic recognition and thumbnails for these filetypes today. Behind the scenes, both formats are essentially just zips with a list of images, so all your zips will be scanned, and if they look like a CBZ or Ugoira, their filetype will change and they will get a thumbnail. Ugoira thumbnails will be x% in, like for other video.

Also, a user is working on true Ugoira rendering now, so I hope we will be able to finally roll this out in the medium term.

Unfortunately, neither format has a particularly definitive/unique specification, so while I have tried to be careful, my tests here are imperfect. We can expect a few incorrect determinations one way or the other. If you get an outrageous false positive or false negative here (e.g. something you know is a Ugoira that stays as a ZIP, or a ZIP of misc files that detects as a CBZ), please send in the details, and I'll see if I can tweak my tests.

next week

I put in extra time this week to figure out CBZ, so I'll let things breathe and just catch up on simple, small work. I have a bunch of interesting quality of life UI items in my immediate todo, so I'll probably focus on that!
Open

R: 1 / I: 0 / P: 2

Version 553

https://youtube.com/watch?v=FhGziRJY730 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v553/Hydrus.Network.553.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v553/Hydrus.Network.553.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v553/Hydrus.Network.553.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v553/Hydrus.Network.553.-.Linux.-.Executable.tar.zst

I had a great week. The issues with gifs are fixed, and slideshows with videos are smoother.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

I was happy with the 'has transparency' work last week, but unfortunately some damaged animated gifs (usually where one frame was borked) were raising a serious error when then were inspected. This error, which was interpreted as a hard drive fault, was a false positive! If you got it (should have been a popup talking about an 'I/O Error', and that maintenance work would halt until the next boot), do not worry--it wasn't a big deal. I have fixed up the error handling and a bunch of other gif issues. Sorry for the trouble!

There are several new slideshow timing options under options->media. You don't need to tweak them unless you really care, but now, when you do a slideshow that involves video or audio, the transitions should happen earlier or a little later in order to line up with where the video naturally ends. Check the changelog and tooltips of the new widgets for full details.

The various 'import options' on downloaders now say '(all default)', '(some set)', and '(all set)' on their labels, so you can quickly, at a glance, see what you have set where.

The thumbnail manage->regenerate menu is now called maintenance, and it now lists all the possible file maintenance commands. Should make it easier to fix weird problems.

next week

I have some server work to do. Hopefully clear out some github bug reports too.

I also just realised we only have about four more releases before the year is done--it feels to me like the second half of this year has flown by. I'll try and fit in one more medium-sized job before then.
Open

R: 1 / I: 0 / P: 2

https://youtube.com/watch?v=VE2NCPnULA0 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v552a/Hydrus.Network.552a.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v552a/Hydrus.Network.552a.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v552a/Hydrus.Network.552a.-.macOS.-.App.dmg
linux
tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v552a/Hydrus.Network.552a.-.Linux.-.Executable.tar.zst

I had a great week. There's a bunch of small fixes and improvements, and the addition of 'system:has transparency' search.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

has transparency

So, the database can now remember if a file has transparency. This might mean a file with fully transparent pixels, or it might just be an area that is semi-transparent. You can now search it under the new 'system:file properties' entry, selecting 'system:has/no transparency' (or you can just type it).

Like 'has exif' and others before it, 'has transparency' will be correct for all new files instantly, but figuring it out for your existing files will take a bit of background maintenance work. It will take some time for the full results to populate. You can review how much it has still to do under database->file maintenance->manage scheduled jobs.

Note that this is not a 'this file is RGBA' test. There are a bunch of files out there with an alpha channel that is all 100% opaque, so my 'has transparency' check actually looks for impactful transparency information in the image. I'm also keeping it simple to start with, so we are scanning all images except jpegs and animated gifs. No Krita or PSD or anything yet--I'm not sure if our various rendering hacks are even able to pass along accurate alpha info.

As always, I'm interested in seeing any unusual files that fail the test. However, while doing this work, I encountered several files that looked normal but still got the 'has transparency' label. When I inspected them closer, I discovered they had a single 98% opaque pixel, or a border with a slight fade, or just one invisible corner pixel. Maybe some of these pixels are secret artist markers, or perhaps they are tool errors or drawing tablet smudges. They probably aren't really what we are interested in finding with a 'system:has transparency', so be on the watch for them and let me know how bad the problem is in an IRL client. Maybe I can fine-tune this system to say 'image is at least 0.3% transparent'.

other highlights

I fixed a stupid logical typo in the folder move/copy tech changes from last week. If you couldn't run an internal backup or migrate your hydrus folders around, sorry for the trouble! Should all work correct again.

I fixed the 'open externally' thumbnails position if your thumbnail supersampling is >100%, and made it so files without thumbs (e.g. zip, epub) will show their filetype thumb or the hydrus icon.

'system:number of character tags > 4' now parses if you type it in (previously, this didn't support the namespace filter). 'unnamespaced' should work too.

'system:has audio' and the 'system:embedded metadata' stuff, which never had good homes, are now all merged under that new 'system:file properties' entry. If you can't find something weird, check there!

next week

Everyone around me is sick, and I can feel myself going, so it may be up in the air! In any case, I think I'd like to take a simple code cleanup week. Nothing too clever or amazing, but should be some small fixes and QoL.
Open

R: 1 / I: 0 / P: 2

Version 551

https://youtube.com/watch?v=J5I00p3KqvE [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v551a/Hydrus.Network.551a.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v551a/Hydrus.Network.551a.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v551a/Hydrus.Network.551a.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v551a/Hydrus.Network.551a.-.Linux.-.Executable.tar.gz

I had a great week mostly fixing bugs.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

First off, thanks to a user, you can now turn off the thumbnail 'fade' transitions under options->thumbnails. This will just make any thumbnail appearance or change happen instantly, in one frame.

Secondly, users who are on PyQt (macOS, and some who run from source), had some bad crashing issues last week. I changed one tiny thing, and it turns out PyQt had a big problem with it--Mr Bones and 'open externally' panels and the ratings hover in the media viewer all displayed wrong and caused program instability. Thanks for your reports, sorry for the trouble, and should be all fixed now!

Other than that, I just fixed a bunch of different little things. Some transparent image handling, a weird timezone calculation, database counting issues, and a ton of file move/copy failure handling. Check the changelog for details.

next week

I'd like to do something new, so maybe 'system:has transparency'.
Open

R: 4 / I: 1 / P: 2

Version 548

https://youtube.com/watch?v=Tf8GG3gwam8 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v548/Hydrus.Network.548.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v548/Hydrus.Network.548.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v548/Hydrus.Network.548.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v548/Hydrus.Network.548.-.Linux.-.Executable.tar.gz

I had a good week. Thumbnail shortcuts are now customisable.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

First off, I cleared out a ton of bad thumbnail keyboard shortcut code and moved it all to the newer system under file->shortcuts. There is a new shortcut set there, called 'thumbnails', that governs opening the media viewer, the archive/delete filter, selecting files, and moving the thumbnail focus. Everything is now customisable (although I'd stay away from the thumbnail focus stuff unless you really want to wade through a big list), and I have added some new commands for 'select inbox', 'select trash' and similar, for more file selection options. This work clears out some really ancient code, and it should be much easier to extend thumbnail shortcuts in future, including hooking mouse-clicks into the customisable system and adding ctrl+selection logic.

Thanks to a user, we now have renderable krita files! We've got it set like PSDs for now, where you'll get 'open externally' on the preview and the full image in the normal viewer. As always, if you have files that don't work or render crazy, please send them in!

An issue with file export drag-and-drops sometimes giving the wrong file when those files share the same export filenames should be fixed!

Just a small thing, but a couple of places that were unintentionally sorting service lists in a random way now sort them alphabetically, as indended. The F9 new page picker was one of these, so if you have multiple local file domains, you may need to learn some new muscle memory!

next week

With luck I'll have some file storage stuff done, and I'd also like to clear a 'thumbnail fill' rewrite I've been planning.
Open

R: 3 / I: 0 / P: 2

Version 550

https://youtube.com/watch?v=yj2g8l1JqWI [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v550/Hydrus.Network.550.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v550/Hydrus.Network.550.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v550/Hydrus.Network.550.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v550/Hydrus.Network.550.-.Linux.-.Executable.tar.gz

🎉 Merry 550 🎉

I had a good week. I fixed some important bugs and reworked 'scale to fill' thumbnails.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

I fixed an issue with the login system where, for certain scripts, a job that was supposed to be waiting on an ongoing login process could nonetheless start early. All jobs are more patient about waiting for their specific login process to completely finish, no exceptions.

I added a maintenance system to explicitly close and clear persistent network connections. I had underestimated how aggressive my underlying libraries were at closing these no-longer-used 'keep-alive' network connections, and a half-dead object was occupying your OS's network slots for longer than desired. Network connections now close promptly five minutes after the last use.

I fixed some thumbnail display bugs, and, particularly, 'scale to fill' thumbnails are much more sensible behind the scenes. Previously, the thumbnail on disk was of the cropped visible region (which was a little complicated to figure out and maintain); now, things are simple--it just stores a full-image thumbnail at the appropriate larger scale and then draws the cropped area of that on demand. There are several technical benefits to this, and it saves some CPU in the long run, fixes some bugs for things like blurhash which need the full image to work properly, and it means an external process like the Client API can now ask for a thumbnail and get the full image rather than an arbitrary crop.

If you are set to 'scale to fill', your thumbnails will regenerate as you browse. Fingers crossed, you will not notice any visual difference. Let me know if the regeneration is annoyingly laggy--I didn't want to schedule a ton of regen CPU for you all this update, but I can figure a 'catch up in the background' thing out if it is a problem.

next week

More small work and bug fixing, I hope. If I have time and energy, some file storage overhaul work.
Open

R: 1 / I: 0 / P: 3

Version 549

https://youtube.com/watch?v=VR9oZZcdYNU [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v549/Hydrus.Network.549.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v549/Hydrus.Network.549.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v549/Hydrus.Network.549.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v549/Hydrus.Network.549.-.Linux.-.Executable.tar.gz

I had a good week, improving performance and adding some advanced tools for future testing.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

I fixed some deleted file records in the database. When you update, it'll take a moment to recalculate some things.

The options->maintenance and processing page has a ton of new settings to tune how hard the various background maintenance systems work and rest. These settings are mostly only for debugging, but if you are an advanced user and feel brave, you can play with them. If you have the 'memory explosion on PTR processing' problem, please check the changelog for instructions.

I optimised the taglist sort and update code a bit. Very large pages should be a little less laggy.

If you run from source, the 'easy setup' scripts have a couple new choices for setting up Qt, including choosing to type a specific version in. If you had the issue where your menus were in the wrong position, there is now a new (t)est Qt version (6.6.0) to try out, too.

next week

I got caught up in this 'background' work this week. I'd like to do something more visible and fun next week, but we'll see!
Open

R: 1 / I: 0 / P: 3

Version 547

https://youtube.com/watch?v=Cjkjba-qbhE [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v547/Hydrus.Network.547.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v547/Hydrus.Network.547.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v547/Hydrus.Network.547.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v547/Hydrus.Network.547.-.Linux.-.Executable.tar.gz

I had a good week mostly fixing bugs.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

Most importantly, I banged my head against a persistent knot of mpv crashiness this week, and I actually got somewhere. A variety of situations where mpv could fail to load a weird file, which would then lead to hydrus program instability, are now much much better, usually no crash worries at all. Also, there's a little more error handling, stuff like a placeholder image to show instead of the broken file, and an error popup to tell you what happened. I cannot promise that this improves mpv on any system that already had trouble just getting it working (e.g. macOS, some Linux), but I think that heart-skipping moment of 'will it crash?' lag when an mpv load fails should be solved. Let me know how you get on!

Thanks to a user, we have support for 'djvu' files, which are basically an open source pdf format.

If you have a client with hundreds of search pages open, please let me know if your session is any less laggy, particularly if you get bumps of lag every five minutes or so. I rewrote some ugly code and think I eliminated some background CPU overhead for your situation.

There's a handful of smaller fixes and tweaks too, nothing super important, but check the changelog if you are interested.

next week

I'm sorry to say my past couple of months have been pretty low productivity overall. Just life stuff cutting at my time and energy, mostly. I'm going to keep my head down and not change anything drastic, but it has been on my mind, and I regret not being able to push on larger projects. We'll see how the new year goes, I think. Anyway, I expect I'll clear out some more jobs like this next week. I've also been dabbling with the idea of a new 'hash sort' to preserve the sort of a 'system:hash' or downloader highlight in various situations, so I might explore that.
Open

R: 1 / I: 0 / P: 3

Version 546

https://youtube.com/watch?v=aFx7woWkZbc [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v546/Hydrus.Network.546.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v546/Hydrus.Network.546.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v546/Hydrus.Network.546.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v546/Hydrus.Network.546.-.Linux.-.Executable.tar.gz

I had a simple week. I mostly cleaned code, so there isn't much to talk about except for some bug fixes to recent systems.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

misc

I accidentally screwed up PSD thumbnail colours at the last minute last week while fixing something else. This is fixed, and any borked PSD thumbnails should regenerate themselves soon.

The new file history chart's cancel button works better, and I fixed some of the counting logic in the archive/inbox lines when you have a search filtering the results. Let me know if you still see archive numbers that are way too high.

If you have run into the new 'blurhash' thumbnails and don't like them, hit the new checkbox in options->thumbnails to turn them off!

next week

I really got lost in code refactoring this week. It is always worthwhile work to do, reshaping old bad ideas and preparing for the future, but it is unexciting, and with a janked out codebase like mine it often feels like bailing out the ocean.

For next week, I'd like to get back to my file storage system improvements, ideally getting background migration ready to go.
Open

R: 1 / I: 0 / P: 3

Version 545

https://youtube.com/watch?v=OLJhFROQg5A [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v545/Hydrus.Network.545.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v545/Hydrus.Network.545.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v545/Hydrus.Network.545.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v545/Hydrus.Network.545.-.Linux.-.Executable.tar.gz

I had a good week. I did some small things, and a user contributed some cool things!

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

blurhash

tl;dr: we have some special blurry thumbnails now, but you won't see them much yet.

Thanks to a user who did a bunch of research and work to get this going, the client now generates the 'blurhash' of all files that have a thumbnail. This hash is essentially a micro-thumbnail, only about 34 bytes, that shows a very blurry coloured impression of the general shape of the image or video. They are usually used as placeholders when it may take time to load something--you have probably seen something similar on slower news websites.

You won't see these in hydrus itself much, since thumbnails load fast enough that we don't have to worry about placeholders. In the advanced ways of seeing files you don't actually own, however, I will now show you any file's known blurhash rather than the default 'hydrus' thumb. Same deal for damaged/missing files--if I can't fetch a thumb, I'll now fall back to the blurhash.

The more important thing is these hashes are now available on the Client API, under the normal 'file_metadata' call. If you are implementing a browser or similar and have access to a fast blurhash library, try them out! I've scheduled all users to generate the blurhashes for their existing files in the background, which will take a few weeks/months for users with hundreds of thousands of files (although, if you are working with this and want to hurry it along, remember the queue is manageable under database->file maintenance.

other highlights

The file history chart (help->view file history) now has a search panel, just like Mr Bones! You can search your import, archive, and delete history for creator x, filetype y, or any other query you can think of. Some of the math here got a bit weird, and I am sure I have missed out several 'cannot provide good numbers for this domain' situations, so let me know if you get any really stupid results or outright errors. There's more work to do here, like a button to hide the search panel, which I hope to push on in the near future.

Thanks to the same user above, we also have epub support! No 'num_words' yet, but it turns out epubs are just zip files with some html, so I think it'll be doable in future. Also, some rare incorrect jpeg rotations (for 'MPO' jpeg files) are fixed.

If you right-click on a selection of files, the 'open->similar files' menu now has a 'in a new duplicate filter page' command. This will initialise the filter page just looking at those files, hopefully making it simple to clear out the potential duplicates in any particular selection.

Unfortunately, I am retiring the Deviant Art artist search and login script. DA have been slowly killing their nice old API, and the artist search just went. Individual page URLs still seem to work, but I suspect they will be gone soon. Gallery-dl have a nice DA solution that works with the new API, so if you can't find the same content on a more open booru, that's my best recommendation for now.

next week

I want to take a very simple cleanup week. Nothing too exciting on the changelog front, but I'll refactor some of the worst code into something nicer to work with.
Open

R: 4 / I: 0 / P: 3

Version 544

https://youtube.com/watch?v=T9IjJIX50Ls [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v544/Hydrus.Network.544.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v544/Hydrus.Network.544.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v544/Hydrus.Network.544.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v544/Hydrus.Network.544.-.Linux.-.Executable.tar.gz

I had a good week. File storage locations can now have max size limits. There is a security fix in this release, and users who run from source will want to rebuild their venvs this week.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

max size

This is just for advanced users for now.

So, in migrate database, if you have multiple locations set for your files, you can now, for all but one, set a 'max size', limiting the total mass of files it will store. The idea here is that if you have a 2TB drive and a 500GB one, then rather than having to keep playing with the weights to ensure the 500GB doesn't get too full, you can now set the 500GB location with a 450GB limit and hydrus will allocate that space and no more.

There are a couple of limitations. It isn't perfectly precise, so if your space is tight, give it a little padding (although remember that drives should be at least 10% free space anyway!). Also, it won't yet enforce these rules outside of the 'move files now' button. If you set a tight limit and then import many files, it will blow over that limit (at least until you open up migrate database and hit 'move files now' again to rebalance).

The next step here is to implement automatic background file migration for always-on checking so we can have these rules apply without regular user intervention. My aim is to get rid of the lagtastic 'move files now' entirely so no file migration blocks other use. I hope to have this done in the next few weeks.

webp

A remote execution (very bad) vulnerability was recently discovered in the main webp library. You probably noticed your chrome/firefox updated this week, which was fixing it. Our 'Pillow' image-loading library uses libwebp, and they rolled out an update themselves this week. The builds today incorporate the fix, so if you use them, just update as normal and you are good. If you run from source, reinstall your venv.

However, if you are Windows 7 (or otherwise on Python <=3.7), I am afraid you cannot get the fix! As I understand, it just won't work for you. There is now an additional choice in the 'advanced' setup_venv script to choose the older Pillow version, so you can keep running, but I'm afraid with this and other things you lads are now solidly in the phase of limited security updates. I will keep supporting you as long as it is generally reasonable, but if you still want to use Win 7, I think the train is running out of track.

misc

When images fail to render, they now show something proper in the media viewer. (previously they were just blank forever and made some error popups).

After last week's updates, Mr Bones gets a readability/layout pass.

The client should stop mistaking various text files for mpegs in the pre-import phase!

I think the Docker client wouldn't boot last week, same for anyone else without the PSD-tools library. Sorry for the trouble, should be fixed now!

next week

I'd like to add a file search to the file history chart and get stuck into more file storage infrastructure updates.
Open

R: 1 / I: 0 / P: 3

Version 543

https://youtube.com/watch?v=3zw3sRLVPN0 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v543/Hydrus.Network.543.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v543/Hydrus.Network.543.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v543/Hydrus.Network.543.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v543/Hydrus.Network.543.-.Linux.-.Executable.tar.gz

I had a good week. Mr Bones gets an update. If you run from source, you might like to rebuild your venv today to get some new parsing support.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

Mr Bones

So, the statistics under help->how boned am I? now have an attached file search. The numbers will change based on whatever you search, so you can now see the average size of your pngs, the archive/delete ratio for creator x, how many times you have viewed pictures of sonic the hedgehog, or any other search you can think up. The default search is what the dialog was fixed to before, 'system:everything' on 'all my files'.

This turned out to be surprisingly complicated, and to stop it spiralling in complexity, I culled some of the odder scenarios. If you do a multiple-file-domain search, you won't get 'deleted' or 'earliest import' numbers. And if you search 'deleted from x', you are going to get some jank logic. EDIT: I also disabled the potential dupes count right before release, it was performing awfully on my large IRL database. Keep it simple, and let me know what turns up. I love these sorts of weird statistics.

I'd like to do the same for the file history chart soon. Should be a bit easier!

misc

PSDs should render more quickly and with less memory!

PDFs no longer spam the log on thumbnail generation!

Bitmaps files (.bmp) are now fully supported. Since the program started, these have been converted to png on import, mostly because the bmp format personally annoys me. This was always in conflict with the tenet that hydrus not alter files in any way, and I think it was ultimately a mistake. Bitmaps now stay as-is, and we've happily untied a couple of awkward knots in the code as a result.

If you are a downloader maker, there's a new 'datestring to timestamp (easy)' String Converter step. It uses an excellent external library ('dateparser') that can parse pretty much any datestring you can think of, including stuff like '2 hours ago'. If you run from source, this is the thing you are rebuilding for today. I have left all the existing 'datestring to timestamp' conversion steps in place for now, as '(advanced)', but if this new step integrates well, I think it'll supercede them and I may ultimately convert them all over.

next week

I'm going to have a busy week IRL, so work might be limited. I trudged through more file storage overhaul this week, and while I didn't finish the 'max num bytes' option as I had hoped, it is all ready behind the scenes. I just need to put some UI together and I think we'll have it working.
Open

R: 3 / I: 0 / P: 3

Version 406

https://youtube.com/watch?v=UZr6b4mBoxI [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v406/Hydrus.Network.406.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v406/Hydrus.Network.406.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v406/Hydrus.Network.406.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v406/Hydrus.Network.406.-.Linux.-.Executable.tar.gz
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v406.tar.gz

I had a good week. It is now easy to deduplicate queries in subscriptions.

subscriptions

Managing multiple subscriptions can get complicated. Figuring out which queries are in which sub, whether and where you have duplicates, is tedious. A recent bug with query pasting also could have introduced some duplicates within the same sub. This week should make it simple to manage.

The main manage subscriptions dialog now has a 'deduplicate' button. It lights up any time your subs of a particular downloader have the same query, whether that includes dupes within the same sub or across different subs. Click it, and it launches a thorough process to determine what you want to do:

- First, it asks if you want to deduplicate by case or not--e.g. whether "Samus_Aran" and "samus_aran" should be considered dupes, or only exact matches.
- Then it summarises which downloaders appear to have dupes, letting you choose which to work on.
- Then you have the option of selecting which of the queries you wish to deduplicate--e.g. if you want to do two runs to dedupe, say, artists to one sub and characters to another.
- Then you select which subscription(s) will retain the deduplicated queries.

Every step explains itself. Mostly it is simple, but there are some odd situations. When a sub dedupes queries within itself, it keeps the one(s) with the most files. I hope I have accounted for everything reasonable.

Adding to caseless matching, also added is a 'lowercase' button, which will coerce all the queries in the selected subscriptions to lowercase, just to help keep things neat. And lastly, when you paste queries into a sub, the 'already in the sub' test is now caseless.
Open

R: 3 / I: 0 / P: 3

Version 534

https://youtube.com/watch?v=wpUEdGDdaTQ [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v534/Hydrus.Network.534.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v534/Hydrus.Network.534.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v534/Hydrus.Network.534.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v534/Hydrus.Network.534.-.Linux.-.Executable.tar.gz

I had a good week. There's lots of smaller fixes and improvements, and ratings are added to the Client API.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

Thanks to a user submission, hydrus now recognises and supports importing .sai2 files! Basic metadata for now, but I'll see about resolution and thumbnail support in future.

It turns out PNG and WEBP files can have EXIF data, and our EXIF scanner works on them! All newly importing PNGs and WEBPs will be scanned, and on update, you'll have a yes/no dialog asking if you want to scan your existing PNGs. If you have a gigantic database, or your files are on a remote cloud storage, you might want to defer it, but it isn't really a big deal.

I banged my head against the media viewer 'notes hover' layout code, and I was actually really successful this time. Many of the borked layout issues are fixed, and I think I've completely fixed the overlapping hover problems in the duplicate filter.

I fixed a bunch of weird bitmap paste problems in the new 'similar files' paste button. Let me know if you still have any issues!

Just a small thing, but the 'loading' page in hydrus, when you refresh a search, is no longer grey. It is the same colour as the normal thumbgrid, so the loading flicker is gone.

Twitter is gone from the downloader defaults for all new users. All existing users should look to migrate away, too. I hope to roll out some downloader engine improvements in the near future that'll make nitter (on multiple domains) easier to set up, but there aren't a lot of excellent solutions.

I also added ratings to the Client API. The service object now talks about min/max stars and star shape, the file metadata call gives every file's ratings, and you can set and clear ratings too with a new command and permission. It is all fairly simple, but there's several possible data types depending on the rating type, so check out the Client API help for details and examples.

next week

I regret that I again did not find time for PTR janitor workflow improvements this week. Now that ratings on the Client API is clear, this is top priority. Depending on how things go, I may take a two-week release, since I definitely want some solid work done here.

R: 2 / I: 0 / P: 3

No discord hydrus shenanigans for now.

Locked out of discord because of their shitty mobile verification, so doing my part tagging paywalled furry art ¯\_(ツ)_/¯

> https://youtube.com/watch?v=SMTz9nIUkGc [Embed]

R: 3 / I: 0 / P: 3

unable to keep my tags visible and overall finicky

whattup,

I've been fiddling with hydrus for a bit but cannot figure out how to keep my tags visible and usable at all times. I have two tags so far, but expect to use dozens, if not hundreds in total in the future.

The whole thing isn't easy for me introductory tutorial that I've gone over so far has basically been "select image, press f3, do ur tag stuff".

Can I see all my own created tags without selecting an image and pressing f3 and be able to edit/delete them? Also, how do I keep tags in the "selection tags" section at all times?

Thanks in advance.
Open

R: 4 / I: 0 / P: 4

Version 451

https://youtube.com/watch?v=Cv87NJ2xX0k [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v451/Hydrus.Network.451.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v451/Hydrus.Network.451.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v451/Hydrus.Network.451.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v451/Hydrus.Network.451.-.Linux.-.Executable.tar.gz

I had a great week cleaning code and fixing bugs. If you have a big database, it will take a minute to update this week.

all misc this week, most bug fixes

I fixed a critical bug in tag siblings. It was causing some tag siblings to be forgotten, particularly on local tag services, and particularly when a tag had a sibling removed and added (e.g. 'delete A->B, then add A->C') in the same transaction. I will keep working here and will trigger a new sibling reprocess in the future just as 450 did so we can fix more PTR issues.

The new content-based processing tracking had a couple more issues. Some Linux users, in particular, it just broke for, due to a SQLite version issue. I have fixed that, and I have fixed some issues it caused for IPFS. There are new unit tests to make sure this won't happen again.

I fixed an issue with sessions recently not saving thumbnail order correctly!

I fixed issues with some of the new Client API file search parameters not working right!

Big video files should import a bit faster, and will show more status updates as they do their work.

We have had more anti-virus problems in recent weeks. We'd hoped building on github would eliminate them, but it hasn't completely. A user helped me trace one issue to the Windows installer. We have 'corrected' it (believe it or not, it was removing the 'open help' checkbox on the final page of the install wizard, and yes the reasons for why this was causing problems are ridiculous), so with luck there will be fewer problems. Thank you for the reports, and let me know if you have more trouble!

full list

- stupid anti-virus thing:
- we have had several more anti-virus false positives just recently. we discovered that at least one testbed used by these companies was testing the 'open html help' checkbox in the installer, which then launched Edge on the testbed, and then launched the Windows Update process for Edge and Skype, which was somehow interacting with UAC and thus considered suspicious activity owned by the hydrus installer process, lmao. thereafter, it seems the installer exe's DNS requests were somehow being cross-connected with the client.exe scan as that was identified as connected with the installer. taking that checkbox out as a test produced a much cleaner scan. there is a limit to how much of this nonsense I will accomodate, but this week we are trying a release without that 'open help' link in the installer, let's see how it goes
- semi-related, I brushed up the install path message in the installer and clarified help->help will open the help in the first-start welcome popup message
Open

R: 1 / I: 0 / P: 4

Version 542

https://youtube.com/watch?v=vDPoR5PHw7Y [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v542/Hydrus.Network.542.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v542/Hydrus.Network.542.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v542/Hydrus.Network.542.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v542/Hydrus.Network.542.-.Linux.-.Executable.tar.gz

I had a good week. There are improvements to PDFs and system predicate parsing.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

Thanks to a user, PDFs now have thumbnails! I'm happy and surprised with how this turned out. The module it uses also offers raw text access, so I have rejuvenated a I-think-good-enough 'num words' count for PDFs. Also, PDFs can have a little embedded metadata like Author and Title, and this is viewable on the special button at the top of the media viewer. On update, all your existing PDFs will be scheduled for some file maintenance work to figure all this out, so give it a little while to catch everything up. As always, please send in any PDFs that get weird results. I expect we'll do a little more here, particularly pulling the embedded data into hydrus proper as tags etc...

System predicate parsing (e.g. by typing 'system:inbox' into the tag autocomplete) has several improvements. All comma-grouped numbers are now parseable, so you can paste 'system:width = 1,920' and it'll work. 'system:tag as number' has some special fixes, and several tags have presentation updates within the client to make sure that the text they are producing for a label is actually something that parses back to them.

Tag export sidecars now have a 'tag display type' selector to choose between 'storage' and 'display' tags. Until now, all the sidecars were all 'storage', meaning they did not include parent or sibling tags (like the list you edit in manage tags), whereas 'display' includes sibling replacements and implied parents (like the normal file search view shows). Existing tag export sidecars will stay on 'storage', but new ones will default to 'display'.

next week

I did some more file storage updates this week, but it remained pretty boring behind the scenes stuff. I'm going to keep bashing away, and I'd love to have a bit of other fun too, maybe custom searches for Mr Bones and the file history chart, which I have been meaning to do for ages.
Open

R: 1 / I: 0 / P: 4

Version 541

https://youtube.com/watch?v=xJRAubWCkcc [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v541/Hydrus.Network.541.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v541/Hydrus.Network.541.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v541/Hydrus.Network.541.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v541/Hydrus.Network.541.-.Linux.-.Executable.tar.gz

I had a good week. There's a mix of all sorts of work.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

Thanks to a user, we are adding support for two more filetypes today: QOI, which is a png alternative, and Procreate, an Apple image project file. QOIs have full support, Procreate files just have thumbnails.

If you use one of the darkmode styles on Windows and your tooltips suddenly got too dark to read a couple weeks ago, please check out the new 'alternate-tooltip-colour' variants of the styles in the options. It should fix you up for now. Try switching back when we move up to a new version of Qt again--it is a bug on their end.

There's two new checkboxes in options->media to ignore 'uninteresting' import and modified times. By default, if the modified time is very close to when a file was added to 'my files', I hide it in the media viewer. If you would prefer to always see it, you can now set this.

Importing massive apngs is now significantly faster.

The parsing system has a new 'String Joiner', which does string concatenation. It can glue the strings together with any custom text, including the empty string for pure concatenation, and it can join groups in 1-2 1-2 1-2 or 1-2-3 1-2-3 style patterns too.

The Client API can now fetch the siblings and parents of tags. If you want to get into siblings and parents but have never done it before, it can get complicated, so brace for impact.

file storage upgrades

I started on this work this week, but it is all boring behind the scenes stuff so far. I'm happy with the progress though, and I'd like, in the nearish future, to have:

- lower file access latency when you have millions of files
- the ability to say 'store no more than 200GB in this folder'
- background migration of files rather than the current block-the-whole-program-for-ages 'move files now' system

I'll keep chipping away at this over the next few weeks. I feel good about it.

next week

More like this. Catchup on older issues and pushing on file storage.
Open

R: 3 / I: 0 / P: 4

Version 540

https://youtube.com/watch?v=ANDQoEl2StM [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v540/Hydrus.Network.540.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v540/Hydrus.Network.540.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v540/Hydrus.Network.540.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v540/Hydrus.Network.540.-.Linux.-.Executable.tar.gz

I had an ok week. There's a mix of smaller work.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

system:filetype parsing now supports more human-friendly words. You can type 'system:filetype is image' or 'system:filetype is static gif, png' and it should just work. I think that any existing filetype predicate string (e.g. if you right-click->copy it) should now parse ok.

I fixed a bunch of broken 'system:similar files' image bitmap pastes that were not producing the correct pixel hash. Also, I've started a background job today to fix some lingering bad duplicate pixel data for pngs, so if you have recently done a duplicate filter with 'must not be pixel dupes' but got some anyway, I hope this will fix itself soon.

I added links to https://github.com/Garbevoir/wd-e621-hydrus-tagger (and https://github.com/abtalerico/wd-hydrus-tagger, which it is based on) to the Client API help. These use the recent 'I give text, it draws an image' AI tech everyone has been messing around with to do the reverse--you give it an image, and it automatically tags it. This stuff is just getting ready to seriously play with, so if you are interested, give it a go!

next week

I didn't get to the file storage upgrades I wanted this week, so I'll give it another go.
Open

R: 1 / I: 0 / P: 4

Version 539

https://youtube.com/watch?v=htZEYN1TkOE [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v539a/Hydrus.Network.539a.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v539a/Hydrus.Network.539a.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v539a/Hydrus.Network.539a.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v539a/Hydrus.Network.539a.-.Linux.-.Executable.tar.gz

I had an ok week. There's full PSD support and a variety of quality of life improvements.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

PSD files

Just like last week, if you run from source, you might like to rebuild your venv again this week, and you'll get the PSD support. Just run 'setup_venv' again.

Thanks to a user, PSD files should now show in the program just like any other image! Since it takes a little extra CPU to render them, we're starting with conservative view settings--PSD files will show in full in the normal media viewer, but you'll get the cheaper 'open externally' button in the preview panel. If you want to try showing them everywhere, you can change the view settings under options->media.

PSD thumbs are also much nicer now, and your existing PSD thumbnails should regenerate very soon in the background.

As always, if you run across a PSD that has crazy colours or bad transparency--or if it straight up won't load at all, which we think some will--please send them in and I'll see what I can do.

other highlights

I'm splitting 'gif' files into 'static gif' (under images) and 'animated gif' (under animations) this week. It isn't a huge thing, and there aren't all that many static (i.e. non-animated) gifs going around anyway, but it simplifies some view settings as well as some code, and it aligns with how the other formats like png/apng work.

Rating system predicates are now parseable. Type 'system:has rating (service_name)' or 'system:rating for (service name) > 4/5' into the tag autocomplete, and it should prefill the result. It should work for all like/disklike, numerical, and inc/dec rating services, and like the other parseable system predicates, this extends to the Client API. Try it out!

I fixed the duplicate filter's right-hand hover window back in place. It was a mistake to have it 'avoid' the notes hover, since the buttons bounced around too much for easy clicking, so now they can just overlap, with the duplicates hover taking precedence.

next week

I'm due some larger work, and I think I'm going to tackle some file storage improvements. The migrate database dialog sucks, and the file storage system in general groans when it has millions of files, so I'd like to have the first steps towards user friendly storage settings and cleverer storage overall. We'll see!
Open

R: 1 / I: 0 / P: 4

Version 538

https://youtube.com/watch?v=8yXk3_o1Hoo [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v538/Hydrus.Network.538.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v538/Hydrus.Network.538.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v538/Hydrus.Network.538.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v538/Hydrus.Network.538.-.Linux.-.Executable.tar.gz

I had a great week. There's some new libraries and new file support.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

notes

If you get a popup on update about missing indices, don't worry. Something I forgot last week is being fixed. Just let it work.

If you run from source, I recommend you rebuild your venv this week. Just run the 'setup_venv' script again with the same settings you chose before. You'll get new Qt, OpenCV, and the HEIF support. If you are on an older OS and run into trouble with Qt, the setup now has a 'Qt (o)lder' option that gives you something we know is very stable and compatible.

=highlights

The main UI and image libraries that hydrus uses are updated today. We've been in an awkward place with Qt for a while, stuck on a version I regret updating to, so I hope this jump relieves this. It should speed some image load, reduce some flicker, and fix some UI scaling issues. There are no special install instructions, and we've been testing for months on different computers and haven't seen any specific update problems, so I'm not expecting any significant issues. If you do run into anything, then please roll back to your backup and let me know.

Thanks to a user, we now have full support for HEIF, HEIC, and AVIF files! They import and work like any other image. We also support the 'sequence' version of these files, which are essentially like (currently super rare) high quality animated gifs. I'm very new to all these formats, and they are obviously less common, so if you have a weird version or your colours are all messed up, I'll be interested in examples. I don't even have a 'HEIC sequence' to test with!

My deferred delete system is complete. Most users don't have to ever think about it, but it is under database->database maintenance. There's an UI panel to track what still needs to be deleted and some basic settings to control when it does its work. If you delete the PTR or do a large regeneration job, the (very slow) actual delete of the existing data now happens in pieces in the background in the future. I'm really pleased with the performance, and even deleting 30-million-row tables seems to work very well, but if you delete the PTR (2 billiow row table!) and get unacceptable lag every few seconds, check that menu and disable the work during idle/normal time and let me know, and we'll figure it out.

tags->migrate tags has cleverer file filtering. You can now select the tags for files 'deleted from x' or 'currently in y or z' real easy.

next week

I'm generally happy with my current productivity, but I'm falling behind on many things at once. I would like to focus on system predicate parsing and Client API for a week, because it keeps slipping.
Open

R: 1 / I: 0 / P: 4

Version 537

https://youtube.com/watch?v=pg-yeXEvVwA [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v537/Hydrus.Network.537.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v537/Hydrus.Network.537.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v537/Hydrus.Network.537.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v537/Hydrus.Network.537.-.Linux.-.Executable.tar.gz

I had an ok week. There's a mix of bug fixes and a couple of neat improvements.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

First off, the filetype selector, which you'll see in 'system:filetype' and any File Import Options, is completely rewritten to a checkbox tree. It is easier to work with than the old expanding panels system and scrolls smoothly when it gets internally tall. I've been worried about adding more filetype support for a while because of how busy this widget was getting, but this will relieve things.

Also, the 'applications' group is now split up. All the archives like zip and rar are now in 'archives', and all the xcf, psd, and krita formats are in 'image project files'. It doesn't matter much yet, but these groups also exist as separate rows in the viewing settings under _options->media_.

Secondly, I have figured out a 'fast delete' method for very large database objects. Deleting the PTR is now an instant operation! Unfortunately, under the hood this is really just a deferred delete, so the object still uses its hard drive space until it is deleted later in the background in pieces. Unfortunately x2 is that I haven't written the maintenance job that does the delete yet, lol, so if you need to delete some service or do a large database regenerate job, please bear with the bloat for a week.

If you updated last week, jumping up several versions at once, and got a popup that some EXIF scanning update did not work, don't worry. I fixed the problem, and nothing important was actually broken anyway. Thanks for the reports!

I fixed the grid layout on tags->migrate tags, which exploded under my nose. Sorry if you were recently confused by this!

next week

I'll finish this background deletion maintenance system and keep on trucking on smaller fixes. I've got a lot of immediate work like this to do and github issues to catch up on. I'd also love to get some more system predicate parsing, for ratings and more, working.
Open

R: 1 / I: 0 / P: 4

Version 536

https://youtube.com/watch?v=EtVjGWpbE7k [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v536/Hydrus.Network.536.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v536/Hydrus.Network.536.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v536/Hydrus.Network.536.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v536/Hydrus.Network.536.-.Linux.-.Executable.tar.gz

I had a mixed week, but I fixed some things, and there is more filetype support. There are a couple of database fixes this week, so anyone with a large database, including anyone who syncs or has synced with the PTR, can expect the update to take several minutes.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

Thanks to a user, the program now supports gzip (a type of archive) and xcf (GIMP savefiles)! SVG thumbs also look correct now if you have 'fill' thumbnails. Since the 'filetype selector' widget is getting pretty stacked, I'm going to rewrite it very soon, and I think we are going to have one or more groups, maybe spitting raster/vector and grouping 'image' application types like psd, krita, and xcf, together.

I fixed an issue where even if you deleted a tag from all the files somewhere (i.e. reduced its autocomplete count to 0), it wasn't disappearing from autocomplete results. It shouldn't happen again, and one of the updates this week will be scanning through your autocomplete data to remove all existing orphans.

Otherwise it is mostly background work this week! One thing is more responsive, another is more efficient, and some UI protects against mistakes, but none of it is too interesting. There's a clever delete-lock update to the Client API if you handle a lot of advanced file deletions.

future build

There's another test build out today, here: https://github.com/hydrusnetwork/hydrus/releases/tag/v536-future-2

It is the same 536 but uses newer libraries. If you have trouble with the current version of Qt (e.g. borked high UI scale text sizes) or just want to test out newer tech, feel free to have a go. We've had some real problems with the recent crop of Qt updates, but this one looks tentatively good, so I'm very hopeful this jump will put the issues to bed. Let me know if there are any problems!

next week

I'm behind on my github issues, so I'll give them a good look, and otherwise continue pushing at my immediate backlog of smaller jobs.
Open

R: 2 / I: 0 / P: 4

Version 535

https://youtube.com/watch?v=qv6yDjtIl8k [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v535/Hydrus.Network.535.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v535/Hydrus.Network.535.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v535/Hydrus.Network.535.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v535/Hydrus.Network.535.-.Linux.-.Executable.tar.gz

I had a good couple weeks working on better petition-processing workflow for repository janitors. Most users won't see this, so you can ignore most of this release!

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

krita and SVG

Thanks to two different users, hydrus gets Krita (.kra, .krz) and SVG (.svg) support today. Both formats also get thumbnails!

These formats can get quite complicated, so if you have one that won't import right (for Krita, it'll look like a zip; for SVG, probably HTML), please send it in and I'll have a look. Same for if the thumbnails are broken, although there's a limit to how clever we can go there.

Adding SVG also exposed that I don't have a good category/pipeline for handling vector vs raster graphics. Several things like thumbnail size calculations are different with an SVG (which has no true fixed resolution), so I will be doing several waves of background updates in future to handle this and other vectors better. Who knows, we might have beautifully zooming SVG media viewer support with one day.

petitions

Ok, this was almost all of my work this past two weeks. It kind of kicked my ass, but I'm happy it is done. It changes UI that only a handful of users see, so most users can skip over it completely.

For server admins, this new tech needs a server update, and it is not compatible with older clients (nor can newer clients talk to older servers), so please plan to update your server and janitors' clients at roughly the same time. It isn't going to break anything, but the client won't be able to do janitor work and will produce some harmless errors either way.

For janitors, I've made the petition processing page handle multiple petitions at once. There's a new panel, a multi-column list, and when you click the 'fetch petitions' button, it populates with as many petitions as you like. You can see and sort by the action (ADD, DELETE, REPLACE), account id, reason, and total size (e.g. '60 mappings in 12 petitions'), and then double-click any to highlight it in the normal petition UI you are used to.

This is about speed of processing. When you process a petition, it starts uploading in the background and the page moves immediately on to the next one. No more waiting for the server to summon another petition. Also, it gives you a better overview of your queue and even lets you mass-approve/deny a set of petitions when one account or reason keeps making good or bad stuff. You can fire off twenty petitions in one go and quickly move on to the next.
Open

R: 1 / I: 0 / P: 4

Version 533

https://youtube.com/watch?v=x33quKIyVu0 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v533/Hydrus.Network.533.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v533/Hydrus.Network.533.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v533/Hydrus.Network.533.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v533/Hydrus.Network.533.-.Linux.-.Executable.tar.gz

I had an ok week. There's a mix of different work and a fix to the recent macOS crashing.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

all misc this week

Subscription popup windows should update far more smoothly and frequently this week. Some of the 'should I check for new text?' logic was broken when I embedded the popup toaster into the main UI a few months ago--it is fixed.

If you right-click an image that is larger than 1,024 pixels on a side, the share->copy menu now has a second 'copy bitmap' that crops the image down for easier and faster pasting in lookup services like SauceNAO.

I added some info about the not-well-documented tag/predicate mouse shortcuts (e.g. shift+double-click to edit system predicates) to the help here: https://hydrusnetwork.github.io/hydrus/getting_started_searching.html#editing_predicates

I added a link to the Client API help for a neat new tool that finds video duplicates and queues them up in the duplicate filter. Here: https://github.com/appleappleapplenanner/hydrus-video-deduplicator

Since v529, at least some users of the macOS app have had terrible crashing. Last week's attempt didn't do it, but I think we've fixed it now! It was something I never even knew about updating in the background to an incompatible version. Thank you for your patience and help testing, and let me know if you still have any trouble!

If you are an advanced downloader maker, check out the changelog regarding sub-gallery urls. Nothing huge has changed, but they'll inherit and parse tags and some other stuff from their parent parsing step now.

The experimental QtMediaPlayer now initialises with and updates according to the volume and mute settings. Previously it was stuck on 100% always. The volume bar is still hidden on the actual player, but we are moving forward.

next week

I failed to do as I planned this week. I'd hoped to set aside a good whack of time to figure out a PTR tag filtering system, but it didn't happen. I'll see if I can get the time and energy together again this week, and if not, I'll try and kick out some ratings in the Client API instead.
Open

R: 1 / I: 0 / P: 5

https://youtube.com/watch?v=eclMFa0mD1c [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v532a/Hydrus.Network.532a.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v532a/Hydrus.Network.532a.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v532a/Hydrus.Network.532a.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v532a/Hydrus.Network.532a.-.Linux.-.Executable.tar.gz

The 532a is a hotfix, I messed something up with the new pixiv parser first time around.

I had a good week. I've improved stability and added some little features.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

stability

Some damaged videos were causing mpv to dump out in a way Qt was unhappy with, and it lead to black video screens that would crash after a few clicks. I have written a new error-catching system that should recognise these specific files and immediately unload them along with a warning to the user. Also, all other general errors (like missing metadata or chapter marks, which are common for our sort of files) are now printed to your log for future investigation. As we learn more, I expect to add more 'unload this immediately' and 'don't print to log, this error is nbd' hooks here in future.

The macOS release has unfortunately been unstable for some/all users since v530. Clicking on lists can simply not work and then suddenly cause a crashe. With some user help, I have learned an important potential crash in Qt and tightened up all the respective areas in hydrus code. If you are a macOS user and have had trouble, let me know how you get on with today's release. If you still get crashes from simple list interaction (e.g. in manage subscriptions), please revert to v529, which does not appear to have the problem. There have been no big database changes since then, so all you have to do is re-install v529 and you should be good to go.

The above macOS error handling improvements may have fixed some other unexplained crashes. What once caused a crash should now appear as a normal popup error. Let me know what you see.

other highlights

When you hit one of the 'open these files in a new page' commands, the new page is now search-enabled and starts with a 'system:hashes' of the files it opened with. This lets you quickly add more search predicates to narrow down the selection!

When you right-click a tag, the 'search' menu now lets you open the tag(s) in a new duplicate filter page!

I updated the pixiv downloader again. A couple weeks ago, Pixiv decided that if you aren't logged in, you can't get the 'original' quality of an image. I updated the downloader before my vacation by telling it to get the resized version, but as this was importing low quality files silently, it was not a great idea. I have given it another go, and now, if you aren't logged in to pixiv, it sets those files as 'ignored' and the 'note' on the file log mentions you need to be logged in to get the original quality. If you want the previous 'get the medium quality' version, it is now selectable as a different parser for 'pixiv file page api' under network->downloader components->manage url class links.

next week

I'd like to sink my teeth into a medium size job. It'd be nice to finally catch up on some serverside tech for more PTR updates.
Open

R: 1 / I: 0 / P: 5

Version 531

https://youtube.com/watch?v=7ACn4nCY4lQ [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v531/Hydrus.Network.531.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v531/Hydrus.Network.531.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v531/Hydrus.Network.531.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v531/Hydrus.Network.531.-.Linux.-.Executable.tar.gz

NOTICE! For everyone but macOS, Version 527 had special update instructions. If you are updating from 526 or earlier, please consult the post here: https://github.com/hydrusnetwork/hydrus/releases/tag/v527

I had a good week working on simple fixes and cleanup before my break.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

I also made another future build for v531 here: https://github.com/hydrusnetwork/hydrus/releases/tag/v531-future-1

all misc this week

I fixed editing favourite searches, which I accidentally broke last week!

If you right-click a tag or a file and it has a billion siblings/parents/children/known URLs, the menus now truncate the giga list. Previously, child-heavy tags like pokemon would make a menu that would laggily take up your entire screen. Now it lists about ten items and says '438 more...'.

I fixed the Pixiv parser for guest logins, but it no longer gets the 'original' quality. It seems like they hide this for non-logged-in users now. If you do a lot of Pixiv downloading, figure out a login solution with Hydrus Companion or cookies.txt imports.

If you have some downloaders that were inexplicably halted on 'waiting for a work slot', even after client restart, check them again this week--I cleaned up some bad status reporting.

The Client API gets an overhaul to service info this week. '/get_services' and '/get_files/file_metadata' are affected. If you do Client API work, please check the changelog and the updated help here https://hydrusnetwork.github.io/hydrus/developer_api.html#services_object . The old method of giving service names and types still works, so nothing breaks today, but it is now deprecated and will be deleted in early 2024. Please plan to move over!

next week

I am now on vacation! I'm off for a week to shitpost not-E3. I'll be back at it in a week, and I want to start pushing on my larger projects again. v532 should be out on the 21st. Thanks for the support everyone!
Open

R: 3 / I: 0 / P: 5

Version 530

https://youtube.com/watch?v=Gp1dzdFmwmc [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v530/Hydrus.Network.530.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v530/Hydrus.Network.530.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v530/Hydrus.Network.530.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v530/Hydrus.Network.530.-.Linux.-.Executable.tar.gz

NOTICE! For everyone but macOS, Version 527 had special update instructions. If you are updating from 526 or earlier, please consult the post here: https://github.com/hydrusnetwork/hydrus/releases/tag/v527

I had a good week. You can now type system predicates in a normal search, and highlighting big downloaders is nicer.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

The tag autocomplete text inputs on search pages get a couple of neat improvements this week:

First off, you can now type most system predicates, and they'll appear in the dropdown. You type the whole complete predicate, like 'system:imported before 2021-03-09'. This is the same system predicate parser that has been available in some advanced places, now for everyone. Note that it can't do every predicate yet, and while I try to cover several grammatical variants, some it is still quite inflexible. Let me know how you get on, and I'll keep working on it!

Second, the autocomplete now has a paste button. It accepts multiple newline-separated tags and will understand -negation, wildc*rds, namespace:* wildcards, and the new system predicates.

For downloaders, I made the 'highlighting' when you double-click a downloader or thread watcher asynchronous--it now loads the files in the background, only presenting them when the results are ready. This means if you highlight a big 6,000 file downloader, it won't hang the UI for several seconds. You can cancel a load by un-highlighting or highlighting something else, and a file-load that takes more than a few seconds will make a popup with its progress.

next week

I've just got one more week before my vacation, so I'll just do some code cleaning and little fixes again. Thanks everyone!
Open

R: 2 / I: 0 / P: 5

Version 529

https://youtube.com/watch?v=zIY17xqTzmk [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v529/Hydrus.Network.529.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v529/Hydrus.Network.529.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v529/Hydrus.Network.529.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v529/Hydrus.Network.529.-.Linux.-.Executable.tar.gz

NOTICE! For everyone but macOS, Version 527 had special update instructions. If you are updating from 526 or earlier, please consult the post here: https://github.com/hydrusnetwork/hydrus/releases/tag/v527

I had a good week. There's a new quick lookup system that lets you search for a file's source without importing it.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

sauce

Every now and then, I am looking at a file outside of hydrus and I can't remember who the artist/character is, or I want to know if I have the artist subbed. When it isn't something the common online source platforms support, I usually download the file, import it to my client, and then do a 'open->similar looking files' on it to see everything I have that looks like it to get more info. I'm basically doing SauceNAO but on my own client. I wanted a way to do this faster.

So, this week, check out the renamed 'system:similar files' predicate on a fresh search page. It now has two tabs. The latter is the normal 'system:similar to' that takes file hashes, if you need to do some manual lookup between imported files, but the first panel is now essentially a paste button. If you copy image data or a file path to your clipboard and paste it there, it'll calculate the similar files info and embed it into the search predicate, letting you search for anything that looks similar to what you have in your clipboard. Give it a go!

I also added a search cache to the main 'similar files' search system. Repeat searches should be massively faster, and searches you re-run with increased distance, like 0 to 4 to 8, and the background similar files search should also accelerate naturally as the cache populates. My 10k file test client's similar files search sped up about 3-4x! I'm not sure what a million+ file client will do, but let me know what you see.
Open

R: 1 / I: 0 / P: 5

Version 528

https://youtube.com/watch?v=I2lgy4_Quko [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v528/Hydrus.Network.528.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v528/Hydrus.Network.528.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v528/Hydrus.Network.528.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v528/Hydrus.Network.528.-.Linux.-.Executable.tar.gz

NOTICE! Version 527 had special update instructions. If you are updating from 526 or earlier, please consult the post here: https://github.com/hydrusnetwork/hydrus/releases/tag/v527

I had an ok week. A bit short on work time, but there's some simple quality of life and bug fixes. The macOS App is also fixed.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

If you regularly use a custom slideshow time, you can now edit the quick-select times shown in the media viewer menu under options->media. It is hacky comma-separated numbers, like the zoom intervals, but have a fiddle and you'll get it.

I improved a bunch of the file search cancel tech. If you put in a search with 'system:imported since 6 months ago', and you then update it before it finishes, it should cancel that ongoing search and start the new one very quickly. This _should_ also vastly improve Client API browsing latency for these situations, since the cancel tech is supposed to fire when a search request is dropped, but let me know how it works IRL.

The time-based system predicate parsing is also more flexible. You can type 'since' or 'before' rather than having to figure out which '<' or '>' you mean, and putting a colon in or 'ago' on the end should parse more reliably. My ideal is to get the text that a system predicate copies to clipboard perfectly parseable. I'd also, fairly soon, like to plug system predicate parsing into the normal file search autocomplete so more users can play with this.

The macOS App had a couple problems last week, with the client->hydrus_client change, and it wasn't booting correctly. Thanks to the users who helped figure this out. Please just install as normal, you still have no special update instructions for the v527 step (it'll just work now!).

next week

Mostly more simple work like this. I'm toying with the idea of a 'paste file/bitmap onto the client, it then does a similar-files search without importing the file' process, basically a 'sauce?' command from your clipboard, which might actually be real easy to add, but we'll see.
Open

R: 1 / I: 0 / P: 5

Version 527

https://www.youtube.com/shorts/xK5SJ6CuezY
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v527/Hydrus.Network.527.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v527/Hydrus.Network.527.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v527/Hydrus.Network.527.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v527/Hydrus.Network.527.-.Linux.-.Executable.tar.gz

I had a good week working on important updates to the build and a new media player prototype.

This build has special update instructions! All users who update from 526 or earlier to 527 or later will have to do this once!

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

build update

A couple of weeks ago, I tried updating OpenCV (our image library) for everyone, but it caused a dll conflict and the program wouldn't boot. I am trying again this week, and since we have to do this change, I am also rolling in a long-planned change to the main exectuable names. 'client' is now 'hydrus_client' and 'server' is now 'hydrus_server'. Some users will have to do a 'clean install', and most will have to update their shortcuts or launch scripts.

As always--but particularly this week--please make a backup before you update. The following instructions are simple, but if anything goes wrong, you can always roll back to a safe backup.

Ok, the instructions, by situation, are:
- If you use the Windows installer, update as normal. It will do the clean install for you. Your start menu 'hydrus client' shortcut should also be overwritten with one to the new executable, but if you use a custom shortcut, you will need to update that yourself.
- If you use one of the normal extract builds, do a clean install. You also need to update any program shortcuts.
- If you use the macOS app, there are no special instructions. Update as normal.
- If you run from source, git pull as normal. If you haven't already, feel free to run setup_venv again to get the new OpenCV. Update any custom launch scripts to point at the new hydrus_client.py file.

A 'clean install' is as here: https://hydrusnetwork.github.io/hydrus/getting_started_installing.html#clean_installs
You are basically going to be deleting everything except your database (to clear out old dll files) and then updating as normal.

To update your shortcut or launch script, simply edit it, and where it says 'install_dir\client.exe', just rename it to 'install_dir\hydrus_client.exe'. For Linux users, that'll be 'install_dir/client'.

If you have a custom firewall rule for the Client API, you'll need to update it for the new executable name too.

And that's it! Let me know if you have any trouble.

QMediaPlayer

This is for advanced users only. Also only users who run from source and use Qt6!

I integrated Qt's built-in media player this week. I hope that in future it may be a fallback for users who cannot run mpv. Unfortunately, while it worked surprisingly well when it worked, it was also very crashy. I've got it stable when running from source, but it seems to insta-crash when in a build, so that's something to figure out later.

If you are an advanced user who runs Qt6 from source, you will now see the new experimental Qt Media Player as a selectable viewer under options->media for audio, animations, and video. It has a number of limitations (check the changelog for the full list), but it does work in most cases. Have a play with it and let me know how you get on.

next week

This week was a lot of work for what felt like little outcome. I was disappointed that QMediaPlayer crashed in the build. In any case, I'm short on getting small things done, so I'll return to that. Just some little fixes and quality of life.

I am taking my vacation week on the 8-14th of June, so I have four releases left before then. I won't go for anything huge, but try, if I can, to tie up some loose ends.
Open

R: 1 / I: 0 / P: 5

Version 526

https://youtube.com/watch?v=2_fxNRtmyAE [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v526/Hydrus.Network.526.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v526/Hydrus.Network.526.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v526/Hydrus.Network.526.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v526/Hydrus.Network.526.-.Linux.-.Executable.tar.gz

I had a mixed week. I was a little short on time, so this is a simple bugfix and quality of life release.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

When you petition in the manage tags/siblings/parents panel on a repository, it typically asks you for a reason for the change. There are some fixed reasons, but now the system will remember the last five custom reasons you enter too. I hope this saves some typing/pasting work for people with clever petitions to make. You can change the number of recent reasons saved (including setting to 0 to disable) under options->tags.

Every multi-column list in the program now has a 'reset the column widths for this type of list' entry in their header right-click menus. It also resizes immediately (an old debug routine that did this for all lists needed a restart). If your thread watcher list goes crazy wide, it should be fixable now! Let me know how it goes.

I added the basic media keys you find on many keyboards to the shortcut system--previous, next, play/pause, volume up, volume down, and mute. On Windows, these keys seem to have a global hook that fires regardless of how I capture it too, so they may not be so useful yet, but feel free to play around with them.

I fixed a stupid logical typo in my new date calculating code that was breaking system time predicates with the same number of months as the current month (like "system:modified time before 2 years 5 months ago" in May). Sorry for the trouble!

next week

There is going to be an important update next week!

Last week, I tried to update OpenCV for everyone, but it caused a dll conflict in the updated build and I quickly rolled it back in a hotfix. We are going to try again next week, which will require 'extracting' users to perform a clean install. Since we are making that change, I have decided to also roll in a long-planned renaming of the program executables. 'client' will become 'hydrus client' and 'server' 'hydrus server'. This part will require updating of custom shortcuts and launch scripts. I will reiterate all this with specific instructions for all platforms in next week's release, and I'll drop reminders for several weeks after.

Next week is also a 'medium size' job week. If I have time and energy, I'd really like to see if I can get the Qt media player working as an mpv fallback.
Open

R: 2 / I: 0 / P: 5

Version 525

https://youtube.com/watch?v=MgNCjYXCxOc [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v525a/Hydrus.Network.525a.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v525a/Hydrus.Network.525a.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v525a/Hydrus.Network.525a.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v525a/Hydrus.Network.525a.-.Linux.-.Executable.tar.gz

There was a problem with the initial v525, so I rolled back the OpenCV update in this new v525a. If you installed v525 and cannot boot, please delete 'install_dir\cv2\cv2.cp39-win_amd64.pyd' or just do a clean install! If you are an advanced Linux user and use the built release above, please check the note at the top of the old v525 release here: https://github.com/hydrusnetwork/hydrus/releases/tag/v525 . Thank you!

I had an ok week. There's a mix of small work and some library updates for everyone.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

new library versions

Some advanced users have been testing new library versions for Qt (user interface), OpenCV (image processing), and, on Windows, mpv (audio/video display). There haven't been any reported problems so far, so I am rolling these new versions into the regular build and automatic setup scripts for source users. If you use one of the builds above, you don't have to do anything special to update--with luck, you'll simply get some subtly improved UI, images, and video. If you are a source user, you might like to run the 'setup_venv' script again this week after pulling, and it'll update you automatically. If you are a Windows source user, you might also like to get the new mpv dll, which requires a simple rename to work, as here: https://hydrusnetwork.github.io/hydrus/running_from_source.html#built_programs

I don't expect this to happen, but if you are on an older OS, there is a chance you will not be able to boot this release. If this is you, make sure you make a backup before updating, and then if you have problems, roll back and let me know, and we'll figure out a solution.

If you can't boot after an update, but a fresh extract on your desktop works fine, then you probably have a dll conflict, so run a clean install: https://hydrusnetwork.github.io/hydrus/getting_started_installing.html#clean_installs

other highlights

I fixed some more time/calendar stuff. Some longer time durations now compensate for leap years and span month-deltas more precisely.

In the media viewer, clicking the 'exif and other embedded metadata' button up top on jpegs now shows their subsampling (e.g. 4:4:4) and if they are progressive.

I fixed an awful bug related to deleting the new inc/dec rating services. If you deleted one of these and found you couldn't load many files, sorry for the trouble--the underlying problem is fixed, and updating will correct any bad records!

The Sankaku downloader gets some new download objects this week to handle their recent shift to md5-based Post URLs. Unfortunately, this is going to cause any sank subscriptions to go slightly bananas on their next check, hitting their file limits since they think they are seeing new URLs. There's no nice solution to this, so you will pretty much just want to wait them out.

next week

More small stuff like this I think. Maybe I'll squeeze some API work in.
Open

R: 1 / I: 0 / P: 5

Version 524

https://youtube.com/watch?v=UdiwBiw5dyo [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v524/Hydrus.Network.524.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v524/Hydrus.Network.524.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v524/Hydrus.Network.524.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v524/Hydrus.Network.524.-.Linux.-.Executable.tar.gz

I had a good week working on bug fixes, quality of life, and more timestamps.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

timestamps

I've added timestamps to the sidecars system. You can import/export any timestamp the file has, so it is now possible to migrate various times (particularly archive and imported) from one client to another. I still need to add some more things to sidecars, e.g. inbox status and ratings, but we are almost at the point where we can do complete client-to-client transfers of all the major metadata. I have plans to make this ultimate goal simple with templated one-click jobs and no need to repeat the file import--just a raw JSON file metadata import/export.

Also, in the duplicate system, file modified dates can now be merged, and by default they will--for 'this file is better', then an earlier timestamp on the worse file will be applied to the better file, and for 'they are the same', both files will get the earlier of the two. If you don't want this to happen, check your duplicate merge options and set 'make no changes'.

Also in duplicates, if you have URLs set to merge, then any associated domain timestamps with those URLs will be earliest-merged too.

Lastly, the program can broadly support pre-1970 timestamps now. 1970-01-01 is a special date in computing, and supporting times before it can be tricky, but I've fudged some general support. It is a bit of a meme, but maybe it'll be neat to apply '1503' to old paintings. The code I was working with says it is good back to 1 AD, but it seems like it can actually go further, ha ha. Anyway, have fun, and let me know if it fails. I know that the '1,504 years 2 months ago' strings are a little inaccurate--it turns out I haven't been counting leap-year days in those calculations, and it shows on the longer durations.

other highlights

Export Folders now have popups while they work. You can turn them off for regular runs, but now Import and Export Folders force-show their working popups when you manually start them from the file menu. These popups have stop buttons, so you can cancel Export Folders mid-job now too. Also, I removed the stupid and confusing 'paused' vs 'run regularly?' duality. Export Folders now just have 'run regularly?'.

I added some special predicate labels, 'system:ratio is square/portrait/landscape', for =/taller/wider than 1:1. They are in the quick-select list for system:dimensions too.

next week

I pushed it too hard this week and exhausted myself. Thankfully, next week is due to be code cleanup, so I'll try and take it easy and just dive into some simple refactoring work and do some small jobs here and there.
Open

R: 1 / I: 0 / P: 5

Version 523

https://youtube.com/watch?v=LGs_vGt0MY8 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v523/Hydrus.Network.523.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v523/Hydrus.Network.523.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v523/Hydrus.Network.523.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v523/Hydrus.Network.523.-.Linux.-.Executable.tar.gz

I had a great couple of weeks working on file timestamps.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

timestamps

Unfortunately, although I did a ton of work here, not much of it is interesting or even visible! Essentially, two weeks ago, the different timestamps your files have--stuff like import time and archived time--were stored all over the place, and now they are stored together. When they are consulted, it now all happens down the same pathway, and I have written a proper update pipeline using this new system.

The user upshot is that you can now right-click->manage->times on a file and edit all its timestamps. This means archived, imported, deleted, previously imported (for undelete), file modified, domain modified, and last viewed times. This new dialog is basic and only for single files for now, but we can now see these things any time and edit little problems. If you end up using this dialog a lot, there's a media shortcut action, 'manage file times' for it.

I've also added 'system:archived' for the 'system:time' stub. It works just like the other time predicates.

I had hoped to integrate this new timestamp manager and update pipeline into the sidecar and Client API systems, but there was so much cleanup to do that I just ran out of time. I'm very glad I did this work, and there's now a much nicer base to work with and extend time tracking in future--like if I want to figure out a 'fill in some good guesses for retroactive archived times', it wouldn't be so much of a hack now--but there is more to do.

other highlights

All multi-column lists across the program now show ▲ or ▼ on the column header they are currently sorted by. This is something I meant to add for ages; now it is done.

All menus across the program now send their commands' longer description text to the main window's status bar. They also show it on tooltips now. These descriptions have been in the program the whole time, but many were difficult or impossible to see. Let's see how annoying the tooltips are--I'll add an option or turn them off if they keep getting in the way.

I fixed a stupid bug from v521 that was causing the advanced deletion dialog to always set the default reason. Sorry for the trouble--it was a testing issue!

next week

A mix of small work, just general catchup. I'd like to also get timestamps working in sidecars and/or the Client API, but we'll see.

R: 207 / I: 17 / P: 6

Q&A Thread

Please feel free to ask questions about hydrus here.

As a reminder, the help and getting started guide is here:

https://hydrusnetwork.github.io/hydrus/help/index.html
Open

R: 2 / I: 0 / P: 6

Version 522

https://youtube.com/watch?v=2AjrLci_AL8 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v522/Hydrus.Network.522.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v522/Hydrus.Network.522.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v522/Hydrus.Network.522.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v522/Hydrus.Network.522.-.Linux.-.Executable.tar.gz

I had a good week mostly working on technical things for advanced users.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

notes in sidecars

You can now import and export file notes with sidecars! There's a technical compromise I had to make here, which is that a note in a sidecar comes in the form 'name: text', both for import and export. If you want to import a whole bunch of notes, you'll need to wangle them into this format, or, if you can, use string processors to get them into that format. There's an escape sequence, ':\ ', for if your name has a colon in, too. Have a play with it and you'll see how it works.

The changelog and a new section in the sidecars help talks about it more, including issues about multi-paragraph notes. I suspect I'll be revisiting this, so let me know how it works for you!

neat user submission

This is advanced.

A user contributed two cool new things for the parsing system:

First, there's a new content parser type, 'http headers', which lets you parse something to be submitted in all subsequent URLs created by the current job. It should let us figure out some unusual tokens and pseudo-login issues.

Second, there's a new String Conversion type that lets you calculate the hex hash of any text for the normal hydrus suite of hashes--md5, sha1, sha256, sha512.

more http header stuff

This is advanced.

On my end, I added full custom http header management to the Client API this week. Basically anything global/domain related that you see under network->data->review http headers can be done via API now. This has been a long time coming, and I am glad it is finally out the door. It should allow for setting up some complex site access via the API.

next week

Next week is a 'medium size job' week. I'm not sure what I want to do. I've been thinking of a complete pass to let us edit timestamps, but there's a million other things in my list, so I'll think about it. I'm still a little busy with some IRL stuff, so it might end up being a two-week release.
Open

R: 1 / I: 0 / P: 6

Version 521

https://youtube.com/watch?v=F7gXLqMe5Bw [Embed]
Sorry these links were for v520 by accident earlier!
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v521/Hydrus.Network.521.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v521/Hydrus.Network.521.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v521/Hydrus.Network.521.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v521/Hydrus.Network.521.-.Linux.-.Executable.tar.gz

I had a good week mostly cleaning things and fixing some unusual issues.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

Building on last week's custom 'sibling' connection, options->tag presentation now lets you edit the 'OR' connector too, and you can now set a specific colour for them both using the existing namespace colours.

If you don't like file viewing statistics tracking your viewing in the archive/delete filter, you can now turn it off. Also, when you are looking at media with duration, the 'max' time it can record for that video or audio is now the larger of your 'max' setting and five times' that duration. So, if you are capped to ten minutes max normally, but you loop or skip around on a twenty minute vid for thirty mins, it'll save the full thirty mins. I'd like to add some pause tech in future too, to account for minimising and pausing media with duration. Thanks to the user who provided feedback on this stuff.

The file delete dialog should be better, in simple and advanced mode, at not overwriting existing delete reasons, which happens most when you manually delete from the trash. If you use this stuff a lot, let me know how it goes.

The rest is a bit advanced and niche. Hydrus parsers can now support unnamespaced tags with colons in, essentially an extension of the existing ':p' tags, and users can enter these tags too by starting a tag with a colon. If you make parsers, check out the changelog for how tag content parsers can now support 'any' vs 'unamespaced' namespace. I also fixed some weird file domain handling in importers.

next week

I really didn't have time for everything I wanted this week, so I'd like to figure out custom/remembered 'reasons' for the petition workflows, more file viewing stuff, http header editing in the API, and maybe notes support in the sidecar system.
Open

R: 2 / I: 0 / P: 6

Version 520

https://youtube.com/watch?v=outcGtbnMuQ [Embed]
If you are an engineer but didn't see the above vid yesterday, I strongly recommend it, the whole thing. We're clearly in the inflection point of this tech right now, and if you aren't experimenting with integrating some variant of it in some way into your work workflow, I think it is time. I just started using it for some scripting jobs I had been putting off this week and it honestly really helped out. There's problems aplenty in the whole space, but I'm pretty AI-pilled right now.
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v520/Hydrus.Network.520.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v520/Hydrus.Network.520.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v520/Hydrus.Network.520.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v520/Hydrus.Network.520.-.Linux.-.Executable.tar.gz

I had a good week. There's a mix of bug fixes and improvements to quality of life, mostly in the tag autocomplete interface.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

So autocomplete has a couple new things. First off, there's a new 'all files ever imported/deleted' entry under the file domain button. This searches everything your client has ever seen, runs super fast, and works with 'all known tags'. Also, when editing tags, I've tightened up the selection rules and reduced the sibling spam--now, you should get the 'ideal' of what you typed at the top, then what you actually typed (if that differs), and then normal results. Also, the various paste buttons tucked into edit autocompletes now only ever add items--if any of what you paste is already in the accompanying list, it won't be removed now.

And I changed the '(displays as xxx)' sibling indicator. It is now a simple unicode arrow, and because I wanted to play with gradients, I made it fade its colour from one namespace to another. If you don't like the text or the fade, you can change them under options->tag presentation.

There's a new Deviant Art downloader, thanks to a user's work, that pulls the 'original' quality from the 'download' button, which some creators turn on for logged-in users. There's also five new File URL URL Classes, which you can search with system:known url, that will categorise your existing DA downloads according to their quality. Enterprising advanced users might like to play around with this!

next week

Next week is slated for code cleanup. I want to do some boring database breakup work!
Open

R: 1 / I: 0 / P: 6

Version 519

https://youtube.com/watch?v=NNNRlPfoC7I [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v519/Hydrus.Network.519.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v519/Hydrus.Network.519.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v519/Hydrus.Network.519.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v519/Hydrus.Network.519.-.Linux.-.Executable.tar.gz

I had a good couple of weeks prototyping a new rating service.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

inc/dec ratings

So, under services->manage services, there's a new 'inc/dec rating' (for increment/decrement). It works pretty much like all the other ratings, and goes in the same places, but it is a simple number counter--left-click to add, right to subtract. Middle-click lets you edit directly. I've wired it up for system:rating, too, with only one oddness--since every file starts at count 0, there's no concept of 'not rated' for this service.

If you want to count how often an image makes you smile or anything else, give it a go! I mostly did this for fun, but let me know how it works out.

other highlights

The manage siblings and parents dialogs now both try to show more of the full 'chain' of related pairs as you enter new tags. The manage parents one also has a new checkbox that let's you decide if you want to really show everything, including cousins (which can be overwhelming on the bigger tags), or just (grand)children and (grand)parents. I'm planning an overhaul to these dialogs, one where they don't have to load everything on boot, and this is a first step.

I fixed my native animation renderer drawing things too small when your UI scale is >100%!

Server janitors get a nicer account modification dialog this week, and a return of the old 'superban' functionality, which deletes everything an account has uploaded. Check the changelog for full details!

next week

I'm sorry to say I'm exhausted and lagging a bit. I'll clear some small work, nothing too special!
Open

R: 3 / I: 1 / P: 6

Version 518

https://youtube.com/watch?v=lDWfgUjlEMk [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v518/Hydrus.Network.518.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v518/Hydrus.Network.518.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v518/Hydrus.Network.518.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v518/Hydrus.Network.518.-.Linux.-.Executable.tar.gz

I had an ok week. Tag autocomplete search should be a bit nicer to work with today!

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

tag autocomplete

I rewrote the part of tag autocomplete search that handles showing you early results. The whole thing works better and is smoother.

First off, every time you type something like 'cat' and the search is taking a while, you should now always get 'cat' and 'species:cat' and anything else that might match, with counts, and your selection will be preserved when the full results come in. Quick-entering a tag on both search pages and the manage tags dialog should be faster, easier, and more reliable. This is roughly how things used to work, a long time ago, but the various updates and new options I've added over the years made it patchy and buggy--it should be working again, and with some new quality of life improvements.

Secondly, I've improved sibling lookup in the manage tags dialog. You should now always see siblings, on the pre-results above and the final results, even if the siblings have no count. I think we might want some options for handling how these display and rank (e.g. 'always put the ideal of what I typed at the top of the list'), so see what you think.

Lastly, when you search for tags on a page of thumbnail results, you now get some pre-fetch results very quickly! The full results, which have extra sibling lookup data, will fill in a few seconds later, but, fingers crossed, there will be no more huge waits for anything on thumbnail tag lookups!

misc

The system:time predicates now support hh:mm, like '10:30pm', on the calendar search! Should be easy to cut a day in half now.

I believe I have optimised some content update code. If you have a client with a large page of thumbnails, particularly if that page uses collect-by, let me know if your subscriptions and/or 'upload pending mappings' run a bit smoother.

If you are an advanced Windows user, please check out the bottom of the changelog for an mpv test. I'd like to test out a new dll on a wide range of machines.

next week

My ongoing family IRL stuff is winding down. I'm still going to have extra responsibilities for another month or so, but it shouldn't be too bad. I'd like to have a medium-size job week, and I want to have some fun, so I'm going to try to make a 'click to increment' integer rating service.
Open

R: 1 / I: 0 / P: 6

Version 517

https://youtube.com/watch?v=q4veLQfXFLI [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v517/Hydrus.Network.517.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v517/Hydrus.Network.517.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v517/Hydrus.Network.517.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v517/Hydrus.Network.517.-.Linux.-.Executable.tar.gz

I had another short week, but I'm generally happy with the work, which was a mix of different things.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

Thanks to a user's work, Export Folders now support exporting to symlinks! If you are on Windows, you may need to run hydrus in Admin mode for any symlink exporting to work (or, on Windows Pro, I believe you can mess around with Group Policy Editor to give normal user accounts the permission).

The newly upgraded 'Related Tags' shouldn't suggest superfluous siblings or parents any more! I'm pretty sure I got the logic right here, but let me know if it is still suggesting the wrong things somewhere.

Hydrus can now show a checkerboard pattern behind a transparent image instead of the normal background colour. There's two new checkboxes, in options->media and ->duplicates, to control it--by default I have set it just to show in the duplicate filter, where it is useful to have transparency pop when you are comparing two similar files.

next week

Yet again, I'm a little busy with IRL again next week, but otherwise should be normal. I'll do more work like this! Thanks for your patience!
Open

R: 1 / I: 0 / P: 6

Version 516

https://youtube.com/watch?v=d1zDNBIswek [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v516/Hydrus.Network.516.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v516/Hydrus.Network.516.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v516/Hydrus.Network.516.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v516/Hydrus.Network.516.-.Linux.-.Executable.tar.gz

I had a good week that was unfortunately short again. There's some options to tune the new 'related tags' search and some misc little work.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

related tags

I have added three new settings to options->tag suggestions for related tags. They are fairly advanced and deal with the technical details of the search, so I only recommend poking around if you have been enjoying this system and want to change how it works.

First off is a 'concurrence threshold'. This is a percentage throttle that determines how 'related' a suggestion tag needs to be to its search tag to be included. The higher you make it, the fewer--but more relevant--suggestion tags you will get.

Next are two controls that boost or reduce the power of different namespaces in the search. If you want the character tags already on the file to do most of the searching, or if you want to see creator tags most of all, you can alter the weights here. You can also easily set a zero weight to say 'I don't want to see any filename: tags suggested' and so on. I have set them up with some reasonable defaults, so tweak what I've put in and see what you like.

I also fixed a stupid bug where you couldn't hide related tags while in side-by-side mode. Sorry for the trouble!

misc

APNGs that play only once now do that!

The new sidecar system now has import/export/duplicate buttons, so you can save and transfer your work! A favourites system will come in the future too, I'm sure.

The Client API has some more 'king' info in the 'get_file_relationships' command.

next week

The big IRL thing that I had to do is now behind me. I've got about six more weeks of lighter 'hydev has to do some family responsibility stuff', and then I am back to normal schedule. Next week I would like to continue on the small jobs, just to catch up on things that I have recently missed.
Open

R: 0 / I: 0 / P: 6

Get Professional Voiceovers in 3 Clicks with Speechelo

Tired of spending hours searching for voiceover artists? Save time and money with Speechelo. Instantly transform any text into a human-sounding voiceover with just 3 clicks.

Speechelo uses AI to generate professional-quality voiceovers in various languages and accents. Choose from male or female voices, and get your voiceovers done in minutes, not hours or days.

Transform your text into a professional voiceover now with Speechelo.

https://3e9bfyidnydw3k8l34n2v1snci.hop.clickbank.net

Best regards
Thor Andersson
Open

R: 1 / I: 0 / P: 6

Version 515

https://youtube.com/watch?v=Q4brYfcHQR0 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v515/Hydrus.Network.515.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v515/Hydrus.Network.515.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v515/Hydrus.Network.515.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v515/Hydrus.Network.515.-.Linux.-.Executable.tar.gz

I had a challenging week with little work time, but I polished last week's related tags work.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

related tags

I've improved the 'related tags' test I did last week and brushed up the UI, and I'm happy enough with it that I am making it default-on for all users. You will see it after you update in your manage tags dialogs. If you do not like it, you can turn it off or customise its timings under options->tag suggestions!

For the technical details, I've cleaned up the new algorithm a bit, made it run faster in all cases and massively faster for very large-count tags, and now it works on the same 'cancel after 1,200ms' tech the old buttons ran on. Those buttons now use the new algorithm exclusively, there's a new label that reports how particular searches went. There's also a 'my files' vs 'all known files' on/off button to change the search domain ('new 1' vs 'new 2' in the test last week).

Also, if your suggested tags are set to show 'side-by-side', they now get nicer boxes around them, and if they are set to show in a notebook, you can set which is the default page under options->tag suggestions. Let me know how you get on with it!

next week

Unfortunately some stressful IRL landed on my head this week, and I'll have some extra responsibilities for the next 6-8 weeks. Nothing dreadful, but it will cut into my work time a bit. Please expect some more light and possibly skipped releases for the next bit, thanks!

Next week I'll see if I can just do some simple code cleaning and more small jobs.
Open

R: 4 / I: 0 / P: 7

Version 514

https://youtube.com/watch?v=PjfkAuUUq5w [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v514/Hydrus.Network.514.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v514/Hydrus.Network.514.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v514/Hydrus.Network.514.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v514/Hydrus.Network.514.-.Linux.-.Executable.tar.gz

I had a good couple of weeks doing a variety of work. The changelog is long, mostly fixes and improvements that don't change much, but if you use the twitter downloader or the Client API, make sure to read before you update.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

twitter

Twitter changed their policies regarding third-party interfaces recently, and it seems as part of that they took down the 'syndication' API our nice twitter downloaders were using. None of them work any more!

A user has kindly figured out a patch, which today's update will apply. It will throw up a yes/no dialog asking whether to do this, which almost everyone will want to click 'yes' on.

Unfortunately, this downloader can only get the first 0-20 tweets in the latest 20 tweets and retweets an account makes!! That's it! When you update, please check your twitter subs to make sure they moved to this new downloader correctly, and you might like to tell them to check more frequently.

I expect twitter to continue in this direction. I don't know if nitter or any other mirrors will be a viable alternative. If any of your twitter subs get posted in any normal booru, I strongly recommend you pursue those instead!

Client API

I am retiring some old data structures from the Client API today. If you use hyshare, please make sure you update to 0.11.2 or newer. If you use hyextract, please update to 0.4.1. If you use Hydrus Companion, I think you'll be good with the latest release for everything. The main changes on my end are to do with tag-viewing and tag-editing, so if you do a lot of that, make sure you have a backup and do a test of everything after you update.

If you have custom scripts, check the changelog. Mostly services are going to be referred to with service_keys in future, so I am removing 'service_name' structures and only supporting 'service_name' arguments in a legacy state. Please move to the /get_service(s) calls to figure out the service keys you need to use, as I will slowly retire the last legacy support later this year. Note that all users can now copy their service keys from review services.

As a side thing, I also rewrote how v413's /set_file_relationships call works. The old 'list of lists' parameter should still work, but a user reminded me that an Object with proper names would be a lot easier to extend in future, so I just rewrote it. This call is sufficiently new and advanced-user-only that I'm ok changing so quick, but if you set something up for it already, check the docs!
Open

R: 4 / I: 0 / P: 7

Version 513

https://youtube.com/watch?v=D5T9RByz9i8 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v513/Hydrus.Network.513.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v513/Hydrus.Network.513.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v513/Hydrus.Network.513.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v513/Hydrus.Network.513.-.Linux.-.Executable.tar.gz

I had a good week focusing on adding the duplicates system to the Client API. There is nothing else, so today's release is only for advanced users.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

duplicates with Client API

The main duplicate commands are now available. Obviously check the changelog and Client API help out for full explanation, but the general gist is 1) There is a new 'manage file relationships' permission to gain access, and 2) You can fetch and create relationships, but you can't delete them yet. I will add these commands in the future.

https://hydrusnetwork.github.io/hydrus/developer_api.html#managing_file_relationships
https://hydrusnetwork.github.io/hydrus/duplicates.html#duplicates_advanced

There's some new enums to grapple with, and you can be dealing with two searches at once when fetching potential pairs. Please have a read, try things out, and let me know where it goes wrong. I expose the ugly technical side of things, so if it isn't clear what a king is, I may need to brush up the duplicates help etc...

I expect to keep working on this in the near future, including adding more search tech as I do related 'mulitple local file services' expansions to the Client API, and in the more distant future as I expand alternates. While I was in here this week, I realised I could brush up and clarify some of the edge-case logic too, as I wasn't sure myself on every situation. Again, let me know what seems confusing or wrong for you, and we'll iterate.

next week

A mix of small jobs, and it would be nice to add some remove/dissolve commands for this new duplicates Client API.

On a side note, I migrated to a new dev machine this week, upgrading from a Minis Forum X35G to a TH80. I'm pretty sure the old machine was a 2-core CPU while the new is 8, ha ha ha, so my working session feels super smooth now. I recommend these little machines if you need a decent office computer (the old one also drove two 4k@60Hz monitors no problem), or a younger family member's first big boy computer.
Open

R: 1 / I: 0 / P: 7

Version 512

https://youtube.com/watch?v=jkYg2eV3xBc [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v512/Hydrus.Network.512.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v512/Hydrus.Network.512.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v512/Hydrus.Network.512.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v512/Hydrus.Network.512.-.Linux.-.Executable.tar.gz

I had a great week back at work. There's a mix of different things, but particularly duplicate search improvements.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

duplicates

You can now set a search in the duplicates processing page for 'one file comes from search A, the other comes from search B'! It is now super easy to, say, find png vs jpeg pixel duplicates.

I also improved the overall accuracy and precision of the duplicate search logic. When I looked closer at this code, I wasn't always getting pixel duplicates or search distance correct when actually fetching results to show you, or I was selecting bad representatives of larger groups in the final step. This was particularly true in 'show some random potential pairs', which was showing too many files, often outside the search scope. There's some other bells and whistles that are subtle but should make the results feel more 'human'--for instance, in 'show some random potential pairs', if you are set to 'at least one file matches the search', then the first of the random group will definitely match the search, and if you are set to 'both files match different searches', then the first file matches the first and the others match the second.

I have made several changes here, sometimes significantly increasing the CPU load. In testing, things generally seemed to be running at ok speed, but let me know if your IRL complicated situation gets laggy. Also let me know if you discover any miscounts.

misc

When you select thumbnails or right-click on them, the labels in the menu and status bar are a bit nicer--it can now say 'x images' or 'x mp4s' instead of the flat 'x files' in more places, and any time you have a selection of files with duration, it will say the total duration.

v511's supersampled thumbnails now look good on any UI scale. If you have a multi-UI-scale setup, on multiple monitors or not, you should now set the new setting (under options->thumbnails) to your maximum UI scale and hydrus should look good everywhere!

If you want to import or export a .txt sidecar with a separator character other than newline (e.g. comma, for CSV format), you can now do this! You can set any separator characters you like.

The Client API can now refresh pages.

next week

Next week will be a medium-sized job week. I want to spend extra time adding the duplicates system to the Client API.
Open

R: 4 / I: 0 / P: 7

Version 511

https://youtube.com/watch?v=_3ZC45Q82pg [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v511/Hydrus.Network.511.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v511/Hydrus.Network.511.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v511/Hydrus.Network.511.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v511/Hydrus.Network.511.-.Linux.-.Executable.tar.gz

𝕸𝖊𝖗𝖗𝖞 𝕮𝖍𝖗𝖎𝖘𝖙𝖒𝖆𝖘!

I had a good week fixing some things and adding some new options.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

First off, if you uploaded to v510 and had problems uploading to the PTR, sorry for the trouble! I missed something last week with an object update, and it impacted the network protocol. Thanks for the reports--I figured out a serverside patch and we updated the PTR.

I also fixed some logic in 'file import options' where it uses URLs and parsed hashes to predict if a file is 'already in db' or 'previously deleted' before a download. Some edge cases, particularly where many urls were mapped to one file, were not working quite right, and hashes tended to be overly dispositive. Ultimately, this thing should now be better at saving you bandwidth. The options here, which can be seen by advanced users, have updates too--check the changelog and tooltips for explanations.

If you have a large monitor and UI scale >100%, your thumbnails have looked bad for a while. Please check the new 'Thumbnail UI scale supersampling %' option under options->thumbnails. Set it to your monitor's UI scale, and your thumbnails should regen to be crisp. Let me know how it goes!

If you are into clever tag searching, check out the new 'unnamespaced input gives (any namespace) wildcard results' option under services->manage tag display and search. This is a first attempt to bring back the old 'unnamespaced search tags search namespaced variants' tech we used to have. It makes it so typing in an unnamespaced tag will give you the '*:subtag' variants quickly. These wildcards are labelled specially now, too.

next week

I am taking my Christmas vacation week! I'm going to spend some time with family and get lost in vidya. The week after, I'll see about sinking my teeth back into the serverside tag filter. v512 should be out January the 4th.

Thanks everyone!
Open

R: 4 / I: 0 / P: 7

Version 508

https://youtube.com/watch?v=81tEdqBvHVs [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v508/Hydrus.Network.508.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v508/Hydrus.Network.508.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v508/Hydrus.Network.508.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v508/Hydrus.Network.508.-.Linux.-.Executable.tar.gz

I had a great week fixing some bugs, adding hash lookup to the Client API, and finally attaching a Tag Filter to the PTR.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

tag filter

For normal users: the PTR will start filtering out some bad tags automatically, you don't have to do anything.

The big job I wanted to do this week, adding a Tag Filter to the PTR (and other tag repositories), went a lot better than I expected. I split the work into four pieces and got the first one done. If you are a tag repository administrator, you can now edit a Tag Filter for your repo under the services->administrate services->tag repo->edit tag filter menu. Your tag repo will silently discard any tag mapping uploads that do not pass the filter. So, if you don't want 'tagme' or 'source:some-borked-url' tags, you can now filter them at the point of upload and no one has to create or process any petitions in future.

Regular users will sync to the set filter and can see it under services->review services. Users in advanced mode will also get popups on specific rule changes. The next step here will be to expose this filter more, showing it in manage tags and similar UI, not even trying to pend tags that don't match the filter, and retroactively fixing mappings that already exist on the server (including hard-replacing since-sibling'd tags). I'd like to get at least some of this done before the end of the year, but I'm sure some will spill into 2023.

other highlights

There is a new 'file relationships: show x' action in the 'media' shortcut set, which will show a file's potential dupes, actual dupes, alternates, or false positives in a new page, just like the 'show x' actions that are buried in the thumbnail menu. This works in the media viewer too.

I fixed a recent stupid bug in the duplicate filter that meant if you did a batch of work with only file deletes, they were not going through. Sorry for the trouble! The thing now calculates and reports duplicate decisions and manual file deletes separately. I also added a (+50%, -33%) label to the file size comparison statement, just as a test. The numbers are correct, but I'm not sure I like it--let me know what you think.

If you work with the downloader system, URL Classes get two new checkboxes this week, 'do not allow any extra path components/parameters', which stop matching URLs that have too-long paths or parameters. It should help fix up some situations where hydrus was having trouble figuring out which URL Class to match sets of nested URLs to, particularly when the Gallery URL is 'longer' than the Post URL.

With the new
/get_files/file_hashes
call, the Client API can now do lookups between sha256, md5, sha1, and sha512 hashes. If you provide some hashes in one of these types, the API can now give you known matches for any of the other types. Let's say you have a bunch of sha1 hashes and want to see if you have them imported, or you have a bunch of sha256 hashes for files you have and want to get their known md5s for an external site lookup, this is now possible. The client typically knows all four hashes for every file it has ever imported, so you can do your own 'already deleted' checks too.

next week

More like this. Some small work and some more janitor/serverside stuff. Only three more releases this year, so I'll try to keep things simple.
Open

R: 4 / I: 0 / P: 7

Version 510

https://youtube.com/watch?v=8H40p44oo1U [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v510/Hydrus.Network.510.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v510/Hydrus.Network.510.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v510/Hydrus.Network.510.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v510/Hydrus.Network.510.-.Linux.-.Executable.tar.gz

I had a good week improving notes and UI quality of life. It is Hydrus's 11th birthday today!

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

File notes got several updates this week, mostly for copying notes from one file to another. First off, the 'duplicate metadata merge options' finally support notes! All users will get some basic 'yes please, merge my notes' defaults set, copying from worst to best and copying both ways when the files are the same quality. If you do a lot of duplicate filtering and have been waiting for this, double check your options (under the duplicates processing page) to make sure you are happy. Relatedly, the notes dialog now has copy/paste buttons for copying from one file to another. To get this all to work, these actions use some Note Import Options tech to handle conflicts, generally by extending existing sub-notes where possible and renaming on conflicts. If you want to work with notes in the Client API, this 'easy merge' tech is also optionally available in the Client API. Let me know how this works for you--I think I am going to have to adjust the logic to handle two-way related note merges a little better, so feedback on IRL situations would be great.

I significantly improved how the taglist handles shift-selections and drag-selections. A bunch of position and logic jank is cleaned up, and now you can 'undo' a multi-selection, just like I did for thumbnails recently. Best way to test this is just click and drag down the list, and then move your mouse up--it undoes the selection as you move up, and it remembers what was originally selected before! Also, holding ctrl down to start a deselection drag is cleaner and more human. Give it a go, and you'll feel the difference.

next week

Just one more week of work and then I am on my vacation. I'll do some more cleanup/small jobs and try to catch up on github bug reports, which I have been neglecting.
Open

R: 6 / I: 0 / P: 7

Version 509

https://youtube.com/watch?v=pLlAvGG6AYg [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v509/Hydrus.Network.509.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v509/Hydrus.Network.509.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v509/Hydrus.Network.509.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v509/Hydrus.Network.509.-.Linux.-.Executable.tar.gz

I had a good week mostly fixing bugs.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

important fixes

When adding the tag filter last week, some users got errors appearing on shutdown. Thankfully, the problem was mostly harmless, but it was a surprise, and annoying, and I regret it. It believe in a legacy issue that some users had a different internal datatype that made it difficult to save the synced tag filter. I am confident I have fixed it this week, but let me know if you have any more trouble.

The macOS app has had trouble booting since v505. Thanks to help from several users, we finally pinned down the issue and got a fix. Also, we are switching the macOS release to a slightly different version of Qt, the UI library, and the .dmg is now half the size (548MB->258MB!).

other highlights

There are several menu-spawning buttons across the program that can be changed by scrolling the mouse over them. Good example is the 'ascending/descending' buttons for file sort or the taglist sort. If you find this behaviour annoying, you can now turn it off under options->gui!

If you run from source and are willing to test new things out, you might like to run setup-venv again this week. The advanced install route now has a '(t)est' Qt version that tries out 6.4.0. This new release has a heap of bug fixes, but it caused problems when it first launched, and I'd like to get some feedback on it before again trying to roll it into the built releases.

I fixed a bunch of bad thumbnail handling/rendering, but there is more work to do.

next week

A cleanup week. More small fixes and UI tweaks, nothing too dramatic.
Open

R: 4 / I: 0 / P: 7

Version 507

https://youtube.com/watch?v=21uh28Z77Xg [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v507/Hydrus.Network.507.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v507/Hydrus.Network.507.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v507/Hydrus.Network.507.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v507/Hydrus.Network.507.-.Linux.-.Executable.tar.gz

I had an ok week. There are some bug fixes, more Client API data, and a new tool for server janitors.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

This week is a little slim on updates for normal users. Mostly bug fixes, and updates to the 'running from source' setup scripts, and there is a neat fix/improvement when you apply set-duplicate actions to many thumbnails at once. Check the changelog if you want to read more!

For advanced users, the Client API gets some more of the service name, type, and type_pretty variables, and 'all known tags' appears on the new 'tags' structure in the 'file_metadata' call (I forgot to add it last week!).

For hydrus repository admins and janitors, there is an update to the petition workflow. The petitions page now has an 'account id' field where you can paste an account id and see petition counts (and petitions) just from that account. Servers have to update for this to work.

next week

It felt good to clean some ancient server code this week. There's plenty more I want to get done before the end of the year, so I'll keep focusing on it. This is scheduled to be a 'medium size job' week, so if I can figure it out, I'd really like to have a basic tag filter implemented so the server can auto-discard tags on upload. We'll see!
Open

R: 1 / I: 0 / P: 7

Version 506

https://youtube.com/watch?v=YJu0iXn-T_U [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v506/Hydrus.Network.506.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v506/Hydrus.Network.506.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v506/Hydrus.Network.506.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v506/Hydrus.Network.506.-.Linux.-.Executable.tar.gz

I had a great week catching up on a variety of small cleanup, fixes, client api, sidecars, and ui quality of life.

Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html

highlights

system:rating gets the same overhaul system:time had recently. Some nicer labels and clearer interaction. If you use this a lot, have a poke around, because the click logic has changed a little--to say 'is not rated', you now say the rating 'is' 'nothing'.

The media viewer has some new shortcut actions for 'zoom: 100/canvas/default'.

If you right-click on a file that has more than one recorded modified date, you can now see them all on the top submenu. This should be true for anything downloaded inside the client in the past few months. Notice how that the typical modified date you see is the most reasonable minimum value of all recorded.

The new expanded sidecar import/export system gets more options for the sidecar filenames. You can now take out the '.jpg' component, or, if you need to, go crazy with a full String Converter on the filename. There's also some test UI so you can enter an example media filename and see the expected sidecar filename.

The Client API's '/get_files/file_metadata' gets several updates this week. If you do API stuff, please check it out. There's a new combined 'tags' structure, and I am retiring the old 'service_blah_to_blah_tags' structures over the next eight weeks, so watch out! EDIT: I did one part of this too quick. If you use hydrus web, hyextract, or hyshare, or hydrus companion, wait a couple of days before you update to 506--the guys who work on these are updating them to catch the change I made here.

The 'running from source' help and scripts get some more fixes and updates for unusual situations. Thank you everyone for the reports and recommendations. If you had trouble with this stuff previously, please try again!

next week

The end of the year is coming fast, and the one thing I wanted to do--janitor workflow improvements--isn't anywhere near where I want it. I am going to try to focus on serverside stuff until I feel better about this, so normal user features may be a little barebones for a bit.
Open

R: 5 / I: 1 / P: 7

Version 465

https://youtube.com/watch?v=QgNEYyOlST0 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v465/Hydrus.Network.465.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v465/Hydrus.Network.465.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v465/Hydrus.Network.465.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v465/Hydrus.Network.465.-.Linux.-.Executable.tar.gz

I had a good week. I focused on background improvements to the database. If you have a big client, it will take several minutes to update the database. My 2.4 million file PTR-syncing IRL client took 8 minutes.

misc

First off, some quick fixes: I fixed unnamespaced wildcard tag searches (e.g. 'sam*s'), which were recently not producing namespaced results. I also improved handling in the new ICC profile system when the ICC profile with an image being loaded was completely borked.

Also, it seems the login-required 'gelbooru favourites by user id' downloader recently(?) broke severely--as well as pulling favourite links, it was also parsing and visiting the 'remove from favourites' link and deleting them! I fixed the gelbooru gallery parser to never pull a delete link again, but if you used this parser to grab all your favourites, please check your favourites list, and if you can, dig that downloader back out of hydrus and do a mass 'open sources' on the file log or the thumbnails so you can re-favourite any files that were dropped. Thanks to the users who noticed what was going on here and figured out what needed to be fixed.

I added some new options. The 'default tag service in manage tags' choice is reset this week, and it now starts off working different: it now remembers the last used service the next time you open a tag service dialog. Let's see if this works out, but if you don't like it, you can go back to a fixed default under options->tags. There's also a new checkbox in options->search that lets you default new file search pages to 'searching paused' rather than 'searching immediately'.

ipfs and deleted files search

This is mostly database prep for future multiple local file service expansions.

If you have an IPFS service, you can now search it in a normal search page. Just switch 'my files' on the autocomplete dropdown to your IPFS service and you should be able to search it with tag counts and everything. IPFS works a little differently to a normal file service in hydrus, so this will need some more work to get those workflows integrated. Also, while an IPFS service in hydrus only knows about your pins atm, in future I would like hydrus to harvest more info from external sources so this search space could potentially populate with remote files that you could then command the client to download.

In a related but quieter move, I did the same thing here for a new 'deleted files' umbrella domain. It'll take a few minutes to calculate this search cache on update. This will be of use in the near future when I let advanced users start searching deleted files.
Open

R: 2 / I: 0 / P: 8

Version 505

https://youtube.com/watch?v=Z1MV0WbrNwY [Embed]
Windows release got a hotfix! If you got 505a right after the release was posted and everything is a bad darkmode, get the new one!
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v505b/Hydrus.Network.505b.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v505b/Hydrus.Network.505b.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v505a/Hydrus.Network.505a.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v505a/Hydrus.Network.505a.-.Linux.-.Executable.tar.gz

I had a great couple of weeks fixing bugs, exposing EXIF and other embedded metadata better, and making it easier for anyone to run the client from source.

Full changelog (big one this week): https://hydrusnetwork.github.io/hydrus/changelog.html

EXIF

I added tentative EXIF support a little while ago. It wasn't very good--it never knew if a file had EXIF before you checked, so it was inconvenient, and non searchable--but the basic framework was there. This week I make that prototype more useful.

First off, the client doesn't just look at EXIF. It also scans images and animations for miscellaneous 'human-readable embedded metadata'. This is often some technical timing or DPI data, or information about the program that created the file, but, most neatly, for the new AI/ML-drawn images everyone has been playing with, many of the generation engines embed the creation prompt in the header of the output png, and this is now viewable in the client!

Secondly, the client now knows ahead of time which files have this data to show. A new file maintenance job will be scheduled on update for all your existing images and animations to retroactively check for this, and they will fill in in the background over the next few weeks. You can now search for which files have known EXIF or other embedded metadata under a new combined 'system:embedded metadata' predicate, which works like 'system:dimensions' and also bundles the old 'system:has icc profile' predicates.

Also, the 'cog' button in the media viewer's top hover window where you would check for EXIF is replaced by a 'text on window' icon that is only in view if the file has something to show.

Have a play with this and let me know how it goes. The next step here will be to store the actual keys and values of EXIF and other metadata in the database so you can search them specifically. It should be possible to allow some form of 'system:EXIF ISO level>400' or 'system:has "parameters" embedded text value'.

running from source

I have written Linux (.sh) and macOS (.command) versions of the 'running from source' easy-setup scripts. The help is updated too, here:

https://hydrusnetwork.github.io/hydrus/running_from_source.html

I've also updated the setup script and process to be simpler and give you guidance on every decision. If you have had trouble getting the builds to work on your OS, please try running from source from now on. Running from source is the best way to relieve compatibility problems.

I've been working with some users to get the Linux build, as linked above, to have better mpv support. We figured out a solution (basically rolling back some libraries to improve compatibility), so more users should get good mpv off the bat from the build, but the duct tape is really straining here. if you have any trouble with it, or if you are running Ubuntu 22.04 equivalent, I strongly recommend you just move to running from source.
Open

R: 6 / I: 0 / P: 8

Version 504

https://youtube.com/watch?v=W-59sWasI98 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v504/Hydrus.Network.504.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v504/Hydrus.Network.504.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v504/Hydrus.Network.504.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v504/Hydrus.Network.504.-.Linux.-.Executable.tar.gz

I had a great week focusing on improving sidecar import/export. Importing and exporting tags and URLs with files is now easy.

Qt5

As a reminder, I am no longer putting out Qt5 builds. If you are on Windows 7 (and I heard one instance of 8.1), or a similarly old OS, you probably cannot run the hydrus builds any more. You can still update, but you need to run from source from now on. The good news is that this is now much easier to do on Windows--please check the help here: https://hydrusnetwork.github.io/hydrus/running_from_source.html

Windows source users should also get a nice hydrus taskbar icon this week.

sidecars

There's a semi-advanced feature in hydrus where you can tell the client to look for (on file import) or create (on file export) additional files (sidecars) along with your media. Until now, this has been fixed to 'filename.jpg.txt' files that would include tags for the file separated by newlines. This system expands radically this week.

What was static before is now flexible. You can set up multiple workflows for multiple sidecars in one task, grab or send URLs as well as tags, parse or output formatted JSON instead of .txt, and transform or filter the text at every stage.

This is available on manual imports and exports, and import folders and export folders. For manual imports, it becomes a new 'sidecars' page tab on the 'add tags before import' window, and for all the others it is now a button on the UI. If you are interested in this tech, please check it out and let me know what is confusing. It is quite powerful now, and users who are familiar with the concepts involved (and the parsing system) should find it obvious, but I think I'll need to add some help text and a full help guide for users new to it.

All existing 'neighbouring .txt' settings on import folders, export folders, and the manual file export window should be migrated to the new system when you update. Let me know if there are any problems!

There are many potential next steps here. I'd like a 'favourites' system first, so it is possible to save/load/share workflows, and then we can think about more content types (ratings, notes) and .xml parsing/export.
Open

R: 11 / I: 2 / P: 8

Version 503

https://youtube.com/watch?v=VFyu1qvub7A [Embed]
windows
Qt5 zip: https://github.com/hydrusnetwork/hydrus/releases/download/v503/Hydrus.Network.503.-.Windows.Qt5.-.Extract.only.zip
Qt6 zip: https://github.com/hydrusnetwork/hydrus/releases/download/v503/Hydrus.Network.503.-.Windows.Qt6.-.Extract.only.zip
Qt6 exe: https://github.com/hydrusnetwork/hydrus/releases/download/v503/Hydrus.Network.503.-.Windows.Qt6.-.Installer.exe
macOS
Qt5 app: https://github.com/hydrusnetwork/hydrus/releases/download/v503/Hydrus.Network.503.-.macOS.Qt5.-.App.dmg
Qt6 app: https://github.com/hydrusnetwork/hydrus/releases/download/v503/Hydrus.Network.503.-.macOS.Qt6.-.App.dmg
linux
Qt5 tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v503/Hydrus.Network.503.-.Linux.Qt5.-.Executable.tar.gz
Qt6 tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v503/Hydrus.Network.503.-.Linux.Qt6.-.Executable.tar.gz

I had a good week improving how the program works behind the scenes. The build is more reliable, and it is now easy to run the program from source in Windows.

reliability

tl;dr: The program breaks less now and in the future.

Just as I was hotfixing a fresh boot problem last week, in the intervening time yet another boot problem arrived! We've been hit by several of these in the past couple of months, and I hate causing people this stress, so I decided to put some time into the way the official release is put together. I really hope this will not happen as much in future.

This is mostly technical, but the requirements.txts that prep the program environment are simpler, laid out for easier reading and maintenance, and have frozen version numbers for important libraries. I also put time into the build script itself, doing some cleanup and anti-deprecation work. The issues with PySide6 and dateutil are fixed.

I want the program to be more stable going forward when I click 'publish'. It has been duct-tape-and-a-prayer for too long. Thank you to those who report problems, and please keep them coming if and when this new system fails. I want to keep improving things here, especially too as more users run from source.

running from source

Absent any more issues, I expect this to be the last week I put out a Qt5 build. The Qt6 build is stable with no giant outstanding problems. Users on Windows 7 (and some similarly old Linux and macOS versions) cannot run Qt6, so they will need an alternate solution.

It is now easy to run the program from source in Windows. I have written several batch files to make it simple, no technical knowledge required beyond the ability to open a zip file, and the help has a brand new full walkthrough. If you want to run Qt5, the way to do so is equally simple and clearly labelled. Updating, too, is very similar to how you already update with the Windows extract zip.

https://hydrusnetwork.github.io/hydrus/running_from_source.html

The process is also simpler and clearer for advanced users who want to use 'git', 'venv', and 'pip' manually. If you have been running hydrus from your system python, you might want to check out the help to learn how to set up a venv--it is really easy these days, and you can save yourself a real headache.

I'm obviously interested in reports if any of this fails for you; or if there is anything you find confusing. There is more work to do here, and once we have a workflow people are happy with, I would appreciate if some advanced Linux and macOS users can translate my new .bat files roughly to .command and .sh (I just don't have the familiarity/expertise in these platforms to do this well), so we have easy 'running from source' support for all platforms.
Open

R: 3 / I: 0 / P: 8

Version 502

https://youtube.com/watch?v=SkOAc_eU9b0 [Embed]
windows
Qt5 zip: https://github.com/hydrusnetwork/hydrus/releases/download/v502/Hydrus.Network.502.-.Windows.Qt5.-.Extract.only.zip
Qt6 zip: https://github.com/hydrusnetwork/hydrus/releases/download/v502/Hydrus.Network.502.-.Windows.Qt6.-.Extract.only.zip
Qt6 exe: https://github.com/hydrusnetwork/hydrus/releases/download/v502/Hydrus.Network.502.-.Windows.Qt6.-.Installer.exe
macOS (updated links, first version had boot trouble)
Qt5 app: https://github.com/hydrusnetwork/hydrus/releases/download/v502a/Hydrus.Network.502a.-.macOS.Qt5.-.App.dmg
Qt6 app: https://github.com/hydrusnetwork/hydrus/releases/download/v502a/Hydrus.Network.502a.-.macOS.Qt6.-.App.dmg
linux
Qt5 tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v502/Hydrus.Network.502.-.Linux.Qt5.-.Executable.tar.gz
Qt6 tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v502/Hydrus.Network.502.-.Linux.Qt6.-.Executable.tar.gz

I had a good week fixing bugs and improving UI quality of life.

highlights

First off, the tag autocomplete dropdown has some important behind the scenes changes. The way it is stored and drawn is better, the same updates the popup toaster and media viewer hover windows have had recently. In general, it means a lot less flicker and jank. If you have set autocompletes to embed in normal search pages because of bugs, try letting them float under options->search this week!

Unfortunately, as a consequence of this change, I am disabling autocompletes floating in dialogs. Outside of normal search pages, they now always embed. There are hard technical reasons for this, and design ones too--I never liked how they overlapped apply/cancel--so if you were floating before and liked it, please bear with it for now and let me know if and where it is most inconvenient.

I put some work into the system:time edit predicate dialog. Rather than the confusing '<' and '~=' radiobox, it now says proper english 'since' and 'before' and so on. Some of the calculations behind this are also improved.

'sort files by import time' now works between files that were imported in the same second. If you had a some small-image manga you wanted to sort by import time because it lacked page tags, this now works properly.

I fixed some stupid mistakes that broke a part of the Client API (and Hydrus Web) in v501. Sorry for the trouble!
Open

R: 5 / I: 1 / P: 8

Version 501

https://youtube.com/watch?v=4UeFP0al1IY [Embed]
windows
Qt5 zip: https://github.com/hydrusnetwork/hydrus/releases/download/v501/Hydrus.Network.501.-.Windows.Qt5.-.Extract.only.zip
Qt6 zip: https://github.com/hydrusnetwork/hydrus/releases/download/v501/Hydrus.Network.501.-.Windows.Qt6.-.Extract.only.zip
Qt6 exe: https://github.com/hydrusnetwork/hydrus/releases/download/v501/Hydrus.Network.501.-.Windows.Qt6.-.Installer.exe
macOS
Qt5 app: https://github.com/hydrusnetwork/hydrus/releases/download/v501/Hydrus.Network.501.-.macOS.Qt5.-.App.dmg
Qt6 app: https://github.com/hydrusnetwork/hydrus/releases/download/v501/Hydrus.Network.501.-.macOS.Qt6.-.App.dmg
linux
Qt5 tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v501/Hydrus.Network.501.-.Linux.Qt5.-.Executable.tar.gz
Qt6 tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v501/Hydrus.Network.501.-.Linux.Qt6.-.Executable.tar.gz

I had a good week working on a mix of important bug fixes and UI quality of life.

highlighted fixes

First off, I have rewritten the guts of the popup toaster. It looks the same, but now it is more 'embedded' into the main gui window and I completely overhauled its layout system. I strongly hope that this will eliminate a variety of jank that has occured over the years, from odd activation/focus issues to weird width sizing to unusual popup locations. If you have had trouble with popup messages, let me know how things work today!

Next, due to a change behind the scenes in another library, the Windows build wouldn't boot last week and needed a hotfix. The Linux build seems to have suffered from a similar issue, so I have fixed both. If you had trouble booting v500 Linux, I am sorry--please try again today. The macOS release uses a different system and was unaffected.

I fixed a bug (and sometimes crash) when opening file selection dialogs for users running from source with PyQt6. Thank you for the reports!

I fixed a crash that was affecting many users (including myself a couple times, I now realise) when performing some options-saving events such as pausing/resuming subscriptions, repositories, or import/export folders. I was finally able to reproduce this reliably as I was putting the release together today. Thank you very much to the users for the reports here, I would not have figured it out but for what you discovered. It turns out it was the collect-by control doing it, so its ability to update itself without a client restart is disabled for now.

I fixed a bad bug in the manage ratings dialog, where any numerical ratings that started and ended on dialog ok in the 'mixed' (dark grey colour) state were not leaving them alone but resetting those files' ratings to the minimum allowed value (0 or 1 stars). This was an unusual bug, and I regret it a lot. I think it came in in recent weeks, when I rewrote how some rating states are handled internally.

The Deviant Art downloader broke last week. It seems like they locked away the API we were using. I have reset us to the original html parser, which still seems to work ok. Please check your DA subs and tell them to retry ignored, and let me know if there are any glaring problems.
Open

R: 6 / I: 0 / P: 8

Version 500

https://youtube.com/watch?v=UGZKKXTNcE8 [Embed]
The first Windows build was broken, if you got that please check again--the links have been updated to a hotfix.
windows
Qt5 zip: https://github.com/hydrusnetwork/hydrus/releases/download/v500a/Hydrus.Network.500a.-.Windows.Qt5.-.Extract.only.zip
Qt6 zip: https://github.com/hydrusnetwork/hydrus/releases/download/v500a/Hydrus.Network.500a.-.Windows.Qt6.-.Extract.only.zip
Qt6 exe: https://github.com/hydrusnetwork/hydrus/releases/download/v500a/Hydrus.Network.500a.-.Windows.Qt6.-.Installer.exe
macOS
Qt5 app: https://github.com/hydrusnetwork/hydrus/releases/download/v500/Hydrus.Network.500.-.macOS.Qt5.-.App.dmg
Qt6 app: https://github.com/hydrusnetwork/hydrus/releases/download/v500/Hydrus.Network.500.-.macOS.Qt6.-.App.dmg
linux
Qt5 tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v500/Hydrus.Network.500.-.Linux.Qt5.-.Executable.tar.gz
Qt6 tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v500/Hydrus.Network.500.-.Linux.Qt6.-.Executable.tar.gz

I had a good week-and-a-bit returning to normal hydrus schedule after my personal issues. There are some important bug fixes, particularly for windows crashes, and some neat updates to tag search logic.

Those who use the Windows Qt5 release will want to perform a 'Clean Install' this week: https://hydrusnetwork.github.io/hydrus/getting_started_installing.html#clean_installs

crashes and bugs

I messed up the new mpv version in v499. My golden rule is to never put out bleeding-edge library updates, but without thinking I gave Windows users a dll from late August. This caused instability for a variety of installs, but thanks to some great reports and user testing, we were able to figure out the problem and solution. I regret I wasn't able to roll out an official fix until now, but I will remember this issue for future--never fold in the latest build of anything.

There are two fixes here. Windows Qt6 users simply get a more stable mpv-2.dll today. You don't have to do anything special, just install as normal and your hydrus should be more stable. Windows Qt5 users will be rolling back to mpv-1.dll, so if you are a Qt5 user who updated to v499, you should perform a 'Clean Install', as here: https://hydrusnetwork.github.io/hydrus/getting_started_installing.html#clean_installs . Just follow the guide and you should be good again, but let me know if you have any trouble!

I also fixed a critical issue that was affecting a couple of users with damaged similar file search trees. If you have had 'similar file tree rebalancing' maintenance that seemed to go on forever before locking up your client, this is now fixed. Some related simple errors when the maintenance routine ran into a damaged or looped tree are also fixed.

The Client API now handles disconnects more gracefully. Some logspam is cleared up, and very slow file and tag searches via API now cancel on disconnect just like in the UI (e.g. when you type a new character in autocomplete tag search, it'll cancel the older slower search and start a newer faster one). If you run a busy Hydrus Companion or another Client API application that really hammers your client, let me know how you get on.
Open

R: 18 / I: 0 / P: 8

Version 499

https://youtube.com/watch?v=OIFASfPkw9g [Embed]
windows
Qt5 zip: https://github.com/hydrusnetwork/hydrus/releases/download/v499/Hydrus.Network.499.-.Windows.Qt5.-.Extract.only.zip
Qt6 zip: https://github.com/hydrusnetwork/hydrus/releases/download/v499/Hydrus.Network.499.-.Windows.Qt6.-.Extract.only.zip
Qt6 exe: https://github.com/hydrusnetwork/hydrus/releases/download/v499/Hydrus.Network.499.-.Windows.Qt6.-.Installer.exe
macOS
Qt5 app: https://github.com/hydrusnetwork/hydrus/releases/download/v499/Hydrus.Network.499.-.macOS.Qt5.-.App.dmg
Qt6 app: https://github.com/hydrusnetwork/hydrus/releases/download/v499/Hydrus.Network.499.-.macOS.Qt6.-.App.dmg
linux
Qt5 tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v499/Hydrus.Network.499.-.Linux.Qt5.-.Executable.tar.gz
Qt6 tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v499/Hydrus.Network.499.-.Linux.Qt6.-.Executable.tar.gz

I had a great week catching up on bug reports.

highlights

The changelog is quite long this week. It is mostly smaller bug fixes that aren't interesting to read. Some users have reported crashes in Qt6, and I got one crash IRL myself, which is very rare for me. I am not totally sure what has been causing these, but I fixed some suspects this week, so let me know if things are better.

I am also updating mpv for Windows users this week. It is a big jump, about a year's worth of updates, and I feel like it is a little smoother. It is also more stable and supports weirder files.

And thanks to a user's work, there is an expansion to the twitter downloader. You can now download a twitter user's likes, and from twitter lists, and--if you can find any--twitter collections.

If you are a big 'duplicates' user, there is a subtle change in the potential duplicates search this week. Check the changelog for full details, but I think I've fixed the 'I set "must be pixel dupes" but I saw a pair that wasn't' issue.

full list

- mpv:
- updated the mpv version for Windows. this is more complicated than it sounds and has been fraught with difficulty at times, so I do not try it often, but the situation seems to be much better now. today we are updating about twelve months. I may be imagining it, but things seem a bit smoother. a variety of weird file support should be better--an old transparent apng that I know crashed older mpv no longer causes a crash--and there's some acceleration now for very new CPU chipsets. I've also insisted on precise seeking (rather than keyframe seeking, which some users may have defaulted to). mpv-1.dll is now mpv-2.dll
- I don't have an easy Linux testbed any more, so I would be interested in a Linux 'running from source' user trying out a similar update and letting me know how it goes. try getting the latest libmpv1 and then update python-mpv to 1.0.1 on pip. your 'mpv api version' in _help->about_ should now be 2.0. this new python-mpv seems to have several compatibility improvements, which is what has plagued us before here
- mpv on macOS is still a frustrating question mark, but if this works on Linux, it may open another door. who knows, maybe the new version doesn't crash instantly on load
Open

R: 2 / I: 0 / P: 8

Version 498

https://youtube.com/watch?v=D5T9RByz9i8 [Embed]
windows
Qt5 zip: https://github.com/hydrusnetwork/hydrus/releases/download/v498/Hydrus.Network.498.-.Windows.Qt5.-.Extract.only.zip
Qt6 zip: https://github.com/hydrusnetwork/hydrus/releases/download/v498/Hydrus.Network.498.-.Windows.Qt6.-.Extract.only.zip
Qt6 exe: https://github.com/hydrusnetwork/hydrus/releases/download/v498/Hydrus.Network.498.-.Windows.Qt6.-.Installer.exe
macOS
Qt5 app: https://github.com/hydrusnetwork/hydrus/releases/download/v498/Hydrus.Network.498.-.macOS.Qt5.-.App.dmg
Qt6 app: https://github.com/hydrusnetwork/hydrus/releases/download/v498/Hydrus.Network.498.-.macOS.Qt6.-.App.dmg
linux
Qt5 tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v498/Hydrus.Network.498.-.Linux.Qt5.-.Executable.tar.gz
Qt6 tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v498/Hydrus.Network.498.-.Linux.Qt6.-.Executable.tar.gz

I had a good week. I got started on some long-delayed serverside janitor improvements, and I got so stuck it in was all I worked on! This release has nothing important for regular users, so if you are not involved in running a hydrus server, you can skip it entirely!

server stuff

Only for server admins and janitors!

Improving the janitor workflow is now my 'big job' for the rest of the year. This is the first step of what I'd like to fit in to spare days over the coming weeks and months.

This week I update the repositories so they cache counts of their various metadata. They can now quickly say 'I have 2,124,543 mappings' and so on. If you have a large server, it will take a few minutes to update as it counts everything up for the first time.

Once you are booted, make sure you are in help->advanced mode, and on review services you will have a new 'service info' button. Click it and you can see all the numbers, including the current lengths of the petition queues. Anyone with an account can see these for now. If you want more privacy, I can figure something out, but tbh I think it is probably good for users to be able to see everything here.

The numbers in the petition processing page are fed by this too. No longer will it manually count up and max out at 1,000 petitions--it will deliver the actual number real fast.

Also, a sibling petition can now have both ADD and DELETE rows. This happens if the same account gives the same reason (like 'replacing a->b with a->c') to a sibling petition and a sibling pend. You now see those together, with that shared reason, and action it as one item.

I suspect we'll need some more UI clientside to encourage using the same reason, but for now I have updated manage tag siblings to give the same 'reason' when you replace an existing sibling. Previously, this is where it would give a 'AUTO-CONFLICT...' style reason. Now, those things should be bundled into the same thing you see.

This stuff changes some of the hydrus network protocol. Normally, I would update the network version number, but that requires everyone to update. Since this only affects advanced users, and I expect I'll be doing more in coming weeks, I am not updating the version number. An old client will run into errors if it tries to pull petitions from a new server, but I think a new client will be able to work with an old server. In any case, if you are a server admin or janitor, please update your clients and servers at roughly the same time this week, or you'll get some harmless but annoying parse/404 errors.

As a side thing, as a server admin, if the service info numbers ever get borked, please hit 'regen service info' in your 'administrate services' menu. I've added extensive testing this week to ensure the update routines are mostly good, but I'm sure there are some complicated situations where the counting logic is dodgy. Let me know how you get on with it!
Open

R: 6 / I: 1 / P: 8

Version 497

https://youtube.com/watch?v=0KrpZMNEDOY [Embed]
windows
Qt5 zip: https://github.com/hydrusnetwork/hydrus/releases/download/v497/Hydrus.Network.497.-.Windows.Qt5.-.Extract.only.zip
Qt6 zip: https://github.com/hydrusnetwork/hydrus/releases/download/v497/Hydrus.Network.497.-.Windows.Qt6.-.Extract.only.zip
Qt6 exe: https://github.com/hydrusnetwork/hydrus/releases/download/v497/Hydrus.Network.497.-.Windows.Qt6.-.Installer.exe
macOS
Qt5 app: https://github.com/hydrusnetwork/hydrus/releases/download/v497/Hydrus.Network.497.-.macOS.Qt5.-.App.dmg
Qt6 app: https://github.com/hydrusnetwork/hydrus/releases/download/v497/Hydrus.Network.497.-.macOS.Qt6.-.App.dmg
linux
Qt5 tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v497/Hydrus.Network.497.-.Linux.Qt5.-.Executable.tar.gz
Qt6 tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v497/Hydrus.Network.497.-.Linux.Qt6.-.Executable.tar.gz

I had a good week working on a variety of different things.

The Windows Installer is now Qt6. You do not have to do anything special to update, just do your normal routine. If you are a Windows 7 user, it is very likely you will not be able to run it. You can use the Qt5 extract release for another week or two, but please plan to either stop updating hydrus, move to a newer OS, or consider running hydrus from source.

highlights

I bulked out the 'star' rating shape. The pentagram was a little thin, so I've fattened it back up while still keeping the coordinates good. If you liked the thinner star, you can now set it explicitly as a new shape choice under services->manage services.

Manage tag siblings/parents now has a proper delete button, which should make removing many rows at once easy.

I did some more note parsing work this week, and I updated the Hentai Foundry downloader to grab artist description notes. If you download from HF, you should see new files get notes. I would like to slowly update most of the client default downloaders with note parsing support, so let me know where all this succeeds and fails, and I'll adapt things as we go.

Since we are doing more note work, I also improved the size calculations for the media viewer's note hover window. It still isn't perfect in all cases, but it'll clip the last line of text less often.

New clients now start with ctrl+page up/down as 'move page selection left/right'.

multi-line parsing

This is only for advanced users who make downloaders.

The parsing system now has basic multi-line support. Any formula on a 'notes' content parser or the formulae that do subsidiary page parser 'splitting' will now no longer remove newlines when it grabs text. It makes it possible to parse a multi-paragraph artist comment and have it all end up nicely formatted in the file note you end up with.

I have hardcoded in some additional formatting rules, too. When you select 'string' as the parsing result from a nested tree of html tags, I now insert newlines on 'p' or 'br' tags. I also strip leading and trailing whitespace from every line of a note, and I only allow two consecutive newlines to clip very large paragraph breaks.

The main remaining frustration is the string processing UI has mixed multi-line parsing support in its testing UI. Some of it shows and converts well, but most of it collapses multi-line test content down to a single line.

I have updated the Hentai Foundry parser this week to grab notes. It is ultimately pretty simple, if you want to check it out as an example. I'd like to know what is most frustrating about this. I would like to chip away at the string processing test UI (and any rules that simply do not work well on multi-line content) in future.
Open

R: 3 / I: 0 / P: 8

Version 496

https://youtube.com/watch?v=2jzugX2NMnk [Embed]
windows
Qt5 zip: https://github.com/hydrusnetwork/hydrus/releases/download/v496/Hydrus.Network.496.-.Windows.Qt5.-.Extract.only.zip
Qt6 zip: https://github.com/hydrusnetwork/hydrus/releases/download/v496/Hydrus.Network.496.-.Windows.Qt6.-.Extract.only.zip
Qt5 exe: https://github.com/hydrusnetwork/hydrus/releases/download/v496/Hydrus.Network.496.-.Windows.Qt5.-.Installer.exe
macOS
Qt5 app: https://github.com/hydrusnetwork/hydrus/releases/download/v496/Hydrus.Network.496.-.macOS.Qt5.-.App.dmg
Qt6 app: https://github.com/hydrusnetwork/hydrus/releases/download/v496/Hydrus.Network.496.-.macOS.Qt6.-.App.dmg
linux
Qt5 tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v496/Hydrus.Network.496.-.Linux.Qt5.-.Executable.tar.gz
Qt6 tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v496/Hydrus.Network.496.-.Linux.Qt6.-.Executable.tar.gz

I had a great week. There's more Qt6 work, and the first version of note parsing is finished.

UI Scaling and Qt6

The media viewer now understands UI scale and will draw images with all available pixels and state the current zoom correctly. Previously, if you were on 150% UI scale, it would draw at the smaller 100% size and then nearest-neighbour scale up. Now it compensates for the UI scale completely, drawing at that size, so if you look at a 4k image on a 4k screen, it'll look the same no matter what your UI scale is.

Some funny math went into this, particularly as I stitched the image tiles together through several different coordinate spaces, so if you are >100% UI scale and see a grey or white line flickering anywhere as you pan an image around, let me know your details.

After last week's nicer thumbnails at >100%, a user let me know that the thumbnail banner text was actually looking pretty bad at 100%. Looking at screenshots of the two situations made it obvious. I chased down what was going on, which turned out to be an annoying technical problem with no easy solutions, and figured out an ok solution. If all your monitors are 100% UI scale, the thumbnail banner text should now be antialiased. It isn't perfect (and even looks blurry with some semi-transparent background colours), so I may revisit this, and it may be the underlying Qt issue causing the ugly font gets fixed one day anyway.

The rating shapes (circles, squares, stars) are now antialiased too, and I re-did the star shape to look better.
Open

R: 4 / I: 0 / P: 9

Version 495

https://youtube.com/watch?v=qBJRJvHYS_k [Embed]
windows
Qt5 zip: https://github.com/hydrusnetwork/hydrus/releases/download/v495/Hydrus.Network.495.-.Windows.Qt5.-.Extract.only.zip
Qt6 zip: https://github.com/hydrusnetwork/hydrus/releases/download/v495/Hydrus.Network.495.-.Windows.Qt6.-.Extract.only.zip
Qt5 exe: https://github.com/hydrusnetwork/hydrus/releases/download/v495/Hydrus.Network.495.-.Windows.Qt5.-.Installer.exe
macOS
Qt5 app: https://github.com/hydrusnetwork/hydrus/releases/download/v495/Hydrus.Network.495.-.macOS.Qt5.-.App.dmg
Qt6 app: https://github.com/hydrusnetwork/hydrus/releases/download/v495/Hydrus.Network.495.-.macOS.Qt6.-.App.dmg
linux
Qt5 tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v495/Hydrus.Network.495.-.Linux.Qt5.-.Executable.tar.gz
Qt6 tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v495/Hydrus.Network.495.-.Linux.Qt6.-.Executable.tar.gz

I had a great week working on more Qt6 support and getting Note Import Options ready for advanced users to try out.

Qt6

The Qt6 launch last week went generally well. There were a couple of little typo bugs as expected, but most users reported nothing drastic.

I have fixed several issues and also improved graphical quality at >100% UI scale. Qt6 handles UI scale tech much better, but that also exposed all the better where my custom UI was failing. Thumbnails at 125% were looking pretty ugly, with nearest-neighbour scaling, so I knuckled down and did my homework on how all this is supposed to work, and I think I have it fixed. Thumbnails should look ok at any UI scale in Qt6, and their banner text too. My fixes apply to Qt5 too, but as far as I can tell that only really works comprehensively for 100%/200% scale. I will try to tackle the media viewer next week.

If you are an experienced user with a backup, please feel free to try Qt6 out on your real install. If both Qt5 and Qt6 are available, the client will now default to Qt6, so you shouldn't need to do a 'clean install' like last week. My test of this went fine, but if there is some odd dll conflict when you try to boot, check here on how to clear things out and either revert to Qt5-only or try Qt6-only:

https://hydrusnetwork.github.io/hydrus/getting_started_installing.html#clean_installs

Any reports on failures here would be useful so I can write in any needed 'delete these old files' rules to the Qt6 Win installer.

As a reminder, afaik Windows 7 cannot run Qt6, so don't try it out if that's you. I will switch over to Qt6 exclusively in a few weeks, at which point I'll update the help and talk more about your future options, which will be: stop updating the client; move to a newer OS; run in Win7/Qt5 from source.

Note Import Options

Unfortunately I couldn't fit all this in again, but I've done work I'm happy with and have parts of it ready for advanced users to play with. Fingers crossed, the first simple version of this will be completely integrated next week.

The File Import Options update last week went ok. I messed something up with the Presentation Options, so highlighting a gallery or watcher with a default FIO was always showing the same thing (new or all files) instead of what the default actually said. This is fixed.

I have pushed FIO and TIO further together this week. File Import Options and Tag Import Options are now merged into one button and one dialog across the program. The dialog is tabbed, so you edit both sets of options at the same time, and in future, any new Import Options will work through the same interface.
Open

R: 3 / I: 0 / P: 9

Version 494

https://youtube.com/watch?v=85bvcndvEpo [Embed]
windows
Qt5 zip: https://github.com/hydrusnetwork/hydrus/releases/download/v494/Hydrus.Network.494.-.Windows.Qt5.-.Extract.only.zip
Qt6 zip: https://github.com/hydrusnetwork/hydrus/releases/download/v494/Hydrus.Network.494.-.Windows.Qt6.-.Extract.only.zip
Qt5 exe: https://github.com/hydrusnetwork/hydrus/releases/download/v494/Hydrus.Network.494.-.Windows.Qt5.-.Installer.exe
macOS
Qt5 app: https://github.com/hydrusnetwork/hydrus/releases/download/v494/Hydrus.Network.494.-.macOS.Qt5.-.App.dmg
Qt6 app: https://github.com/hydrusnetwork/hydrus/releases/download/v494/Hydrus.Network.494.-.macOS.Qt6.-.App.dmg
linux
Qt5 tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v494/Hydrus.Network.494.-.Linux.Qt5.-.Executable.tar.gz
Qt6 tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v494/Hydrus.Network.494.-.Linux.Qt6.-.Executable.tar.gz

I had a great week with two big changes. First, there is an optional 'Qt6' version of the program for advanced users to try out, and second, File Import Options has some important updates.

Qt5 and Qt6

If you are a regular user, stick with the Qt5 versions of the release this week. It is the same as before.

Hydrus is moving up to a new version of its UI library, Qt. The new version has a ton of bug fixes and generally better support for newer OS concepts like UI scaling. I am going to be putting out releases for both 5 and 6 for a month or two, testing it with advanced and then normal users, and then will switch to 6 exclusively. Everything seems to be going well, and you don't have to do anything.

If you are an advanced user though, please try out the Qt6 builds. They work exactly the same as the old ones. Just to be careful, I recommend you not try them on your real database first off, and doubly so if you do not have a great backup. I am not worried about database damage, but you never know, and if there are problems, I don't want to give you inconvenience on your main install. Try a fresh extract on your desktop first to make sure it boots ok, and then delete that extract. Then, if you want to try it on your real database, try doing a 'clean install' as here: https://hydrusnetwork.github.io/hydrus/getting_started_installing.html#clean_installs

If you don't do a clean install, you'll still have Qt5 dlls in your install folder and the client will default to the older version (for now). If you are a macOS user, you don't have the concept of 'clean install', so just run the App as normal, but make sure you have a backup of your database first. There's also no Windows Qt6 installer yet. You can check help->about to see what version of Qt is currently running.

So far, the update has been remarkably smooth for me, with very few bugs. A user has been watching the situation for me and kindly provided a patch to deal with the most important syntax changes, so moving over has not been a massive pain in the neck. I've been using it IRL for a few days now and I think things are just that bit smoother and less flickery.

I am particularly interested in Linux and macOS users' feedback. So far, the main limitation I know about is that Windows 7 can't run Qt6 (it is just too old), but there may be other issues in other platforms. Let me know, and we'll see if we can iron them out.

I am going to keep hydrus Qt5 compatible, so anyone who needs to stay on it but wants to keep updating will have the option of running from source.
Open

R: 5 / I: 1 / P: 9

Version 493

https://youtube.com/watch?v=5VygwTBgph4 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v493/Hydrus.Network.493.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v493/Hydrus.Network.493.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v493/Hydrus.Network.493.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v493/Hydrus.Network.493.-.Linux.-.Executable.tar.gz

I had a great week working on some fixes and prototype EXIF support.

EXIF

EXIF are metadata labels embedded in some media files, usually JPEG. They might say the make of camera that took a picture, or its aperture/ISO, the GPS coordinates it was taken at, how an image is rotated, or the DPI of a logo for printing purposes. There are many different possible fields. I have wanted to add support to view and even search this data for a long time, and this week we start with something simple but not user-friendly.

The media viewer now has a 'cog' icon on the top hover. On JPEGs, it lets you check for EXIF data. Most files don't have it, but if it does (usually this is photos or professional art exported from Photoshop etc...), it now throws up a little window to see every field.

The duplicate filter now actively scans for EXIF data and says if one or both files have it, just like the recent addition for ICC Profiles. Many websites strip EXIF data on upload, so if you have two exact dupes, the one with EXIF data is probably closer to the 'original' version.

Now I have this framework, I would like to extend it. Beyond general polish like replacing the cog icon with something nicer, and only enable it when I know there is some EXIF to show, I want to cache 'has exif data yes/no' in the database and allow you to search by that. I expect I'll also add the actual EXIF data itself to the database one day, so you'll be able to search all your pictures for iPhone 6 photos or whatever.

So, if you are interested in EXIF, please give this a go and let me know what you think. This feature was taking so long to happen that I decided to just spam out a rough v1.0 that I can keep improving.

full list

- EXIF:
- in the first step of 'official' EXIF support, the media viewer now has a 'cog' button on the top hover, enabled when looking at a jpeg, that will check the file for EXIF data. if found, it will throw it up on a simple new window that shows EXIF id, label, and value. this is a hacked-together prototype, not super user-friendly, but it works. let me know what you think, and please send me any files that have weird EXIF that doesn't parse right but you think should. I already discovered a file with a null character that wouldn't display in UI, that sort of thing
- GPS EXIF values are also parsed and extracted
- made it so you can double-click a row in this new window to copy an EXIF value to clipboard
- in the duplicate filter, if one or both files have exif data, this is now noted in the comparison statements, just like ICC profile! (issue #469)
- obvious future extensions here will be storing 'has exif' in the database and allowing its presence to be searchable and enabling the cog button (or a nicer 'exif' button) only when there is known data to see. a subsequent step would be actually caching the data in the database for full EXIF search
- as a side thing, we're now set up on the hydrus end to pull TIFF EXIF, but PIL doesn't seem to offer it, so we'll have to wait for a different solution there
Open

R: 2 / I: 1 / P: 9

Is there a way to change the upper limit on gifs from 32?
I'm trying to save a 52Mb sized gif and I got prompted with this message.

R: 26 / I: 3 / P: 9

Suggestion Threads

Drop your suggestions here
http://archive.fo/heSey
Open

R: 1 / I: 0 / P: 9

Version 492

https://youtube.com/watch?v=N9LFp_brHvE [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v492/Hydrus.Network.492.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v492/Hydrus.Network.492.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v492/Hydrus.Network.492.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v492/Hydrus.Network.492.-.Linux.-.Executable.tar.gz

I had a good week mostly cleaning code and adding some things for advanced users.

highlights

If you are in advanced mode, the file sort and collect controls now have a 'tags' button. This lets you determine which tag service that particular sort/collect applies to. If you are this tag-clever, let me know how it works for you. This tag button is the same thing that autocomplete dropdowns use, and I expect it to soon get a 'multiple location' makeover like the file button did with multiple file services.

The archived file delete lock (under options->files and trash) gets a pass this week. If you try to delete files that are currently locked, it now makes a popup with a button to see those specific files, so you can decide what to do. The duplicate filter also handles the different situations (like 'archive both files' + 'delete the worse file') better.

The duplicate filter also now shows if one or both files have an ICC profile.

The shortcut actions to 'move page selection home/left/right/end' now try to stay on the same level if you hit them several times. If you use these actions, try them out through a mix of page of pages and you'll see how it works now. It remembers the current level within three seconds of the last move event. This was requested by several users, and there isn't a nice way to do it so I hacked an answer, so let me know what you think.

full list

- sort and collect updates:
- for big brain users, the collect control now has a tag domain button. it only shows if you are in advanced mode (issue #572)
- the sort control also has a tag domain button hidden behind advanced mode. it applies to system:num tags and namespace sorting
- the collect control now appears on all import pages
- .
- archived file delete lock:
- the duplicate processing action code now no longer archives files that are due for deletion right before that deletion. this was hitting the archive delete lock
- if archive delete lock is on and the 'other' file in the duplicate filter is archived, the option to 'this is better, delete the other' is now disabled
- if you attempt to delete a delete-locked file during normal browsing, or if an automatic system like export folders wants to but fails on some, a popup is now made with a button to show the files that were filtered out so you can review the situation and fix it if you want
- I am considering adding a dialog to say 'hey, this is locked, want to send back to inbox?' to fix these situations in a nice way, but I think this is probably a bad idea in terms of workflow, design, and my sanity given all the edge cases and potential future expansions of lock rules. maybe I'll add a simple 'delete and override lock checks' option, but a lock is a lock tbh. for now, I will focus on this better UI feedback of currently delete-locked files and make it simpler for humans to remove any locks
Open

R: 3 / I: 0 / P: 9

Version 491

https://youtube.com/watch?v=OQEDWiM-QRI [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v491/Hydrus.Network.491.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v491/Hydrus.Network.491.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v491/Hydrus.Network.491.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v491/Hydrus.Network.491.-.Linux.-.Executable.tar.gz

I had a good couple of weeks doing a behind the scenes overhaul and a variety of quality of life work. I was ill last week and put off the release, but I am feeling great now.

metadata import/export

As is often the case, an important overhaul makes few actual changes front end. I've been trying to do this for a while, but now the 'export/import tags in a neighbouring .txt file' routine now works on completely new tech. Rather than hardcoded tags into .txt files, it now uses a modular system that I will be able to expand in future to support filtering and string processing, more metadata types like ratings and URLs, JSON and XML as the file formats, and will even allow funky migrations like converting tags to URLs.

As a side benefit, Export Folders now support tags .txt export.

For now, all the UI front-end looks the same, but please expect this to change in future. I'll be writing some nice unified panels and dialogs to handle the new objects as I write them.

advanced user highlights

The 'OR*' advanced tag input now supports system predicates! It uses the same system predicate parser as the Client API, so you can now type or copy/paste most system predicate text and get something both useful and complicated. The text that parses isn't always exactly the same as the predicate label, so check out the big list of example parseable system predicates here: https://hydrusnetwork.github.io/hydrus/developer_api.html#get_files_search_files

Also, if you are a parser creator, String Processing now has a Tag Filter processing step. Let's say you can grab all the tags from somewhere but you need to filter out a handful of non-tag text like '+' and '?', or you are able to create hydrus namespaced tags and want to filter by namespace, just insert this into your string processing and it should be much easier than messing around with long regexes.

R: 0 / I: 0 / P: 9

Moving Hydrus library between computers

I'm running Hydrus on a Mac, and tried to move the .app of my library (23GB) onto another Mac, but found it didn't work. Is there an easy way to do this?

Thanks!
Open

R: 3 / I: 0 / P: 9

Version 490

https://youtube.com/watch?v=LBzE9JMoCeE [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v490/Hydrus.Network.490.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v490/Hydrus.Network.490.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v490/Hydrus.Network.490.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v490/Hydrus.Network.490.-.Linux.-.Executable.tar.gz

I had a good week working on a variety of small jobs.

highlights

While working with a user, I discovered I had recently messed up the initialisation of the image caches, causing them to use default values and be too small for many power users. This is fixed, so I hope you will get some smoother media viewing, especially when it comes to very large images.

I tidied up some weird file service bugs and annoyances that came from the multiple local file services transition. Fixed some weird 'delete from x' entries on the thumbnail menu, stopped spamming 'all my files' in places it wasn't helpful, figured out better service ordering, just some simple workflow stuff.

And I think I have fixed another PTR processing bug some users had. If you have had 'this update file was missing' errors that wouldn't fix themselves, please try again and let the automatic maintenance run one more time--it should repair your records this time.

And just a fun thing--Mr Bones now has more numbers, and more neatly laid out.

full list

- misc:
- fixed a stupid bug that meant the image caches were initialising with default values (as under _speed and memory_) until you opened and OKed the options dialog (or did some other options-refresh events). sorry for the trouble, please enjoy some smoother image browsing.
- mr bones now shows more numbers, and in a neater table. it should be clearer what the percentages are for now, too
- the _manage->regenerate_ thumbnail menu has additional quick maintenance commands for presence and integrity checks and regenerating data in the similar files system
- wrote a new 'special duplicate' button for the edit shortcut set dialog. the list on this dialog doesn't allow duplicates (which meant the old 'duplicate' button was doing nothing), so this duplicates the current actions with 'incremented' shortcut keys. 'a' becomes 'b', 'ctrl+5' becomes 'ctrl+6', and so on. it doesn't always work, but if you want to make ten shortcuts for setting rating 1-10, this should help
- fixed an issue where the thumbnail banner text and the media viewer background text was not changing size or font according to QSS stylesheet rules (issue #1173)
- SIGTERM should now cause a clean program exit (previously it killed the GUI App but left some daemon threads alive for thirty seconds or more). unlike SIGINT, it will not ask you if you are sure you want to exit or if you would like to do shutdown maintenance--it just closes the client promptly
- fixed a bug in last week's importer page status improvements--the hard drive import page wasn't showing all the updates it should have
- brushed up some backup help
- .
- file services:
- fixed a bug where advanced users could set 'all known files'/'all known tags' on a search dropdown. this search domain is not supported
- in the archive/delete filter, if the current location is 'all my files' and the files being deleted are only in one local file domain, the surplus 'all my files' will no longer appear at the top of the filter's commit dialog
- the file services in the thumbnail select/remove menu are now sorted in the same order as the file domain button in search dropdowns
- the thumbnail select/remove menus now exclude 'all my files' and 'all local files' if those choices are redundant (e.g. if you only have files in 'my files', 'all my files' will be hidden)
- fixed some incorrect 'delete from x' actions appearing in thumbnail right-click menus
Open

R: 2 / I: 0 / P: 9

Version 489

https://youtube.com/watch?v=6rboksqjPy4 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v489/Hydrus.Network.489.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v489/Hydrus.Network.489.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v489/Hydrus.Network.489.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v489/Hydrus.Network.489.-.Linux.-.Executable.tar.gz

I had a good week getting back into the swing of things. I fixed some important bugs and improved some UI.

highlights

All the downloader pages--gallery, watcher, urls, and simple--have a revamped status system. All the text that shows how file or gallery downloads are going is now generated in a better way, with more error states (e.g. it will tell you when your gallery stopped because it hit the file limit, or when one of the emergency pause states under the network menu has kicked in), and logic in edge cases is improved. Everything is unified now, so the texts are the same across all pages. Also, if a gallery query or watched thread is 'pending', its text now reports that it is waiting for a work slot, rather than staying blank. There _shouldn't_ be any situations now where a downloader is unpaused with work to do but has blank status.

If you use the multiple local file services system, the archive/delete filter now presents more options when you are done. If the files are in more than one local file service, you can choose where you delete them from, including all applicable. This was confusing and opaque before, so I hope this makes it more clear what is happening and gives you more choice.

I _believe_ I have fixed an important bug some users were having with PTR processing. There was an annoying issue about a 'definitions' file being seen as a 'content' file, or vice versa, that the automatic maintenance could not fix. I finally managed to reproduce the issue and fixed it. I schedule a fix in the update this week, so if you have been hit by this, please wait for one more round of file maintenance 'metadata' scans, and then unpause the PTR one more time. Essentially, I think I fixed the automatic maintenance. Let me know how you get on!

full list

- downloader pages:
- greatly improved the status reporting for downloader pages. the way the little text updates on your file and gallery progress are generated and presented is overhauled, and tests are unified across the different downloader pages. you now get specific texts on all possible reasons the queue cannot currently process, such as the emergency pause states under the _network_ menu or specific info like hitting the file limit, and all the code involved here is much cleaner
- the 'working/pending' status, when you have a whole bunch of galleries or watchers wanting to run at the same time, is now calculated more reliably, and the UI will report 'waiting for a work slot' on pending jobs. no more blank pending!
- when you pause mid-job, the 'pausing - status' text is generated is a little neater too
- with luck, we'll also have fewer examples of 64KB of 503 error html spamming the UI
- any critical unhandled errors during importing proper now stop that queue until a client restart and make an appropriate status text and popup (in some situations, they previously could spam every thirty seconds)
- the simple downloader and urls downloader now support the 'delay work until later' error system. actual UI for status reporting on these downloaders remains limited, however
- a bunch of misc downloader page cleanup
Open

R: 1 / I: 0 / P: 10

Version 488

https://youtube.com/watch?v=AQOfIENN2tk [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v488d/Hydrus.Network.488d.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v488d/Hydrus.Network.488d.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v488d/Hydrus.Network.488d.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v488d/Hydrus.Network.488d.-.Linux.-.Executable.tar.gz

I had a good simple week making a clean release before my vacation.

Everything is misc this week, nothing earth-shattering, just a bunch of cleanup and little stuff. If you have any wavpack files, try importing them!

full list

- the client now supports 'wavpack' files. these are basically a kind of compressed wav. mpv seems to play them fine too!
- added a new file maintenance action, 'if file is missing, note it in log', which records the metadata about missing files to the database directory but makes no other action
- the 'file is missing/incorrect' file maintenance jobs now also export the files' tags to the database directory, to further help identify them
- simplified the logic behind the 'remove files if they are trashed' option. it should fire off more reliably now, even if you have a weird multiple-domain location for the current page, and still not fire if you are actually looking at the trash
- if you paste an URL into the normal 'urls' downloader page, and it already has that URL and the URL has status 'failed', that existing URL will now be tried again. let's see how this works IRL, maybe it needs an option, maybe this feels natural when it comes up
- the default bandwidth rules are boosted. the client is more efficient these days and doesn't need so many forced breaks on big import lists, and the internet has generally moved on. thanks to the users who helped talk out what the new limits should aim at. if you are an existing user, you can change your current defaults under _network->data->review bandwidth usage and edit rules_--there's even a button to revert your defaults 'back' to these new rules
- now like all its neighbours, the cog icon on the duplicate right-side hover no longer annoyingly steals keyboard focus on a click.
- did some code and logic cleanup around 'delete files', particularly to improve repository update deletes now we have multiple local file services, and in planning for future maintenance in this area
- all the 'yes yes no' dialogs--the ones with multiple yes options--are moved to the newer panel system and will render their size and layout a bit more uniformly
- may have fixed an issue with a very slow to boot client trying to politely wait on the thumbnail cache before it instantiates
- misc UI text rewording and layout flag fixes
- fixed some jank formatting on database migration help

next week

I am now off for a week. I think I need it! I'm going to play a ton of vidya, shitpost the big streams that are happening, fit some Wagner in, and get on top of outstanding IRL stuff. I'll be back to catch up my messages on Saturday the 18th.

Thanks everyone!
Open

R: 2 / I: 0 / P: 10

Version 487

https://youtube.com/watch?v=VKuGYKkH3oA [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v487/Hydrus.Network.487.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v487/Hydrus.Network.487.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v487/Hydrus.Network.487.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v487/Hydrus.Network.487.-.Linux.-.Executable.tar.gz

I had an ok week. I was a unexpectedly short on time, so I couldn't get everything I wanted done, but I cleared out some small work.

highlights

The big update last week, which I recommended only for advanced users, went well. There don't seem to be any obvious problems with the logic of the new search cache, so I now recommend it for everyone. You will be presented with a popup just before the update runs, giving you an estimate of how long it thinks it will take. Most users should take 5-10 minutes, but if you have millions of files, it will be longer. Just let it run and some things will run a bit faster and neater in the background. If you have played with 'multiple local file services', then check out the new 'all my files' domain you will see--this is basically an efficient umbrella of all your local file services. It works super fast for things like the duplicates system.

I also put some time into the duplicate filter this week. The logic of the queue is improved again, so some rare errors when reaching the end of a batch should be fixed. I also integrated manual file deletes into the queue processing: now, when you manually delete a file, or both, the deletes will not happen until you commit--just like the other decisions you are making--and they are undoable if you select 'forget' or go back a pair. You also won't see a file you manually deleted again in a batch (it'll auto-skip if that file comes up again).

Also, the duplicate filter now has a little 'send pair to page' button, which publishes the current pair to the duplicates page that made the filter, just in case you want to save them for some extra processing after you are done filtering. You can do this with multiple pairs and they'll just stack up in the page.

A couple other neat things happened in last week's advanced-user-only release, which I will repeat here:

The 'media viewers' shortcut set has three new zoom actions: 'switch between 100% and max', 'switch between canvas and max', and 'zoom to max'.

When you enter pairs in the tag sibling dialog, it shouldn't complain about loops anymore, but instead automatically break them, just like how it will auto-petition an A->B, A->C conflict.
Open

R: 3 / I: 0 / P: 10

Version 486

https://youtube.com/watch?v=ZUrcYKghr-Y [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v486/Hydrus.Network.486.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v486/Hydrus.Network.486.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v486/Hydrus.Network.486.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v486/Hydrus.Network.486.-.Linux.-.Executable.tar.gz

I had a great week working on a variety of smaller issues and some important database updates.

The release this week is only recommended for advanced users. I make an important change, and I want to make sure the update works quickly and without problems before I roll it out to everyone. If you are not an advanced user, please check back in next week!

The update will also take a few minutes this week.

all my files

So, I have made a new virtual service, 'all my files', which covers the union of all your local file services. This service is very similar to 'all local files', but it does not include trash or repository files. It provides a bunch of tools across the program for quick and precise searching of all the files that have value and are worth looking at.

When you update, this new service will be created and populated. It will take a few minutes, longer if you have millions of files and tags. My 2.8-million-file ptr-syncing client took 32 minutes. There are progress updates on the splash window.

Once you are booted, you will see 'all my files' in review services and the file domain selector if you have more than one local file domain. Feel free to play around with it--it will run a lot faster than previously going 'multiple locations' and unioning all your local file services.

The code is working really well on my end, and I am not afraid of anything being damaged, but if something goes wrong, it may require some clever/slow regeneration to fix. The main things I would like to know are:

1) Did your update take significantly longer than ~100k files/minute? Did it get held up on anything?
2) After some use, have you noticed any file/tag miscounting with 'all my files'?

As always, make a backup before you update.

other highlights

The 'media viewers' shortcut set has three new zoom actions: 'switch between 100% and max', 'switch between canvas and max', and 'zoom to max'.

When you enter pairs in the tag sibling dialog, it shouldn't complain about loops anymore, but instead automatically break them, just like how it will auto-petition an A->B, A->C conflict.

The database now cleans up after itself more thoroughly. Some users have been having trouble with very large 'WAL' files, some getting to be multiple GB, and perhaps seeing bloated memory use along with it. A set of new maintenance routines now force write-flushing at regular intervals. In my testing, there is no lag related to this, but I will be interested to hear if anyone gets new commit hang-ups during very heavy work. If you have had a huge WAL, let me know if this helps!
Open

R: 10 / I: 1 / P: 10

Version 484

https://youtube.com/watch?v=OtPsKtUyGxg [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v484/Hydrus.Network.484.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v484/Hydrus.Network.484.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v484/Hydrus.Network.484.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v484/Hydrus.Network.484.-.Linux.-.Executable.tar.gz

I had an ok week. I fixed some things, improved some quality of life, and made internal file migration a bit easier.

highlights

Last week's debut of multiple local file services went well! As far as I know, no one who tried it out had any big problems, and my main concerns--mostly that it needs some better migration tools and workflows and 'this file is in here' UI--proved true. So, I know what I have to do and will keep working. Multiple local file services remains for advanced users for now, but I hope to launch it properly for everyone, with nice help, next week.

However, while doing this work, I did accidentally break the simple version of the 'delete files' dialog when files were in the trash--rather than say 'delete these permanently?', it just wouldn't appear. This was due to a logical oversight where it wasn't testing and counting up 'trash' status correctly. It is fixed now.

Also, there was a problem with the new duplicate filter queue for users who have done a good bit of processing. A certain function that in complicated situations automatically skips some pairs was failing whenever it hit the end of a batch. This is also fixed now, thank you for the great reports on this.

For multiple local file services, I updated the UI code, fixing some little bugs and improving the workflow when you have complicated situations, and I integrated the shortcuts system and the media viewer. You can now create 'add/move to service x' actions in the 'media' shortcut set, and the media viewer has the same add/move menu on right-clicks.

The media viewer has several other improvements: I think I fixed that annoying bug where a fullscreen borderless view of media that exactly fits the screen would sometimes not resize when you went back to normal window mode! Also, scrolling the mouse over the taglist hover window should no longer ever cause a 'previous/next media' event. And I have implemented a 'short and simple' version of the video/audio scanbar to show (instead of completely hiding it) when your mouse is away--just a few pixels to show things 'at a glance'. Even though it covers a few pixels of video at the bottom, I liked this so much that I set it as the default for all users. If you don't like it, you can hide it again with the new setting under options->media.
Open

R: 1 / I: 0 / P: 10

Version 485

https://youtube.com/watch?v=AKgjOCuW_MU [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v485a/Hydrus.Network.485a.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v485a/Hydrus.Network.485a.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v485a/Hydrus.Network.485a.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v485a/Hydrus.Network.485a.-.Linux.-.Executable.tar.gz

I had a good week. The multiple local file services system is now ready for all users.

multiple local file services

I have written some proper help for this new system to talk about what it is and how to use it. The basic idea is you can now have more than one 'my files', which lets you compartmentalise things for privacy or workflow reasons. The help is here:

https://hydrusnetwork.github.io/hydrus/advanced_multiple_local_file_services.html

All users can try this out--you no longer have to be in advanced mode--but in terms of experience level, I recommend it to people who are at least comfortable with tag siblings and parents.

This system is fundamentally feature complete. The outstanding immediate problems are that it file location doesn't show up in UI very well yet, the Client API should plug into it better, and it needs some en masse controls to do large file migrations and client merging. I hope to work on these in the coming weeks.

If you give this a go, let me know what you think!
Open

R: 4 / I: 0 / P: 10

Version 483

https://youtube.com/watch?v=ymI1g2VjyCY [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v483/Hydrus.Network.483.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v483/Hydrus.Network.483.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v483/Hydrus.Network.483.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v483/Hydrus.Network.483.-.Linux.-.Executable.tar.gz

I had a good couple of weeks doing some regular work and getting 'multiple local file services' ready for testing.

multiple local file services

This is not ready for everyone yet! Advanced users only for now please.

I turned multiple local file services on in debug mode last week, just to see how things were looking, and it turned out suprisingly great, no big problems. For several months now I have been doing prep work for it, and that seems to have paid off. I decided to finish the last important things and get a v1.0 out.

So, it is now possible to have multiple 'my files' services in your client, and to search, import to, and migrate files between them. These services are completely blind to each other, so searching for autocomplete tags in one will not return suggestions from another. The hope is this will allow fairly good sfw/nsfw-style separations in clients and open up interesting new contained workflows.

I am recommending this only for advanced users for now, and moreso only those who have been following this feature. I have not yet written up nice help for this, and some of the UI/workflow is still not user friendly, so what I would like is for people who are enthusiastic to try it out and let me know what they think. I really haven't run into any massive errors, but I won't encourage you to go crazy on a real client yet. Go nuts on a new empty test client, or experiment carefully on a real client, just in case something goes wrong, and I will keep polishing the experience.

The basics are: you can now make a new 'local file domain' in manage services. file import options now lets you import to different or multiple local file domains, and thumbnail right-click lets you copy or move files between them too. The normal search page dropdown lets you jump between local services just like searching trash, and of course it now supports multiple domains if you want to do a union. The delete and undelete commands are similarly a little more powerful when you start adding new services. Check out the changelog for more specific details.

Next step I think is to make it more obvious when thumbnails/files are in certain services, since at the moment you have to scan the text on the status bar, top media hover, or thumbnail menu. Maybe custom icon rules (e.g. 'when the file is in "sfw" domain, give it a flower icon'). Then general polish like shortcut integration, maybe some more search tech, and then I really want to write a nice help document for it all to introduce normal experienced users to the idea, and some 'merge these clients' tech would be great, so users who have been using two or more clients for years can finally combine them into one.
Open

R: 3 / I: 0 / P: 10

Version 482

https://youtube.com/watch?v=PGEZutQ-tCM [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v482/Hydrus.Network.482.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v482/Hydrus.Network.482.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v482/Hydrus.Network.482.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v482/Hydrus.Network.482.-.Linux.-.Executable.tar.gz

I had a great week doing cleanup and other simple work.

highlights

I fixed the problem where clicks on a scrolled taglist were going to the wrong location. I was cleaning up some ancient wx->Qt code hacks and it seems I rarely scroll and click when working, so I never noticed the problem. I have a new test to make sure this does not happen again. Sorry for the trouble!

The URLs in the top-right hover menu are now styled better. No longer underlined, and now colourable by QSS. I have updated all the default stylesheets that come with the client (you can set these under options->style) to have some decent colours. If you have your own custom QSS, check my default to see how to set it yourself.

You can now set duplicate action options to 'always archive both files', if you want to play with making the duplicate filter do some of the work of the archive/delete filter.

Also, the duplicate filter now has improved image prefetch. There should be less flickering when you switch from A to B the first time and when you action a pair and move to the next. Please note that if you still get flicker for 4k images, try boosting the image cache size under options->speed and memory (I boosted the default up to 384MB this week, so you might like to give it some more too).

full list

- misc:
- fixed the stupid taglist scrolled-click position problem--sorry! I have a new specific weekly test for this, so it shouldn't happen again (issue #1120)
- I made it so middle-clicking on a tag list does a select event again
- the duplicate action options now let you say to archive both files regardless of their current archive status (issue #472)
- the duplicate filter is now hooked into the media prefetch system. as soon as 'A' is displayed, the 'B' file will now be queued to be loaded, so with luck you will see very little flicker on the first transition from A->B.
- I updated the duplicate filter's queue to store more information and added the next pair to the new prefetch queue, so when you action a pair, the A of the next pair should also load up quickly
- boosted the default sizes of the thumbnail and image caches up to 32MB and 384MB (from 25/150) and gave them nicer 'bytes quantity' widgets in the options panel
- when popup windows show network jobs, they now have delayed hide. with luck, this will make subscriptions more stable in height, less flickering as jobs are loaded and unloaded
- reduced the extremes of the new auto-throttled pending upload. it will now change speed slower, on less strict of a schedule, and won't go as fast or slow max
- the text colour of hyperlinks across the program, most significantly in the top-right media hover window, can now be customised in QSS. I have set some ok defaults for all the QSS styles that come with the client, if you have a custom QSS, check out my default to see what you need to do. also hyperlinks are no longer underlined and you can't 'select' their text with the mouse any more (this was a weird rich-text flag)
- the client api and local booru now have a checkbox in their manage services panel for 'normie-friendly welcome page', which switches the default ascii art for an alternate
- fixed an issue with the hydrus server not explicitly saying it is utf-8 when rendering html
- may have fixed some issues with autocomplete dropdowns getting hung up in the wrong position and not fixing themselves until parent resize event or similar
Open

R: 3 / I: 0 / P: 10

Version 481

https://youtube.com/watch?v=nShSEUBKe3o [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v481/Hydrus.Network.481.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v481/Hydrus.Network.481.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v481/Hydrus.Network.481.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v481/Hydrus.Network.481.-.Linux.-.Executable.tar.gz

I had a great week. Lots of different small jobs done.

notes and hover windows

I'm happy with last week's work making notes show in media viewers, but I introduced some little bugs while rewriting hover windows. I have now fixed the bad text colour behind the top hover, the problem where clicking on tags or greyspace was propagating up to the archive/delete and duplicate filters, the bad hover panel colour on non-default stylesheets, and some note window position and size issues.

Also, for notes, you can now right-click them to collapse them in the hover window. Right-click again on the name to expand again. This is a test, really, just to see if it helps navigating files with many long notes. Double-clicking on the note tab in the edit dialog lets you rename, and a checkbox under the new options->notes now lets you choose whether the text caret starts at the beginning or end of the document when editing.

Furthermore, I have updated all the icon buttons in all the hovers to no longer take focus when you click on them. They were previously stealing arrow key and space after a click (to do button-to-button form navigation), which meant you couldn't click on, say, a duplicate filter action button and then go back to arrow keys to navigate. Now you should be able to mix clicks and arrow keys without trickery. If this affects you, let me know how it goes!

other highlights

If you didn't like the recent 'ctrl- and shift-clicks no longer show files in the preview viewer' change, check out the new checkboxes under options->gui pages. You can make either click type focus for all files again or just files with no duration--if you don't want noisy videos being annoying while you ctrl-click.

The 'advanced mode' autocomplete dropdown now has two 'OR' buttons. The left one opens a new empty OR edit dialog, the right one opens the advanced text parsing input as before.

full list

- fixes and improvements after last week's hover and note work:
- fixed the text colour behind the top middle hover window
- stopped clicks on the taglist and hover greyspace being duplicated up to the main canvas (this affected the archive/delete and duplicate filter shortcuts)
- fixed the background colour of the hover windows when using non-default stylesheets
- fixed the notes hover window--after having shown some notes--could then lurk in the top-left corner when it should have been hidden completely
Open

R: 2 / I: 0 / P: 10

Version 480

https://youtube.com/watch?v=R1t6iNG28zI [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v480/Hydrus.Network.480.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v480/Hydrus.Network.480.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v480/Hydrus.Network.480.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v480/Hydrus.Network.480.-.Linux.-.Executable.tar.gz

I had a good week. Notes now display on the media viewer.

notes

Notes have always been a slightly hidden system, a bit like ratings were. Today is a step forward to exposing them. Any file that has notes (you can start adding notes to a file by hitting manage->notes on their right-click menu) will now show them in the media viewer, just below the top-right hover window. They get their own hover window too, if you mouse over them. If you click on a particular note, the 'edit notes' dialog opens on it.

This is a first version, and a little ugly, but I'm happy we now have something I can iterate on in future. If you are a big notes person, please let me know how it works best and worst for you. If you have unusual font style, size, or colour, let me know if it goes crazy or sizes too short or tall.

While working on this, I rewrote the media viewer's hover windows to be more sensible, something I have been planning for a long time. They are now 'embedded' into the parent canvas, which should reduce a variety of jank behaviour--particularly, if you now click a hover, the main media viewer window no longer loses focus. There is still some hackery in the system to clean up, but I hope it'll work better overall for you.

Unfortunately, I just did not get to note merge in the duplicates system or note parsing. That'll have to be for the future.

the rest

A user is working on a neat 'gallery share' system that plugs into the Client API, here: https://github.com/floogulinc/hyshare . It looks like a great replacement for my old 'local booru', so if you are interested in sharing groups of files straight from your client with friends over an attractive booru-like interface, check it out!

I copied the 'file log' and 'search log' button menus, where you can do en masse actions like 'retry all failed' and 'export all to clipboard', to both the log review panels and the downloader/watcher list right-click menus. It is now possible to big actions on logs without highlighting anything.

Just a small thing, but when you select a gallery in the gallery downloader page, the focus moves straight to the query text input, so you can start typing immediately.
Open

R: 3 / I: 0 / P: 10

Version 479

https://youtube.com/watch?v=P7MsTw9s03o [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v479a/Hydrus.Network.479a.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v479a/Hydrus.Network.479a.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v479a/Hydrus.Network.479a.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v479a/Hydrus.Network.479a.-.Linux.-.Executable.tar.gz

I had an ok week doing a mix of work.

highlights

I made it so when you shift-select some thumbnails, you can now move 'back' to deselect what you just selected. This also remembers what was previously selected before the shift-select started, so it works basically like an undo. I like how this works, but as part of it I had to make every thumbnail 'hit' focus in the preview viewer, which is not how all selects worked before. I already find this annoying, so I think I am going to make the system more clever and add some options around this behaviour.

I think I improved the duplicate filter's zoom locking, particularly when one of a pair is portrait and the other is landscape. It should generally be more 'stable' now, but let me know if you still have any trouble.

WebPs should now show transparency correctly!

The new file history chart has another round of better math, an 'archived files' line, and you can hide the deleted files line if it is too big.

I reworked a little help to make some 'ok, I know the basics, what next?' things clearer to find. If you missed learning about the autocomplete dropdown, tag wildcards, or OR searching, please check here: https://hydrusnetwork.github.io/hydrus/getting_started_searching.html

full list

- misc:
- when shift-selecting some thumbnails, you can now reverse the direction of the select and what you just selected will be deselected, basically a full undo (issue #1105)
- when ctrl-selecting thumbnails, if you add to the selection, the file you click is now focused and always previewed (previously this only happened if there was no focused file already). this is related to the shift-select logic above, but it may be annoying when making a big ctrl-selection of videos etc.. so let me know and I can make this more clever if needed
- added file sort 'file->hash', which sorts pseudorandomly but repeatably. it sounds not super clever, but it will be useful for certain comparison operations across clients
Open

R: 0 / I: 0 / P: 11

Nijie working?

Does Nijie work for anyone else anymore? I get 503 errors every time, no matter the user-agent or cookies I use.
Open

R: 2 / I: 0 / P: 11

Version 478

https://youtube.com/watch?v=eGybwV3U9W8 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v478/Hydrus.Network.478.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v478/Hydrus.Network.478.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v478/Hydrus.Network.478.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v478/Hydrus.Network.478.-.Linux.-.Executable.tar.gz

I had a good week mostly fixing some bugs and cleaning things up behind the scenes. There's nothing super big to highlight, but I did improve the new file history chart (help->view file history). The axes are a bit nicer, and I fixed a small counting logic bug in the 'inbox' line.

full list

- misc:
- if a file note text is crazy and can't be displayed, this is now handled and the best visual approximation is displayed (and saved back on ok) instead
- fixed an error in the cloudflare problem detection calls for the newer versions of cloudscraper (>=1.2.60) while maintaining support for the older versions. fingers crossed, we also shouldn't repeat this specific error if they refactor again
- .
- file history chart updates:
- fixed the 'inbox' line in file history, which has to be calculated in an odd way and was not counting on file imports adding to the inbox
- the file history chart now expands its y axis range to show all data even if deleted_files is huge. we'll see how nice this actually is IRL
- bumped the file history resolution up from 1,000 to 2,000 steps
- the y axis _should_ now show localised numbers, 5,000 instead of 5000, but the method by which this occurs involves fox tongues and the breath of a slighted widow, so it may just not work for some machines
- .
- cleanup, mostly file location stuff:
- I believe I have replaced all the remaining surplus static 'my files' references with code compatible with multiple local file services. when I add the capability to create new local file services, there now won't be a problem trying to display thumbnails or generate menu actions etc... if they aren't in 'my files'
- pulled the autocomplete dropdown file domain button code out to its own class and refactored it and the multiple location context panel to their own file
- added a 'default file location' option to 'files and trash' page, and a bunch of dialogs (e.g. the search panel when you make a new export folder) and similar now pull it to initialise. for most users this will stay 'my files' forever, but when we hit multiple local file services, it may want to change
- the file domain override options in 'manage tag display and search' now work on the new location system and support multple file services
- in downloaders, when highlighting, a database job that does the 'show files' filter (e.g. to include those in trash or not) now works on the new location context system and will handle files that will be imported to places other than my files
- refactored client api file service parsing
- refactored client api hashes parsing
- cleaned a whole heap of misc location code
- cleaned misc basic code across hydrus and client constant files
- gave 'you don't want the server' help page a very quick pass
- .
- client api:
- in prep for multiple local file services, delete_files now takes an optional file_service_key or file_service_name. by default, it now deletes from all appropriate local services, so behaviour is unchanged from before without the parameter if you just want to delete m8
- undelete files is the same. when we have multiple local file services, an undelete without a file service will undelete to all locations that have a delete record
- delete_files also now takes an optional 'reason' parameter
Open

R: 3 / I: 1 / P: 11

Version 477

https://youtube.com/watch?v=mVG77xTPH6E [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v477/Hydrus.Network.477.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v477/Hydrus.Network.477.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v477/Hydrus.Network.477.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v477/Hydrus.Network.477.-.Linux.-.Executable.tar.gz

I had a good week. There is a mix of small work, an expansion to the Client API, and a bit of fun.

misc

The network engine now pulls source time directly from file downloads if the server provides a date. This means a whole bunch of sites that haven't provided a good source time until now suddenly do, which improves the new aggregate modified time and also subscription and watcher check timings.

With our new apng parsing tech, I fixed up apng duration parsing, which was until now relying on a fallback default of 24 fps if ffmpeg couldn't figure it out. All apngs are scheduled for another scan.

I fixed an important precedence bug in the network engine that matches URLs to URL Classes. If you have been making a downloader and had Gallery URLs matching as Post URLs, please give it another go. Sorry for the trouble!

Client API

A user has written a cool expansion to the Client API, which I appreciate. You can now fetch, set, and delete file notes! If you are an API dev, check out the documentation for the new calls (fetching notes is now a parameter on file_metadata).

He also made technical improvements. The Client API now supports far longer GET requests, up to 2MB of URL if needed, and the whole API has tentative and experimental support for CBOR instead of JSON if you wish.

file history chart

Another user has for some time been playing around with drawing charts of a client's file history in matplotlib using raw database data. You may have run his script yourself. We have been talking for a while about integrating this into hydrus, and this week I finally got around to implementing it in QtCharts.

Please hit 'view file history' on the help menu to see the new chart. This is a simple, first attempt on my end, but it should show you a cool history of how many files you have had. If you have been using the client for any time, the lines for deleted files and inbox will be very incomplete, but this data will fill out in time.

This was fun to do, and I learned a bit more about QtCharts. I fixed a couple of ugly things in the bandwidth bar chart I made before, and I think I'll do some more here too. I have a thought to start drawing some of our other data, let's say file size or number of file views, and seeing if pareto or normal distributions pop out. Anyway, let me know what you think, and feel free to share your file history chart!
Open

R: 3 / I: 0 / P: 11

Version 476

https://youtube.com/watch?v=tigTaObQORM [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v476/Hydrus.Network.476.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v476/Hydrus.Network.476.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v476/Hydrus.Network.476.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v476/Hydrus.Network.476.-.Linux.-.Executable.tar.gz

I had a good week integrating two new features: autocomplete tag search in the client api, and saved 'post times' from downloaders.

post times

'Modified time' is neat, but it isn't super useful for downloaded files--since the file was only just added to your hard drive, it'll always be the same as import time. This week I integrate the 'source time' we parse from various websites to improve the modified time for downloaded files. The objective is to make 'modified' a fairly decent 'this file was completed around this time' number for searching and sorting purposes.

I'm being careful not to overwrite anything. The client now saves its best 'source time' for every different site it downloads from and then the earliest of those + modified date is used as the aggregate modified date. You don't have to do anything, but with luck you will see your new watcher and gallery files start to get some nicer modified times in the media viewer and thumbnail right-click menu.

There are many potential future expansions here. I can grab better post times from sites, show and edit every stored timestamp in UI, allow clever search and sort of those specifically (e.g. 'sort all these files by their danbooru post time), and most importantly make some sort of maintenance system to retroactively fetch a good post time for all the files we downloaded before post times were saved. This is just a first step.

I integrated the new 'archive time' too during this work. This now shows in the media viewer and thumbnail right-click menu similarly and can be sorted by. Search will come soon. I also want to think about optionally filling in some estimate dummy data here for all the files we archived before timestamps were tracked.

client api autocomplete search

A user has helpfully written Client API routines for autocomplete tag search, which is something I have had trouble fitting in. I appreciate the work. This should let the various tools that use the Client API do more tag browsing in future.

The documentation is here: https://hydrusnetwork.github.io/hydrus/developer_api.html#add_tags_search_tags

There are several ways to expand this too, so if you are an API dev interested in it, let me know how it goes.
Open

R: 3 / I: 0 / P: 11

Version 475

https://youtube.com/watch?v=rNFLCB_T2hA [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v475/Hydrus.Network.475.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v475/Hydrus.Network.475.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v475/Hydrus.Network.475.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v475/Hydrus.Network.475.-.Linux.-.Executable.tar.gz

I had a good couple of weeks. There's a long changelog of small items and some new help.

new help

A user has converted all my old handcoded help html to template markup and now the help is automatically built with MkDocs. It now looks nicer for more situations, has automatically generated tables of contents, a darkmode, and even in-built search.

It has been live for a week here:

https://hydrusnetwork.github.io/hydrus/

And is with v475 rolled into the builds too, so you'll have it on your hard disk. Users who run from source will need to build it themselves if they want the local copy, but it is real easy, just one line you can fold into an update script:

https://hydrusnetwork.github.io/hydrus/about_docs.html

I am happy with how this turned out and am very thankful to the user who put the work in to make the migration. It all converted to this new format without any big problems.

misc highlights

I queue up some more files for metadata rescans, hopefully fixing some more apngs and figuring out some audio-only mp4s.

System:hash now supports 'is not', so if you want to paste a ton of hashes you can now say 'but not any of these specific files'.

Searches with lots of -negated tags should be a good bit faster now.

I fixed a bug that was stopping duplicate pages from saving changes to their search.

pycharm

I moved to a new IDE (the software that you use to program with) this week, moving from a jank old WingIDE environment to new PyCharm. It took a bit of time to get familiar with it, so the first week was mostly me doing simple code cleanup to learn the shortcuts and so on, but I am overall very happy with it. It is very powerful and customisable, and it can handle a variety of new tech better.

It might be another few weeks before I am 100% productivity with it, but I am now more ready to move to python 3.9 and Qt 6 later in the year.

full list

- new help docs:
- the hydrus help is now built from markup using MkDocs! it now looks nicer and has search and automatically generated tables of contents and so on. please check it out. a user converted _all_ my old handwritten html to markup and figured out a migration process. thank you very much to this user.
- the help has pretty much the same structure, but online it has moved up a directory from https://hydrusnetwork.github.io/hydrus/help to https://hydrusnetwork.github.io/hydrus. all the old links should redirect in any case, so it isn't a big deal, but I have updated the various places in the program and my social media that have direct links. let me know if you have any trouble
- if you run from source and want a local copy of the help, you can build your own as here: https://hydrusnetwork.github.io/hydrus/about_docs.html . it is super simple, it just takes one extra step. Or just download and extract one of the archive builds
- if you run from source, hit _help->open help_, and don't have help built, the client now gives you a dialog to open the online help or see the guide to build your help
- the help got another round of updates in the second week, some fixed URLs and things and the start of the integration of the 'simple help' written by a user
- I added a screenshot and a bit more text to the 'backing up' help to show how to set up FreeFileSync for a good simple backup
Open

R: 6 / I: 0 / P: 11

Version 474

https://youtube.com/watch?v=JYGb9HRCCyg [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v474/Hydrus.Network.474.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v474/Hydrus.Network.474.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v474/Hydrus.Network.474.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v474/Hydrus.Network.474.-.Linux.-.Executable.tar.gz

I had a great week working on small fixes and improvements. There's nothing earth-shattering here to highlight, but just a mix of little work.

full list

- command palette:
- the guy who put the command pallete together has fixed a 'show palette' bug some people encountered (issue #1060)
- he also added mouse support!
- he added support to show checkable menu items too, and I integrated this for the menubar (lightning bolt icon) items
- I added a line to the default QSS that I think fixes the odd icon/text background colours some users saw in the command palette
- .
- misc:
- file archive times are now recorded in the background. there's no load/search/sort yet, but this will be added in future
- under 'manage shortcuts', there is a new checkbox to rename left- and right-click to primary- and secondary- in the shortcuts UI. if you have a flipped mouse or any other odd situation, try it out
- if a file storage location does not have enough free disk space for a file, or if it just has <100MB generally, the client now throws up a popup to say what happened specifically with instructions to shut down and fix now and automatically pauses subscriptions, paged file import queues, and import folders. this test occurs before the attempt to copy the file into place. free space isn't actually checked over and over, it is cached for up to an hour depending on the last free space amount
- this 'paused all regular imports' mode is also now fired any time any simple file-add action fails to copy. at this stage, we are talking 'device disconnected' and 'device failed' style errors, so might as well pause everything just to be careful
- when the downloader hits a post url that spawns several subsidiary downloads (for instance on pixiv and artstation when you have a multi-file post), the status of that parent post is now 'completed', a new status to represent 'good, but not direct file'. new download queues will then present '3N' and '3 successful' summary counts that actually correspond to number of files rather than number of successful items
- pages now give a concise 'summary name' of 'name - num_files - import progress' (it also eli...des for longer names) for menus and the new command palette, which unlike the older status-bar-based strings are always available and will stop clients with many pages becoming multi-wide-column-menu-hell
- improved apng parsing. hydrus can now detect that pngs are actually apngs for (hopefully) all types of valid apng. it turns out some weird apngs have some additional header data, but I wrote a new chunk parser that should figure it all out
- with luck, users who have window focus issues when closing a child window (e.g. close review services, the main gui does not get focus back), should now see that happen (issue #1063). this may need some more work, so let me know
- the session weight count in the 'pages' menu now updates on any add thumbs, remove thumbs, or thumbnail panel swap. this _should_ be fast all the time, and buffer nicely if it is ever overwhelmed, but let me know if you have a madlad session and get significant new lag when you watch a downloader bring in new files
Open

R: 7 / I: 0 / P: 11

Version 473

https://youtube.com/watch?v=F6OR66gZY54 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v473/Hydrus.Network.473.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v473/Hydrus.Network.473.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v473/Hydrus.Network.473.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v473/Hydrus.Network.473.-.Linux.-.Executable.tar.gz

I had a mixed week. Unfortunately some IRL reduced my work time. There's a neat new widget to play with though!

command palette

A user has written a cool widget that helps you navigate the program by keyboard. I have integrated the first version and am interested in feedback. If you have nothing on Ctrl+P for your 'main window' shortcut set, you should get it mapped to that on update.

So, hit Ctrl+P and you'll get a palette where you can type and press up/down and enter to quickly navigate to any of the pages currently open. If you are an advanced mode user, you will also search all of the menubar actions and also the current thumbnail selection menu. This latter part is unfiltered at the moment--you'll just see everything--so be careful. The system needs more polish, including filtering out these more advanced database routines, and proper display for checkbox items 'check' status, and so on.

I can do a lot more with this widget, so give it a go and let me know what you think. I think some of the labels can probably be improved, and I am sure some would like to customise it a little. If you don't like Ctrl+P, just hit up file->shortcuts->the main window and re-map it!

full list

- misc:
- fixed the recent problem with drag and dropping thumbnails to a level below the top row of pages. sorry for the trouble!
- fixed a bug where the client would not load results sorting by 'import time' when the search file domain was a single deleted file domain
- fixed a list display bug in the edit page parser dialog when a subsidiary page parser has two complicated string-match based content parsers
- collections now sort by modified time, using the largest known modified time in their collection
- added sqlite3.exe console back into the windows build--sorry, it was missing since the github build changeover!
- added a note to the help about backing up when tight on space, which I will repeat here: the sqlite database files are very compressible (70GB->17GB on default 7zip settings!), so if you need more space on your backup drive, this is a good way to reclaim it
- .
- command palette:
- a user has written a cool 'command palette' for the program! it brings up a type-and-search interface to navigate to pages or menu entries.
- I have integrated his first version and set the default shortcut to Ctrl+P. users who update will get this shortcut if they have nothing else on Ctrl+P on 'main window' set. if you prefer Ctrl+K or anything else, you can change it under _file->shortcuts->the main window_
- regular users will get a page list they can search and select, advanced users will also get the (potentially dangerous) full scan of the menubar and current thumbnail right-click menu. I will be polishing this latter feature in future to filter out big maintenance jobs and show checkbox status and similar, so if you are advanced, please be careful for now
- try it out, and let me know how it goes. the underlying widget is neat, and I can change its behaviour and extend it significantly
Open

R: 9 / I: 2 / P: 11

Version 472

https://youtube.com/watch?v=tGgmv1yEI-U [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v472b/Hydrus.Network.472b.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v472b/Hydrus.Network.472b.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v472b/Hydrus.Network.472b.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v472b/Hydrus.Network.472b.-.Linux.-.Executable.tar.gz

I had a good week. There are some neat new UI features and several bug fixes.

new stuff

Although it isn't super useful yet, searching in multiple file domains is now complete. If you click the file domain button on a search page's tag autocomplete dropdown, there is now a 'multiple locations' entry. This lets you select a union of several domains. New users will probably only see 'my files' and 'trash', but advanced users will see more, and deleted domains too. Please feel free to play with this. Autocomplete counts should have accurate ranges and the union file searches, limited though they currently are, do work. We are a step closer to fully supporting multiple 'my files' services.

Under options->thumbnails, there is now a dropdown for 'thumbnail scaling'. The default remains 'scale down only', but you can now 'scale to fit' (which scales small things up) or 'scale to fill', which crops and scales up so the thumbnail fills the whole thumbnail space. Give it a go--the animation as one format changes to another is accidentally one of the coolest things I have done.

If you right-click on the page tab bar, there is now a 'pages' submenu, listing all the pages at or below that level, that lets you quickly navigate to them. This is a prototype, basically a copy of the same thing you'll see in a web browser, so let me know what you think--I expect to put more time into it.

bug fixes

I fixed a stupid issue where search pages were refreshing on session load. I regret this slipped through (again!), so I made an explicit test to catch it in future. Sorry for the trouble!

I also fixed an issue with 'do work on shutdown' cancelling repository processing and a couple of other jobs as soon as they started. The 'fast shutdown' logic was working a little too well!

Images with an alpha channel are now tested on load--if the alpha channel is completely opaque, it is stripped. This saves a little memory and CPU, and it means they will be correctly detected as pixel duplicates of their non-alpha-channel'd equivalents.
Open

R: 3 / I: 0 / P: 11

Version 471

https://youtube.com/watch?v=HSdNxvFCZj4 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v471/Hydrus.Network.471.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v471/Hydrus.Network.471.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v471/Hydrus.Network.471.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v471/Hydrus.Network.471.-.Linux.-.Executable.tar.gz

I had a great week. The program can now track the last time you viewed a file.

last viewed time

By default, file viewing statistics are on. You can control how they work and turn them off under options->file viewing statistics and if you like clear all your records under the database menu.

If you have them on, hydrus now records the last time you saw a file in the media viewer and preview viewer. A 'last viewed' here counts with the same rules as recording view time (typically you need to look at the file a few seconds), so if you just scroll through ten files in a second, those won't be recorded as views.

You can see the last viewed time in the thumbnail right-click menu with the other view time numbers. You can also search and sort it, like with import or modified time.

These three 'time' sorts are now grouped into a new 'time' entry in the sort menu, and for searching under a new 'system:time' system predicate.

Let me know how this works for you! The search and sort uses the media viewer number for now, but if you are a preview or combined person and this does not work for you, let me know how you would like your situation supported.

other highlights

I added some new ways to import the downloader-pngs you can drop onto network->downloaders->import downloaders and other places. The program is now ok with different versions of this data, so you can now drag and drop an image from your browser onto Lain and it should still work. You can also do 'copy image' and click the new paste button. Should just be a bit more convenient when you only need to use the file once. I hope to add more export methods too (e.g. export to clipboard bitmap, which you can then paste into a site/discord), so it should be possible soon for downloader makers to share their work direct to a person without either ever touching a file.

For advanced users: The colours of the scanbar I put below videos is now editable in QSS! This is a simple test for now, as this was more complicated than you'd think. You should now be able to set the base, border, and nub colour. Check out the example in default_hydrus.qss and give it a go in your own custom QSS files. I expect to move all the other hardcoded colours except for probably tag namespaces and maybe some custom thumbnail borders to QSS in future.

R: 0 / I: 0 / P: 11

Parser Sharing Thread

Can't see one here, maybe I'm blind, I'll start.
Open

R: 3 / I: 0 / P: 12

Version 470

https://youtube.com/watch?v=_SSPdeoAHsM [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v470b/Hydrus.Network.470b.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v470b/Hydrus.Network.470b.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v470b/Hydrus.Network.470b.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v470b/Hydrus.Network.470b.-.Linux.-.Executable.tar.gz

I had a great week. The big rewrites are finally doing something interesting, and I got some other stuff done as well.

cleverer search

First of all, I finished my file domain rewrites. File search, tag search, and all the UI that displays it now works on n file domains at once rather than just one. Also, any of those n can be the deleted files of a service now. You can't make new local file services yet, obviously, but all search is now ready for it.

Since adding deleted files search was easy, I am adding it this week. Users in advanced mode will now see a list of deleted file domains on the tag autocomplete file domain button menu. While it is rare you ever want to do this, it was never actually possible to search these domains completely before, and definitely not quickly. Give it a go and let me know if you run into any trouble!

The next step here will be to write a new widget, probably some sort of checkbox list, that lets you select multiples of this new list. Then in future, if you had, say, a 'sfw' file service and a 'nsfw' service, you would be able to seach either or both at once easily. Then, I'll have a handful more things to do: an expansion to file import options to determine where imports are going, cleverer trash that supports n locations, and migration tools so you can move/copy between services, and then I _think_ I will be able to just let you add new local file services in manage services.

misc

I spent a little time with weird files this week. I added support for audio-only mkvs/webms and improved the test that checks whether a video with an audio track actually just has a silent one. I know I get annoyed when a video seems to have audio but actually doesn't, so this week will queue up all your videos for a recheck and hopefully fix a bunch. I also fixed the colours of some weird LAB TIFF files. If you have some jank test images in other unusual colourspaces, please send them in, it was fun figuring this one out.

The program should also shut down a little quicker now!
Open

R: 8 / I: 0 / P: 12

Version 468

https://youtube.com/watch?v=UGnbiBDmuBI [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v468/Hydrus.Network.468.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v468/Hydrus.Network.468.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v468/Hydrus.Network.468.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v468/Hydrus.Network.468.-.Linux.-.Executable.tar.gz

I had a good first week back. A mix of bug fixes and little improvements.

couple of highlights, otherwise all misc this week

Thanks for the quick issue reports over the holiday. The scanbar had an annoying thing where the new single pixel border was making it awkward to drag when in borderless fullscreen--you'd move your mouse to the bottom of the screen, but then you'd click the border and not the scanbar, and the whole video would move--this I have now fixed. Also an issue with new imports' 'pixel duplicates' data not being saved correctly--I have fixed the problem and scheduled all affected files to regen their pixel duplicate data.

I also put some time into multiple local file services. It was more background work, this time to do with system predicate fetching, so there isn't much neat stuff to show off unless you are an advanced user. Almost all the background work is done now, though, so I hope to start spinning up more complex search UI and actually adding new file services pretty soon. One neat thing, now that I have new file filtering tools, I was able to expand 'system:file service' to search for deleted and petitioned files.

full list

- misc:
- fixed an issue where the one pixel border on the new 'control bar' on the media viewer was annoyingly catching mouse events at the bottom of the screen when in borderless fullscreen mode (and hence dragging the video, not scanning video position). the animation scanbar now owns its own border and processes mouse events on it properly
- fixed a typo bug in the new pixel hash system that meant new imports were not being added to the system correctly. on update, all files affected will be fixed of bad data and scheduled for a pixel hash regen. sorry for the trouble, and thank you for the quick reports here
- added a 'fixed font size example' qss file to the install. I have passed this file to others as an example of a quick way to make the font (and essentially ui scale) larger. it has some help comments inside and is now always available. the default example font size is lmao
- fixed another type checking problem for (mostly Arch/AUR) PyQt5 users (issue #1033)
- wrote a new display mappings maintenance routine for the database menu that repopulates the display mappings cache for missing rows. this task will be radically faster than a full regen for some problems, but it only solves those problems
- on boot, the program now explicitly checks if any of the database files are set as read-only and if so will dump out with an appropriate error
- rewrote my various 'file size problem' exception hierarchy to clearly split 'the file import options disallow this big gif' vs 'this file is zero size/this file is malformed'. we've had several problems here recently, but file import options rule-breaking should set 'ignore' again, and import objects should be better about ignore vs fail state from now on
- added more error handling for broken image files. some will report cleaner errors, some will now import
- the new parsing system that discards source urls if they share a domain with a primary import url is now stricter--now discarding only if they share the same url class. the domain check was messing up saving post urls when they were parsed from an api url (issue #1036)
- the network engine no longer sends a Range header if it is expecting to pull html/json, just files. this fixes fetching pages from nijie.info (and several other server engines, it seems), which has some unusual access rules regarding Range and Accept-Encoding
Open

R: 2 / I: 0 / P: 12

Version 467

https://youtube.com/watch?v=MrweXsImVhg [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v467/Hydrus.Network.467.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v467/Hydrus.Network.467.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v467/Hydrus.Network.467.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v467/Hydrus.Network.467.-.Linux.-.Executable.tar.gz

I had a good week cleaning some things up for end of year.

couple of scanbar things

I polished the new autohiding video scanbar. A bunch of the layout and coordinate detection code of the scanbar and the media canvas behind it is less jank. If you had some flicker or weird mouse-popup behaviour last week, I hope it is better now!

If you are a macOS user however, I held back one of the changes. There's a background-erasing hack I put in a couple years ago because without it macOS media viewers went to 100% CPU. I hope this is no longer true, but I'm not certain, so you still have the flag on by default. Please hit _help->debug->gui actions->macos anti-flicker test_ and then try browsing some images and video in a new viewer. Does your client lock up, or is it ok now? If everything is good, I'll remove the flag and you can get some nicer anti-flicker tech too.

full list

- new scanbar cleanup:
- the media container's scanbar and volume control are now combined on the same widget, meaning they now show/hide in sync and faster. their layout calculation is also more sensible. the new controls bar also has a thin border to make it pop better against a background video
- improved the way some auto-hide anti-flicker tech on the scanbar now works. it all hides a frame faster sometimes
- figured out some new anti-flicker tech to reduce/eliminate a frame of stretch when flicking from a static image to an mpv video, particularly for the first or second time in a session
- fixed a bug where clicking the global mute/unmute on an mpv player meant that certain shortcut keys (usually those with arrow keys) would not work on that player again. (it was a focus issue on the button, which then captured some form navigation keys but they had nowhere to go)
- brushed up some mouse coordinate testing logic across the program. some linux clients had trouble with the new animation scanbar popping up over mpv, I think I improved it!
- fixed another type problem with newer python/PyQt5 on Arch, also in scanbar coordinate testing
- fixed some dodgy colours in the scanbar initialisation and volume control border
- macOS users: I undid a long-time paint hack on the media container and the static image canvas. Qt is responsible for clearing the background again, which allows me to remove some jank anti-flicker tech. HOWEVER, the original reason for this hack was because without it, old macOS went to 100% CPU whenever the media viewer was showing something. therefore, to be safe, this option is still on for macOS users for now. you'll get a little flicker when browsing. please try hitting _help->debug->gui actions->macOS anti-flicker test_ and do some mixed video/image browsing. does your whole damn client lock up?

R: 0 / I: 0 / P: 12

Malformed Drive -- Help

I've got multiple drives I've tested Hydrus on, and can't figure out why I'm having issues. All my drives are healthy and run without problems. At first I thought Window's built in drive compression might be fucking with Hydrus, but I decompressed my whole SSD, and I'm still getting errors. Can someone suggest a possible reason for my issue? Will provide logs if needed, Thanks in advance.
Open

R: 4 / I: 0 / P: 12

Version 466

https://youtube.com/watch?v=kWXAL9bHciQ [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v466/Hydrus.Network.466.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v466/Hydrus.Network.466.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v466/Hydrus.Network.466.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v466/Hydrus.Network.466.-.Linux.-.Executable.tar.gz

I had a great week. The duplicate filter now supports filtering by pixel duplicates, and videos now fit better in the media viewer.

better duplicate search

Now we have pixel duplicate data stored in the database, I can search it. The duplicate filter page now has a dropdown that lets you select 'must be pixel dupes', 'can be pixel dupes', and 'must not be pixel dupes'. It does exactly what you think, and with luck it should be pretty fast no matter what you select.

Note that you may not have all your old pixel dupe data calculated yet. I started it for everyone last week, but even if it is doing 10,000 files a day, if you have a big client it will take a little longer. It may also still be working on the ICC profile queue. Hit up database->file maintenance->manage scheduled jobs to see how it is doing. You can rush it there if you like.

As I was working in this system, I discovered that at some point I had started recording at what search distance potential duplicates were matched. This is to say if you were set up to search at distance 4 and two files were matched as potential duplicates with a distance of 2, it would save that 2 into the database. I am not totally sure how retroactively accurate this data is, but I've added a control for it to the filtering panel too. You can now tell the filter to only present files that are 'exact match' at distance 0 and it should work mostly ok.

Some pretty complicated database work went into this. The most complicated search joins seven different things together. I know it is fast on my test machine, but if you have a really large client with a lot of searched dupes, some search types may be unbearably slow. Please let me know how you get on, and I'll optimise what I can.

scanbar

The audio/video scanbar in the media viewer is now embedded inside the media frame. It shows and autohides when your mouse moves closer and away. It now means you can now go borderless fullscreen on a 16:9 display and finally have a 16:9 video fit perfectly!

I have wanted to do this for a really long time, but some of the layout code here is really awkward, and getting widgets to pop on top of each other can be tricky. Thankfully it turns out Qt has a nice way to do it, so I've now hacked this together. There is still a small amount of jank--the scanbar and volume control are currently separate objects, so sometimes they'll show/hide in separate frames, and you might also see the scanbar nub pop in a frame late, but I can work on these issues in future. I will also add some options so you can change the size of the show/hide activation area around the scanbar. But for now, I am pretty happy with this.

If you are a keyboard user, please check out the new shortcut in the 'global' set that flips on/off a 'force the animation scanbar to show' mode. I don't really want to bring back the old always-on hanging-below scanbar, so I hope this will be a good enough substitute. But let me know, and if you really hate this new scanbar, we'll see what we can do.
Open

R: 3 / I: 0 / P: 12

Version 464

https://youtube.com/watch?v=mn34GKTVqMM [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v464/Hydrus.Network.464.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v464/Hydrus.Network.464.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v464/Hydrus.Network.464.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v464/Hydrus.Network.464.-.Linux.-.Executable.tar.gz

I had a good week. Some images get better colour, and the software now deletes files more neatly.

ICC profiles

tl;dr: Some image colours are better now.

Some images come with an 'ICC profile'. A scan of one ICC-heavy client showed that 10% of all images had them. ICCs are basically colour correction data like 'actually, the blue should be this much more intensely blue than is stored' in a variety of sometimes complicated ways. Some nice cameras attach a fast ICC based on lighting conditions or known defects in the camera, and some image editing software does similar. ICCs are also used for broader and more complicated colour conversion tasks like in the migration from sRGB to HDR formats if you have a new tv/monitor.

Hydrus now has basic ICC tech. It recognises when an image has an ICC and will apply it, converting the rendered colour to 'normal'. This happens right on image load, so it should apply in every media viewer and all new thumbnails. Should work for the server, too. With luck, this will be one of those things where you don't really notice any difference, but you should just have some nicer pictures, mostly photos, in your client. Let me know if you have any files that you know are still not rendering right!

This work took a long time to get done, but a user helped me out with some great examples and more background info on how to handle it all properly. The next step will be to add more indication of when an image has an ICC and add the ability to flip it on and off. Another topic is 'display ICCs', where some artists and other enthusiasts will have an ICC calibrated for their particular monitor or for different display conditions (e.g. 'how would this look like under this lighting?' or '...from this printer?'). Since we have the tech now, I think I should be able to support this sort of ICC conversion too.
Open

R: 3 / I: 0 / P: 12

Version 463

https://youtube.com/watch?v=Xm3mrwyR2pw [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v463/Hydrus.Network.463.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v463/Hydrus.Network.463.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v463/Hydrus.Network.463.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v463/Hydrus.Network.463.-.Linux.-.Executable.tar.gz

I had a great week. I was able to catch up on the work I wanted and do some neat stuff besides.

import presentation

I completely overhauled the importer 'presentation' options, which you can see under all 'file import options'. These have always been a bit advanced and confusing, so I rewrote it to specifically say a more human-friendly summary like 'show new files' and added text and tooltips to better explain what was going on. It also works a bit faster and cleaner.

You can also now say 'show inbox files', regardless of 'successful' or 'already in db', which I think may be neat as the default for a watcher page that you process and revisit several times--every time you were to re-highlight a watcher, it would filter out anything you previously archived.

And I updated the right-click menu on downloader and watcher pages' lists, where it said "show importers' new files", to use the new options. It now tucks it into a submenu, adds the 'inbox only' option, and if only one importer is selected it says what the default actually is and removes any duplicate action.

other highlights

The client now recognises ogg files with video as ogv! Previously, they were always considered audio, but now it will make thumbnails and parse resolution and duration and let you zoom properly with the video player and so on. All existing ogg files will be scheduled for a rescan when you update, so your ogvs should just pop into place.

I think I fixed the deviant art tag search downloader. They changed their tag search routine recently, so I had to change some things behind the scenes. Everyone will get it on update, and the existing 'deviant art tag search' downloader should just work again.

The manage ratings dialog now has copy/paste buttons for quick copying of multiple ratings across sets of files. This is a prototype, so give it a go and let me know what you think. It copies 'null' ratings at the moment, but maybe we'll want a second option too that only copies set ratings.
Open

R: 2 / I: 0 / P: 12

Version 462

https://youtube.com/watch?v=Gu2Iayi3AvM [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v462/Hydrus.Network.462.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v462/Hydrus.Network.462.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v462/Hydrus.Network.462.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v462/Hydrus.Network.462.-.Linux.-.Executable.tar.gz

I had a good week with a variety of fixes and optimisations. New users also get a couple of user-friendly services to start with.

highlights

A recent update to the mpv seekbar accidentally caused bad errors during audio playback. I have now fixed it. I regret the regression, and I have added tests to make sure this does not happen again.

All 'file log' import queues now search and count themselves faster. Any heavy import session, like a very large downloader or many medium size downloaders working at once, should use less CPU just to do normal work.

I sped up some file searches that were running super slow. The general problem file search was typically a mix of 'has multiple search predicates' and 'includes a short unnamespaced tag or namespace:anything'. This combination was taking two minutes or more rather than a handful of seconds due to an overly complicated database query that wasn't running well. Should be fixed now, but let me know if you have any more of this sort of trouble!

new user services

To help new users, new clients now start with two extra services:

First is a local tag service called 'downloader tags', which the initial default tag import options is now set to get tags for. I hope this will reduce some of the confusion around default tag import options, merged downloader/personal tags in 'my tags', and having to retroactively fill in missed tags for files downloaded early on.

Second is a like/dislike rating service called 'favourites'. Many users are unaware of the existence of ratings services at all, so now everyone will start with a simple clickable star. When you update, if you currently have no rating services, you will be asked if you want to get this new default. You don't have to, but if you haven't tried ratings out yet, give it a go!

full list

- misc:
- fixed a recent serious regression that could cause a crash when playing audio in mpv (issue #1007)
- the main importer file log now does 'get next/all/count imports with status y' calls significantly faster, particularly on very large lists. these calls happen all the time for different status text changes and general 'which import to try next?' work. all busy downloader situations should see CPU gains to regular and background work
- fixed a problem where importing with the min/max file resolution options set would give a typo error when the size was violated rather than 'ignored'
- I think I have fixed an issue with subscriptions not wanting to run a query if by random accident that query has an invalid URL selected as the query's 'example url' for various pre-work login and bandwidth tests
- hydrus can now capture duration/fps of videos that specify two very close fps, e.g. 60 and 59.99. previously, these would default to the 24 fallback and could cause some weirdness in mpv
- replaced the default pixiv artist page api parser with one that fetches the newer url format, matching the tag search. existing users will see no automatic change but will receive the new parser, so if you are a big pixiv user, you might like to switch 'pixiv artist gallery page api' to the 'new urls' parser variant under _network->downloader components->manage url class links_. note that if you do this, your pixiv artist subscriptions will do a mini-resync (which involves some wasted time/bandwidth) as their urls change!
Open

R: 3 / I: 0 / P: 12

Version 461

https://youtube.com/watch?v=oiP5bKlJ0fM [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v461/Hydrus.Network.461.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v461/Hydrus.Network.461.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v461/Hydrus.Network.461.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v461/Hydrus.Network.461.-.Linux.-.Executable.tar.gz

I had a great week doing a lot of background cleanup work, and I also fixed some things and improved some quality of life.

highlights

I brushed up last week's .clip file metadata parsing. Some clip files were getting too-small resolutions and thumbnails due to a 'canvas unit' issue, basically some clips store resolution in mm or inches etc... rather than pixels. Thanks to a user who also provided some great examples, we managed to pin down a solution to get actual pixel resolution, and I also figured out duration and number of frames for animated clips. All clips will reparse again on update and generate nicer size thumbs if they need it.

Also, if you ever got a webm that seemed normal except it somehow had a crazy 7 hour duration, I think I figured out the problem here too. All superlong-but-actually-small files will be reparsed on update and should get a more accurate duration.

When you right-click on tags in the taglist, the 'search' submenu now provides actions to require/exclude 'namespace:anything' if your selection all shares the same namespace.

Also, as a little test, I am trying out a new quiet shortcut on the file selection taglist. If you now hold down control when double-clicking or hitting enter on tag(s) here, they now go up to the search negated. Double-click = 'add tag to search', ctrl+double-click = 'add -tag to search'. Of course ctrl+click is awkward since it can cause a deselection, so I think this could do with some more work, and eventual integration into the shortcuts system so you can change it as you like, but let me know what you think. A user recommended adding the +/- buttons as you see on some boorus, so this is my interim step towards that.

When you paste query texts into the edit subscription dialog, if any of the texts you paste are already in the subscription and currently DEAD, the dialog now revives those subs (basically the same as 'check now'). I have personally wanted this for ages and kept forgetting to add it. The sub will just do another quick check on what you paste, just in case that query started getting things again recently, rather than swallowing your input as forevermore DEAD.

I moved some autocomplete options from 'gui pages' options panel to 'search'. If you are a Linux user who has trouble with the floating autocomplete dropdown window, check it out and see if 'embedding' the dropdown works better for you.
Open

R: 2 / I: 0 / P: 12

Version 460

https://youtube.com/watch?v=gP5LXd6DFx4 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v460/Hydrus.Network.460.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v460/Hydrus.Network.460.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v460/Hydrus.Network.460.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v460/Hydrus.Network.460.-.Linux.-.Executable.tar.gz

I had an ok but light week of various work.

misc

When I improved the 'apng' file parsing the other week, there was a mistake in how I hooked it up and not all the file metadata was being propagated to the database. A couple of users got some weird '1 frame' apngs as a result. I have fixed this problem, and improved the parsing to deal with more sorts of apngs, and scheduled a complete rescan of all your apngs to get some nicer 'number of frames' counts.

In a related thing, a user sent in a way to get some cool metadata from 'clip' files. These now get resolution and a rough thumbnail. I have scheduled all your clip files, if you have any, to update too.

The Client API now supports a better way to refer to tag services when editing tags. If you are a Client API developer, please check out the updated help--you can now user service keys instead of service names. I am in no rush to delete the 'names' parameters, but I expect I will eventually. 'names' was always an early hack that has flaws as an identifier, so when it is convenient, please try to migrate to /get_services and service keys.

A new help file in install_dir/db directory, 'help my mpv crashes with WASAPI or ASIO audio', now has a neat couple lines to add to your mpv.conf if you have this problem. Many thanks to the user who figured this out!

full list

- client api:
- /add_tags/add_tags now supports 'service_keys_to_tags' and 'service_keys_to_actions_to_tags'.
- /add_urls/add_url now supports 'service_keys_to_additional_tags'
- /get_files/file_metadata now returns with duplicates of the tag structures using 'service_keys_to_statuses_to_(display_)tags'
- added unit tests for the above
- updated the client api help for the above
- I recommend you move from 'service_names_to...' to 'service_keys_to...' when convenient. 'names' was an ugly old hack, and while I am not in a rush to delete it from the client api, I think I will eventually
- client api version is now 21
- .
- apngs and clips:
- fixed a problem where the new apng metadata parsing was not completely hooked up, so num_frames was not being parsed correct for the final metadata row even when 'apng' filetype was, leading to some odd '1 frame apng' situations
- apng parsing now recognises more kinds of apngs--if one has an unusual scaling chunk in the header, this is recognised and the correct animation chunk searched for
- clip files now get resolution on import and a thumbnail! big thanks to the user who dug up how to extract this--it actually comes from a sqlite file embedded in the clip! (issue #996)
- on update, all apngs and clips will be scheduled for metadata rescan, and all clips will be scheduled for thumbnail generation
Open

R: 3 / I: 0 / P: 13

Version 459

https://youtube.com/watch?v=gZgbF2TRK2k [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v459.2/Hydrus.Network.459.2.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v459.2/Hydrus.Network.459.2.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v459.2/Hydrus.Network.459.2.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v459.2/Hydrus.Network.459.2.-.Linux.-.Executable.tar.gz

Hey, the release I put out earlier had a problem! The links above point to the fixed version. If you got the old one within an hour of me putting this post up, please download and install/extract again and you'll be able to boot. Sorry for the trouble!

I had a great week mostly clearing out bugs.

highlights

I gave the new duplicate filter zoom locking calculation another pass. Thank you to the users who submitted examples of my last attempt not working right. It is now more careful and targeted towards fixing the 'a watermark banner spilled over my monitor edge' problem, and it deals with different combinations of portrait and landscape images and viewers better. Let me know if you still have any borked zooms!

The client's downloader system can now deal with some 'partial/interrupted download' situations and automatically resume from where it left off. If you ever saw a network job widget error out with 'was expecting 21.3MB but got 458KB', this will try to fix that!

The program should be better about saving the work of subscriptions that are running when you exit the program. The way their jobs ran was a little hacky, previously, so I moved them to the main worker pool everything else runs on and brushed up their shutdown signals. Let me know if you encounter any situations of them not shutting down nice and fast with client shutdown.

To help with debugging and other screenshotted reports, the UI now puts the program version number in the title of windows and dialogs. Also, if you previously set a custom 'main gui title' under options->gui, I have reset this this week and renamed it to 'application display name'--it now shows on all windows.

Big Sur macoS clients have had a problem for a while where menus spawned from dialogs can't be clicked. It seems to be a weird Qt bug. Until there is a better fix, I hacked together a new window to display the menu items instead. If you are a Big Sur user, let me know how it works for you. You can turn it off if you need to under options->gui.

I wrote a new help document, 'help my media files are broke.txt', in the install_dir/db directory. This collects the different repair routines I have developed with users who have run into trouble with broken drives or incomplete backup restores. If you are missing many files--or you know you have many surplus files--and were not sure how to fix the situation, please check it out!

I fixed the new 'advanced file deletion' checkboxes under options->files and trash! Sorry, they weren't hooked up right in 458!
Open

R: 4 / I: 0 / P: 13

Version 458

https://youtube.com/watch?v=45pF6YtMvQ4 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v458/Hydrus.Network.458.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v458/Hydrus.Network.458.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v458/Hydrus.Network.458.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v458/Hydrus.Network.458.-.Linux.-.Executable.tar.gz

I had a good week. I fixed some more bugs and improved some quality of life.

highlights

I fixed an html parsing problem that started in the past few weeks in the release build. I understand it broke Newgrounds searching, but it should be fixed now, so if this affected you please give it another go.

You can now set the client to always boot with network traffic paused under network->pause. Useful if you have an unusual network/vpn situation that is often not connected!

I have improved the duplicate filter zoom lock calculation. Previously, it forced the width of the two images to be the same--now it chooses height if it thinks that fits things better. This should stop 'B' images with a watermark banner on the bottom accidentally spilling that watermark difference below the screen.

advanced user highlights

I improved some repository update error handling and auto-healing. If you have had an issue processing the PTR related to 'this file was the wrong type' that did not seem to fix by itself, please unpause processing and let it try one more time.

If you use the advanced file deletion dialog, you can now choose to remember the last used reason and action under options->files and trash. I also fixed a bug where some number of files were forgetting their given deletion reason once they were permanently deleted from the trash. I am sorry for the trouble with this bug, and I will make sure to add unit tests so it cannot happen again.

File import objects now track 'primary' URLs (that directly lead to a file download) separately from parsed 'source' URLs. They are now split under the new 'file log' URLs menu, which also now shows the referral URL. Furthermore, file import options now lets you choose whether primary URLs are associated with imports (which you might want to turn off if you are doing some clever one-time proxy download etc...).
Open

R: 3 / I: 0 / P: 13

Version 457

https://youtube.com/watch?v=T4D3Qmn0nuY [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v457/Hydrus.Network.457.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v457/Hydrus.Network.457.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v457/Hydrus.Network.457.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v457/Hydrus.Network.457.-.Linux.-.Executable.tar.gz

I had a good week. Some UI jank should be much better!

highlights

I reworked how the menubar menus (file, network, pending etc...) update this week. They update labels and swap items in and out more efficiently. Most importantly, the 'pending' menu should now update with no flicker at all, so if you have a bunch of downloaders/subscriptions running and continually pending new tags, you should be able to commit and upload with no trouble! Let me know how it works for you.

I also removed some ancient and now obsolete safety checks that stop the client from updating certain UI while the program is minimised. With luck, when you restore the client--and more significantly, when you restore it from minimised to system tray--there should be less judder, and hopefully no freezing, since the client no longer has to catch up with hundreds or thousands of updates. If you minimise to system tray and have had hangs, let me know if the situation is better now. Also, if you suddenly get popup message sizing problems or similar UI jank after minimising or putting the client on a virtual desktop or similar, please let me know.

The client now uses much less CPU to parse and import png files!

I updated the mid-tier file searching help here: https://hydrusnetwork.github.io/hydrus/help/getting_started_more_files.html
Open

R: 2 / I: 0 / P: 13

Version 456

https://youtube.com/watch?v=MxaD8_Vazpo [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v456/Hydrus.Network.456.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v456/Hydrus.Network.456.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v456/Hydrus.Network.456.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v456/Hydrus.Network.456.-.Linux.-.Executable.tar.gz

I had an ok week. With luck, the client should have less UI lag, and I also fixed a bunch of stuff and improved some quality of life.

basic highlights

I removed some very hacky memory management code this week. It was eating far more CPU than it was worth, particularly for large clients. If you have a very heavy client, particularly if it has lots of heavy subscriptions, please let me know if you A) have fewer UI lockups, and B) see any crazy memory spikes while running subs. As bad as the old routine was, it was aggressive and effective at what it did, so I may have to revisit this.

All multi-column lists across the program now catch enter/return keystrokes and trigger an 'activate' call, as if you had double-clicked. Should be easy to navigate and highlight a downloader page list just with the keyboard now.

advanced highlights

The 'file log' window now lets you see and copy an import item's hashes, tags, and urls from its right-click menu. I hope this will help debug some weirder downloader problems and generally inform on how the downloaders work.

The Edit URL Class dialog has had a layout makeover. Also, URL Classes now support matching and normalising 'single value' parameters (this is where you have a token/keyword parameter rather than the traditional key=value pair).

The hydrus server now remembers custom update and anonymisation periods! Previously, it was resetting to defaults on a restart! Thank you for the reports here--I apologise for the inconvenience and delay.

I added an 'mpv report mode' to the debug menu. If mpv loads for you but you have silent audio or similar (and perhaps some crashes, but we'll see if this catches useful info in time), this'll dump a huge amount of mpv debug information to the log.
Open

R: 5 / I: 1 / P: 13

Version 455

https://youtube.com/watch?v=Gs069dndIYk [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v455/Hydrus.Network.455.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v455/Hydrus.Network.455.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v455/Hydrus.Network.455.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v455/Hydrus.Network.455.-.Linux.-.Executable.tar.gz

I had an ok couple of weeks. Some quality of life is improved, the UI should save sessions a bit quicker, and the database repairs itself more of itself.

Your client may warn you about a missing index on first boot. If it does, that's great--some new repair code is working!

session save

Some users with large sessions (typically around 2-3+ million 'weight') are experiencing severe UI lockups and general lag. This week I deal with part of that problem by greatly reducing the amount of CPU needed to save most sessions. Now pages know if they have been changed since being loaded, and if they have no changes, they can skip a bunch of session save work.

There's nothing you have to do here, but if you have been experiencing 12+ second freezes every few minutes, please let me know how you get on now. I can't promise things will be perfect, but if you have any large and inactive downloader pages, you should notice things running a bit smoother (exiting the program too, should be faster).

There is still a problem of lag and lockups when there are 200+ downloaders in the session. This is a different and unfortunately more severe problem that I have relieved in recent weeks but needs some other large work to completely fix. Best solution for now is to clear out finished downloaders if you can.

misc

Many of the simple system predicates, like height and filesize, now support the '≠' (not equal) operator! This is still in the early stages, but I'd like to add some options of the thumbnail menu so you can say 'find files that (do not) have this file's precise resolution/duration/size' etc...

You can now copy thread subjects to clipboard in the watcher downloader page list!

The advanced file deletion dialog now remembers the last reason you chose!

The network job widget (which shows download progress, you see it on any downloader) now shows the current URL it is working on under its cog menu. You can click it to copy to clipboard. May help to debug some odd network errors etc...

The way data sizes over 1KB are rendered has been overhauled. It should now generally be to three significant figures, so 3.11MB instead of 3.1MB, 12.3GB instead of 12GB. See if you like the change, and if not, try tweaking it to 2sf under the new EXPERIMENTAL option in options->gui.
Open

R: 3 / I: 0 / P: 13

Version 454

https://youtube.com/watch?v=Ch24PpzMKog [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v454/Hydrus.Network.454.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v454/Hydrus.Network.454.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v454/Hydrus.Network.454.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v454/Hydrus.Network.454.-.Linux.-.Executable.tar.gz

I had an ok week mostly working on making images look better.

warped tiles

tl;dr: Images look better now.

I put a lot of time into the tiled renderer this week, mostly fixing 'stretch-warped' tiles. These are most obvious in the duplicate viewer when you are flipping between two very similar images that have slightly different resolution--you would see one part of one of the images stretched one pixel horizontally or vertically compared to the other.

This turned out to be complicated to fix, and moreso for unusual zooms (like a 124.65% style canvas fit zoom), but I think I have fixed almost all warping, and tile join artifacts, for all zooms below about ~6000%, with the exception of some bottommost and rightmost tiles.

I also added an option to options->speed and memory to govern how large the tiles should be. This is 768-pixel squares by default, but if you still have trouble with warping in the duplicate viewer or other high zoom situations, please try boosting it a bit. Larger numbers will be less memory and CPU efficient, so I don't recommend 4096 to start off, but let's see what we find in real situations.

I like the tiled renderer, so I don't want to abandon or completely replace it. I thought seriously of ditching it this week, when I could not figure out a solution to warping, but I slept on it and figured it out. If I can keep patching remaining problems, I'll be happy, but I do have a backup plan as well. Let me know how you get on!

the rest

I think I fixed the issue where, after some serious network errors, the downloaders could get in a 'traffic jam' and all stay on 'pending', with nothing going to 'working'.

Finishing a big archive/delete filter should result in less UI lag. If you are a user who hits F5 right after committing archive/delete, let me know how this works for you!

The green/blue/red texts in the duplicate filter's right-hand hover window are now moved to the style system, so QSS stylesheet files (as under options->style) can finally change their colour. I set up some basic dark/light blues for all the default QSS files, so it should be a bit easier to read the blue text in dark mode styles now. These were the last hardcoded text colours in the program, so we are now one step closer to moving everything to QSS.
Open

R: 3 / I: 0 / P: 13

Version 453

https://youtube.com/watch?v=YA6A8xohpxc [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v453/Hydrus.Network.453.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v453/Hydrus.Network.453.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v453/Hydrus.Network.453.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v453/Hydrus.Network.453.-.Linux.-.Executable.tar.gz

I had a great week. I ended up mostly fixing bugs. If you have a large client, the update may take a few minutes this week.

bug fixes

I discovered a semi-important tag processing bug last week--for some users, files that were imported before they added the PTR were not getting all the correct tag info and would not appear in some tag searches. I fixed the underlying file-tracking bug and have added a retroactive 'gap-filler' to this week's update. If you used the client for a long time but added/reset the PTR recently, you'll have more gaps so your update will take longer.

Another persistent issue has been the 'tiling artifacts' with my new tile-based image renderer, where at certain zooms an image would have one or more horizontal or vertical blurred/janked line of pixels. You'd notice it on cleaner vector images where a smooth curved black line would suddenly jag at one zoom. I had trouble reproducing this at first, but some users got it quite often, and with their help I was able to figure it out. I have rewritten the part of my renderer that was failing, and I think I now have the artifacts completely fixed for all 'normal' zooms. Let me know how you get on!

misc

Network job widgets have a little user-friendly update this week. I brushed up some of the status texts, made the text lay out better, and when the widget counts down waiting for something (like free bandwidth), if it doesn't get it because another downloader did, it says so before resetting the countdown clock.

A helpful user wrote a new darkmode style called OledBlack. Check it out under options->style!

I added OR predicates to Client API file search!

If you are an advanced user, you might like to read this idea I had for setting up multi-tag processing workflows: https://hydrusnetwork.github.io/hydrus/help/advanced_parents.html#parent_favourites I suddenly thought of it a couple weeks ago and it worked so well for me that I decided to write it up. It basically involves using local parent tags to group PTR tags into a pseudo-OR search that you can easily edit.
Open

R: 2 / I: 0 / P: 13

Version 452

https://youtube.com/watch?v=iePWOFOSl_U [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v452/Hydrus.Network.452.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v452/Hydrus.Network.452.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v452/Hydrus.Network.452.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v452/Hydrus.Network.452.-.Linux.-.Executable.tar.gz

I had an ok week. I ended up doing a lot of boring behind the scenes prep, but there is also some nice new quality of life. If you sync with the PTR, update will take a few minutes this week.

all misc this week

After a very long delay, I finally have shortcuts for seeking video back/forwards in the media viewer. New users will get ctrl+left/right to seek back 2.5 seconds or forwards 5 seconds. Existing users need to add their own--please check it under the 'media viewers - all' shortcut set. You can set whatever seek distance you like, and even set multiple with different distance jumps. Sorry for how long it took to get this in--I had to update my shortcut system first!

I did some behind the scenes tag work this week. The database can now handle larger 'tag as number' searches. Update will take a few minutes to load these bigger numbers into the fast search cache.

If you have a subscription that is not meant to completely sync (e.g. it pulls a sample from a gallery with unusual sort order, or you have the 'normal file limit' intentionally small), there is a new checkbox that suppresses the 'hey, there was a gap, click here to fill it in' popup windows.

full list

- misc:
- my 'simple' shortcut commands can now store additional variables. to start things off, I have finally added 'seek video' shortcut commands that have back/forwards and second+millisecond values. these should work on the native video viewer and mpv, audio and video. existing users will have to add their own (do it to the 'media viewers - all' set), but new users will get ctrl+left/right for 2.5s back and 5s forwards as the new defaults. let me know if you have any trouble!
- the maximum number tracked by 'tag as number' system predicate is expanded from -99999999->99999999 to -2^63 -> 2^63 - 1. tag caches will be regenerated on update to store these, it will take a few minutes. the input ui for the system predicate is temporarily limited to -/+2^31, but I'll expand it
- subscriptions now have a checkbox for 'do not worry about subscription gaps'. if you have a subscription that gets files randomly, or gets an intentionally small sample, this will disable the 'hey, there was a gap, click here to fill it in' popup messages
- you can now set negative values for the duplicate score weights in options->duplicates
- also added a weight for the 'nice ratio' duplicate comparison

R: 69 / I: 13 / P: 13

Bug Reports

Please feel free to submit any bug reports here.
Open

R: 3 / I: 0 / P: 13

Version 450

https://youtube.com/watch?v=fLLKwIfs1NM [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v450/Hydrus.Network.450.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v450/Hydrus.Network.450.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v450/Hydrus.Network.450.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v450/Hydrus.Network.450.-.Linux.-.Executable.tar.gz

🎉🎉 MERRY 450! 🎉🎉

I had an ok week. Last week's experimental release went well, so I have polished that, and I otherwise caught up with a variety of small work.

If you sync with the PTR, update will take a couple of minutes.

all misc this week

So, the update storage change went ok! There were a couple of little sync bugs to clear up, but overall it works--tag repositories now track their processing progress by mappings, siblings and parents separately. You don't have to do anything, and this doesn't matter much for day to day work, but it allows for individualised reset and reprocessing. All 448 users will have their siblings and parents reset and reprocessed, which will take a couple of minutes to do on update, and about fifteen minutes on your next processing job to fill back in, and which should eliminate some bad siblings and parents due to years-old processing bugs that long term users have been dealing with (leaving only current bugs, which I am also working on). The reset will not delete any pending siblings or parents you have, so no worries if you have a bunch waiting to be uploaded.

Advanced Users: The PTR sibling and parent reset will however remove any siblings and parents you uploaded that were then denied by jannies (which your client would have added to itself anyway). Everyone is reset to a 'clean' sync with this change, so if you know you have a ton of surplus denied siblings you rely on, perhaps from years ago that we agreed I would deny on the PTR to help you hack in an overwrite in the old system, you might like to hold off updating and first figure out a PTR sibling/parent backup to a local service using tag migration.

I fixed some things with Mr Bones. His numbers are accurate to your 'my files' again, and he now talks about your total deleted files and also your earliest file import time. I divided the ugly growing stack of numbers into tabs, which I am sort of happy with, sort of not. People like to take screens of Mr Bones, but they have different preferences on what to show, so I may replace this with expand/collapse buttons or similar, so you can show everything if you want.

If you use the export files window to get a lot of files out, it now makes a progress popup. You can close the window while it is exporting and still see and cancel the job.

The Client API file search now supports file and tag domain selection (like the 'my files' and 'all known tags' buttons on a normal search page), and also file sort for searches. I know the Client API guys have been waiting on this, so with luck we should see some neat new search options in the Client API programs in the near future.
Open

R: 6 / I: 0 / P: 14

Version 448

https://youtube.com/watch?v=VEwVAV3VPw4 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v448/Hydrus.Network.448.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v448/Hydrus.Network.448.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v448/Hydrus.Network.448.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v448/Hydrus.Network.448.-.Linux.-.Executable.tar.gz

Hey, I am sorry, Endchan has been down or not allowed login/posting the past couple weeks when I needed to get release out, so there was a gap.

I had an ok couple of weeks. I was pretty ill in the middle, but I got some good work done overall. .wav files are now supported, PSD files get thumbnails, vacuum returns, and the Client API allows much cleverer search.

client api

I have added some features to the Client API. It was more complicated than I expected, so I couldn't get everything I wanted done, but I think this is a decent step forward.

First off, the main 'search for files' routine now supports many system predicates. This is thanks to a user who wrote a great system predicate text parser a long time ago. I regret I am only catching up with his work now, since it works great. I expect to roll it into normal autocomplete typing as well--letting you type 'system:width<500' and actually getting the full predicate object in the results list to select.

If you are working with the Client API, please check out the extended help here:

https://hydrusnetwork.github.io/hydrus/help/client_api.html#get_files_search_files

There's a giant list of the current supported inputs. You'll just be submitting system predicates as text, and it handles the rest. Please note that this is a complicated system, and while I have plenty of unit tests and so on, if you discover predicates that should parse but are giving errors or any other jank behaviour, please let me know!

Next step here is to add file sort and file/tag domain.

Next there's a routine that lets you add files to arbitrary pages, just like a thumbnail drag and drop:

https://hydrusnetwork.github.io/hydrus/help/client_api.html#manage_pages_add_files

This is limited to currently open pages for now, but I will add a command to create an empty file page so you can implement an external file importer page.

misc

.wav files are now supported! They should work fine in mpv as well.

Simple PSD files now get thumbnails! It turns out FFMPEG can figure this out as long as the PSD isn't too complicated, so I've done it like for .swf files--if it works, the PSD gets a nice thumbnail, and if it doesn't it gets the file default icon stretched to the correct ratio. When you update, all existing PSDs will be queued for a thumbnail regen, so they should sort themselves out in the background.

Thanks to profiles users sent in, I optimised some database code. Repository processing and large file deletes should be a little faster. I had a good look at some slow session save profiles--having hundreds of thousands of URLs in downloader pages currently eats a ton of CPU during session autosave--but the solution will require two rounds of significant work.

Database vacuum returns as a manual job. I disabled this a month or so ago as it was always a rude sledgehammer that never actually achieved all that much. Now there is some UI under database->db maintenance->review vacuum data that shows each database file separately with their current free space (i.e. what a vacuum will recover), whether it looks like you have enough space to vacuum, an estimate of vacuum time, and then the option to vacuum on a per file basis. If you recently deleted the PTR, please check it out, as you may be able to recover a whole ton of disk space.

I fixed Mr Bones! I knew I'd typo somewhere with the file service rewrite two weeks ago, and he got it. I hadn't realised how popular he was, so I've added him to my weekly test suite--it shouldn't happen again.
Open

R: 7 / I: 0 / P: 14

Version 446

https://youtube.com/watch?v=asojeparbK0 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v446/Hydrus.Network.446.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v446/Hydrus.Network.446.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v446/Hydrus.Network.446.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v446/Hydrus.Network.446.-.Linux.-.Executable.tar.gz

I had an ok week. The client does not have a huge changelog this week, but the server has a neat privacy improvement.

The PTR is going to be doing a heap of maintenance this week. It will be 'busy' a lot, approximately one hour of busy and then three hours free. Please bear with it, and if you have a million mappings to upload, I recommend you just give it a break and come back later. I am not totally sure how long it will take. Best case it is a day, worst case it might take four or five.

null account

tl;dr: The hydrus server is now even more anon. You don't have to do anything.

As the PTR has moved to multiple accounts, we've had several good discussions about privacy. Separate accounts, despite being anon, could potentially leave a fingerprint of preferences on the server. If the server were to fall into bad hands many years from now, someone could mine those records--maybe mixed with one time you casually said 'yeah, I added that sibling'--and perhaps derive something from it.

There is no technical need to remember which account uploaded what long term, so now all hydrus servers completely anonymise their content after a certain duration, default 90 days. A new non-useable 'null' account takes possession of files, tag mappings, siblings, or parents after the delay, letting the original uploaders be forgotten. Janitors will still have time to work on recent account-based problems, but the historical record works just like the old shared public account: all merged together.

I have updated the privacy document in the help to talk a little about it. As long as you do not tag your own face in pictures or something, I think we are pretty great now, especially if you use a VPN.

https://hydrusnetwork.github.io/hydrus/help/privacy.html#account_history

It will take some time to anonymise the PTR or any other big server, as it has to go through its whole historical record to catch up. Please bear with it.
Open

R: 3 / I: 0 / P: 14

Version 445

https://youtube.com/watch?v=ulvvQHD4184 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v445/Hydrus.Network.445.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v445/Hydrus.Network.445.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v445/Hydrus.Network.445.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v445/Hydrus.Network.445.-.Linux.-.Executable.tar.gz

I had a great week mostly working on optimisations and cleanup. A big busy client running a lot of importers should be a little snappier today.

optimisations

Several users have had bad UI hangs recently, sometimes for several seconds. It is correlated with running many downloaders at once, so with their help I gathered some profiles of what was going on and trimmed and rearranged some of the ways downloaders and file imports work this week. There is now less stress on the database when lots of things are going on at once, and all the code here is a little more sensible for future improvements. I do not think I have fixed the hangs, but they may be less bad overall, or the hang may have been pushed to a specific trigger like file loads or similar.

So there is still more to do. The main problem I believe is that I designed the latest version of the downloader engine before we even had multiple downloaders per page. An assumed max of about twenty download queues is baked into the system, whereas many users may have a hundred or more sitting around, sometimes finished/paused, but in the current system each still taking up a little overhead CPU on certain update calls. A complete overhaul of this system is long overdue but will be a large job, so I'm going to focus on chipping away at the worst offenders in the meantime.

As a result, I have improved some of the profiling code. The 'callto' profile mode now records the UI-side of background jobs (when they publish their results, usually), and the 'review threads' debug dialog now shows detailed information on the newer job scheduler system, which I believe is being overwhelmed by micro downloader jobs in heavy clients. I hope these will help as I continue working with the users who have had trouble, so please let me know how you get on this week and we'll give it another round.

the rest

I fixed some crazy add/delete logic bugs in the filename tagging dialog and its 'tags just for selected files' list. Tag removes will stick better and work more precisely on the current selection.

If you upload tags to the PTR and notice some lag after it finishes, this should be fixed now. A safety routine that verifies everything is uploaded and counted correct was not working efficiently.

I fixed viewing extremely small images (like 1x1) in the media viewer. The new tiled renderer had a problem with zooms greater than 76800%, ha ha ha.

A bunch of sites with weird encodings (mostly old or japanese) should now work in the downloader system.

Added a link, https://github.com/GoAwayNow/Iwara-Hydrus, to Iwara-Hydrus, a userscript to simplify sending Iwara videos to Hydrus Network, to the Client API help.

If you are a Windows user, you should be able to run the client if it is installed on a network location again. This broke around v439, when we moved to the new github build. It was a build issue with some new libraries.
Open

R: 4 / I: 0 / P: 14

Version 444

https://youtube.com/watch?v=0OmOQs0ziSU [Embed]
This whole vid is great. Watching a ton of this stuff was a decent whack of my vacation.
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v444/Hydrus.Network.444.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v444/Hydrus.Network.444.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v444/Hydrus.Network.444.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v444/Hydrus.Network.444.-.Linux.-.Executable.tar.gz

I had an ok week getting back to speed. There's some new help regarding the new PTR accounts and I fixed some bugs.

accounts and help

If you use the PTR, you may have seen a note appear on the public account that it is now 'read only' and can now only download updates. If you would like to upload tags, siblings, or parents, please generate your own account either by clicking the popup button you get when trying to upload, or just hit up the PTR on services->manage services and click the easy button there. This pretty much completes the long project me and the guys running the PTR have been working on: moving all uploading users to their own still-anon-but-seperate accounts so the janitors can work through groups of petitions and undo future mistakes without it all being merged together in one account. I have given both of these pages a complete pass to represent the changes and updated screenshots etc... to be more modern:

https://hydrusnetwork.github.io/hydrus/help/access_keys.html
https://hydrusnetwork.github.io/hydrus/help/privacy.html

The privacy page now compiles the various conversations I have had with users over time and most recently about this account change, and my current understanding of exactly how private a hydrus repo is (really good), and what you can do to make it almost perfect, even against esoteric problems or an untrustworthy administrator (basically using a VPN is always a good idea when you are doing anything fun online, and don't tag your private photos on the PTR with your real name lol). I have tried to be comprehensive and even tried to imagine some advanced future problems, so it is a long read, but I'll try to keep updating all my thoughts to there so it can be a good reference point in future. If you are interested in this stuff, please read it and let me know what I have missed or if you have any other ideas!

Also, if you are a Linux user and get OOM (out of memory) crashes, please check the new thorough user-written guide linked in this new 'running' section here:

https://hydrusnetwork.github.io/hydrus/help/getting_started_installing.html#running
https://hydrusnetwork.github.io/hydrus/help/Fixing_Hydrus_Random_Crashes_Under_Linux.md

all misc otherwise this week

I rewrote a heap of thread-UI object interactions that should improve stability on some Linux flavours and also get rid of the 'QtDeadWindow' errors some users have seen (usually in manage tags).

The new popup message button that fills in subscription gaps would sometimes not assign the correct downloader (usually when the button was opening a new page). This should be fixed!

A variety of error reporting is more reliable and presents better text.

An odd error when loading the downloader .png files is fixed, so if you were unable to import new downloaders in the past few weeks, please try again and let me know how you get on!
Open

R: 3 / I: 0 / P: 14

Version 443

https://youtube.com/watch?v=NgYIIPszZjA [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v443/Hydrus.Network.443.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v443/Hydrus.Network.443.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v443-macos/Hydrus.Network.443-macos.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v443/Hydrus.Network.443.-.Linux.-.Executable.tar.gz

I had a great week doing nice cleanup and quality of life work.

Hey, we had a problem getting the macOS release to build this week. The macOS link above goes to a build using a simpler and faster method. It should work fine, but please let me know if you have any trouble. As always, back up before you update!

highlights

Popup messages can now launch complex jobs from a button. The first I've added is when a subscription hits its 'periodic' file limit. The situation itself is now better explained, and a button on the popup will create a new downloader page with the specific query set up with an appropriate file limit to fill in the gap. The second is if you try to upload some content to a repository that your account does not have permission for (this is affecting sibling- and parent-uploading PTR users as the shared public account is changing), the popup message that talks about the issue now has a button that takes you straight to the manage services panel for the service and starts automatic account creation.

Subs should now be more careful about determining when they have 'caught up' to a previous sync. Small initial file limits are respected more, and the 'caught up' check is now more precise with sites that can give multiple files per URL or very large gallery pages.

I gave options->speed and memory a full pass. The layout is less crushed and has more explanation, the options all apply without needing a client restart, and the new, previously hardcoded cache/prefetch thresholds are now exposed and explained. There's a neat thing that gives an example resolution of what will be cached or prefetched, like 'about a 7,245x4,075 image', that changes as you fiddle with the controls.

The client has recently had worse UI lag. After working with some users, the biggest problems seemed to come in a session with lots of downloaders. I traced the cause of the lag and believe I have eliminated it. If you have had lag recently, a second or two every now and then, please let me know how things are now.

If you use the Client API a lot while the client is minimised, you can now have it explicitly prohibit 'idle mode' while it is working under options->maintenance and processing.
Open

R: 5 / I: 0 / P: 14

Version 442

https://youtube.com/watch?v=bpEFn3MFyfA [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v442/Hydrus.Network.442.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v442/Hydrus.Network.442.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v442/Hydrus.Network.442.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v442/Hydrus.Network.442.-.Linux.-.Executable.tar.gz

I had a great week. An important part of GUI Sessions is overhauled, which should save a lot of hard drive time for larger clients.

gui sessions

I always encourage a backup before you update, but this week it matters more than normal. If you have a client with large sessions with many important things set up, make sure you have a backup done before you update! I feel good about the code, and I try to save data on various failures, but if your situation gives errors for an unforeseen reason, having the backup ready reduces headaches all around!

Like the subscriptions and network objects breakups I've done in the past year, I 'broke up' the monolithic GUI Session object this week. Now, when your session has changes, only those pages that have changed will be saved, saving a ton of CPU and HDD write I/O. Furthermore, sessions that share duplicate pages (this happens all the time with session backups), can now share that stored page, saving a bunch of hard drive space too. Like with subscriptions, some users are pushing multiple gigabytes of session storage total, so there is a good amount of work to save here.

You don't have to do anything here. Everything works the same on the front end, and all your existing sessions will be converted on update. Your client should be a little less laggy at times, and client shutdown should be a bit faster.

If any of your old sessions fail to load or convert, a backup will be made so we can check it out later. Let me know if you have any trouble!

Advanced stuff:

Another benefit is the old limit of 'sessions fail to save at about 500k session weight' now applies to pages individually. Please don't immediately try to nuke your sessions with five million new things, but if you do end up with a big session, let me know how other performance works out for you. Now this bottleneck is gone, we'll start hitting new ones. I believe the next biggest vulnerability is thread starvation with many simultaneous downloaders, so again please don't paste-spam a hundred now queries (for now).

If you have been tracking session weight (under the pages menu), I am rebalancing the weights. Before, the weight was file = 1, URL = 1, but after all our research into this, I am setting it to file = 1, URL = 20. In general, I think a page will fail to save at the new weight of about 10 million. If you are in advanced mode, you can now see each page's weight on page tab right-clicks. Let's get a new feeling for IRL distribution here, and we can aim for the next optimisation (I suspect it'll eventually be a downloader-page breakup, storing every query or watcher as a separate object). Since URLs seem to be the real killer, too, see if you can spread bigger downloads across multiple download pages and try to clear out larger completed queries when you can.
Open

R: 2 / I: 0 / P: 14

Version 441

https://youtube.com/watch?v=EJLNLWv-nmM [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v441/Hydrus.Network.441.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v441/Hydrus.Network.441.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v441/Hydrus.Network.441.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v441/Hydrus.Network.441.-.Linux.-.Executable.tar.gz

I had an ok week. Not as much as I wanted, but there are some nice Client API improvements.

all misc this week

The test builds from last week seem to work ok, so they are now master. The built clients now use Python 3.8, and the security libraries (like OpenSSL) are all much newer--and will reliably stay up to date in future--so a whole bunch of things across the client should have slightly better performance. There are no special install instructions, they seem to work on an existing install just as normal. Let me know if you do run into any problems!

I fixed some more bad tiles calculations for the new tiled image renderer. Some files that seem to have little black lines on an edge at some zooms, or previews that just turn up black, should be fixed! Error reporting is also nicer.

The Client API can now do a couple more things. Particularly, it can now set your client's global User-Agent, which should help fix some difficult CDN and login problems in future. Please watch this space.

For advanced users, if you have help->advanced mode on, then setting a namespace file sort now allows you to choose which 'tag context' the sort works on. If you hide certain tags in single or multiple media view (as set in tags->manage tag display and search), then those hidden tags will not count for the sort. This is obviously advanced, so if you hadn't thought of it, you can just set 'display tags' to keep 'normal' behaviour.

full list

- misc:
- after successful testing, all the master builds are now made on github rather than my home dev situation. the clients now work off python 3.8, and several security libraries (e.g. OpenSSL) are now always going to be latest, so there should be several quiet performance and reliability improvements across the program. there are no special install instructions--normal update seems to go fine--but let me know if you do have any trouble. big thanks to the user who did the leg work on developing the workflow build scripts here
- if you are in advanced mode, namespace file sorting now allows you to set the 'tag display context' on which it will sort. this appears as a new menu button or a button list selection dialog wherever you edit namespace file sorts. if you are not in advanced mode, the default is the 'display tags' I switched to last week (i.e. before any tags are hidden by your tag display options)
- namespace sort has some related code cleanup. the 'defaults' object is updated and moved to the newer options object
- the new tiled renderer now checks for rounding errors in zoom calc, which in some cases was giving a single extra (non-existing) native pixel row or column on rightmost or bottommost tile samples
- the new tiled renderer now double-checks clip regions for validity before attempting to crop
- improved the reported error information when a tile fails to render
- when pasting an uneven number of tags into manage siblings/parents, the error is now a nicer popup dialog. I'm pursuing a related error here--if you get this a bunch, please let me know what more info you discover
- when repositories fail to fetch the update hashes to process, they now force a metadata resync. any processing error should force a metadata resync now
- added a default url class for the new pixiv _artist_ page format
- fixed a recent typo bug with ipfs pinning
Open

R: 3 / I: 0 / P: 14

Version 440

https://youtube.com/watch?v=mO1j6xx2HhQ [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v440/Hydrus.Network.440.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v440/Hydrus.Network.440.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v440/Hydrus.Network.440.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v440/Hydrus.Network.440.-.Linux.-.Executable.tar.gz

I had an unfortunately short week, but I did some good work. The tiled renderer has nice fixes.

tiled renderer

I regret the tiled renderer, while good most of the time, had crashes when it did go wrong. To stop this with any new errors that might pop up, the whole block now has an additional layer of error catching around it. If a tile fails to render for any reason, you now get a black square, and if some coordinate space cannot be calculated or a portion of the image is truncated, the system will now print some errors but otherwise ignore it.

A particular problem several users encountered was legacy images that have EXIF rotation metadata but were imported years ago when the client did not understand this. Therefore, hydrus thought some old image was (600x900) when it then loaded (900x600). In the old system, you would have had a weird stretch, maybe a borked rotation, but in my new tiled system it would try to draw tiles that didn't exist, causing our errors-then-crashes. The client now recognises this situation, gives you a popup, and automatically schedules metadata regeneration maintenance for the file.

some misc

You can now set a custom 'namespace' file sort (the 'series-creator-volume-chapter-page' stuff) right on a page. Just click the new 'custom' menu entry and you can type whatever you like. It should save through your session and be easy to edit again. This is prep for some better edit UI here and increased sort/collect control, so if you do a lot of namespace sorting, let me know how you get on!

I prototyped a new 'delete lock' mode, which prohibits deletion of files if they match a criteria. I am starting with if a file is archived. You can turn this mode on under options->files and trash. It mostly just ignores deletes at the moment, but in future I will improve feedback, and maybe have a padlock icon or something, and eventually attach my planned 'metadata conditional' object to it so you'll be able to delete-lock all pngs, or all files with more than four tags, or whatever you can think of.
Open

R: 3 / I: 0 / P: 14

Version 439

https://youtube.com/watch?v=CyPKxkH3vB8 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v439/Hydrus.Network.439.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v439/Hydrus.Network.439.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v439/Hydrus.Network.439.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v439/Hydrus.Network.439.-.Linux.-.Executable.tar.gz

I had an ok week. The new tiled renderer is improved.

tiled renderer

The new image drawing system generally worked well! There were a couple of bugs, and it still has some limitations, but in general it really improved zoom and precache performance.

For the bugs, first of all, there was a rare crash, I think triggered by loading a very unlucky coincidence of tile and image size. Then clipboard bitmap copy threw an error, tiny images could not deal with extremely small zoom, and clients under heavy load could sometimes have trouble initialising the viewer. I have fixed them all, but let me know if you have any more trouble!

There was also a problem with PyQt5, an alternative version of the Qt UI library that some 'running from source' users use. It was an object handling difference between PyQt5 and PySide2 that broke the tile caching system. I think I have fixed it, so if you are running PyQt5, please give this version a go.

Beyond bugs, there were tiling artifacts visible at higher zooms. Essentially, where the tiles lined up, there were small disagreements in resize math, resulting in little lines of mismatching colour gradients along tile borders. I worked on the tiling algorithm and have significantly mitigated the problem--I mostly only see artifacts at extreme zooms now, about 2000%.

Since people are suddenly zooming more, users who have mouse-centered zooming were having more images accidentally flying off screen too. I've hacked in off-screen rescue after a zoom, sliding it back to the nearest border, so the image should always stay in view. If people like it, I may patch this in for all media for dragging events too. There's not much need for non-visible media, and when it does happen it can sometimes be a pain dragging around to find where it went.

I hope this basically makes the tile render a complete '1.0' now. In the future, I would like to rejigger some of the virtual geometry, since at the moment a limit in Qt means I cannot zoom higher than a 'virtual' 32,768x32,768 canvas (e.g. 4k at about 800% zoom). I'll also replicate the tiling for my native Animation widget, which displays gifs and video when mpv is not available.
Open

R: 4 / I: 0 / P: 14

Version 438

https://youtube.com/watch?v=LE8QKcriHH4 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v438/Hydrus.Network.438.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v438/Hydrus.Network.438.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v438/Hydrus.Network.438.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v438/Hydrus.Network.438.-.Linux.-.Executable.tar.gz

Hey, this causes errors if you are running from source and using PyQt5 (PySide2 is fine)! All the releases above are PySide2, so they are ok! I will fix this for next week, so if you are source+QtPy5, please hold off for now.

I had a great week overhauling the media viewer's image rendering. Zooming and navigation should be a lot smoother now!

image tiles

tl;dr: the media viewer now zooms and navigates with less lag and flicker

Zooming in a long way, particularly for large images, has been pretty hellish for as long as the program has existed. Historically, the client drew the whole image in memory at the zoom you desired so it could quickly show you the bit it needed on screen. Of course this meant zooming in to 400% on anything above 4k was suddenly taking a very long time to set up and eating a lot of memory to do it. As images have naturally grown over time, the problem has occurred more often and was starting to affect other systems.

My plan to fix this has been to break the image into tiles that then render on demand. The parts of the image off-screen are never drawn, saving CPU and memory and allowing arbitrary zoom. This is a significantly more complicated idea, and rewriting the whole rendering pipeline was always expected to be a multi-week 'big job'. I originally planned to just optimise and tweak the secondary systems and add in some sanity brakes this week, but I ran a couple of small tiling tests and realised if I went bonkers it would be possible to hack in a prototype. So I did!

In the media viewer, images now draw in tiles. It works a little like a browseable satellite map, where when you zoom in and pan about you see squares of data fading in (except in hydrus they appear instantly). You should now be able to zoom in as far as you like on an image pretty quick and you won't have any sudden memory needs.

Furthermore, I have written a cache for these image tiles. This saves CPU when revisiting different images or zooms, so when you flick back and forth between two normal things, it should now be instant! It still takes 20-200ms to view or zoom most images the first time, but going back to that view or zoom within a minute or so should be really smooth. The cache starts at a healthy 256MB this week. I think that will cover most users very well (in screen real estate, it works out to about 35 x 1080p worth of tiles), but you can alter it under the settings at options->speed and memory.

And I did some misc work improving the rendering pre-fetch logic when you browse in the media viewer. Huge files won't stomp all over the image renderer cache any more, which should make browsing through a series of giant images far less jank. If you are feeling advanced, you can now edit the prefetch timing and distance settings too, also under options->speed and memory.
Open

R: 3 / I: 0 / P: 15

Version 437

https://youtube.com/watch?v=FfBdUvVQpNQ [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v437/Hydrus.Network.437.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v437/Hydrus.Network.437.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v437/Hydrus.Network.437.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v437/Hydrus.Network.437.-.Linux.-.Executable.tar.gz

I had a good week mostly fixing bugs and optimising. It will take a couple of seconds to update this week.

all misc this week

I reduced a heap of UI lag on clients that have pages with a lot of collected (like 'collect by creator') media. If you often have five or ten thousand files collected and you noticed your client was getting choppy just when running some downloaders, I hope this improves things. Let me know how you get on!

I started real work on multiple local file services. Most of it is boring behind the scenes stuff, but as part of it, I overhauled the trash system this week. A heap of logic is improved, it is ready for more than one 'my files' service, and now hydrus remembers when files are deleted. Delete timestamps have never been recorded clientside, and unfortunately we cannot recover old information retroactively, but it is stored for all deletes from now on. Whenever the client wants to say 'this file was deleted', it should now have 'at an unknown time' or a nicer '3 days ago' suffix.

A second neat thing with the improved deleted files storage is I hope in the nearish future to let you search deleted files. This is a rare, clever query, like 'all known files', but there will be some kind of button you can press to flip your 'my files' or 'all local files' search to go through what has ever been removed from them. 'system:file service' will get similar improvements.

To reduce confusion, I renamed some hydrus network concepts across the program. The 'access key' (secret password for account), 'account key' (account identifier), and 'registration key' (one time token to create an account) are now known as 'access key' (i.e. no change), 'account id', and 'registration token'. There is more work to do here, particularly improving server setup and account management workflows to suit the user (rather than my technical ease), so I will keep at it.

In a related topic, the PTR is updating its accounts. The public account is moving more towards a 'read-only' account, and accounts that can upload siblings and parents (and perhaps tag petitions, eventually) will be individual to you and freely auto-creatable in manage services. This is mostly an attempt to make janitorial decisions easier and more accurate, since at the moment everything on that side is merged due to the shared account. Permissions have not been used much in hydrus network yet, and the workflows and user notifications here are bad and buggy. Please bear with me as I iron out the problems and make it all nicer to use.
Open

R: 3 / I: 0 / P: 15

Version 436

https://youtube.com/watch?v=i_u3hpYMySk [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v436/Hydrus.Network.436.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v436/Hydrus.Network.436.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v436/Hydrus.Network.436.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v436/Hydrus.Network.436.-.Linux.-.Executable.tar.gz

I had a great few days mostly cleaning and fixing things. If you sync with the PTR, update will take a minute this week.

macos release polish

I cleaned up the new macOS release. It seems to have launched and otherwise generally worked last week, but there was a bug in finding the specific database location macOS users are used to. Without the '--d' launch parameter, it was creating an empty new db inside the app, in the 'db' dir hydrus would normally use (and the really old App used to use, if you remember that), and hence would say 'hey, this looks like the first time you are running the program...' on boot. I have fixed the 'I am running in an app' detection and the ~/Library/Hydrus database path calculation routine, so everything should be back to normal.

It also has the old readme and Applications shortcut in the dmg, and the filename should be fixed too. I expect this to be the only macOS release I put out from now on. Let me know if you have any more trouble!

miscount fix

Last week, I made the number in the 'pending (1,234)' menu title add up in a more efficient way. Rather than counting raw mapping rows every time, it uses a table of pre-computed numbers, the same used for autocomplete results. It turns out there were some legacy (from a long time ago) miscount bugs in there for some users. This resulted in a 'sticky' number that would not go away even after committing. A maintenance routine exists to fix this, but it is a sledgehammer when we need a scalpel.

So, I have written a maintenance routine to regen this pending data efficiently and correct these old bugs. It is basically the same as I did a few months ago with the 'display' caches during the siblings and parents work, but for a deeper level of tags. It will be run on update, along with a new thing that forces the menu's count to regen, both of which can now be accessed from database->regenerate menu in case we need them again in future. If you sync with the PTR, it may take a minute or so to finish.

I hope this will fix the issue completely, but if you still have a bad count, or if your count drifts off zero again over time, please let me know!

underscores

After discussion with some users, I have added an experimental setting to options->tag presentation that replaces all underscore characters in tags with space characters, as long as you are in 'front-facing' UI like regular search pages or the media viewer. It works on the same system as the 'hide namespace' option--and siblings--in that you still see the raw truth in manage tags and other edit locations.

This setting is experimental since it will add a bit of CPU lag to tag presentation and may result in some seemingly duplicate rows. I have long planned to fix the underscore issue with a really nice system, but I was convinced that adding a hacky system in the meantime would be a good thing to play with. If you care about this issue, give it a go and let me know if you run into any problems.
Open

R: 3 / I: 0 / P: 15

Version 435

https://youtube.com/watch?v=-ZdLbiF9gCo [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v435/Hydrus.Network.435.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v435/Hydrus.Network.435.-.Windows.-.Installer.exe
macOS
NEW app: https://github.com/hydrusnetwork/hydrus/releases/download/v435/HydrusNetwork-v435.dmg
OLD app: https://github.com/hydrusnetwork/hydrus/releases/download/v435/Hydrus.Network.435.-.macOS.old.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v435/Hydrus.Network.435.-.Linux.-.Executable.tar.gz

I had a good week. I continued last week's server work and have a macOS Big Sur build available for testing.

new macOS build

This was delayed far longer than I wanted, but I think I have it figured out now. Thanks to work from Suika and ReAnzu on github figuring out the script, the macOS build is now put together on github itself, on Catalina. This new build thus works on Big Sur, and should generally have less compatibility jank on any newer macOS machine. I have included my old 10.12-built release as well today, just in case I messed something up, but if you are a macOS person, please give the new one a try and let me know how you get on.

Assuming this works out ok for the majority of people, I'll be moving to it as the official macOS build. Anyone who is still on an old macOS will have to run from source to stay updated with hydrus. The new release is just out of testing, so I'll clean up the filename and add the readme.rtf and so on so it looks prettier. I will also explore reducing the size (300MB vs 137MB wew), but I am told this may be unavoidable due to some new macOS thing that bundles both old and new versions of libraries in the App.

editable repository update periods

Just in case, it is probably a good idea to back up your server before you try this!

Hydrus repositories now have an editable update period! This was an oft-requested feature for users running a server on a LAN or between just a few friends, so if this is you, please give it a go. There is a new 'edit options' permission for accounts, which any account with 'manage account types' (admins basically) will inherit on update. If you have this, the 'admin services' menu will have a new 'edit update period' entry, which launches a simple time delta widget where you can change it.

Changes occur immediately. The server will create any updates that are now due, and your client will do a resync to keep up to date with the new numbers, so if you have 'review services' open when you make the change, you should see everything update in realtime. Other clients that sync with the server will catch up with the new check times when they next do their normal sync, unless they too force a resync with the new 'review services' button I added last week.

You can change the update several times, up or down, and it shouldn't be a big deal--the client no longer cares if one set of updates were short or long--so please feel free to play around with this a bit, just don't go crazy with super short or long times. The minimum is 10 minutes, but I recommend you only go down to about 60 minutes if you really want to be that fast, at least as we experiment with this. The system was originally built with 100,000 seconds in mind, so I think super short periods will add overhead bloat over time (not to mention, you only see content after processing, which atm only happens in idle time anyway).
Open

R: 2 / I: 0 / P: 15

Version 434

https://youtube.com/watch?v=f1nvZ74OY_I [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v434/Hydrus.Network.434.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v434/Hydrus.Network.434.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v434/Hydrus.Network.434.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v434/Hydrus.Network.434.-.Linux.-.Executable.tar.gz

I had a great three weeks working on long overdue network updates. Most of the changes this week restore old account management UI for server admins, but there are a couple of fixes and improvements for regular users as well.

The network version goes up to 20 today. Clients and servers can only talk to each other if they are on the same version, so if you want to talk to the PTR, you will want to update at some point. No rush, but if you try to talk to the server on an older client, you will get a polite error message.

regular user stuff

On review services, you can now pause repository account sync, update downloading, and update processing separately. If you don't want to process right now but still want the client to be grabbing update files for later, or you want to pause all service network traffice for a while but still process your backlog, it should now be easy to set up.

I fixed an annoying bug where going from 'random' file sort to one with asc/desc sort order by loading a favourite search would always default the asc/desc to the top value. Guess what hydev's default sort is.

Server administrators can now set messages to accounts. If you get a message, your client should get the update within a day or two, at which point it will appear in a popup box and be viewable on review services.

Server administrators can now set some account types to be user-creatable--no need to mess around with registration keys or access keys yourself, the client can fetch them for you from the server. A new button on the manage services edit panel checks this and can do it for you. I understand the PTR is likely to move to this in future for accounts with sibling and parent petition permissions, since grouping these by account will be useful for janny petition workflow. Having separate accounts for these complicated uploads will also allow better account management (including the above rudimentary messaging) for users who make common errors with sibling or parent submissions.

I believe I fixed a bug in repository processing that caused some newer users to get stuck with siblings and other data that was later deleted. This should not happen again, and I am now planning ways to efficiently fix the problem retroactively. (If you were hit by this and want the non-efficient fix, hit 'reset processing->reprocess content' for the repo under review services).

Some users recently got some scary errors about an invalid repository update file. This was the result of the client becoming confused about which update files it should have for a repository. I am still investigating why this happened, but to fix the immediate issue I have added a maintenance routine (under the new review services->reset downloading) that triggers a full resync of update metadata and delete surplus records. This function will fire automatically in future if the service encounters borked update files of any kind. If you got these errors, please try firing it and let me know how you get on.
Open

R: 5 / I: 0 / P: 15

Version 433

https://youtube.com/watch?v=twOFs6zJq8Q [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v433/Hydrus.Network.433.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v433/Hydrus.Network.433.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v433/Hydrus.Network.433.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v433/Hydrus.Network.433.-.Linux.-.Executable.tar.gz

I had a good week doing simple cleanup and fixes. This is intended to be a 'clean' release. There will be no release for about three weeks.

all misc this week

Thanks to the work of a user (who was also very patient as I slowly added the features he needed), we now have a new native twitter downloader! It gets video now! (Including gif, which on twitter is just a small mp4!) It is called 'twitter syndication' and uses a new access method. Its main limitation seems to be that its search can only reach about 500 tweets into the past. Please check it out, and if it works well for you, please move your subscriptions over from nitter to save them some bandwidth. This is the first version, so it may well have future updates.

I have written a new 'scrollable' menu button, which works like the file sort selector--instead of a dropdown, you get a menu, and you can scroll through it in either direction with your mouse. I have converted several old small choice dropdowns to this new button: file sort ascending/descending, collect-by 'collect/leave unmatched', and all the new tag sort dropdowns I added last week. Try putting your mouse over and scrolling them! This is a little thing, but I hope it smooths out the workflow here. I'll likely use it on more things.

Some more mouse changes: Middle-clicking on the collect-by dropdown now clears it. Middle-clicking or right-clicking the 'page tab bar' to open the 'new page' chooser or the page right-click menu should now only work on the tab bar's empty greyspace (previously, this was confusingly happening on a click on greyspace in any child panel, like some random downloader page panel).

Namespace file sort (the 'series-creator-title...' stuff) now support a-z/z-a ordering. Files that have no match are still counted as 'less than a' in a-z terms, but I can change this stuff easier these days. I don't use this myself much, so if you do, let me know what you'd like in future.

Network job controls (the little panel that shows your downloads) now let you edit their active job's bandwidth rules from the cog menu! It will show all the network contexts and which are currently waiting on bandwidth. If the job is using the 'default' rules, it will say so and allow you to set specific ones just for that domain.
Open

R: 3 / I: 0 / P: 15

Version 432

https://youtube.com/watch?v=_ZDUHbIVaz8 [Embed]
^ After years of seeing all sorts of smaller edits using this footage, I finally discovered this source vid this week. If you like fine art, please watch it in full, it is eight minutes of unbroken kino. It gets a bit heavy at the end, not appropriate for kids. If you don't like old art, no worries, I was just chuffed this week to have finally found this thing.

windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v432/Hydrus.Network.432.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v432/Hydrus.Network.432.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v432/Hydrus.Network.432.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v432/Hydrus.Network.432.-.Linux.-.Executable.tar.gz

I had a great week. There are a bunch of different little fixes and improvements this week, and the tag sort dropdown is now easier to use.

tag sort

Rather than a big list, the tag sort dropdown is now a dynamic control that splits the different variables (sort type, asc/desc, group by) into different mini-dropdowns that show and hide when available. Confusing labels like 'lexicographic' and 'ignore namespace' are replaced with clean 'sort by tag/subtag/count'. I hope it is a bit easier to work with now. Give it a go and let me know what you think.

If you don't like your default tag sort, you can set it under options->tags. Your existing default should be converted to the new format on update.

I also added the option to sort by sibling text or the actual tag text when you have the manage tags dialog open.

There is more work to do. One thing I don't like is if, say, on the asc/desc dropdown, you have 'a-z' selected, you can hover your mouse over and scroll down to get 'z-a', but you can't scroll up to get it. I prefer to have scroll work like 'flip' when there are only two values, where either scroll direction will make a change, so I'll figure that out, and in the file sort/collect controls, so there is even less to think about when you edit. I'll also see about saving the tag sort in the session.

I also updated the back-end of tag sorting here, so it'll be easier to update in future. Some users have asked for custom namespace sorts (e.g. creator, character, series, then the rest (a-z), then unnamespaced), so I'd like to start thinking about that properly.

misc

As some users have not realised they were getting a big session that was lagging them, the client now makes a once-per-boot gentle warning popup if your session exceeds 500k weight. If you cannot conveniently reduce your session size, you can turn this warning off under options->gui pages.

Subscriptions now work more efficiently with a very small 'first run' file limit and a larger 'normal' file limit. Previously, the second run was often charging right through the handful of files caught on first sync and getting more than it should, but now it should better recognise it has 'caught up' and stop at the right position.
Open

R: 3 / I: 0 / P: 15

Version 431

https://youtube.com/watch?v=CU5NKzKiAGU [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v431/Hydrus.Network.431.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v431/Hydrus.Network.431.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v431/Hydrus.Network.431.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v431/Hydrus.Network.431.-.Linux.-.Executable.tar.gz

I had a challenging week, but I got some decent fixes in.

all misc this week

Pretty much everything was little fixes this week. Nothing hugely significant, but check the changelog if you have been waiting on something.

The e621 parser now pulls 'lore' tags, which seems to be some new 'canonical' tag. I have put them in 'lore' namespace, so you can exclude them easily if you want. One user suggested they are useful for certain blacklists.

For advanced users, I added a 'sorter' to string processing. It has a neat thing where you can sort by a regex'd substring of the main string. I was happy about this, but then I realised this morning that it is only half of the job I wanted to do. I will add a String Slicer next week so you can just sample the first item or the last three items or whatever you like, once you have your sort figured out.

full list

- misc:
- when parents are hidden in the edit/write taglists (e.g. in manage tags), there is now a '(n parents)' suffix
- thread watchers that are DEAD or 404 but still have files downloading now report 'working' status until that is done
- search terms with ';' like 'steins;gate' should now work in downloaders. sorry for the trouble!
- fixed an issue where un-ideal tags were sometimes becoming non-searchable when they were entirely replaced in manage tags with their 'ideal' siblings (i.e. their autocomplete count went to 0). this was due to overzealous deletion in the new tag definitions cache not filtering out sibling/parent chain members. a small routine will run on update to resynchronise affected tags
- fixed an issue when loading up files in the main 'import files' dialog where a critical error (as opposed to a nice 'couldn't figure it out, sorry') in mime detection would cause the whole job to hang
- that main 'import files' dialog now counts 'missing' files separately in the error count
- fixed tags not updating on the filename tagging dialog when double-clicking to remove from the simple taglists
- fixed the sort on the manage tags dialog's suggestion taglists--they now preserve their original sort, rather than alphabetising once sibling/parent data is populated

R: 0 / I: 0 / P: 15

e621

tried to login after changing password and I get this message about remember cookie missing in post login step
it says in logged in when i look at the manage logins
Open

R: 3 / I: 0 / P: 15

Version 430

https://youtube.com/watch?v=Giy5iQ9gGvc [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v430/Hydrus.Network.430.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v430/Hydrus.Network.430.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v430/Hydrus.Network.430.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v430/Hydrus.Network.430.-.Linux.-.Executable.tar.gz

I had an ok week. I mostly fixed and improved things. If you sync with the PTR, it will take a minute or two to update.

all misc this week

When I did the autocomplete lookup speedup recently, we unfortunately lost the ability to match unusual characters with short tag inputs. Entering 'a' would not return a match of '/a/', and in fact there was no way to look that tag up without some slow cleverness like wildcards. I have fixed this this week. If you sync with the PTR, it will take a minute or two to update while a small new cache is constructed.

You can now hide/show the 'hanging' parents in taglists (in manage tags dialogs etc...) from the list right-click menu. You can also set in options->tags whether they should start shown/hidden for normal taglists and autocomplete dropdown results.

Taglists that show '(displays as xxx)' sibling info now sort by that sibling for lexicographic purposes. I had thought to make this optional, but when I tried it out, it felt great to me, so I left it as default. Let me know what you think.

I fixed an issue that meant some texts across the program were staying blank for a few seconds when they should have been updating. This mostly affected the 'highlighted downloader' panel in gallery download pages. This was personally driving me nuts IRL.

I fixed the tumblr downloader. Sorry for the problems here, I misunderstood initial reports. This was due to a logical change in hydrus that just happened to hit tumblr's URL format. If you have tumblr subscriptions, please go into them and see if any recent URLs have the status 'Found 0 new URLs.'. Right click them and say to 'try again' and they should fix up.
Open

R: 6 / I: 1 / P: 15

Version 429

https://youtube.com/watch?v=ezC4ybLJsPM [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v429/Hydrus.Network.429.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v429/Hydrus.Network.429.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v429/Hydrus.Network.429.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v429/Hydrus.Network.429.-.Linux.-.Executable.tar.gz

I had a good simple week just working on smaller jobs and cleanup. If you have a large client, it may take a minute for you to update this week.

all misc this week

The new taglists seem to have gone generally well. There was one display/update error that hit some users, which I have fixed, along with some little menu logic cleanup. I will work on more display and sort options for them in future.

Since I added 'pending' status for thread watchers last week, you may have seen some apparently stuck on that status. This was a harmless legacy issue, just old check jobs that were never closed out correctly, which I have fixed. I also cut out the 'checking in' prefix from the waiting-to-check status text, let's see if that fits better.

Dialogs with multi-column lists should be less of a pain this week. I think I fixed the issue where the last column could sometimes shrink to a few pixels on the next dialog launch, and I think I also fixed the issue where columns would sometimes get a few pixels thinner every launch. Just set them once more to the size you want, and they should stick now!

'.clip' files, those for Clip Studio Paint, are now recognised and importable. Too complicated for me to support thumbs, but they have their own icon. I cleaned up the behaviour of the increasingly tall filetype selection widget, so I am happy to start adding simple new filetypes for a bit. If you would like a new media filetype added to hydrus, please send an example of that file to me with a bit of info about it. Nothing too obscure, please, and I can't promise I can add everything quick, but many are actually fairly simple. I'd like to see about .sai and .wav, and maybe start on an xml scanner so I can at recognise things like .svg.

I took a bit of time to check out animated webp this week. Unfortunately, gathering duration and frame info about webps seems to be much more difficult than actually making webps, so I am afraid we will have to wait for an update to FFMPEG or one of the other libraries we rely on to get animated webps going.

A new cache speeds up a bunch of file database lookups this week. If you have a large client, it may take a minute to update as this is populated.
Open

R: 11 / I: 0 / P: 16

Version 428

https://youtube.com/watch?v=RNPnBVHSeZc [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v428/Hydrus.Network.428.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v428/Hydrus.Network.428.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v428/Hydrus.Network.428.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v428/Hydrus.Network.428.-.Linux.-.Executable.tar.gz

I had a good couple weeks working on the taglist code and some other jobs.

If you are on Windows and use the 'extract' release, you may want to do a 'clean' install this week. Extra notes below.

taglists

So, I took some time to make taglists work a lot cleaner behind the scenes and support more types of data. A heap of code is cleaner, and various small logical problems related to menus are fixed. The tag right-click menu is also more compact, quicker to see and do what you want.

The main benefits though are in the manage tags dialog. Now, the '(will display as xxx)' sibling suffix colours in the correct namespace for the sibling, and parents 'hang' underneath all tags in all the lists. It is now much easier to see why a parent or sibling is appearing for a file.

This is a first attempt. I really like how these basically work, but it can get a bit busy with many tags. With the cleaner code, it will be much easier to expand in future. I expect to add 'expand/collapse parents' settings and more sorts, and maybe shade parents a bit transparent, in the coming weeks. Please let me know how it works for you IRL and I'll keep working.

the rest

The main nitter site seems to be overloaded. They have a bunch of mirrors listed here: https://github.com/zedeus/nitter/wiki/Instances

I picked two roughly at random and added new downloaders for them. If you have Nitter subs, please move their 'sources' over, and they should start working again (they might need to do a bit of 'resync' and will complain about file limits being hit since the URLs are different, but give them time). If you would rather use another mirror, feel free to duplicate your own downloaders as well. Thanks to a user who helped here with some fixed-up parsers.

I gave the recently borked grouped 'status' sort in thread watchers and downloader pages another go, and I improved the reporting there overall. The 'working' status shouldn't flicker on and off as much, there is a new 'pending' status for downloaders waiting for a work slot, and the 'file status' icon column in galleries now shows the 'stop' symbol when files are all done.

The menu entry to 'open similar-looking files' is now further up on thumbnails' 'open' submenus.

The duplicate filter has its navigation buttons on the right-hand hover window rearranged a bit. It is silly to have both 'previous' and 'next' when there are only two files, so I merged them. You can also set 'view next' as a separate shortcut for the duplicate filter, if you want to map 'flip file' to something else just for the filter.

windows clean install

If you use the Windows installer, do not worry, these issues are fixed automatically for you from now on.

I updated to a new dev machine this week. Some libraries were updated, and there is now a dll conflict, where a dll from an older version is interfering with a new one. As it happens, the library that fails to load is one I made optional this week, so it doesn't seem to actually stop you from booting the client, but it will stop you from running the Client API in https if you never did it before (the library does ssl certificate generation).

It is good to be clean, so if you extract the Windows release, you may want to follow this guide this week: https://hydrusnetwork.github.io/hydrus/help/getting_started_installing.html#clean_installs

Remember: Don't delete your db folder.
Open

R: 1 / I: 0 / P: 16

e621 downloader doesn't grab "lore" tags

they added a new "lore" tag section, for tags that are all on my blacklist. incest, tranny, """"""nonbinary"""""", etc.

since they're apparently separate from the other tags, my blacklist on hydrus isn't catching them.

this is a bug report, i guess. i can't use github bc they don't accept cock.li emails, can't use discord bc they require a phone number. hope you see this
Open

R: 4 / I: 0 / P: 16

Version 427

https://youtube.com/watch?v=E97ftpTjeC4 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v427/Hydrus.Network.427.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v427/Hydrus.Network.427.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v427/Hydrus.Network.427.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v427/Hydrus.Network.427.-.Linux.-.Executable.tar.gz

I had an ok week. I mostly fixed and cleaned things.

more ghosts

There was another 'ghost' pending tag issue. Sometimes, on committing pending tags with siblings up to the PTR, the pending record and its count was not clearing properly. I reproduced the problem and fixed it, and a new test ensures it will not happen again. Thank you to the reports about this, they helped me to figure it out.

Since we have had some trouble with pending tags in particular, I have written a new maintenance routine to recalculate pending tags and their counts efficiently without having to wait for siblings/parents re-sync. This will run on update, so with luck, your tags and autocompletes should be fixed up (until the next problem!).

the rest

There is a new shortcut set under file->shortcuts to handle tag autocomplete shortcuts. Any 'switch searching immediately' that was previously on 'main gui' will be migrated over to this on update, and I have also moved several previously hardcoded and undocumented shortcuts to the new shortcut system, so you can now edit them! The new actions (with their defaults) are: forcing tag autocomplete lookup now (ctrl+space), flipping an IME-friendly input mode on/off (insert), navigating autocomplete results tabs left/right when input is empty (left/right arrow), navigating the service pages in manage tags dialog left/right when input is empty (up/down arrow), and navigating a parent media viewer previous/next when input is empty (page up/down).

I fixed a bad auto-setup link in the gelbooru downloader last week that messed up some users. Also rolled in is a new URL definition that should fix 420chan thread watching for their newer url format.

full list

- ghost pending tags:
- fixed another ghost pending tags bug. this may have been new or there since the display cache started, I am not sure, but it shouldn't happen again. it was occuring when a pending tag was being committed to 'current' and another tag in its sibling group already existed as a current tag for that file. the pending tag and its count would not clear for non-'all known files' domains, causing ghosts to appear in search pages but not typically manage tags. you may have noticed ghost tags hanging around after a pending commit--this was it. the true problem here was in the 'rescind pending' action that occurs just before an add/commit. a new unit test tests for this situation both for two non-ideal tags being pend-merged, and a non-ideal tag being pend-merged into the existing ideal
- wrote a routine to regenerate _pending_ tag storage and autocomplete counts from scratch for the combined and specific display tag caches. this job is special in that it regens tags instantly without having to reset sibling/parent sync. you can run the job from the database->regen menu. this is the start of 'fix just this tag' maintenance ability
- the pending regen routines will occur on update. it shouldn't take long at all, unless you have five million tags pending, where it could be a couple minutes

R: 1 / I: 0 / P: 16

Is it me or is this UI text ant-sized? I've been using hydrus on ubuntu and got used to bigger windows but now im on w10 and this is some small print, so small I dont know how I first got used to it and almost think I've done something wrong. I'm just on 1920x1080. Can I do something about this?
Open

R: 4 / I: 0 / P: 16

Version 426

https://youtube.com/watch?v=SpXsBNBtqP0 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v426/Hydrus.Network.426.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v426/Hydrus.Network.426.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v426/Hydrus.Network.426.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v426/Hydrus.Network.426.-.Linux.-.Executable.tar.gz

I had a great week. I mostly fixed and cleaned things.

8kun board deleted

The 8kun /hydrus/ board is now deleted. I was planning last week to move to Endchan as the new primary, but Codexx of 8chan.moe kindly offered to host me on 8chan.moe /t/, in a new Hydrus Network General. The thread is here https://8chan.moe/t/res/2219.html . The old /hydrus/ board is also migrated to 8chan.moe /hydrus/, if you would like to search it or recover something important to you. All the links across the program and help are updated to reflect the changes. I appreciate the help from Codexx in getting this going. I have never been a comfortable or great BO, so this arrangement is a better fit for my situation for the primary place to talk Anon about Hydrus. Endchan will remain as a bunker, and I will keep making release posts here. Please feel free to use whichever you are most comfortable with.

downloaders from the future

It seems gelbooru changed their markup recently, and our default downloader stopped pulling tags, at least for some. A user helpfully created an update--which is rolled into today's release if you missed it--but unfortunately users who were running older clients ran into an unusual versioning bug. The updated downloader would only work correctly on above v422 or so, leading older clients to get a variety of annoying errors about 'md5', 'hex' or 'additional_info' when either booting or trying to inspect the broken object.

I have fixed the problem here. Anyone who recently imported a 'downloader from the future' should have it fixed today. Furthermore, the various ways you import downloaders now try to check against versions your current client can support, and if they are too advanced, you now get a nice non-spammy popup message and the too-complex objects are discarded from the import. If the advanced object is buried deeper inside the downloader, it may not be able to discard neatly yet, and you will just get a 'soft' popup message about it, but I hope to tighten this up in future.

This version checking is also applied more softly across the whole program. It is less likely that one of your internal objects will be from the 'future', but the client will now make a popup if this is so. Please let me know if you get a whole bunch of these. With luck, this is ultimately a rare problem and I can toughen up this error and actually stop clients from running if their objects ever turn invalid.

I also fixed up error recovery in the downloader system UI and multi-column lists across the program. When some borked object enters them, they now try to show an appropriate 'cannot render' style text, rather than raising an error.
Open

R: 10 / I: 0 / P: 16

Version 425

https://youtube.com/watch?v=r1nn-tp26KE [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v425/Hydrus.Network.425.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v425/Hydrus.Network.425.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v425/Hydrus.Network.425.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v425/Hydrus.Network.425.-.Linux.-.Executable.tar.gz

I had a good week. I optimised and fixed several core systems.

faster

I messed up last week with one autocomplete query, and as a result, when searching the PTR in 'all known files', which typically happens in the 'manage tags' dialog, all queries had 2-6 seconds lag! I figured out what went wrong, and now autocomplete should be working fast everywhere. My test situation went from 2.5 seconds to 58ms! Sorry for the trouble here, this was driving me nuts as well.

I also worked on tag processing. Thank you to the users who have sent in profiles and other info since the display cache came in. A great deal of overhead and inefficient is reduced, so tag processing should be faster for almost all situations.

The 'system:number of tags' query now has much better cancelability. It still wasn't great last week, so I gave it another go. If you do a bare 'system:num tags > 4' or something and it is taking ages, stopping or changing the search should now just take a couple seconds. It also won't blat your memory as much, if you go really big.

And lastly, the 'session' and 'bandwidth' objects in the network engine, formerly monolithic and sometimes laggy objects, are now broken into smaller pieces. When you get new cookies or some bandwidth is used, only the small piece that is changed now needs to be synced to the database. This is basically the same as the subscription breakup last year, but behind the scenes. It reduces some db activity and UI lag on older and network-heavy clients.

better

I have fixed more instances of 'ghost' tags, where committing certain pending tags, usually in combination with others that shared a sibling/parent implication, could still leave a 'pending' tag behind. This reasons behind it were quite complicated, but I managed to replicate the bug and fixed every instance I could find. Please let me know if you find any more instances of this behaviour.

While the display cache is working ok now, and with decent speed, some larger and more active clients will still have some ghost tags and inaccurate autocomplete counts hanging around. You won't notice or care about a count of 1,234,567 vs 1,234,588, but in some cases these will be very annoying. The only simple fixes available at the moment are the nuclear 'regen' jobs under the 'database' menu, which isn't good enough. I have planned maintenance routines for regenerating just for particular files and tags, and I want these to be easy to fire off, just from right-click menu, so if you have something wrong staring at you on some favourite files or tags, please hang in there, fixes will come.
Open

R: 18 / I: 1 / P: 16

Version 388

https://youtube.com/watch?v=X-vdjur459c [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v388/Hydrus.Network.388.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v388/Hydrus.Network.388.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v388/Hydrus.Network.388.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v388/Hydrus.Network.388.-.Linux.-.Executable.tar.gz
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v388.tar.gz

I had a great week. The client can now save and load searches.

favourite searches

Every tag autocomplete input text box that searches for files--the most obvious being the one on normal search pages--now has a star icon button beside it. Click this, and you get a menu to save your current search, manage your saved searches, or load up one that is saved!

Currently, the saved information is the list of search terms (the tags and system predicates), the current file and tag domains (e.g. my files/all known tags), whether the system is 'searching immediately' or waiting, and optionally the current sort and collect.

Each saved search has a name and optionally a folder, for easy grouping. I have started all clients with an example inbox processing search, just as an example.

If you, like me, keep ten or fifteen empty search pages open (for me it is mostly different creator+inbox searches), I hope this system lets you collapse it down, making for a lighter and simpler session. It should also help for more unusual workflows like duplicate filtering and even file maintenance jobs.

There may be a couple of bugs in the system, something like a collect-by not being saved or updated correctly. Let me know how you get on!
Open

R: 6 / I: 1 / P: 16

Version 424

https://youtube.com/watch?v=YnU_j_ZA-tc [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v424/Hydrus.Network.424.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v424/Hydrus.Network.424.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v424/Hydrus.Network.424.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v424/Hydrus.Network.424.-.Linux.-.Executable.tar.gz

I had a good week. There are some quality of life improvements and faster tag search across the board.

The update will take some time this week to update a cache. If you do not sync to the PTR, it will be just a few seconds. If you sync to the PTR, expect about 5-15 minutes.

faster tag search

In the second half of 2020, I tried several times to tune the database for different sorts of wildcard tag search, which is used in all autocomplete lookups and many file searches. I was sometimes able to get small clients always running well, or complicated large systems running, well, but I failed to get it good for all situations with code alone--the structure of the database tag lookup cache made the tuning difficult.

So, I have updated how that cache works. Rather than always searching one big master table, the client can now 'zoom' in on the appropriate search context based on the type of search page or manage tags dialog or whatever.

Pretty much anything related to autocomplete and tag-based file searches is faster. Most importantly, the worst-case time for these searches is greatly improved. Complicated searches, like a 'namespace:*anything*' file search, should no longer have sudden gigantic lag spikes. These searches may still take ten seconds or more when searching millions of tags and files, but they won't accidentally lag out for two minutes on some tiny 'my tags' search with only 60 results.

The only exception in my testing is 'number of tags' searches still have bad cancelability. It is better, but not great. I'll keep working here.

The cache replaces an existing one. It will take some time to build it on update. If you do not sync with the PTR, it should just be a few seconds. If you sync with the PTR on an SSD, it should be 5-15 minutes (on my heavy client with a nice SSD, it was 7 minutes). If you sync with the PTR on an HDD, it will take significantly longer, so please plan for it. If you sync with the PTR, you will see some numbers count up as it builds the different parts of the cache. There will be some deletion work to start, then counting up to perhaps a million, and then up to 16 million or so, at about 30,000 a second.

I have more plans here, and more work to do to optimise the tag display system, but I will let this new cache breathe for a bit before going back in here with a machete.
Open

R: 4 / I: 1 / P: 16

Version 423

https://youtube.com/watch?v=SvsHVu3xt6A [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v423/Hydrus.Network.423.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v423/Hydrus.Network.423.-.Linux.-.Executable.tar.gz
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v423/Hydrus.Network.423.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v423/Hydrus.Network.423.-.Linux.-.Executable.tar.gz

𝕸𝖊𝖗𝖗𝖞 𝕮𝖍𝖗𝖎𝖘𝖙𝖒𝖆𝖘!

I had a good week making some small fixes and improvements to finish up the year. This is the last release of the year. There is a large poll on what 'big thing' to work on next:

poll

Here is the poll on what large work to go for next:

https://www.survey-maker.com/poll3310902xA574481e-102

You can vote on multiple items. Please don't worry about the seriousness of it too much--I have a good idea of what is likely to win already, and if there are obviously jank votes, I'll reserve the right to discount a result--but I am particularly interested to know what is and is not popular further down the list. I'll take the results on the 6th of January.

The end of 2020 has come quick for me. I still have the network updates 'big job' to do, so that's first for 2021 Q1, but after that, I will plan out and hack away at the top item(s) on the poll.

If you were not aware, a team of users is doing great work managing the Github issue tracker for hydrus. There is also a process there for bumping issues with reaction votes, viewable for big jobs like so:

https://github.com/hydrusnetwork/hydrus/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc+milestone%3A%22Major+jobs%22

I regret that am doing pretty terribly at consuming my Github work queue, but we'll see how 2021 goes. I'd like to start relying on that more to better prioritise my work.
Open

R: 5 / I: 0 / P: 16

Version 422

https://youtube.com/watch?v=AHBJ68icJ_4 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v422/Hydrus.Network.422.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v422/Hydrus.Network.422.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v422/Hydrus.Network.422.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v422/Hydrus.Network.422.-.Linux.-.Executable.tar.gz

🎉🎉 It was hydrus's birthday this week! 🎉🎉

I had a great week. I mostly fixed bugs and improved quality of life.

tags

It looks like when I optimised tag autocomplete around v419, I accidentally broke the advanced 'character:*'-style lookups (which you can enable under tags->manage tag display and search. I regret this is not the first time these clever queries have been broken by accident. I have fixed them this week and added several sets of unit tests to ensure I do not repeat this mistake.

These expansive searches should also work faster, cancel faster, and there are a few new neat cache optimisations to check when an expensive search's results for 'char' or 'character:' can quickly provide results for a later 'character:samus'. Overall, these queries should be a bit better all around. Let me know if you have any more trouble.

The single-tag right-click menu now always shows sibling and parent data, and for all services. Each service stacks siblings/parents into tall submenus, but the tall menu feels better to me than nested, so we'll see how that works out IRL. You can click any sibling or parent to copy to clipboard, so I have retired the 'copy' menu's older and simpler 'siblings' submenu.

Top