icc profile and pixel hash

This is mostly database prep for future duplicate system expansions.

The client database now records whether still images have an ICC profile, and it also saves data for 'these images are exact pixel duplicates' decisions. On update, all your existing files will be queued for scans to fill in this data in the background. Anything with an ICC profile will also regenerate its thumbnail. You don't have to do anything, this will all happen automatically over the coming week(s).

In time, you'll be able to search for images with ICC profiles with the new 'system:has icc profile' search predicate. This predicate is weird and advanced, so I think I'll hide it away soon under an umbrella for advanced stuff.

The 'exact pixel duplicate' data will be useful in the near future, when I expand the duplicate system to find (and optionally automatically merge) certain pairs that are perfect visual dupes.

full list

- misc:
- fixed a recent bug in wildcard search where 't*g' unnamespaced wildcards were not returning namespace results
- sped up multi-predicate wildcard searches in some unusual file domains
- the file maintenance manager is now more polite about how it works. no matter its speed settings, it now pauses regularly to check on and wait until the main UI is feeling good to start new work. this should relieve heavier clients on older machines who will get a bunch of new work to do this week
- 'all local files' in _review services_ now states how many files are awaiting physical deletion in the new deferred delete queue. this live updates when the values change but should be 0 pretty much all the time

- 'has icc profile' now cached in database:
- the client database now keeps track of which image files have an icc profile. this data is added on file import
- a new file maintenance task can generate it retroactively, and if a file is discovered to have an icc profile, it will be scheduled for a thumbnail regeneration too
- a new system predicate, 'system:has icc profile', can now search this data. this system pred is weird, so I expect in future it will get tucked into an umbrella system pred for advanced/rare stuff
- on update, all your existing image files are scheduled for the maintenance task. your 'has icc profile' will populate over time, and thumbnails will correct themselves
- .
- pixel hash now cached in database:
- the client database now keeps track of image 'pixel hashes', which are fast unique identifiers that aggregate all that image's pixels. if two images have the same pixel hash, they are pixel duplicates. this data is added on file import
- a new file maintenance task can generate it retroactively
- on update, all your existing image files are scheduled for the maintenance task. it'll work lightly in the background in prep for future duplicate file system improvements
- .
- boring search code cleanup:
- fixed a bug where the tag search cache could lose sibling/parent-chained values when their count was zeroed, even though they should always exist in a domain's lookup
- fixed up some repository reset code that was regenerating surplus tag search data
- with the new deleted files domain, simplified the new complex domain search pattern
- converted basic tag search code to support multiple location domains
- cleaned up some search cache and general table creation code to handle legacy orphan tables without error

next week

I'll take Christmas week off, so I only have two more proper weeks in the year. I would like to have basic pixel duplicate search working before then. Just a dropdown on the duplicates page for 'pair must/must not be pixel dupes' or similar. So, I will work on that and see if we can aim for a 'clean' release for end of year.

birthday

My todo list reminded me yesterday that I put out the first non-experimental beta of hydrus on December 14th, 2011. This is the rough 'start date' of the project and its birthday now. It will be ten years this week, which is pretty crazy.

Like a lot of people, 2021 was an odd year for me. I changed some lifestyle stuff, dropping some unhealthy habits, and also had some unexpected stress. After looking back though, I am overall happy with my work. Although I completed fewer big new projects than I hoped, and at times I felt bogged down in rewrites and fixes, the general performance of the client grew significantly this year. As well as a variety of new tag search and display options, the sibling and parent system was completely overhauled on several fronts, with the improved virtualised storage in the database and asynchronous real-time application calculation, and with that the autocomplete search finally supported 'perfect' sibling+parent adjusted tag counts in very fast time. Years-old sibling and parent application bugs were finally drilled down to and fixed. The tag lists across the program gained better sibling and parent display and commands. Wildcard searches became much faster too, and all sorts of tag and file search improved on smaller domains, sometimes by a factor of a thousand, even when a client had the whole PTR lurking in the background. We also moved to automatic repository account creation and improved serverside privacy, tags became easier to sort, file and database maintenance gained multiple new commands that saved a ton of time and inconvenience, the database learned to repair much of itself, system predicates became parseable in the Client API and editable in main UI, we moved from my duct-taped dev machines to github-built releases, the image renderer moved to a tiled system that allowed very fast zoom, sessions could grow much larger without CPU death and could save to disk with a fraction of their old write I/O, and a whole ton of other little fixes and quality of life improvements to every system.

I get a lot out of working on hydrus, and I hope to continue just like this. I appreciate everyone's feedback and support over the years. Thank you!

If you would like to further support my work and are in a position to do so, my simple no-reward Patreon is here: https://www.patreon.com/hydrus_dev

I had a great week. I managed to improve the duplicate filter search more than I thought, adding the ability to filter based on pixel duplicates and also pair similarity, and then I was able to rework the video scanbar so it sits inside the video frame and autohides based on mouse position.

The release should be as normal tomorrow.

dfcgvbhjnk

misc highlights

If you are interested in changing page drag and drop behaviour or regularly have overfull page tab bars, check the new checkboxes in options->gui pages.

If you are on Windows and have the default options->style, booting the client with your Windows 'app darkmode' turned on should magically draw most of the client stuff in the correct dark colours! Switching between light and dark while the program is running seems buggy, but this is a step forward. My fingers are crossed that future Qt versions improve this feature, including for multiplatform.

Thanks to a user, the twitter downloader is fixed. The twitter team changed a tiny thing a few days ago. Not sure if it is to do with Elon or not; we'll see if they make more significant changes in future.

I fixed a crazy bug in the options when you edit a colour but find simply moving the mouse over the 'colour gradient' rectangle would act like a drag, constantly selecting. This is due to a Qt bug, but I patched it on our side. It happens if you have certain styles set under options->style, and the price of fixing the bug is I have to add a couple seconds of lag to booting and exiting a colour picker dialog. If you need to change a lot of colours, then set your style to default for a bit, where there is no lag.

next week

I pushed it hard recently, and I am due a cleanup week, so I am going to take it easy and just do some refactoring and simple fixes.

I had a great week working on a variety of small cleanup, fixes, client api, sidecars, and UI quality of life.

The release should be as normal tomorrow.

- importer/exporter sidecar expansion (boring stuff):
- split the importer/exporter objects into separate importers and exporters. existing router objects will update and split their internal objects safely
- all objects in this system can now describe themselves
- all import/export nodes now produce appropriate example texts for string processing and parsing UI test panels
- Filename Tagging Options objects no longer track neighbouring .txt file importing, and their UI removes it too. Import Folders will suck their old data on update and convert to metadata routers
- wrote a json sidecar importer that takes a parsing formula
- wrote a json sidecar exporter that takes a list of dictionary names to export to. it will edit an existing file
- wrote some ui panels to edit single file metadata migration routers
- wrote some ui panels to edit single file metadata migration importers
- wrote some ui panels to edit single file metadata migration exporters
- updated edit export folder panel to use the new UI. it was already using a full static version of the system behind the scenes; now this is exposed and editable
- updated the manual file export panel to use the new UI. it was using a half version of the system before--now the default options are updated to the new router object and you can create multiple exports
- updated import folders to use the new UI. the filename tagging options no longer handles .txt, it is now on a separate button on the import folder
- updated manual file imports to use the new UI. the 'add tags before import' window now has a 'sidecars' page tab, which lets you edit metadata routers. it updates a path preview list live with what it expects to parse
- a full suite of new unit tests now checks the router, the four import nodes, and the four export nodes thoroughly
- renamed ClientExportingMetadata to ClientMetadataMigration and moved to the metadata module. refactored the importers, exporters, and shared methods to their own files in the same module
- created a gui.metadata module for the new router and metadata import/export widgets and panels

I was ill this week and am short on work time. I will spend tomorrow doing some more normal work instead of the release.

505 should be on the 9th of November. Thanks everyone!

----

I did just commit changes for users who run from source. I updated the easy-setup scripts and wrote variants for Linux and macOS. My test options are limited, so if you are an advanced Linux or macOS user, please do test these new scripts and let me know where things break or are otherwise confusing. The updated help walks through it all: https://hydrusnetwork.github.io/hydrus/running_from_source.html

I have thumbnails hidden by default on new pages, I have to do "pages - management & previevs - show" every time. Any way to keep them shown?

Are you on Qt5? You just see a widely stretched left-side panel, and the thumbnail panel is 0 pixels wide on the right? I am going to investigate a bug where in various situations, the thumbnail panel is disappearing in Qt5. Please let me know if I fix your issue in v505.

I had a great couple of weeks working on a variety of features. I fixed a heap of bugs, made it possible to search for files that have EXIF and other human-readable file metadata, and updated and translated my easy setup scripts, so it is now easy for any user on any platform to run the program from source.

The release should be as normal tomorrow.

I had a great week focusing on a full-scale update of the 'neighbouring .txt' import/export system. When you import or export files, you can now send URLs as well as tags; you can use .json instead of .txt; you can transform and filter text as it passes through; and you can set up workflows to multiple files at once. Everything is presented and edited in new UI.

The release should be as normal tomorrow.

>>1398
I was not familiar with what 'proxy pac' was, but it is a kind of javascript proxy configuration file? I don't support this, but if you run from source there may be a way to hack it in. I did a little search, and it looks like the main network library I use, 'requests', does not support this format easily:

https://stackoverflow.com/a/31534229

Some custom libraries referred in that same question seem able to do it. They may work well, but I do not know.

Whenever someone has a complicated proxying request, my general answer is to first see if your program supplies an OS-level or application-level wrapper, since that is always going to be easier and faster and more reliable than anything I can hack together. Do you have anything like that, where you can say 'any traffic from 'hydrus client', please wrap in this proxy'?

>>1399
Thank you for this report. It looks like the media panel (the thumbnails area) is not being swapped correctly. If and when this happens again, can you please hit pages->management and preview panels->show/hide? That normally hides the panel on the left. Maybe it fixes your layout, maybe it reveals more information. It might be the media panel is there, but it is 0 pixels wide on the right. There's a 'restore' command on that same menu that might be helpful in fixing it, too. Let me know what you see. And if you can figure out what is more likely to trigger this bug, so I can try reproducing it reliably on my end, that would be helpful.

>>1400
>>1400
>'any traffic from 'hydrus client', please wrap in this proxy
I want any traffic from 'these sites' through these proxies, according to these rules, for all apps. I imagined solution like this.

>>1397
Right-clicks in the main screen are broken (thumbnails and tags). The menu goes on top of the screen. Tags in file / tag viewer seem to work.
Testing is bit difficult because everything works on boot but breaks after using Hydrus for a while, I don't know what triggers it. If I find anything I'll let you know.

Thanks again!

>>1401
Sorry to say my network engine isn't clever enough for this yet. I hope to have individual domain proxy settings in the next time I update that system. It might be we are in a position then to suck up your configuration file here and auto-populate the settings, but this may be a long time in the future.

>>1402
Thanks. Sorry about the breaking after a while, keep me updated!

>>1400
>Thank you for this report. It looks like the media panel (the thumbnails area) is not being swapped correctly. If and when this happens again, can you please hit pages->management and preview panels->show/hide? That normally hides the panel on the left. Maybe it fixes your layout, maybe it reveals more information. It might be the media panel is there, but it is 0 pixels wide on the right. There's a 'restore' command on that same menu that might be helpful in fixing it, too. Let me know what you see. And if you can figure out what is more likely to trigger this bug, so I can try reproducing it reliably on my end, that would be helpful.
It took awhile but I waited long enough for it to happen again, its still pretty rare. I used the show/hide option in the pages menu and it fixed it or at least made it go back to its normal page. Nothing else out of the ordinary. Searches and images load fine.

full list

- autocomplete dropdown:
- the floating version of the autocomplete dropdown gets the same backend treatment the media hovers and the popup toaster recently received--it is no longer its own window, but now a normal widget floating inside its parent. it should look pretty much the same, but a variety of bugs are eliminated. clients with many search pages open now only have one top level window, rather than potentially hundreds of hidden ones
- if you have turned off floating a/c windows because of graphical bugs, please try turning them back on today. the checkbox is under _options->search_.
- as an additional consequence, I have decided to no longer allow 'floating' autocomplete windows in dialogs. I never liked how this worked or looked, overlapping the apply/cancel buttons, and it is not technically possible to make this work with the new tech, so they are always embedded in dialogs now. the related checkbox in _options->search_ is gone as a result
- if you ok or cancel on the 'OR' buttons, focus is now preserved back to the dropdown
- a bunch of weird interwindow-focus-juggling and 'what happens if the user's window manager allows them to close a floating a/c dropdown'-style code is cleared out. with simpler logic, some flicker jank is simply eliminated
- if you move the window around, any displaying floating a/c dropdowns now glide along with them; previously it updated at 10fps
- the way the client swaps a new thumbnail grid in when results are loaded or dismissed is faster and more atomic. there is less focus-cludge, and as a result the autocomplete is better at retaining focus and staying displayed as changes to the search state occur
- the way scroll events are caught is also improved, so the floating dropdown should fix its position on scroll more smoothly and capably
- .
- date system predicates:
- _this affects system:import time; :modified time; and :last viewed_
- updated the system:time UI for time delta so you are choosing 'before', 'since', and '+/- 15% of'
- updated the system:time UI for calendar date so you are choosing 'before', 'since', 'the day of', and '+/- a month of' rather than the ugly and awkward '<' stuff
- updated the calendar calculations with calendar time-based system predicates, so '~=' operator now does plus or minus one month to the same calendar day, no matter how many days were in that month (previously it did +/- 30 days)

- misc:
- 'sort files by import time' now sorts files correctly even when two files were imported in the same second. thanks to the user who thought of the solution here!
- the 'recent' system predicates you see listed in the 'flesh out system pred' dialogs now have a 'X' button that lets you remove them from the recent/favourites
- fixed the crash that I disabled some code for last week and reactivated the code. the collect-by dropdown is back to refreshing itself whenever you change the settings in _options->sort/collect_. furthermore, this guy now spams less behind the scenes, only reinitialising if there are actual changes to the sort/collect settings
- brushed up some network content-range checking logic. this data is tracked better, and now any time a given 206 range response has insufficient data for what its header said, this is noted in the log. it doesn't raise an error, and the network job will still try to resume from the truncated point, but let's see how widespread this is. if a server delivers _more_ data than specified, this now does raise an error
- fixed a tiny bit of logic in how the server calculates changes in sibling and parent petition counts. I am not sure if I fixed the miscount the janitors have seen
- if a janitor asks for a petition and the current petition count for that type is miscounted, leading to a 404, the server now quickly recalculates that number for the next request
- updated the system predicate parser to replace all underscores with whitespace, so it can accept system predicates that use_underscores_instead_of_whilespace. I don't _think_ this messes up any of the parsing except in an odd case where a file service might have an underscore'd name, but we'll cross that bridge if and when we get to it
- added information about 'PRAGMA quick_check;' to 'help my db is broke.txt'
- patched a unit test that would rarely fail because of random data (issue #1217)
- .
- client api:
- /get_files/search_files:
- fixed the recent bug where an empty tag input with 'search all' permission would raise an error. entering no search predicates now returns an empty list in all cases, no matter your permissions (issue #1250)
- entering invalid tags now raises a 400 error
- improved the tag permissions check. only non-wildcard tags are now tested against the filter
- updated my unit tests to catch these cases

I had a good week fixing some bugs and improving how the program works as a build and from source. It is now easy to run the program from source on Windows.

The release should be as normal tomorrow.

full list

- misc:
- the Linux build gets the same 'cannot boot' setuptools version hotfix as last week's Windows build. sorry if you could not boot v500 on Linux! macOS never got the problem, I think because it uses pyoxidizer instead of pyinstaller
- fixed the error/crash when clients running with PyQt6 (rather than the default Qt6, PySide6) tried to open file or directory selection dialogs. there was a slight method name discrepancy between the two libraries in Qt6 that we had missed, and it was sufficiently core that it was causing errors and best, crashes at worst
- fixed a common crash caused after several options-saving events such as pausing/resuming subscriptions, repositories, import/export folders. thank you very much to the users who reported this, I was finally able to reproduce it an hour before the release was due. the collect control was causing the crash--its ability to update itself without a client restart is disabled for now
- unfortunately, it seems Deviant Art have locked off the API we were using to get nice data, so I am reverting the DA downloader this week to the old html parser, which nonetheless still sems to work well. I expect we'll have to revisit this when we rediscover bad nsfw support or similar--let me know how things go, and you might like to hit your DA subs and 'retry ignored'
- fixed a bad bug where manage rating dialogs that were launched on multiple files with disagreeing numerical ratings (where it shows the stars in dark grey), if okayed on that 'mixed' rating, rather than leaving them untouched, were resetting all those files back to the minimum allowed star value. I do not know when this bug came in, it is unusual, but I did do some rating state work a few weeks ago, so I am hoping it was then. I regret this and the inconvenience it has caused
- if you manually navigate while the media viewer slideshow is running, the slideshow timer now resets (e.g. if you go 'back' on an image 7 seconds into a 10 second slideshow, it will show the previous image for 10 seconds, not 3, before moving on again)
- fixed a type bug in PyQt hydrus when you tried to seek an mpv video when no file was loaded (usually happens when a seek event arrives late)
- when you drop a hydrus serialised png of assorted objects onto a multi-column list, the little error where it says 'this list does not take objects of type x' now only shows once! previously, if your png was a list of objects, it could make a separate type error for each in turn. it should now all be merged properly
- this import function also now presents a summary of how many objects were successfully imported
- updated all ui-level ipfs multihash fetching across the program. this is now a little less laggy and uses no extra db in most cases
- misc code and linter warning cleanup
- .
- tag right-click:
- the 'edit x' entry in the tag right-click menu is now moved to the 'search' submenu with the other search-changing 'exclude'/'remove' etc.. actions

- client api:
- /add_tags/search_tags gets a new parameter, 'tag_display_type', which lets you either keep searching the raw 'storage' tags (as you see in edit contexts like the 'manage tags' dialog), or the prettier sibling-processed 'display' tags (as you see in read contexts like a normal file search page)
- /get_files/file_metadata now returns 'ipfs_multihashes' structure, which gives ipfs service key(s) and multihashes
- if you run /get_files/search_files with no search predicates, or with only tags that do not parse correctly so you end up with no tags, the search now returns nothing, rather than system:everything. I will likely make this call raise errors on bad tags in future
- the client api help is updated to talk about these
- there's also unit tests for them
- client api version is now 33
- .
- popup messages:
- the background workings of the popup toaster are rewritten. it looks the same, but instead of technically being its own window, it is now embedded into the main gui as a raised widget. this should clear up a whole heap of jank this window has caused over the years. for instance, in some OSes/Window Managers, when a new subscription popup appeared, the main window would activate and steal focus. this annoying thing should, fingers crossed, no longer happen
- I have significantly rewritten the layout routine of the popup toaster. beyond a general iteration of code cleanup, popup messages should size their width more sensibly, expand to available space, and retract better after needing to grow wide
- unfortunately, some layout jank does remain, mostly in popup messages that change height significantly, like error tracebacks. they can sometimes take two frames to resize correctly, which can look flickery. I am still doing something 'bad' here, in Qt terms, and have to hack part of the layout update routine. let me know what else breaks for you, and I will revisit this in future
- the 'BUGFIX: Hide the popup toaster when the main gui is minimised/loses focus' checkboxes under _options->popups_ are retired. since the toaster is now embedded into the main gui just like any search page, these issues no longer apply. I am leaving the two 'freeze the popup toaster' checkboxes in place, just so we can play around with some virtual desktop issues I know some users are having, but they may soon go too
- the popup toaster components are updated to use Qt signals rather than borked object callables
- as a side thing, the popup toaster can no longer grow taller than the main window size

next week

The popups now work better on Wayland but there are still some issues. The tag roster is misaligned and the right click menu spawns way above the cursor.

Anyway keep up the good work and don't burn yourself out!

Thanks. I am glad we are moving forward. I am now planning to do to the tag dropdown the same thing I did last week to the popup toaster. Having it as a separate window is too buggy in edge cases, and it doesn't necessarily need to be one, so I will rewrite its backend and embed it. Not sure I can fit it into this week, but I want to get this done.

I had a good week working on a variety of bug fixes and UI quality of life. The tag autocomplete dropdown gets an important de-janking overhaul, and the system:time search predicates are easier to work with.

The release should be as normal tomorrow.

full list

- crashes:
- I messed the mpv update up in v499. my golden rule is never to put out bleeding-edge library updates, but without thinking I gave everyone a dll from late august. it turns out this thing was pretty crashy, and many users were getting other unusual behaviour as well. it seems like people on very new versions of Windows were mostly ok, but a little instability, whereas some older-Windows users were unable to start the client or could boot but couldn't load mpv at all. these latter cases were plagued with other problems. thanks to user help, we discovered it was the newer mpv dll causing all the problems, and an older one, from early May, seems to be fine
- so, I am rolling back the mpv in the windows releases. the 'v3' 2022-08-29 I bundled in 499 was causing several users serious problems, possibly because of the advanced 'v3' chipset instructions or related advanced compiler tech. for the Qt6 release, we are going back to 2022-05-01, which several users report as stable, and for the Qt5 we are rolling back to the 498 version, 2021-02-28, which is back to mpv-1.dll. Since Qt5 users are increasingly going to be Win 7, we'll go super safe. THEREFORE, Qt5 extract users will want to perform a clean install this week: https://hydrusnetwork.github.io/hydrus/getting_started_installing.html#clean_installs
- (you can alternately just delete the now-surplus mpv-2.dll in your install directory, but a full clean install is good to do from time to time, so may as well)
- updated the sqlite dll in the windows release to 2022-05, and the exe in the db directory to 2022-09
- rewrote how some internal MPV events are signalled to Qt. they now have their own clean custom event types rather than piggy-backing on some bad old hydrus pubsub code
- I either fixed a rare boot crash related to the popup messaging system, maybe exclusively on macOS, or I improved it and we'll get a richer error now
- .
- tag sibling search:
- if you search explicitly for a tag that has a better sibling (one way this can happen is when loading up an old favourite search), the client will now auto-convert that tag to the ideal in the search code and give you results for the siblinged tag
- this started off as a predicted five minute thing and spilled out into a multi-hour saga of me realising some tag sibling search code was A) wrong in edge cases and B) slow in edge cases. I have subtly reshaped how core file-tag search works in the client so that it consults each tag service in turn based on its siblings and its mappings, rather than mixing them together. this does not matter for 99.98% of cases, but if you have some weird overlapping siblings across different services, you should now get the correct results. also, some optimisations are more effective, so any instance of searching for tags on small tag services on 'all known tags' is now a bit quicker
- big brain: please note the logic here is complex, and I have not yet updated autocomplete counting to handle this situation. if you type 'cat' and get 'cat (3)' from the three 'cat' tags on 'my tags', but 'cat' is siblinged to 'species:feline' on a big service like the PTR, it will still say (3), rather than (403) or whatever from the auto-corrected PTR results. I have a plan to fix this in a future cleanup round

- tag subtags and namespace wildcards:
- searching for 'samus aran' no longer delivers files that have 'character:samus aran'. the subtag->namespace logic no longer applies. this was a fun idea from the very start of the program, but it was never all that useful as default behaviour and added several headaches, now eliminated. if you wish to perform this search going forward, please enter '*:samus aran', which is now an acceptable wildcard input
- tag lookup is unaffected. typing 'samus aran' will still provide 'character:samus aran' as a tag to choose from
- a heap of rinky-dink counting logic went along with this, such as providing tag search results like ('character:samus aran (100)', 'samus aran (100-105)'), where it tried to predict how many results would come with the unnamespaced search. this no longer exists, and a decent bit of CPU is now saved in any large tag search
- wildcard searching works on similar rules now, so if you enter 'sa*s ar', you will see 'character:samus aran' as a result in the tag list, but searching for it will not give results with 'character:samus aran'. again, enter '*:sa*s ar*' to search for all namespaces (which is now provided as a quick suggestion any time you enter an unnamespaced wildcard), or enter 'character:sa*s ar*' explicitly
- 'system:tag as number' also now follows similar rules, so if you leave the namespace field blank, it will search unnamespaced numbers. it now supports namespace wildcards, so you can enter '*' to get the old behaviour. the placeholder text on the namespace input now states this
- 'system:number of tags' now uses the same UI as 'system:tag as number', where you enter '*' as the namespace to mean all namespaces, rather than checking a box
- .
- misc:
- all tag, namespace, and wildcard search predicates are now properly editable from the active search box. shift+double-click or select from the right-click menu, and you now get a simple text input alongside any system predicate panels. previously, this would only offer you a button to invert the tag to -tag and _vice versa_. now, you can add or remove the '-' and '*' characters yourself info to freely convert between tags, namespace:anything, and wildcard search predicates (issue #1235)
- thanks to a user, you can now add '{#}' to an export filename pattern to get the '#' column in your filename (useful if you want to export files in the order they are currently in on the page)
- furthermore, if you delete items from the manual file export window, the '#' column now recalculates itself to stay contiguous and in order (previously, it left gaps)
- fixed a bug when deleting siblings on a local tags service. sorry for the trouble!
- on manage siblings, when you remove, add, or replace a pair on a local tags service, you will now get a simple 'note' reason informing you more on what is going on. the 'REPLACEMENT:' thing recently added to tag repositories should now work for you too
- when a downloader or similar adds files to a page, and you have at least one existing file selected, the status bar now updates correctly

- fixed a critical issue that was affecting some users with damaged similar file search trees. when starting similar file search tree rebalancing maintenence, their client would go into an infinite loop and spool the cyclic branch into an ever-growing journal file in their temp directory until their system drive briefly ran out of space. sorry for the trouble, and thank you for the excellent reports that helped to figure this out (issue #1239)
- the similar files search tree rebalance maintenance now detects more sorts of damaged trees and handles them gracefully, and the full tree regeneration clears out any damaged maintenance information too
- fixed another problem with the tree branch maintenance system when the root was accidentally queued for branch rebalance
- when you right-click->copy a wildcard search tag, it now copies the actual wildcard text, not the display text with (wildcard search) over the top
- I added ',' to the list of non-decodable characters in the hacky URL Class encoding/decoding routine. sites that use an encoded comma (%_2C) for regular path components or query parameters should now work
- a user has fixed a regex parsing problem in the predicate parser for system:hash
- OR search predicates now sort their sub-predicates on construction/editing, meaning the label is always of set order, and they can now compare with and hence reliably nullify each other
- the manage logins dialog now boots a little taller
- the main gui tab bar may look a bit nicer/more appropriate in macOS
- updated the help text on gui pages where it talks about overflowing rows of tabs, which auto-scroll even worse in Qt6, hooray
- .
- client api:
- the client api now handles request disconnects better. the hydrus server code benefits from the same engine improvements
- the 'twisted.internet.defer.CancelledError' logspam is cleaned up!
- if a client disconnects before a client api autocomplete tag search or a file search is complete, that database job is now cancelled quickly just like when you type new characters in the client UI or stop a slow search
- if you are a client api dev, please let me know how this works out IRL. I'm not 100% sure what a 'disconnect' means in this context, but if you want to develope autocomplete quick lookup as the user types, and you have a way clientside to cancel/kill an ongoing request before it is complete, please give it a go and let me know if this all works. cancelled requests don't make a log record right now, but you should see the client's db lock free up instantly. at the very least, I have the proper infrastructure for this now, so I can add more/better 'cancel' hooks as we need them
- .

next week

I was not able to get to many things I wanted to this week. Things have piled up, so I'm just a bit buried right now. I will just continue working on urgent issues and smaller issues and see how we are on the crashing.

I'm stressed about my Dad, more than anything because there has just been a ton of energy-draining stuff to do, but not as upset as I thought. As I said before, we had a great relationship, so there are no huge regrets. I'm sure it will kick in more in a couple months. Since the hydrus userbase trends young and my Dad died a bit early, I don't expect many of you have organised a funeral. My serious advice is A) talk to your parents now about what they want in a funeral, and B) make sure they have a will. We were good on both fronts, and it has made the whole thing so much easier. Almost all children bury their parents, so get it done now, while it is easy.

As for hydrus, getting out 500 versions is pretty cool. I've been at this for ten years, and the codebase is now 10 MB over almost 300 files. I still enjoy working on it, and I want to keep at it as long as there is interesting stuff to do. The hydrus userbase has grown significantly this year, and my todo list is overflowing more than ever, so running short on work is not a worry. Handling stress and burnout has been tricky at times, but assuming I stay healthy on that front, I can comfortably see 750 in the distance. Let's see what machine learning does to us all over the next five years.

Thanks everyone!

I had an ok week. I cleared out some bugs, upgraded the popup toaster, and improved some UI quality of life.

The release should be as normal tomorrow.

>>1361
Yeah, Cloudflare 403s are a pain. I am not sure about Tor and Cloudflare, that may just be a step too far in complexity, but the 'normal' way to get through a 403 Cloudflare block is to visit the site in your browser (which does the clever problem solving) and then use Hydrus Companion to copy the solved cookies over to Hydrus.

https://gitgud.io/prkc/hydrus-companion

However, this may not work with Tor. Tor probably sets its own User-Agent and I guess is using different IP addresses regularly, and those are the two things that need to be the same as your browser for the Hydrus Companion trick to work. Maybe if you did the work in the Tor browser, but then copying cookies across may be even more difficult.

That said, some sites are just strict when it comes to Cloudflare rules, so if your VPN or whatever network solution is frowned upon by Cloudflare this week, then you'll just have trouble with that site. My own answer is mostly just to navigate away from those sites and stick to what I know works for me. I'm already overloaded with stuff to process, so no point spending 50% effort trying to fetch 5% more stuff that I missed.

For updating, no worries, the normal method is to extract the release on top of your old install folder. The extract straight from the zip file has no 'database' files, so it extracts safely over what you already have without overwriting anything user-created. Please check the help here:

https://hydrusnetwork.github.io/hydrus/getting_started_installing.html#updating

I personally make a backup, extract the fresh release on top, and boot. My update code will investigate your database, notice it is an older version, and apply several rules to update it in place. You will keep the same session and everything as when you last closed the client. The idea of the backup at that time is that if the update fails for some reason (rare, but it happens to a handful of users a few times a year), or more likely I messed something up, I can roll back immediately to where I was, no danger.

If you update using the normal routine and keep your old database, it will remember all its old known urls and be able to skip repeated downloads.

>>1367
>>1368
Sorry for the trouble with the new release. Some users are getting several crashes, and I think the new mpv is the culprit. Since you are on Windows 7, I suspect the new mpv is incompatible with it in the same way things like Qt6 are too. Can you try going in to your install directory and deleting the 'mpv-2.dll' file? You should still have mpv-1.dll from a previous install, and that should work on its own.

If that still gives you trouble, can you try using the client as normal but avoid:

A) mpv (so don't load up any movies into the preview or media viewer)
B) the duplicate filter (I have some reports this is giving trouble too)

If the program does not crash until one of these is loaded, that is good evidence that it is mpv after all. Please avoid them for now, or if you have a backup, roll back to v498, and I will work on these problems and see if I can fix some things for v500.

If you go to "install_dir/db/client (date).log", you might like to scroll down to the bottom of the one for September. Are there any meaty-looking errors in there? They might be called 'traceback'. Most crashes do not leave any log information (they break too quick), but if there are some lead-up errors in there, that would also be useful to know.

>>1370
Thanks. Yeah, I am going to be careful and not try to do three things at once. I'm actually feeling better emotionally than I thought I would. It sucks, a lot, but everyone has been wonderful and neighbours and so on are stepping up to help out. Of all the ways my Dad could go, he didn't want it drawn out, so there are more than a few silver linings here. A lot of people have to see their parents waste away from cancer over years, or to Alzheimers, so we've been saved that. I expect I will start missing him more in a few months, when his absence has a proper shape to it, but a friend of mine wisely said not to make any big decisions, so I am keeping my head down and pushing on with practical help for the situation. The good news is the most urgent funeral preparation was finished last night, so now everything is lined up. Getting everything sorted needed focus and energy, but now we just have to do picking family up from airports and stuff, and the actual day.

Since most hydrus users are going to be on the young side, and most will not have lost their parents yet, the advice I am going to give from now on is:

>>1371

Thanks for the detailed reply. I'll make sure to check out Hydrus Companion and whether it works with Tor. Yes, generally I'd also just move on to a more available source, but for some things that's difficult.

I'm glad to hear that updating is that simple and the db is cleanly divided from the actual program. I set out updates since I messed up once and read about the big update situation. But I'll try that again, one version at a time to be sure and see how it goes.

>>1372

True. It helps a lot to have things worked out and off your mind. I can only recommend to take some dedicated time to face the emotional trouble inside you and go through with it, no distraction or pushing it away. You'll cry like a bitch, but you will feel a lot better afterwards. Unresolved trauma will eat you up from the inside. It's like getting a tooth pulled at the dentist, painful but so much relief. Don't worry, you're built to endure that stuff.

>>1372
>Sorry for the trouble with the new release. Some users are getting several crashes, and I think the new mpv is the culprit. Since you are on Windows 7, I suspect the new mpv is incompatible with it in the same way things like Qt6 are too. Can you try going in to your install directory and deleting the 'mpv-2.dll' file? You should still have mpv-1.dll from a previous install, and that should work on its own.

Ok I removed the mpv-2.dll and it was indeed causing the problem of not being able to play animated files(gifs, webms, mp4, etc.). They play normally now and I don't get the "MPV is not available" errors anymore.

As for the crashes I can't remember what tags I used to cause them now lol however, I feel like it was more related to mpv-2 but I'll continue to be on the lookout. The crash reports doesn't really tell you much. Just normal boots and exists and doesn't even mention it crashed other than after booting Hydrus and saying

>"Found and deleted the durable temporary database on boot. The last exit was probably not clean."

And I don't see any "traceback" either in the logs nor in the db folder.

I had a good week-and-a-bit getting back to schedule. I have fixed some critical bugs, including the crashy mpv problem on Windows, cleaned a bunch of code, and improved regular tag searching with new tools, cleaner logic, and automatic sibling substitution.

The release should be as normal tomorrow, probably a little early. Windows Qt5 users will have special install instructions.

full list

almost all the changes this week are only important to server admins and janitors. regular users can skip updating this week
- overview:
- the server has important database and network updates this week. if your server has a lot of content, it has to count it all up, so it will take a short while to update. the petition protocol has also changed, so older clients will not be able to fetch new servers' petitions without an error. I think newer clients will be able to fetch older servers' ones, but it may be iffy
- I considered whether I should update the network protocol version number, which would (politely) force all users to update, but as this causes inconvenience every time I do it, and I expect to do more incremental updates here in coming weeks, and since this only affects admins and janitors, I decided to not. we are going to be in awkward flux for a little bit, so please make sure you update privileged clients and servers at roughly the same time
- .
- server petition workflow:
- the server now maintains an ongoing fast count of its various repository metadata, such as 'number of mappings' and 'number of petitions of type x'. when you fetch petition counts, no longer will it count live and max out at 1,000, it'll give you good full numbers every time, and real fast
- you can see the current numbers from the new 'service info' button on review services, which only appears in advanced mode. any user with an account key can see these numbers, which include number of petitions in the queue. I can make this more private if you like, but for now I think it is good if advanced users can see them all
- in the petition processing page, sibling and parent petitions will now include both delete and add rows if the account and reason are the same. I'm aiming to get better 'full' coverage of a replace petition, so you can see and approve/deny both the add and the remove parts in one go. for fetching, these combined petitions count as 'delete' petitions, and won't appear in the 'add' petition queue
- when users encounter an automatic conflict resolution in the manage siblings dialog, those auto-petitioned pairs are now assigned the same reason as the original conflicting pended pairs. they _should_ show up together in the new petition processing UI
- as part of this, sibling and parent petitions are no longer filtered by namespace. you will see everything with that same account and reason in one go. let's try it out, and if it is too much, I will add filters clientside or something. since we are now starting to see add and remove together, we'll want to at least have the option to see everything

- boring server stuff:
- the petition object is updated to handle multiple actions per petition, and the clientside petition UI is updated appropriately
- the server tracks 'actionable' petition counts as separate to the number of raw petition rows. some of this was happening before, but the logic is improved, including clever counting of the new petitions that include both add and delete rows
- for when my count-update logic inevitably fails, there is now a 'regen service info' entry in the 'administrate services' menu for all repositories. numbers generated will be printed to server log
- some unusual repo upload logic is cleaned up, e.g. if a user with 'create permission' uploads a sibling or parent, any pending rows for that content will now be properly cleared)
- fixed a stupid swap logical bug where janitors who could only moderate siblings (and not parents) were only being given parent numbers and vice versa
- all server services now respond to /busy check. it requires no authentication and just returns 1 or 0 depending on the current lock state
- fixed a bug where tag siblings or parents that were denied would still make a new definition record for the child/bad tag
- with all the fine number changes, fleshed out the server unit tests with more examples of submitting and altering content and then checking for numbers afterwards. now checked are: file add, file admin delete, mapping add, mapping admin delete, mapping petition, mapping petition approve+deny, parent add, parent admin delete, parent pend, parent pend approve+deny, parent petition, parent petition approve+deny
- significant refactoring of the tail end of server content update pipeline. more things now go through logic-harmonised update methods that ensure count is reliable
- did some misc server db and constant enum code cleanup
- .
- misc:
- to match the new change in the server, in the client, tag and rating services now store their 'num_files' service info count as the new 'num_file_hashes'. existing numbers will be converted over during update
- fixed a probably ten year old bug where 'num pending/petitioned files' had the same enum as 'num pending/petitioned mappings'. never noticed, since no service has done both those things
- if the upload pending process fails due to an unusual permission error or similar, the pending menu should now recover and update itself (previously it stayed greyed out)

- multiline parsing:
- the parser now supports limited multiline parsing. the main changes are hardcoded: the formulae beneath note content parsers and those that do subsidiary page parser splitting no longer remove newlines when they parse. all the parsing UI and the test panels and so on are now aware of this and set flags in all the right places, and parsed notes are now washed through the new trimming/cleaning method, and everything _seems_ to basically work. the main remaining problems is the complicated string processing UI has mixed single/multi-line testing support. some looks great, most gets coerced to single-line just for the previewed test results
- as an example, the default hentai foundry downloader now grabs the artist description as a multi-line note
- the parsing sub-system that extracts cohesive strings from complex html blocks now inserts newlines at 'p' and 'br' tags
- trying to parse clean multiline notes still caused several formatting issues this week, so I have updated the automatic note-washing routine to standardise hydrus notes in several new ways that I hope will not be too disruptive to manually written notes:
- the note washing routine now coerces all newline characters to 'backslash-n', regardless of platform
- the note washing routine now trims each line, so no leading or trailing whitespace anywhere. I am open to changing this in future, maybe for handwritten notes where you really want an indent somewhere, but parsing from complex nested html tags is making a heap of weird extra whitespace, for which this is a clean solution
- the note washing routine now trims newline gaps that are greater than two-newlines. you can split paragraphs by one empty line, but no more
- there may be other issues figuring out cleanly formatted strings from nested html tags--so give it a go and let me know what you think. maybe p and br blocks should always make two newlines, so we have separated paragraphs, maybe I need to parse more blocks, like h1 and friends. any specific example html blocks would also be helpful
- .
- cleanup:
- refactored ClientGUIParsing to its own 'parsing' module and split everything into four less tangled files
- cleaned up a bunch of taglist text presentation code, mostly simplicity and clarity in prep for future updates
- updated the checker options button to use a Qt signal instead of a callable

next week

Qt6 version is kinda broken on Wayland / Fedora 36, all pop-ups are in top left corner. Is this a Qt or Hydrus issue?

I can force Hydrus to run on X so it isn't a huge problem.

>>1350
Thank you for this report. I am not sure, as I haven't heard this problem yet, but this is probably a hydrus issue. I expect Qt6 is delivering the coordinates in a different way in Wayland / Fedora 36, or one of the conversion routines the translates coordinates from global space to relative-to-main-window space is not working. The popups are working fine on other platforms, so it probably is fundamentally a Qt interface problem, but it would still be nice if I could fix it.

I'll have a quick look at the logic here, see if I can see an obvious way to fix it.

I'm actually planning to rewrite how the popup toaster works in the nearish future, some behind the scenes technical stuff, that may completely fix this naturally anyway, so please keep me updated on how this works for you in future.

>>1351
Will do. Pic related what's currently happening.

I had a good week, finally getting around to some server updates I had been planning for a long time. Unfortunately, I didn't have time to get to anything else, so tomorrow's release is only for users who run a hydrus repository, and their janitors. The server now reports some fast counts of its content, and petition processing has a couple small improvements.

The release should be as normal tomorrow.

>>1352
Thanks, I've saved this. I completely ran out of time this week, but I will work on this.