icc profile and pixel hash

This is mostly database prep for future duplicate system expansions.

The client database now records whether still images have an ICC profile, and it also saves data for 'these images are exact pixel duplicates' decisions. On update, all your existing files will be queued for scans to fill in this data in the background. Anything with an ICC profile will also regenerate its thumbnail. You don't have to do anything, this will all happen automatically over the coming week(s).

In time, you'll be able to search for images with ICC profiles with the new 'system:has icc profile' search predicate. This predicate is weird and advanced, so I think I'll hide it away soon under an umbrella for advanced stuff.

The 'exact pixel duplicate' data will be useful in the near future, when I expand the duplicate system to find (and optionally automatically merge) certain pairs that are perfect visual dupes.

full list

- misc:
- fixed a recent bug in wildcard search where 't*g' unnamespaced wildcards were not returning namespace results
- sped up multi-predicate wildcard searches in some unusual file domains
- the file maintenance manager is now more polite about how it works. no matter its speed settings, it now pauses regularly to check on and wait until the main UI is feeling good to start new work. this should relieve heavier clients on older machines who will get a bunch of new work to do this week
- 'all local files' in _review services_ now states how many files are awaiting physical deletion in the new deferred delete queue. this live updates when the values change but should be 0 pretty much all the time
- 'all known files' in _review services_ also gets a second safety yes/no dialog on its clear deleted files record button
- updated the gelbooru 0.2.x gallery page parser, which at some point had been pulling 'delete from favourites' links when running the login-required 'gelbooru favorites by user id' downloader!!! it was deleting favourites, which I presume and hope was a recent change in gelbooru's html. in any case, the parser now skips over any deletion url (issue #1023)
- fixed a bad index to label conversion in a common database progress method. it was commonly saying 22/21 progress instead of 21/21
- fixed an error when manage tags dialog posts tags from the autocomplete during dialog shutdown
- fixed a layout issue with the new presentation import options where the dropdowns could grow a little tall and make a sub-panel scrollbar
- added handling for an error raised on loading an image with what seems to be a borked ICC profile
- increased the default per-db-file cache size from 200MB up to 256MB
- .
- some new options:
- the default tag service in the manage tags dialog (and some similar 'tag services in a row of tabs' dialogs) is reset this week. from now on, the last used service is remembered for the next time the dialog is opened. let's see how that works out. if you don't like it, you can go back to the old fixed default setting under the 'tags' options panel
- added a checkbox to the 'search' options panel that controls whether new search pages are in 'searching immediately' or 'searching paused' state (issue #761)
- moved default tag sort from 'tags' options panel to 'sort/collect'
- .
- deleted files and ipfs searchability:
- wrote a new virtual file service to hold all previously deleted files of all real file services. this provides a mapping cache and tag lookup cache allowing for fast search of any deleted file domain in the future
- ipfs services also now have mapping caches and tag search caches
- ipfs services are now searchable in the main file search view! just select them from the autocomplete dropdown file domain button. they have tag counts and everything
- it will take some time to populate the new ipfs and deleted files caches. if you don't have much deleted files history and no ipfs, it will be a few seconds. if you have a massive client with many deleted/ipfs files and many tags, it could be twenty minutes or more

- 'has icc profile' now cached in database:
- the client database now keeps track of which image files have an icc profile. this data is added on file import
- a new file maintenance task can generate it retroactively, and if a file is discovered to have an icc profile, it will be scheduled for a thumbnail regeneration too
- a new system predicate, 'system:has icc profile', can now search this data. this system pred is weird, so I expect in future it will get tucked into an umbrella system pred for advanced/rare stuff
- on update, all your existing image files are scheduled for the maintenance task. your 'has icc profile' will populate over time, and thumbnails will correct themselves
- .
- pixel hash now cached in database:
- the client database now keeps track of image 'pixel hashes', which are fast unique identifiers that aggregate all that image's pixels. if two images have the same pixel hash, they are pixel duplicates. this data is added on file import
- a new file maintenance task can generate it retroactively
- on update, all your existing image files are scheduled for the maintenance task. it'll work lightly in the background in prep for future duplicate file system improvements
- .
- boring search code cleanup:
- fixed a bug where the tag search cache could lose sibling/parent-chained values when their count was zeroed, even though they should always exist in a domain's lookup
- fixed up some repository reset code that was regenerating surplus tag search data
- with the new deleted files domain, simplified the new complex domain search pattern
- converted basic tag search code to support multiple location domains
- cleaned up some search cache and general table creation code to handle legacy orphan tables without error
- misc tag and mapping cache code and combined local files code refactoring and cleanup

next week

I'll take Christmas week off, so I only have two more proper weeks in the year. I would like to have basic pixel duplicate search working before then. Just a dropdown on the duplicates page for 'pair must/must not be pixel dupes' or similar. So, I will work on that and see if we can aim for a 'clean' release for end of year.

birthday

My todo list reminded me yesterday that I put out the first non-experimental beta of hydrus on December 14th, 2011. This is the rough 'start date' of the project and its birthday now. It will be ten years this week, which is pretty crazy.

Like a lot of people, 2021 was an odd year for me. I changed some lifestyle stuff, dropping some unhealthy habits, and also had some unexpected stress. After looking back though, I am overall happy with my work. Although I completed fewer big new projects than I hoped, and at times I felt bogged down in rewrites and fixes, the general performance of the client grew significantly this year. As well as a variety of new tag search and display options, the sibling and parent system was completely overhauled on several fronts, with the improved virtualised storage in the database and asynchronous real-time application calculation, and with that the autocomplete search finally supported 'perfect' sibling+parent adjusted tag counts in very fast time. Years-old sibling and parent application bugs were finally drilled down to and fixed. The tag lists across the program gained better sibling and parent display and commands. Wildcard searches became much faster too, and all sorts of tag and file search improved on smaller domains, sometimes by a factor of a thousand, even when a client had the whole PTR lurking in the background. We also moved to automatic repository account creation and improved serverside privacy, tags became easier to sort, file and database maintenance gained multiple new commands that saved a ton of time and inconvenience, the database learned to repair much of itself, system predicates became parseable in the Client API and editable in main UI, we moved from my duct-taped dev machines to github-built releases, the image renderer moved to a tiled system that allowed very fast zoom, sessions could grow much larger without CPU death and could save to disk with a fraction of their old write I/O, and a whole ton of other little fixes and quality of life improvements to every system.

I get a lot out of working on hydrus, and I hope to continue just like this. I appreciate everyone's feedback and support over the years. Thank you!

If you would like to further support my work and are in a position to do so, my simple no-reward Patreon is here: https://www.patreon.com/hydrus_dev

I had a great week. I managed to improve the duplicate filter search more than I thought, adding the ability to filter based on pixel duplicates and also pair similarity, and then I was able to rework the video scanbar so it sits inside the video frame and autohides based on mouse position.

The release should be as normal tomorrow.

dfcgvbhjnk