better file delete

tl;dr: Files delete from your disk better now.

For a long time, actually physically deleting a file that leaves the client's trash has been a bit hacky. To reduce UI lag, the recycle or delete action has to be deferred to the future, but since there was no permanent record of what to delete, the deferred job was always a bit rushed and in a couple of instances (e.g. if you cleared trash and closed the client soon after) it could leave file orphans behind. The system has always been fail safe, never deleting too much, but it could lead to surplus cruft in file storage. Secondly, it turns out the server never got physical file delete tech added! The file repository has never been used much, and it just never came up.

So, this week I overhauled the whole way it works. Now both client and server keep records of the files they should delete, print delete summaries to the log, use neater delete daemons that smooth out the work to be non-interrupting, have sophisticated 'is this an orphan now?' logic to ensure that edge cases are kept in storage until they are truly no longer needed, and correct themselves in odd cases (like if you re-import a file just a couple of seconds after deleting it from the trash). They are also more aware of repository update files, which is another thing neither have been good at clearing out in the past.

So, you don't have to do anything here, but with luck doing things like 'clear trash' and other large forced physical deletes should just work a bit nicer now. Users who run a server may see it clear out a couple of files after booting, and in a few weeks I'd like to roll out a 'clear orphan files' server database action for the admin menu just like the client's so you can clear out legacy files from deleted services.

full list

- image icc:
- images with embedded icc colour metadata are now normalised (to sRGB) like the rest of media rendering in hydrus. ICC can often mean photos, where a nice camera will apply ICC data to compensate for camera defects or general lighting information, or it can mean normal digital images where the software attached extra colour data when it was saved
- the image will now be rendered with 'fixed' colours in the media viewer, and new thumbnails should be good too. it applies early in image load and should work in all cases hereon, on both client and server
- images with an ICC will take a little longer to initially load. I'd estimate 10-50ms extra for most. one user with many ICC images discovered 10% of their collection had an ICC. I don't think the delay will be terrible IRL, but see how you get on and let me know! maybe giganto patreon pngs will have a fresh surprise for us
- future expansions here will be a database cache of ICC images and system:has icc, perhaps a button to click the ICC application on and off live in the media viewer, and then maybe options to load up and switch an ICC for your display

- better physical file delete:
- both client and server now physically delete files from storage more smoothly and reliably. the 'deferred file delete' list is now saved in the database itself and will survive reboots (and undo itself if a file is re-added before it can be deleted), and the physical delete daemons are able to work at a less spiky pace as a result. physical delete summaries are now logged as well
- the server now physically deletes surplus files from its file storage! this never actually came up before jej--servers were just keeping all files forever
- on update, all servers will scan to see which files it only has deletion records for and will queue them for a deferred delete
- when deleting a service from the server, all its file repository files and/or general repository update files are now queued for deferred deletion if they are now orphaned
- some advanced 'pending upload file delete' logical situations are now tidied up better, for instance if you have a file set to upload to a file repository or IPFS and then delete the file from the trash, the file will hang around until the upload is done and then it will be correctly scheduled for physically deletion. same for if you delete the file repository or clear all its pending. previously, this file would never delete and become an orphan
- thumbnails for non-downloaded file repository files are now removed promptly from a client if a file repository deletes a file
- .
- misc:
- fixed a typo error in last week's file filtering changes when doing wildcard tag searches in 'all known files' domain
- fixed some bad namespace search optimisation also caused by last week's search updates that was making 'system:has x unnamespaced tags' search instead count all tags, not just unnamespaced (issue #1017)
- fixed incorrect file type handling in thumbnail loading that was triggering a safe mode for gif file thumbs (which are actually jpeg/png), it should roughly double thumb load speed for gifs (and .ico too lol)
- .
- boring image stuff:
- wrote some methods to check for and pull ICC profile bytes from an image with PIL
- wrote ICC application in PIL on image load. we had figured out a way to do it with Qt, but this can happen right at the start of the rendering pipeline and will work for the server too
- cleaned up some PIL/OpenCV image load and normalisation code

I had a good week. I fixed a variety of small bugs, added some quality of life options, and made several behind the scenes database improvements that will improve search now and in the future.

The release should be as normal tomorrow.

full list

- misc:
- ogv files (ogg with a video stream) are now recognised by the client! they will get resolution, duration, num frames and now show in the media viewer correctly as resizeable videos. all your existing ogg files will be scheduled for a rescan on update
- wrote new downloader objects to fix deviant art tag search. all clients will automatically update and should with luck just be working again with the same old 'deviant art tag search' downloader
- added prototype copy/paste buttons to the manage ratings dialog. the copy button also grabs 'null' ratings, let me know how you get on here and we'll tweak as needed
- file searches with namespaced and unnamespaced tags should now run just a little faster
- most file searches with multiple search predicates that include normal tags should now run just a little faster
- the file log right-click menu now shows 'delete x yyy files from the queue' for deleted, ignored, failed, and skipped states separately
- the tag siblings + parents display sync manager now forces more wait time before it does work. it now waits for the database and gui to be free of pending or current background work. this _should_ stop slower clients getting into hangs when the afterwork updates pile up and overwhelm the main work
- the option 'warn at this many pages' under _options->gui pages_ now has a max value of 65535, up from 500. if you are a madman or you have very page-spammy subscriptions, feel free to try boosting this super high. be warned this may lead to resource limit crashes
- the 'max pages' value that triggers a full yes/no dialog on page open is now set as the maximum value of 500 and 2 x the 'warn at this many pages' value
- the 'max pages' dialog trigger now only fires if there are no dialogs currently open (this should fix a nested dialog crash when page-publishing subscriptions goes bananas)
- improved error reporting for unsolveable cloudflare captcha errors
- added clarification text to the edit subscription query dialog regarding the tag import options there
- added/updated a bunch of file import options tooltips
- .

- boring presentation import options stuff:
- presentation options are now in their own object and will be easier to update in future
- the 'should I present' code is radically cleaned up and refactored to a single central object
- presentation filtering in general has more sophisticated logic and is faster when used on a list (e.g. when you highlight a decent sized downloader and it has to figure out which thumbs to show). it is now careful about only checking for inbox status on outstanding files
- presentation now always checks file domain, whereas before this was ad-hoc and scattered around (and in some buggy cases lead to long-deleted files appearing in import views)
- added a full suite of unit tests to ensure the presentation import options object is making the right decisions and filtering efficiently at each stage
- .
- boring multiple local file services work:
- I basically moved a bunch of file search code from 1 file services to n file services:
- the file storage module can now filter file ids to a complex location search context
- namespace:anything searches of various sorts now use complex location search contexts
- wildcard tag searches now use complex location search contexts
- counted tag searches now use complex location search contexts
- search code that uses complex location search contexts now cross-references its file results in all cases
- I now have a great plan to add deleted files search and keep it working quick. this will be the next step, and then I can do efficient complex-location regular tag search and hopefully just switch over the autocomplete control to allow deleted files search

next week

I had a good week. I managed to add support for embedded ICCs, which will improve some images' colours, and overhauled how files are deleted from the client and serevr so the system is smoother and more reliable.

The release should be as normal tomorrow.

- network redirect improvements:
- gallery page jobs now give their child 'next gallery page' jobs themselves as a referrer
- when the gallery downloader gets a 3XX redirect, the file import objects and next gallery pages it makes now get the redirected URL as referral url (previously, it used the original gallery url)
- when the post downloader gets a 3XX redirect, the redirected url is now added as a primary source url
- when the post downloader gets a 3XX redirect, child import objects and direct file downloads now get the redirect URL as referral url (previously, it used the original post url)
- when the raw file downloader gets a 3XX redirect, the redirected url is now added as a primary source url
- when the raw file downloader gets a 3XX redirect to a Post URL, it now tries to queue that URL back up in the file log, just like when a gallery fetch comes back with a Post URL. some safety code stops potential loops here as well
- .
- new services:
- a new client now starts with a second local tag service called 'downloader tags'. default tag import options are now initialised in a fresh client to pull all file post tags to this service. this should relieve new users' confusion about setting up default tag import options
- similarly, a new client now starts with a like/dislike rating service called 'favourites'. existing users who have no rating services will be asked if they want to get it on update. many users are unaware of the rating system, so this is a chance to play with it
- the 'getting started with downloading' 'and '...with ratings' help has some updated screenshots and talk about the new default services and parsing
- .
- database fixes:
- fixed a very slow database query that was involved with file search using unnamespaced tags when other search predicates had already limited the scope of search
- fixed a similar slow query when the 'bad' search predicate was 'namespace:anything', particularly when the namespace is a wildcard
- fixed the 'clear orphan tables' database maintenance routine. it had decayed due to bit rot and was deleting new repository processing tracking tables. the routine is now plugged directly into the new database modules system, and any module now can be upgraded to discover surplus service tables. the system has been updated to permit the detection and removal of duplicate tables in the wrong database file, and it also makes a statement if no orphan tables were found

I had a great week. As well as some little fixes and quality of life work, I improved support for ogv files (oggs with video), and completely reworked the 'presentation' options of importers, making the whole system faster and more user-friendly.

The release should be as normal tomorrow.

full list

- misc:
- the text on the animation scanbar is now center-vertically aligned and should look better on taller and thinner scanbars (issue #998)
- the scanbar now reports better frame number and current time for the mpv player when the current video is very short or has very few frames. screamer gifs should now report 2/2 frames if you scan to the right, not like 97/2
- fixed using the mpv player with an embed button (it previously was staying hidden even after embed was clicked) (issue #999)
- the 'search' submenu when you right-click on tags in certain locations now shows add/exclude namespace:anything if all the selected tags share the same namespace
- as an experiment that I think will be bulked out into proper shortcuts later, and maybe actual +/- buttons like you'll see on a booru, if you activate the 'selection tags' listbox (double-click or enter key) while ctrl is down, it now excludes the selected tags from the current query
- when you paste query texts into the edit subscription dialog, those queries already in the sub _and_ DEAD will now be revived (it does a 'check now' on them). the dialog reports this
- when editing subscriptions, the way it waits for the current subs to stop running is improved. it is now separate from the global 'pause subs' variable, so big delays here (e.g. waiting a long time to open the dialog, then hitting 'pause' on the network menu, which was secretly a logically messy unpause) should be less able to run into trouble
- watchers now sort DEAD and 404 separately when sorting the status column (previously they were sorted by their now-defunct 'next check time')
- I think I improved the speed of the new subscriptions guaranteed shutdown. I think I also fixed a shutdown hang on some lagging async jobs. there are a couple of reports of hanging shutdown, so let me know if this changes
- I moved the autocomplete options from 'gui pages' to 'search', and I brushed up the layout and tooltips there generally
- .
- file parsing:
- clip files with canvas size units in mm, cm, inches, or points are now parsed correctly! thanks to the user who helped here! turns out a point is 1/72 (two grossths :^)) of an inch
- clip animations now get the number of frames and duration of the first timeline!

- client api:
- /get_files/file_metadata now has an optional boolean parameter, 'hide_service_names_tags', default False, which will hide the old 'service_names_to...' tag Objects
- a unit test tests hide_service_names_tags
- client api help documentation now talks about hide_service_names_tags
- client api version is now 22
- .
- boring code refactoring and cleanup:
- tl;dr: about 60KB total code moved out of client database!
- moved most combined sibling+parent database code to a new 'tag display' module
- moved autocomplete counts cache database code to a new 'mappings counts' module, and refactored a whole lot of of misc old a/c table creation and reference code into that module, cleaning things up
- the 'mappings counts' module is plugged into new repair code and on error repopulates itself as efficiently as regen code currently allows
- moved tag cache code to a new 'tag search' module, and similar related decoupling refactoring
- the 'tag search' module is plugged into new repair code and on error repopulates itself very efficiently
- the 'local tags cache' module can now regenerate itself on boot
- regenerating the local tags cache now works a little faster and takes less memory
- I _believe_ client.caches.db can now regenerate all of itself automatically, with no subsequent user actions needed
- the boot database repair notifications have some quality of life improvements. modules now say whether they think they can recover everything, and there is more guidance on what to do in the different situations

I had a good week. I fixed some bugs (including the new mpv audio file playback issue), reduced CPU load during heavy import sessions, improved network redirect support in the downloader, added new-user-friendly services to new clients, and massively sped up some complicated file searches that include unnamespaced tags.

The release should be as normal tomorrow.

- misc:
- right-clicking a selection tag and choosing 'select->files with x tag' now obeys the current tag domain (previously, it forced 'all known tags'). so, if you want to quickly select just the files with 'samus aran' on 'my tags', it should be doable
- the new 'partial' download resumer system will now tolerate two successive empty chunks before throwing a 'this chunk was empty' error. it seems some servers will randomly give an empty chunk at times during 206 transfers
- cancelling the slideshow custom time dialog no longer raises an error
- after the build boot problem last week, updated the opencv version in requirements.txts--we are now officially >=4.0.0, <=4.5.3.56. it looks like pyinstaller needs a patch for 4.5.4.58 to work, so we'll wait for that. I am improving my weekly test routine to try to catch this in future
- also, the windows build no longer includes two copies of an opencv dll. turns out PyInstaller finds this dll ok now (putting it in another location) and it doesn't need to be explicitly added during build
- added a new help file to the db dir for users who experience crashes as soon as they load videos in mpv when using WASAPI or ASIO drivers. thanks greatly for the user who figured out the mpv.conf solution here (issue #973)

next week

More small work like this, and I would like to do some behind the scenes database code cleanup too.

I had a great week. As well as a heap of good background cleanup work in the database, I fixed several bugs, improved .clip and video file metadata parsing, added some new quick tag search actions to the taglist, and cleaned up some bad animation display.

The release should be as normal tomorrow.

full list

- main highlights:
- to help debugging from screenshots etc..., the client now puts its version name on every window title, like 'review services - hydrus client 459'. (issue #447)
- the 'main gui title' option is reset and replaced with 'application display name' this week. it now alters the 'hydrus client' part on every window title. the actual 'main gui title' is now "main" lmao
- wrote a new help document, 'help my media files are broke' in the db directory. this collects the different recovery routines I have developed while helping users after drive failure or other problems cause many missing files or a desynced database and file storage structure. I will be pointing people to this in future, please feel free to do the same
- two new file maintenance jobs are added: for 'presence' and 'integrity' checks, you can now do 'if has URL, then try to redownload, else remove record'. this tidily combines the two more specific jobs that are commonly run after a hard drive problem. the 'presence' version is now the default selected job and recommended for most simple situations
- a new easy-select button on the review file maintenance panel lets you select all media files
- I put some more time into the new duplicate filter zoom locking calculation. thank you to users who sent in examples of my code not working well--I have scaled back what it tried to do. now it will tend to heigh-lock for landscape images and width-lock for everything else _unless_ you are currently viewing the default zoom and that roughly fills the canvas on a dimension and doing the default lock would cause the next image to spill over the screen. the 'solution' here hence targets the 'watermark spilled over' problem more specifically and deals with all combinations of landscape/portrait A/B/canvas better. I'd still like to introduce some zoom locking options here for regular browsing, but pinning down what exactly is useful is trickier than I expected
- the edit tag import options panel now shows 'THIS CURRENTLY GETS NO TAGS' warning red text if it is non default and no tags are set to parse and there are no additional tags
- the status bar now shows '1,234 files in 20 collections' when you have just collections or just collections selected (previously, it wrongly said '1,234 collections') (issue #807)
- macOS clients will now show dialog-created menus in a debug dialog unless the new BUGFIX checkbox under 'gui' options page is unchecked. this _should_ help Big Sur users who are unable to interact with menus created in dialogs like manage tags or manage services. I threw this together, so let me know how this works for you! (issue #986, issue #858)
- the program now waits specifically for currently running subscriptions to stop work and save themselves before moving on with further shutdown tasks. hand in hand with this, subscriptions are now faster at stopping work on client exit, even when they have no popup message (through which some hackery dackery shutdown signals are sent otherwise) (issue #790)
- physically deleting thousands of files in one go should no longer lock up the file manager and other systems for ages--physical delete is now serialised and processed on a new threaded mainloop, so it doesn't matter how fast the requests come it, it will chunter at a polite speed and take breaks and should not choke other consumers and freeze up other 'things are great, you can start new work' status checks

- network job improvements:
- hydrus network jobs now try to resume incomplete responses (previously they just dumped out and tried again from the beginning). if a server provides less content than it said it would, or it explicitly gives us a partial response, we now resume at that point! should fix dowmnloading of longer videos on 8chan.moe
- hydrus network jobs now send a range header by default, letting servers return 206 (partial content) if they wish
- SSL errors (cert verification and similar) are now caught in the network engine separately to generic connection errors. they will not be reattempted, and the failure note will display specific error info
- refactored some response header parsing code, cleaning up how some variables are initialised
- greatly improved the job reattempt system, resetting variables more neatly
- improved some response range and content length calculations
- .
- smaller items, mostly bug fixes:
- fixed a recent typo bug that caused the edit url class dialog to always spawn with 'file url' type set. sorry, this was stupid! (issue #982)
- the edit url class dialog now sends the 'normalised' url as the example text for the api and referral string converter edit panels
- fixed the new advanced file deletion 'remember last' checkboxes in _options->files and trash_. they weren't hooked up right, sorry!
- fixed the tag menu's siblings submenu's copy command where it says 'ideal is "xxx" on: yyy'. despite the correct label, this was sometimes copying a different service's ideal (issue #855)
- fixed the 'media zooms' text input under _options->media_ not turning off the 'red' invalid mode once its text is again valid
- when you cancel the 'edit parser' dialog, it shouldn't say 'it looks like you made changes' so much when you didn't make any. the 'has changes' test now ignores some background test data updates that may have happened (issue #875)
- if a JSON parsing formula is given HTML, the 'cannot parse' error now tries to detect this and present a better error text (issue #888)
- I _think_ I fixed a problem in the new bytes rendering calculation (where it goes 1018825 to "995KB") where on some unlucky edge-case numbers it could non-determinitively choose different sig figs (e.g. flipping between 994.9KB and 995KB)

I had an ok week. I fixed some bugs, updated some Client API commands, and improved file parsing for apngs and clip files.

The release should be as normal tomorrow.

full list

- quality of life:
- under _options->files and trash_, you can now govern whether the advanced file deletion dialog remembers action and reason. being able to save action (trash, physical delete, vs physical delete and clear history) is new, default off for now, and won't _always_ save (if you are currently set to trash, but the next dialog doesn't have trash as an option, then it won't overwrite to physical delete). if you try it out, let me know how you like it
- a new option under 'network->pause' now lets you always boot the client with paused network traffic
- the main file import object now stores primary urls (such as post and file url) separately from source url (which is produced by many parsers and typically refers to another website). a new checkbox in 'file import options' (when in advanced mode) now allows you to not associate primary urls separately to source urls (which is useful in some one-time technical jobs that talk to some unusual proxy etc...)
- the new import object right-click menu that shows urls now separates primary and source urls, and also shows any referral url
- when you flip between two images in the dupe filter, the zoom preservation calculation, which previously only locked to the same width, now tries to choose width or height based on the relative ratios of the two images to keep both images completely in view on a canvas zoom start. it should ensure that lower watermark banners stay in view and don't accidentally spill over the bottom of your monitor
- moved popup toaster options from 'gui' options page to the new 'popup' page
- added options for whether the popup toaster should update while the client is minimised and while the mouse is on a different monitor than the main gui window. these options now default to false, so if you have any trouble, please try turning them back on
- a new shortcut action in the 'global' set now flips profile mode on and off. please note for now 'global' only works on main gui and media viewer--I will add a hook to every window in the future!
- .
- bug fixes:
- you now cannot start an 'upload pending' job for a service more than once at a time. the menu is now disabled for each service while uploading is happening
- fixed a bug in media load where if the file was not in a specific domain (i.e. somewhere in all known files), its tags would not show implied parents. for non-specific files, this calculation happens on the fly, and previously it was only doing siblings
- fixed a bug from the somewhat recent file deletion update that meant many files' custom deletion reasons were being overwritten to 'No reason given' when trash was clearing. I am sorry for the inconvenience!
- fixed an issue with parsing 'string' from html 'script' tags (and perhaps some other 'meta' tag types) on recent versions of the built hydrus release. this should fix the newgrounds gallery parser

- boring cleanup:
- cleaned some of the positioning code that places icons and text on thumbnails, removing hardcoded numbers and letting custom icons position better
- cleaned some import url tracking, checking, and association code
- refactored profile mode and query planner mode switching up out of UI code into the controller
- added a hefty unit test to test that siblings and parents are transitively applied to mappings correctly for files outside and inside specific file services, and for tag sync and the normal tag pipeline
- refactored some database file maintenance code to decouple the queue from the worker

next week

I will try to focus entirely on the bug reports on the github issue tracker.

Hey, unfortunately I am super sick. I didn't get much work done, so I will put the release off a week.

469 should be next week, on the 27th!

I had a great week. I fixed many different bugs and improved the network code to better handle and resume interrupted downloads.

The release should be as normal tomorrow.

full list

- smoother menubar updates:
- improved the way the menubar menus update. rather than generating a whole new (e.g. 'pages') menu and replacing the existing out of date one, now there is a static menu skeleton that has subsections or labels updated in place. this means fewer objects changing, less flicker/jank, and should allow you to upload pending even if you have, say, a bunch of subscriptions running
- .
- misc:
- thanks to a user's help, the filetype parser now detects pngs (this mostly happens during import) much faster! the problem previously was determining if a png is actually an apng--figuring out if they are truly apngs is now done with very fast file header scanning, rather than the previous method that booted ffmpeg. this brings filetype parse time for pngs down from 50-150ms to 1-2ms
- getting apng metadata is also now faster. num_frames is now pulled from the file header, it no longer has to be manually counted by ffmpeg
- clicking the session weight item in the 'pages' menu now gives you more detailed info on your session weight, including on currently closed pages in the undo list
- stripped out a lot of ancient wx-era safety code that stops the client from doing certain UI work while it is minimised or minimised to tray. also brushed up some ugly update routines for menus refresh and modal message presentation that could lead to a pile-up of updates as soon as the client was unminimised, causing lag or freezes. with luck, the client should be better about restoring itself from minimised to system tray. if you minimise to tray, feedback on how this works out for you would be appreciated
- when a network job stalls with the 'this domain had some errors recently' message, the cog menu on the widget now allows you to 'scrub domain errors' and try again immediately
- if your search has system:limit, then any tag search you type in the autocomplete will now search the database, not your thumbnails. previously, the hack to enable this behaviour was to flip 'searching immediately' off. let's see if this new behaviour is ultimately confusing/annoying, I am mixed on it and think this subtle search option needs more thought and UI to make it more obvious and user friendly
- if you have autocomplete tag search typed, and results from thumbnails displayed, and you flip 'searching immediately' off, the search will now automatically update and give you full database numbers immediately
- .
- help:
- I moved 'searching with wildcards' from the advanced help to the 'more getting started with files' help here: https://hydrusnetwork.github.io/hydrus/help/getting_started_more_files.html
- I also wrote a more detailed description of what the autocomplete dropdown buttons do in that page

- boring code cleanup:
- cleaned some network job widget update calls
- improved some misc autocomplete search status tracking
- improved some account object permission checking and tests. accounts now never say they have permissions (e.g. if you click the 'see account permissions' button on review services) if they are banned or expired
- file and pages menus now uses the new update routine
- pending menu now uses the new update routine, with an emphasis on anti-jitter so you can interact while it is updating
- database, network, service, and undo menu now use newer async update code and also use the new update routine
- cleaned up help and tags menu init code
- the signal that causes the pending menu to update is now only sent on tag changes if the tag service is a repository (previously, local-only updates were janking this for no reason)
- the pending menu now updates its sibling/parent numbers when repository processing causes a clever row change to stuff you have pending
- also, some menubar items that only show when in advanced mode now update their visibility when advanced mode is flipped on or off
- misc menubar code cleanup and improvements

next week

I've got many little jobs in my immediate todo list, so I'd like to keep working like this. A grab bag of misc items, and some github bug reports.

I had a good week. I fixed a variety of bugs and added several small options to improve quality of life.

The release should be as normal tomorrow.

full list

- misc:
- the client no longer regularly commits a full garbage collection during memory maintenance. this debug-tier operation can take up to 15s on very large clients, resulting in awful lag. various instances of forcing it after big operations complete (e.g. to encourage post-subscription memory cleanup), are now replaced with regular pauses to allow python to clean itself more granularly. this may result in temporary memory bloat for some very subscription-heavy clients, so feedback would be appreciated
- right-clicking on a single url import item in a 'file log' now shows you all the known hashes, parsed urls, and parsed tags for that item. I hope this will help debug some weird problems!
- all multi-column lists across the program now convert an enter/return key press into an 'activate' command, as if you had double-clicked. this should make it easier to, for instance, highlight a downloader or shift/ctrl select a bunch of sibling rows and mass-delete (issue #933)
- the subscription gap filler button now propagates file import options and tag import options from the subscription to the downloader it creates (issue #910)
- a new 'mpv report mode' now prints a huge amount of mpv debug information to the hydrus log when activated
- improved how mpv prints log messages to the hydrus log, including immediate log flushing
- fixed a bug that meant the hydrus server was not saving custom update period or anonymisation period for next boot. thank you for the reports, and sorry for the trouble! (issue #976)
- cleaned up some database savepoint handling after a serious transaction error occurs
- the client api now ignores any parameter with a value of null, as if it were not there, rather than moaning about invalid datatypes (issue #922)
- .
- url classes:
- the edit url class dialog is now broken into two notebook pages--'match rules', which strictly covers how to recognise a url, and 'options', which handles url storage, conversion, and normalisation
- url classes can now support single-value parameters (a parameter with just a value, not a key/value pair). if turned on, then at least one single-value parameter is required to match the url, and multiple are permitted. a checkbox in the dialog turns this on and a string match lets you determine if the url class matches the received single value params
- added unit tests to test the new single-value parameter matching

I had a good week. There's more UI anti-jank work, with a smoother menubar and better minimise-to-tray support, faster png file import, and some quality of life.

The release should be as normal tomorrow.

database repair

tl;dr: Boot checks are better, you don't have to do anything.

I have spent most of my 'cleanup' time this year breaking the client database into smaller modules. These modules are now responsible for their own repair (which happens on boot), and they do much of it in a neater automatic way, using the same set of definitions it uses to initialise on first start to check whether an existing database is missing anything now. Most of my old hardcoded 'oh, you are missing this table' code is now simplified and modularised, works more reliably, and will be much easier to maintain in future.

You may get a popup when you boot saying you are missing one or more indices. There are several legacy reasons why this might be the case, but most often it will be because you once recovered from former hard drive damage and a clone/recover was unable to rescue an index. Whatever the case, these can now be regenerated automatically (they store duplicated data, so can be regenerated 100%), so please let it run. You might even notice PTR processing working faster afterwards.

Many tables can also be regenerated, and some can be completely repopulated. It is now possible (though not a great idea!) to start a client without a client.caches.db or client.mappings.db file and have it recover to a bootable state. Error presentation around this is improved, with instructions on what to do next in the case of critical problems.

I will continue modularising more of the database and fleshing out this repair code to cover more of the schema!

full list

- misc:
- many of the simple system predicates (width, size, duration, etc..) now support the '≠' (not equal) operator! searches only support one of these at once for now (e.g. you can't say height != 640 AND height != 1080--it'll pick one of these pseudorandomly)
- the watcher page list right-click menu now has 'copy subjects', which copies the selected watchers' 'subject' texts to clipboard

- ui freezes:
- session pages can now detect if they have had no saveable changes since a certain time. they use this ability to skip redundant session save CPU time for pages with no changes since the last session save
- for now, since the smallest atom of the session system is a whole page, gallery and watcher pages can only save time if _every_ downloader in the page has had no changes, so in their case this optimisation mostly only applies to completely finished/paused pages. it is still better to have several medium size downloader pages than one gigantic one
- a new database maintenance task ensures that optimisation cannot accidentally lose a page (from something like an unfortunate timing of a session save after many manual session deletes)
- the existing optimisation that skips 'last session' save on no changes now initialises its data as the 'last session' is first loaded (rather than on first save), meaning that if there are no changes while the client is open, no new 'last session's will be saved at all
- misc session save code cleanup
- .
- database repair, mostly boring:
- a client can now boot with client.caches.db missing and will rebuild that file. almost all of its tables are now able to automatically repopulate (issue #975)
- all the new modules I have been working on are now responsible for their own repair. this includes missing indices, missing tables, and table repopulation where possible. modules now know which of their tables are critical to a boot, what version each table and index was added, and now manage both initial and later-created service tables and indices
- essentially, all newer database repair code is now modularised rather than hardcoded. the repair and creation code actually now share the same table and index definitions. the code is more reliable, checkpoints its good work in case of later failure, and will be much easier to maintain and expand in future
- lots of module repair and initialisation code is refactored and generally given a full pass
- the core mappings cache regeneration routine now takes transaction checkpoints throughout its job to save progress and reduce journal size
- master definition critical error detection code is no longer hardcoded!
- mapping storage repair code is no longer hardcoded!
- similar files repair code is no longer hardcoded!
- parent or sibling cache repair repopulation is no longer hardcoded!

I had an ok week. I eliminated more UI lag, added downloader support for a special kind of URL, and fixed some more bugs.

Not sure if this is related to version 455 since I haven't used the site in awhile but I get this error when trying to add links from rule34.xxx about a missing login cookie and I don't see this type of cookie in my browser.

>>1142
Hey, thank you for this report. It may be that the default login script is now out of date. I will investigate this, and if it is an easy fix, I will roll out an update to that login. You may want to deactivate that login for now and maybe try Hydrus Companion to copy your browser login cookies over directly using the Client API:

https://gitgud.io/prkc/hydrus-companion