database repair

tl;dr: Boot checks are better, you don't have to do anything.

I have spent most of my 'cleanup' time this year breaking the client database into smaller modules. These modules are now responsible for their own repair (which happens on boot), and they do much of it in a neater automatic way, using the same set of definitions it uses to initialise on first start to check whether an existing database is missing anything now. Most of my old hardcoded 'oh, you are missing this table' code is now simplified and modularised, works more reliably, and will be much easier to maintain in future.

You may get a popup when you boot saying you are missing one or more indices. There are several legacy reasons why this might be the case, but most often it will be because you once recovered from former hard drive damage and a clone/recover was unable to rescue an index. Whatever the case, these can now be regenerated automatically (they store duplicated data, so can be regenerated 100%), so please let it run. You might even notice PTR processing working faster afterwards.

Many tables can also be regenerated, and some can be completely repopulated. It is now possible (though not a great idea!) to start a client without a client.caches.db or client.mappings.db file and have it recover to a bootable state. Error presentation around this is improved, with instructions on what to do next in the case of critical problems.

I will continue modularising more of the database and fleshing out this repair code to cover more of the schema!

full list

- misc:
- many of the simple system predicates (width, size, duration, etc..) now support the '≠' (not equal) operator! searches only support one of these at once for now (e.g. you can't say height != 640 AND height != 1080--it'll pick one of these pseudorandomly)
- the watcher page list right-click menu now has 'copy subjects', which copies the selected watchers' 'subject' texts to clipboard
- the advanced file deletion panel now remembers which reason you last used. it remembers if you selected the 'default' value up top versus specific reason text, and if you enter custom text, it remembers that for next time too
- the network job widget now shows the current URL as tooltip over the progress gauge, and under the cog menu, where clicking it copies it to clipboard
- the various menu labels across the program should now show ampersand (&) correctly (e.g. in URLs)
- the way byte sizes (like 21.7KB or 1.24GB) above 1KB are rendered to strings has been overhauled. they now generally show three significant figures. a new EXPERIMENTAL option in 'gui' options panel lets you change this, but only 2 or 3 are really helpful
- if a repository clears the message on your account, you no longer get a popup telling you 'hey, new message from server x: ...'
- the new ≠ system preds should be parseable (but be careful, likely buggy) using the client api's new system predicate parser, with '≠', '!=', 'is not', or 'isn't'
- cleaned up some old data presentation methods and improved how client specific options are patched in to base hydrus string conversion code

- ui freezes:
- session pages can now detect if they have had no saveable changes since a certain time. they use this ability to skip redundant session save CPU time for pages with no changes since the last session save
- for now, since the smallest atom of the session system is a whole page, gallery and watcher pages can only save time if _every_ downloader in the page has had no changes, so in their case this optimisation mostly only applies to completely finished/paused pages. it is still better to have several medium size downloader pages than one gigantic one
- a new database maintenance task ensures that optimisation cannot accidentally lose a page (from something like an unfortunate timing of a session save after many manual session deletes)
- the existing optimisation that skips 'last session' save on no changes now initialises its data as the 'last session' is first loaded (rather than on first save), meaning that if there are no changes while the client is open, no new 'last session's will be saved at all
- misc session save code cleanup
- .
- database repair, mostly boring:
- a client can now boot with client.caches.db missing and will rebuild that file. almost all of its tables are now able to automatically repopulate (issue #975)
- all the new modules I have been working on are now responsible for their own repair. this includes missing indices, missing tables, and table repopulation where possible. modules now know which of their tables are critical to a boot, what version each table and index was added, and now manage both initial and later-created service tables and indices
- essentially, all newer database repair code is now modularised rather than hardcoded. the repair and creation code actually now share the same table and index definitions. the code is more reliable, checkpoints its good work in case of later failure, and will be much easier to maintain and expand in future
- lots of module repair and initialisation code is refactored and generally given a full pass
- the core mappings cache regeneration routine now takes transaction checkpoints throughout its job to save progress and reduce journal size
- master definition critical error detection code is no longer hardcoded!
- mapping storage repair code is no longer hardcoded!
- similar files repair code is no longer hardcoded!
- parent or sibling cache repair repopulation is no longer hardcoded!
- the local hashes cache module can now repopulate itself during repair
- the notes fast search table can now repopulate itself during repair
- the similar files search tree table can now rebuild itself during repair

next week

I'm not really happy with my productivity over the past two weeks. This repair and session save worked bogged me down a bit, but I also didn't get as many hours in as I had planned. In any case, I will try to prioritise smaller bug fixes next week. Keep on pushing.

I had an ok week. I eliminated more UI lag, added downloader support for a special kind of URL, and fixed some more bugs.

Not sure if this is related to version 455 since I haven't used the site in awhile but I get this error when trying to add links from rule34.xxx about a missing login cookie and I don't see this type of cookie in my browser.

>>1142
Hey, thank you for this report. It may be that the default login script is now out of date. I will investigate this, and if it is an easy fix, I will roll out an update to that login. You may want to deactivate that login for now and maybe try Hydrus Companion to copy your browser login cookies over directly using the Client API:

https://gitgud.io/prkc/hydrus-companion