/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.

Boards | Catalog | Bottom

Check to confirm you're not a robot
Name
Email
Subject
Comment
Password
Drawing x size canvas
File(s)

Remember to follow the rules

Max file size: 350.00 MB

Max files: 5

Max message length: 4096


Version 388 Anonymous Board owner 03/11/2020 (Wed) 22:23:32 Id: 415127 [Preview] No. 561 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=X-vdjur459c [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v388/Hydrus.Network.388.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v388/Hydrus.Network.388.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v388/Hydrus.Network.388.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v388/Hydrus.Network.388.-.Linux.-.Executable.tar.gz
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v388.tar.gz

I had a great week. The client can now save and load searches.

favourite searches

Every tag autocomplete input text box that searches for files--the most obvious being the one on normal search pages--now has a star icon button beside it. Click this, and you get a menu to save your current search, manage your saved searches, or load up one that is saved!

Message too long. Click here to view full text.

13 posts and 1 image omitted.


Anonymous 04/01/2020 (Wed) 04:25:11 Id: d89d54 [Preview] No.605 del
Is using hydrus for managing a collection that includes doujins a terrible idea?

Or would I simply have to script auto-tagging of those doujins with proper author & title + page index? (Of course I have that info in .json files made from extracted metadata of the source sites)


Anonymous 04/01/2020 (Wed) 04:26:46 Id: d89d54 [Preview] No.606 del
>>605
I suppose I'd need to compile a metadata/hash associative DB before importing anything.


Anonymous Board owner 04/05/2020 (Sun) 19:14:29 Id: af0af2 [Preview] No.617 del
>>605
>>606
It is a mix. It works, but I don't like how hydrus manages paged/chaptered media at the moment.

If you have nice cbr/cbz files, I'd say don't import their files to hydrus for now, and just use another comic reader software. I expect to have actual cbr/cbz support in the next two years, with hydrus able to go into the archive and read through the separate file pages in the media viewer like any other reader. I am now generally of the opinion that treating chapters/volumes as single files is better for comics than treating pages as separate files.

That said, if you can get nice page/chapter/series tags to your files using filename-import-parsing, hydrus does work, and it is usually easy to reverse the operation as you can export the files again with nice filenames. It just takes that extra effort, and you have to get the page tags correct or it is a clusterfuck to try to fix atm.

Maybe you can try importing one chapter or volume, without deleting the original file(s), and seeing how you like to read that in hydrus, with the different sort/collect choices. If it works better than a proper comic reader, then yeah I'd say do more, otherwise hold off a bit.


Anonymous Board owner 04/05/2020 (Sun) 19:33:23 Id: af0af2 [Preview] No.618 del
>>595
Hmm, that's odd. I was writing out the rules here, which involve calculating the 'cull' time period, and realised your fixed scheduling may be producing a weird one. Normally, as long as there are more than 250 urls in a query's file import cache, then urls that have a source time more than twice the sub's death period are deleted. The death period is the '90' part in 'sub query is dead if less than 1 url per 90 days', so typically 180 days.

If your death period is set to 360 days or something, then even if there are a thousand import objects in the sub, they may not be being culled down to 250 because they are probably all still less than two years old and hence relatively new according to the sub's timer and could be involved in future timing decisions.

This of course matters less for you with the static checking timing. It doesn't matter if 17 files came in in the past m days or if 3 did, you are still going to check every n days.

I think I'll do two things here--make the cull period calculation a bit cleverer than just the safe 'twice' the death period, and also cap the cull period to 90 days or so when the sub has static check timing as you do.

If your subs have very high death periods, even if you have the '1' part of '1 url per n days' set to '0' to remove the dead check altogether, I recommend you reduce it to 90 or 180 days, especially for subs that reliably give new files.


Anonymous 01/16/2021 (Sat) 21:36:33 Id: 15a72f [Preview] No.907 del
>>618

I never followed up on this change. It is working now. All my old queries were trimmed to (if my memory serves me correctly) 250 successful URLs plus erroneous.

Thanks!



Version 424 Anonymous Board owner 01/07/2021 (Thu) 02:06:19 Id: 75b7b3 [Preview] No. 890 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=YnU_j_ZA-tc [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v424/Hydrus.Network.424.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v424/Hydrus.Network.424.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v424/Hydrus.Network.424.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v424/Hydrus.Network.424.-.Linux.-.Executable.tar.gz

I had a good week. There are some quality of life improvements and faster tag search across the board.

The update will take some time this week to update a cache. If you do not sync to the PTR, it will be just a few seconds. If you sync to the PTR, expect about 5-15 minutes.

faster tag search

In the second half of 2020, I tried several times to tune the database for different sorts of wildcard tag search, which is used in all autocomplete lookups and many file searches. I was sometimes able to get small clients always running well, or complicated large systems running, well, but I failed to get it good for all situations with code alone--the structure of the database tag lookup cache made the tuning difficult.

Message too long. Click here to view full text.

1 post omitted.


Anonymous Board owner 01/07/2021 (Thu) 02:12:06 Id: 75b7b3 [Preview] No.892 del
- other tag cache info:
- the 'tag text search cache' regeneration routine under the _database->regenerate_ menu is replaced with a service specific routine for the new cache
- on boot, if the client sees any of the new cache tables are missing, it notifies you and regenerates the affected subsection of the cache
- an old method of performing complex wildcard searches was using surplus data and has been eliminated. these searches are now also computationally cheaper beyond the other domain-based optimisations this week
- I have identified the next bottleneck in the tag search pipeline and have a plan to speed all the above up even further, which can all be done in code
- thanks to user feedback, I have also identified other wasteful overhead in tag processing. I'll keep working!
- while the planned 'all known tags' cache will be useful since most file searches are in this domain, it will be a bit of work, so I will first let this new lookup cache breathe for a bit. 'all known tags' will not be nearly as big as the 'all known files/combined file' caches that have hit us with so much CPU recently. I expect it to increase the client.caches.db size by about 5%
- unified all increments or decrements to autocomplete count caches, no matter the service domain, to one location
- unified how autocomplete counts are fetched across different service domains
- optimised specific and combined autocomplete count cache update overhead for new, existing, and deleted tags
- optimised display autocomplete count cache updates for tags with multiple siblings or parents
- optimised the 'local tags cache', which does fast tag text fetching for local files, when new tags or files are added/removed from the 'all local files' domain. this now occurs in the same unified autocomplete count update process. it now also caches pending tags that have no current count
- merged 'exact match' autocomplete tag searching code into generalised wildcard search
- misc autocomplete and other tag code cleanup and harmonisation
- ditched some old mass UNION queries that were not cancelling well


Anonymous Board owner 01/07/2021 (Thu) 02:13:29 Id: 75b7b3 [Preview] No.893 del
- the rest:
- when you paste queries into a sub, the summary 'these were/were not added' dialog now always appears, and if you paste empty whitespace, it now says so
- the manage siblings/parents dialogs now specify which services apply which siblings, whether they are fully synced, the current display tag sync maintenance settings, and ultimately whether you can expect changes to apply quickly after dialog ok
- when a text entry dialog comes with suggestion buttons, it now focuses the text box by default. sorry for the trouble here! (issue #765)
- updated a couple petition reason suggestions in manage tags and parents
- added a shortcut to 'main window' to refresh _manage tags'_ related tags suggestions with 'thorough' duration. in future, these dialog-specific actions will be moved out of 'main window', these have just been a 'temporary' patch
- updated the 'running from source' and 'install' help with some new numbers and info about mpv, and updated the 'server' help with a document helpfully provided by a user explaining that the server does not do what many new users think
- sped up 'has tags' file searches in certain situations, mostly when there are few if any other search predicates
- the default e621 parser now pulls meta tags, thank you to a user for providing this
- the default nitter timeline url classes are updated, thank you to a user for providing this
- the new little hook that takes 'file:///' off of paths pasted into the filename tagging path text now also normalises the path, so if you are on Windows, the URI's slashes will be Windows-corrected to backlashes. it also now removes wrapping quotes
- the hydrus logger again correctly restores stdout and stderr after it is closed on program exit (this was disabled for some reason, but fingers crossed it seems fine now!)
- an issue where automatically started duplicate potentials file search could not cancel when shutdown 'stop work' button was clicked or where idle maintenance mode turned off should be fixed
- the shutdown maintenance work for the first client shutdown now has a little text saying it is just some quick initialisation work
- for hopefully the last and completely final time, I think I fixed the invalid tag repair function for certain sorts of tags applied to currently local files
- improved the way a job thread was pulling new jobs (issue #750)

Message too long. Click here to view full text.



Anonymous 01/07/2021 (Thu) 02:18:52 Id: adbf98 [Preview] No.894 del
(89.86 KB 1280x720 maxresdefault (4).jpg)
>>890
Upset that I'm angry again Modeus? Every fucking time, all alone, literal God tier effort, but you and the rest cannot accept less than the gold. It won't work if you're abusive. You've lost how many good outcomes looking for a constructed ideal instead of ....just, consider the people wiped fro m your extended family tree because of this. I went a full day without this exhausting rage in my chest. Fucks wrong with you? You're making me too jumpy... just... I dunno...don't fuck my life up worse.


Anonymous Board owner 01/08/2021 (Fri) 20:25:26 Id: 3f26ed [Preview] No.895 del
While the new search cache is working great on normal search pages, on 'all known files'/'PTR' domain, which you usually see in 'manage tags' for the PTR, performance is bad. Often 2-6 seconds to fetch results. I regret this, and I am sorry for the inconvenience. I have identified the slowdown to one link in the chain, and will work on it next week.


Release Tomorrow! Anonymous Board owner 01/13/2021 (Wed) 06:32:03 Id: c867bb [Preview] No.896 del
I had a good week. I was not able to fit much interesting fun stuff in, but I fixed the slow tag autocomplete search in the manage tags dialog, sped up tag processing, fixed some 'ghost' tag bugs, and reduced some wasted CPU in the network engine. Overall, the client should be a bit neater and faster in 425.

The release should be as normal tomorrow.



Version 423 Anonymous Board owner 12/23/2020 (Wed) 23:40:02 Id: 2a5e2c [Preview] No. 882 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=SvsHVu3xt6A [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v423/Hydrus.Network.423.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v423/Hydrus.Network.423.-.Linux.-.Executable.tar.gz
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v423/Hydrus.Network.423.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v423/Hydrus.Network.423.-.Linux.-.Executable.tar.gz

๐•ธ๐–Š๐–—๐–—๐–ž ๐•ฎ๐–๐–—๐–Ž๐–˜๐–™๐–’๐–†๐–˜!

I had a good week making some small fixes and improvements to finish up the year. This is the last release of the year. There is a large poll on what 'big thing' to work on next:

poll

Here is the poll on what large work to go for next:

Message too long. Click here to view full text.



Anonymous Board owner 12/23/2020 (Wed) 23:41:11 Id: 2a5e2c [Preview] No.883 del
work is all misc this week

The settings for autocomplete auto-search and character threshold have been moved from options->speed and memory to tags->manage tag display and search. They are also now service-specific! So, you can now set 'all known tags' or 'PTR' to autocomplete after, say, 5 characters but your 'my tags' to always get all results.

I worked on the recently re-activated '*' and 'namespace:*' advanced searches, which were running slow in the new pipeline on larger clients. I have improved some situations, and I think I have reduced the worst case scenario, but some large clients will still have trouble. I am not happy here, nor with other namespace- and subtag- lookup speed across the program, so I have made a plan to make some clever new indices here. As I have simplified and cleaned much of the tag logic over the past year, I now see the next holes that I should fill.

The new siblings and parents tag right-click menu proved a little tall after all, so I have reworked it to group items by all the services that share them. It is denser to look at, but I think it'll highlight unusual exceptions when you are trying to fix application. The 'this is the ideal' line is also removed.

You can now sort a row of pages by name from their right-click menu.

Advanced users: The parser edit UI's test panel now shows more data, and it formats JSON prettily. It isn't anything close to what a browser's developer mode will give you of course, but it is nicer. That panel also detects if you hit a jpeg or something and says so, rather than dumping garbage to the test panel or just throwing an error.

full list

- tag autocomplete searches:
- the 'fetch results as you type' and 'do-not-autocomplete character threshold' options are moved from _options->speed and memory_ to _tags->manage tag display and search_. they are now service specific!
- getting the raw '*' autocomplete is now hugely faster when both file and tag domains are specific (i.e. not 'all known xxx')

Message too long. Click here to view full text.



Anonymous Board owner 12/23/2020 (Wed) 23:42:17 Id: 2a5e2c [Preview] No.884 del
- the rest:
- the siblings & parents tag menu, which proved a little tall after all, is now compressed to group siblings, parents, and children by the shared services that hold them. it takes less space, and odd exceptions should be easy to spot
- this menu also no longer has the 'this is the ideal tag' line
- added 'sort pages by name a-z/z-a' to page right-click menu and tucked the sorts into a submenu
- the parsing test panel now shows up to 64KB of what you pulled (previously 1KB)
- the parsing test panel now shows json in prettier indented form
- when the parsing test panel is told to fetch a URL that is neither HTML or JSON, this is now caught more safely and tested against permitted file types. if it was really a jpeg, it will now say 'looks like a jpeg' and disable parse testing. if the data type could not be figured out, it tries to throw the mess into view and permits parse testing, in case this is some weird javascript or something that you'll want to pre-parse convert
- the dreaded null-character is now eliminated in all cases when text is decoded from a site, even if the site has invalid unicode or no encoding can be found (e.g. if it is truly a jpeg or something and we just end up wanting to throw a preview of that mess into UI)
- the 'enter example path here' input on import folders' filename tagging options edit panel now uses placeholder text and auto-removes 'file:///' URL prefixes (e.g. if your paste happens to add them)
- the 'fix invalid tags' routine now updates the tag row in the local tags cache, so users who found some broken tags were not updating should now be sorted
- added --db_cache_size launch parameter, and added some text to the launch_parameters help about it. by default, hydrus permits 200MB per file, which means a megaclient under persistent heavy load might want 800MB. users with megamemory but slow drives might want to play with this, let me know what you find
- updated to cloudscraper 1.2.50

next week

I am now on vacation. I have some family Christmas things going on, and otherwise I am looking forward to grinding away at lategame X3 TC with some Wagner. I hope you can have a good break as well. I'll be back to normal in the new year, and 424 should be out on the 6th of January.


Release Tomorrow! Anonymous Board owner 01/06/2021 (Wed) 05:40:29 Id: b24a0c [Preview] No.888 del
I had a good work week. I did a variety of small fixes and quality of life improvements, and I finished redesigning the part of the database that does wildcard tag searches. Autocomplete lookups and file searches that rely on complicated tags are running faster across the board, and the old design's sudden lag spikes (e.g. with namespace:*anything* searches) are entirely eliminated.

The release should be as normal tomorrow. There will be some database work on update. I will have a better idea tomorrow, but I estimate users who sync to the PTR with an SSD can expect it to take 5-15 minutes.


Qwertyius 01/06/2021 (Wed) 05:45:31 Id: b16059 [Preview] No.889 del
(20.26 KB 474x352 (Drink Prunes).jpeg)
>>888
Why do Joss Whedon's cronies all post here? Doyle really was the only good man.



Version 422 Anonymous Board owner 12/16/2020 (Wed) 23:38:46 Id: bcfea2 [Preview] No. 876 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=AHBJ68icJ_4 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v422/Hydrus.Network.422.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v422/Hydrus.Network.422.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v422/Hydrus.Network.422.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v422/Hydrus.Network.422.-.Linux.-.Executable.tar.gz

๐ŸŽ‰๐ŸŽ‰ It was hydrus's birthday this week! ๐ŸŽ‰๐ŸŽ‰

I had a great week. I mostly fixed bugs and improved quality of life.

tags

It looks like when I optimised tag autocomplete around v419, I accidentally broke the advanced 'character:*'-style lookups (which you can enable under tags->manage tag display and search. I regret this is not the first time these clever queries have been broken by accident. I have fixed them this week and added several sets of unit tests to ensure I do not repeat this mistake.

Message too long. Click here to view full text.



Anonymous Board owner 12/16/2020 (Wed) 23:39:13 Id: bcfea2 [Preview] No.877 del
misc

Some websites have a 'redirect' optimisation where if a gallery page has only one file, it moves you straight to the post page for that file. This has been a problem for hydrus for some time, and particularly affected users who were doing md5: queries on certain sites, but I believe the downloader engine can now handle it correctly, forwarding the redirect URL to the file queue. This is working on some slightly shakey tech that I want to improve more in future, but let me know how you get on with it.

The UPnPc executables (miniupnp, here https://miniupnp.tuxfamily.org/) are no longer bundled in the 'bin' directory. These files were a common cause of anti-virus false positives every few months, and are only used by a few advanced users to set up servers and hit network->data->manage upnp, so I have decided that new users will have to install it themselves going forward. Trying to perform a UPnP operation when the exe cannot be found now gives a popup message talking about the situation and pointing to the new readme in the bin directory.

After working with a user, it seems that some clients may not have certain indices that speed up sibling and parent lookups. I am not totally sure if this was due to hard drive damage or broken update logic, but the database now looks for and heals this problem on every boot.

parsing (advanced)

String converters can now encode or decode by 'unicode escape characters' ('\u0394'-to-'ฮ”') and 'html entities' ('&'-to-'&'). Also, when you tell a json formula to fetch 'json' rather than 'string', it no longer escapes unicode.

The hydrus downloader system no longer needs the borked 'bytes' decode for a 'file hash' content parser! These content parsers now have a 'hex'/'base64' dropdown in their UI, and you just deliver that string. This ugly situation was a legacy artifact of python2, now finally cleared up. Existing string converters now treat 'hex' or 'base64' decode steps as a no-op, and existing 'file hash' content parsers should update correctly to 'hex' or 'base64' based on what their string converters were doing previously. The help is updated to reflect this. hex/base64 encodes are still in as they are used for file lookup script hash initialisation, but they will likely get similar treatment in future.


Anonymous 12/16/2020 (Wed) 23:42:44 Id: bcfea2 [Preview] No.878 del
birthday

๐ŸŽ‰๐ŸŽ‰๐ŸŽ‰๐ŸŽ‰๐ŸŽ‰

On December 14th, 2011, the first non-experimental beta of hydrus was released. This week marks nine years. It has been a lot of work and a lot of fun.

Looking back on 2020, we converted a regularly buggy and crashy new Qt build to something much faster and nicer than we ever had with wx. Along with that came mpv and smooth video and finally audio playing out of the client. The PTR grew to a billion mappings(!), and with that came many rounds of database optimisation, speeding up many complicated tag and file searches. You can now save and load those searches, and most recently, search predicates are now editable in-place. Siblings and parents were updated to completely undoable virtual systems, resulting in much faster boot time and thumbnail load and greatly improved tag relationship logic. Subscriptions were broken into smaller objects, meaning they load and edit much faster, and several CPU-heavy routines no longer interrupt or judder browsing. And the Client API expanded to allow browsing applications and easier login solutions for difficult sites.

There are still a couple thousand things I would like to do, so I hope to keep going into 2021. I deeply appreciate the feedback, help, and support over the years. Thank you!

If you would like to further support my work and are in a position to do so, my simple no-reward Patreon is here: https://www.patreon.com/hydrus_dev

full list

- advanced tags:
- fixed the search code for various 'total' autocomplete searches like '*' and 'namespace:*', which were broken around v419's optimised regular tag lookups. these search types also have a round of their own search optimisations and improved cancel latency. I am sorry for the trouble here
- expanded the database autocomplete fetch unit tests to handle these total lookups so I do not accidentally kill them due to typo/ignorance again

Message too long. Click here to view full text.



Anonymous 12/16/2020 (Wed) 23:45:00 Id: bcfea2 [Preview] No.879 del
- misc:
- the thumbnail right-click 'copy/open known urls by url class' commands now exclude those urls that match a more specific url class (e.g. /post/123456 vs /post/123456/image.jpg)
- miniupnpc is no longer bundled in the official builds. this executable is only used by a few advanced users and was a regular cause of anti-virus false positives, so I have decided new users will have to install it manually going forward.
- the client now looks for miniupnpc in more places, including the system path. when missing, its error popups have better explanation, pointing users to a new readme in the bin directory
- UPnP errors now have more explanation for 'No IGD UPnP Device' errortext
- the database's boot-repair function now ensures indices are created for: non-sha256 hashes, sibling and parent lookups, storage tag cache, and display tag cache. some users may be missing indices here for unknown update logic or hard drive damage reasons, and this should speed them right back up. the boot-repair function now broadcasts 'checking database for faults' to the splash, which you will see if it needs some time to work
- the duplicates page once again correctly updates the potential pairs count in the 'filter' tab when potential search finishes or filtering finishes
- added the --boot_debug launch switch, which for now prints additional splash screen texts to the log
- the global pixmaps object is no longer initialised in client model boot, but now on first request
- fixed type of --db_synchronous_override launch parameter, which was throwing type errors
- updated the client file readwrite lock logic and brushed up its unit tests
- improved the error when the client database is asked for the id of an invalid tag that collapses to zero characters
- the qss stylesheet directory is now mapped to the static dir in a way that will follow static directory redirects
- .
- downloaders and parsing (advanced):
- started on better network redirection tech. if a post or gallery URL is 3XX redirected, hydrus now recognises this, and if the redirected url is the same type and parseable, the new url and parser are swapped in. if a gallery url is redirected to a non-gallery url, it will create a new file import object for that URL and say so in its gallery log note. this tentatively solves the 'booru redirects one-file gallery pages to post url' problem, but the whole thing is held together by prayer. I now have a plan to rejigger my pipelines to deal with this situation better, ultimately I will likely expose and log all redirects so we can always see better what is going on behind the scenes
- added 'unicode escape characters' and 'html entities' string converter encode/decode types. the former does '\u0394'-to-'ฮ”', and the latter does '&'-to-'&'

Message too long. Click here to view full text.



Anonymous 12/16/2020 (Wed) 23:47:40 Id: bcfea2 [Preview] No.880 del
- in the parsing system, decoding from 'hex' or 'base64' is no longer needed for a 'file hash' content type. these string conversions are now no-ops and can be deleted. they converted to a non-string type, an artifact of the old way python 2 used to handle unicode, and were a sore thumb for a long time in the python 3 parsing system. 'file hash' content types now have a 'hex'/'base64' dropdown, and do decoding to raw bytes at a layer above string parsing. on update, existing file hash content parsers will default to hex and attempt to figure out if they were a base64 (however if the hex fails, base64 will be attempted as well anyway, so it is not critically important here if this update detection is imperfect). the 'hex' and 'base64' _encode_ types remain as they are still used in file lookup script hash initialisation, but they will likely be replaced similarly in future. hex or base64 conversion will return in a purely string-based form as technically needed in future
- updated the make-a-downloader help and some screenshots regarding the new hash decoding
- when the json parsing formula is told to get the 'json' of a parsed node, this no longer encodes unicode with escape characters (\u0394 etc...)
- duplicating or importing nested gallery url generators now refreshes all internal reference ids, which should reduce the liklihood of accidentally linking with related but differently named existing GUGs
- importing GUGs or NGUGs through Lain easy import does the same, ensuring the new objects 'seem' fresh to a client and should not incorrectly link up with renamed versions of related NGUGs or GUGs
- added unit tests for hex and base64 string converter encoding

next week

Last week of the year. I could not find time to do the network updates I wanted to this week, so that would be nice. Otherwise I will try and clean and fix little things before my week off over Christmas. The 'big thing to work on next' poll will go up next week with the 423 release posts.

Note:

It looks like 'namespace:*' searches, despite now working, may be running super slow or doing too much database work, in IRL PTR-syncing situations. I will look into it next week! If you are hit by this, please turn them off for now under tags->manage tag display and search.


Release Tomorrow! Anonymous Board owner 12/23/2020 (Wed) 03:28:32 Id: f29211 [Preview] No.881 del
I had a good week making some small fixes and improvements to finish up the year. Autocomplete works a bit faster, and some quality of life is improved.

The release should be as normal tomorrow. It will have the 'next big job' poll and be the last release for the year.



Version 421 Anonymous Board owner 12/09/2020 (Wed) 23:12:23 Id: 76cc77 [Preview] No. 868 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=m9r7VRNUyR8 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v421/Hydrus.Network.421.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v421/Hydrus.Network.421.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v421/Hydrus.Network.421.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v421/Hydrus.Network.421.-.Linux.-.Executable.tar.gz

I had a good week. I fixed some important bugs, and cleaned up some core access and maintenance code. This week's changelog gets pretty technical, which you can safely ignore. Database works betterโ„ข.

highlights

I fixed a problem importing files with 'only add tags that already exist' checked in tag import options. Sorry, this was a stupid typo. I added a unit test here to ensure it doesn't happen again.

When you search for potential duplicates from the duplicates page, this now happens in a non-interrupting popup in the bottom-right. You can keep searching and browsing while it works. All duplicate pages sync with each other better, too, and share more CPU work.

Message too long. Click here to view full text.



Anonymous Board owner 12/09/2020 (Wed) 23:12:47 Id: 76cc77 [Preview] No.869 del
I played around with core database modes this week. Default values are now tuned a little better for modern use. Unfortunately, the planned new mode for HDD users did not pan out due to architectural problems, but I think that HDD users will experience better write performance nonetheless. For those who want to experiment more, I have finally properly documented the different launch switches for hydrus here: https://hydrusnetwork.github.io/hydrus/help/launch_arguments.html

full list

- misc:
- thanks to a user's contribution, added the export 'filename pattern' to the discord drag and drop mode, under _options->gui_. this lets you auto-rename files in this export mode. I like how this works, but the overall pattern-based filename creation system really needs updating. let me know how this works for you, and I'll finally start the job to update filename generation
- fixed a bug when importing files with the 'only add tags that already exist' filter active, and added a unit test so this should not fail due to a typo again
- fixed an issue where ctrl-selecting on taglists was weird, where any mouse movement during ctrl+click would deselect. drag select and deselect can now only start when the drag crosses two indices
- prototyped a basic profile mode for the client api. it is insufficient (due to the asynchronous nature of twisted), but a start
- when the client catches an invalid tag with the new error handling code, when it shows you that bad tag in a popup, it now clips that to 24 characters (some PTR invalid tags are just a few hundred null characters in a row, wew lad)
- the client now recovers from a repository giving it a new invalid tag definition. all such tags are, for now, called 'invalid repository tag'. a plan to auto-hide these tags clientside and fully eliminate them serverside will come later
- the clipboard url watcher settings should stick a bit more firmly. those users who had trouble, please let me know how you get on
- fixed an issue editing duplicate action options when they contained tag or rating preferences for services that no longer exist
- I think I fixed some issues getting autocomplete results when you type the whole namespace before moving on to the subtag. when you hit 'namespace:', it should invalidate the old cache and start a new search
- when the database is given content updates for services that no longer exist, those content updates filtered out of UI update broadcast
- fixed an issue where URL status check could fail when the url map contained orphan hash_ids. proper orphan clearance will come later
- reduced overhead of tag filtering, which should improve display speed of taglist for very large pages

Message too long. Click here to view full text.



Anonymous Board owner 12/09/2020 (Wed) 23:14:28 Id: 76cc77 [Preview] No.870 del
- duplicates search improvements:
- potential duplicate search now works in the background! it will not interrupt you and is easily cancellable. duplicate search pages disable their search buttons while it is going
- the search distance in duplicates pages is now synchronised across all pages--when one updates, they all do
- all the updates to potential search maintenance numbers are now routed through one cached manager. updates here are repeated less often
- misc cleanup for duplicates page
- .
- database modes:
- a new 'program launch arguments' help page now talks about all the available command line switches, here: https://hydrusnetwork.github.io/hydrus/help/launch_arguments.html
- added the '--db_journal_mode' launch switch to set the SQLite journal mode. default is WAL, permitted values are also TRUNCATE, PERSIST, and MEMORY
- ensured --db_synchronous_override was hooked up correctly
- the old disk cache options under _speed and memory_ are removed, along with various deprecated disk cache load calls and code
- fixed some shutdown maintenance check logic that was saying 'I think a vacuum is due' when it wasn't actually true
- db_journal_mode, synchronous value, and no_db_temp_files is now shown in _help->about_
- .
- technical database nonsense:
- PERSIST is new to hydrus, and _may_ in future versions of SQLite be boost performance for HDD drives with larger databases (e.g. those that sync to the PTR), although unfortunately in our case (which uses multiple ATTACH databases), it seems current SQLite must ultimately treat this as DELETE, as here https://sqlite.org/atomiccommit.html#_clean_up_the_rollback_journals. damn
- hydrus now tries to always trim WAL (and PERSIST, if it worked) journal files down to 1GB after commits (which happen every 30 seconds), so giganto WALs should clear up promptly after big work is done

Message too long. Click here to view full text.



Anonymous Board owner 12/09/2020 (Wed) 23:15:01 Id: 76cc77 [Preview] No.871 del
- when in WAL journal mode, the hydrus db now cleans up any lingering checkpointing work every half hour
- after testing and feedback from users, the database is now default SQLite synchronous 1 (down from 2) when in WAL. the db is still consistent, so sudden program stop (crash, power cut) should not result in software-caused corruption, but the database may lose more than just the last 30 seconds of work. this speeds up tag processing in an SSD test environment by approx 33%
- the 'no_wal' (TRUNCATE) and 'db_memory_journaling' (MEMORY) launch switches remain valid but are now deprecated
- improved launch switch code generally
- boosted cache size for each of the four db files to ~200MB-this will likely become a launch argument in future, along with some other specific db values
- the client and server no longer disconnect from the db to check whether it is possible to vacuum databases

next week

There are two more work weeks in the year. I will continue working on small jobs and cleanup, but the main focus is now to update ancient network and service code to improve client-PTR sync and communications. I'll fix up buggy janitor tools and hopefully add some nice filters so clients and servers can better manage what they sync.

Since a 'next big work' poll is coming up on Christmas, on Saturday I'll post here and in discord about the current big list and ask for more things to add.


Anonymous Board owner 12/12/2020 (Sat) 23:17:19 Id: 14eaa4 [Preview] No.873 del
Hey, here are the preliminary 'big job poll' items. This poll will go up on December 23rd with my release posts, for the new year. You will be able to vote on multiple items. Please feel free to ask questions and request big jobs be added to it.
As many issues are being handled by the github issue team, here is a neat link that lets you review them in more detail:
https://github.com/hydrusnetwork/hydrus/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc+milestone%3A%22Major+jobs%22
The poll:
Add CBZ/CBR support (including framework for multi-page format)
Add Ugoira support (including optional mp4/webm conversion)
Add a carousel for the media viewer to see previous/next files in the current navigation thing
Add an incremental number tagging dialog for thumbnails (for adding page:n etc... to a sequence of files)
Add animated thumbnails for videos (animating on mouseover)
Add blurhash for quick thumbnail presentation
Add duplicate video detection
Add file note parsing to the download system
Add import any file support (giving it 'unknown' mime but preserving file extension)
Add more commands to the undo system
Add multiple local file services (which will enable true nsfw/sfw partition)
Add popular/favourite tag cloud controls for better 'browsing' search
Add ratings import/export, and add 'rating import options' to auto-rate imports

Message too long. Click here to view full text.



Release Tomorrow! Anonymous Board owner 12/16/2020 (Wed) 08:26:59 Id: c9ec18 [Preview] No.875 del
I had a great week. I mostly fixed bugs, including the recently broken 'namespace:*' autocomplete lookups, extended database recovery code, improved some quality of life, and added some advanced tools for downloader makers.

The release should be as normal tomorrow.



Version 420 Anonymous Board owner 12/02/2020 (Wed) 23:06:56 Id: 343d90 [Preview] No. 859 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=gUTOl_KGLEk [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v420/Hydrus.Network.420.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v420/Hydrus.Network.420.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v420/Hydrus.Network.420.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v420/Hydrus.Network.420.-.Linux.-.Executable.tar.gz

I had a great week fixing a whole bunch of bugs.

bugs

I fixed taglist drag-select, which was not moving the to-be selected indices down with the scroll. Sorry for the trouble here. You can now also ctrl+drag-select to deselect.

There was a bug in the new virtual siblings and parents lookup system that meant some grandparents and siblings were not appearing. For instance, for parents, 'samus aran' might have 'metroid', and 'metroid' would have 'nintendo', but 'samus aran' would not have 'nintendo'. Thanks to help from users, I was able to reproduce it and fix the problem. When you update, the client will spend a few seconds regenerating the lookups and finding the missing links. It will queue up a bit more work for the background display sync to do later on. In my test situation, the PTR went from 189,000 sync rows to 192,000.

Message too long. Click here to view full text.



Anonymous Board owner 12/02/2020 (Wed) 23:09:10 Id: 343d90 [Preview] No.860 del
Typing to get autocomplete results in search pages with thumbnails is now significantly faster and more responsive when 'searching immediately' is turned on. This routine gathers count-accurate results based on the thumbnails in front of you and then sends that data down to the database for sibling population. This suddenly got laggy with the new virtuals system, but I have improved how it schedules its searches and performs its tag lookups, and it should be much better now.

An unusual situation with a 'null character' tag that could not be displayed in the client should be better. This tag is detected, the rendering problem caught, and the user notified. The database routine that corrects bad tags now also fixes this tag and forces a tag presentation refresh once it is done.

speed

I spent a bunch of time in the siblings and parents system this week, and autocomplete and tags in general, running profiles on complex data and optimising various calls. I also sped up a critical new optimisation used across the program. Tag searches, autocomplete lookups, tag processing, tag display sync, and anything that runs frequently in small amounts should now be a bit faster. The only thing that may run a bit slower is tag display sync for very complicated tag parent situations, due to the more thorough logic in the fixes above. Thank you to users who helped with profiling here.

Tag display sync now only tries to run every 30 seconds (up from every ~3 seconds) when allowed to run in 'normal' time. It also takes breaks when it thinks a bunch of other work is going on. It was running hotter and faster than we needed, and it made the client too laggy, so I am pulling it back until I can make it more intelligent. I am also considering breaking the main display sync maintenance job into even more granular pieces. I will keep working.


Anonymous Board owner 12/02/2020 (Wed) 23:09:33 Id: 343d90 [Preview] No.861 del
full list

- misc:
- fixed the bad position indexing when drag-selecting taglists that were scrolled down. this also caused some weird selection when scrolled and clicks included a little mouse movement. sorry for the trouble!
- ctrl+drag-select now deselects!
- fetching tag autocomplete results when you have thumbnails and 'searching immediately' on, which has been way too slow recently, now cancels much faster. in some large page situations, it was adding multi-second lag on the first character-press. it also runs faster overall
- hydrus should now deal better with invalid tags that contain the null character (there it one we know about on the PTR, from a decode of botched Shift JIS, which could crash the client from too many errors during critical paint periods). when a tag like this turns up in a taglist, thumbnail, or canvas background, it now renders as an appropriate 'invalid tag' string, and a one-time 'woah, bad tag, run fix tags now' popup appears
- regular tag cleaning now looks for and removes null characters, so all new sources of these bad tags should now be eliminated
- _database->check and repair->fix invalid tags_ now fixes tags with the null character. it also fixes tags so broken that after cleaning they have no subtag left. it also now forces a full media tag reload when it is done for all media
- the 'regen storage mappings', 'regen display mappings', and repopulate from cache' database routines now have an additional step where you can order them only to work on one tag service, so regenning or repopulating local tags, which usually takes a couple seconds, doesn't need to wait two hours for the PTR to go as well
- added some menu help to the 'profile modes' debug menu, and gave 'reducing program lag' help page a pass
- fixed virtual display regeneration on service delete
- .
- parents and siblings:
- fixed situations where some grandparent and sibling relationships would not appear in the virtual system. it was a bug when certain links of a multi-part display 'chain' were updated at different times. when repopulating chain data, the sibling and parent update routines now correctly chase their complete chains both when wiping ideal data and repopulating from raw data, hitting all levels of the chain, ensuring to go back up and down chains when there are multiple grandsiblings/children/parents, and chasing parents where one or both members have better siblings. thank you to the several users who reported and helped figure out this problem, which was not simple to reproduce (issue #725)
- your ideal display data will be regenerated on update, which should not take more than a couple of seconds. it will likely correct some siblings and add some grandparents to be filled in by the siblings/parents sync. my PTR test environment went up from about 189,000 display rows to 192,000
- while sibling and parent lookup is more thorough (and hence more expensive), I also optimised many parts of lookup week. I believe tag display sync and tag processing will be much faster for tags with simple sibling and parent relationships, and slightly slower for tags with complex relationships and many instances to files on your drive. as always, let me know what sort of processing speeds and lag you get, and if you know how to make a db profile, please send them in when it gets bad


Anonymous Board owner 12/02/2020 (Wed) 23:12:04 Id: 343d90 [Preview] No.862 del
- when a 'write' autocomplete results list includes parent expansion rows (as in _manage tags_), parents now show duplicated and properly for all the tags that have them, including siblings and other children/grandchildren (previously, a parent label could only exist once in a list, which meant parents were ending up hanging off the last valid tag for which they applied)
- 'write' autocompletes now show results that exactly match the text entry, and all their siblings, when they do not have count but do have sibling or parent data. so, if you type in 'samus aran', and it has a sibling to 'character:samus aran', but 'samus aran' doesn't actually have count, you now get it and all siblings anyway. this may need tuning, but it solves a persistent and annoying lookup and quick-sibling-access problem in _manage tags_
- copying tags and their indented parents now removes the parent indent whitespace
- tag sync display now takes way longer breaks (now 30 seconds, was 2.5) between 'normal' background work periods. this thing was hammering people far harder than needed and could clog up db write/commit time and nobble UI responsitivity when big bumps collided
- the tag display maintenance manager now also tries to detect when many siblings or parents are streaming in (from a migration or a repository process with a heap of data), and pauses work while that continues
- greatly sped up mass imports of sibling and parent data, either from tag migration or big dialog pastes. what was 40 rows/s should now be about 1,000 rows/s
- fixed the database menu's 'regenerate tag parents lookup cache', which wasn't hooked up
- .
- boring changes:
- gave tag parents and siblings update, regen, and chain fetch a full pass, correcting bad queries to fix the above, fixing raw pair chain level navigation and parent-sibling idealisation, and optimised these lookups as well
- fixed some tag_id vs ideal_tag_id nomenclature (and related bugs) in tag parents cache
- optimised 'all known tags' autocomplete count fetching a little. tag autocomplete and search should be a bit faster in this domain
- reduced display sync pre-processing overhead by about 30% with a better random pair sampling routine
- reduced the overhead of my now very commonly used single integer memory table select optimisation. this now recycles tables after use, which reduces overhead about 50% in small number scenarios. all features of the database will enjoy this speed improvement, particularly small repetitive tag lookup jobs (such as the new display sync and repository tag processing)
- reduced overhead on some sibling chain lookup code
- reduced overhead on the sibling lookup used by manage tag dialog taglist
- reduced overhead on some parent chain lookup code

Message too long. Click here to view full text.



Release Tomorrow! Anonymous Board owner 12/09/2020 (Wed) 08:12:39 Id: 56cba1 [Preview] No.867 del
I had a good week. I fixed several important bugs, improved db write speeds for many sorts of clients, and made the 'potential duplicates' search run in the background, in a way that does not interrupt you.

The release should be as normal tomorrow.



Version 419 Anonymous Board owner 11/25/2020 (Wed) 23:07:50 Id: edb4f4 [Preview] No. 843 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=3xX5B62ENrM [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v419/Hydrus.Network.419.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v419/Hydrus.Network.419.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v419/Hydrus.Network.419.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v419/Hydrus.Network.419.-.Linux.-.Executable.tar.gz

I had a great week finishing editable system predicates and making some long-running search situations faster and snappier.

editable predicates

As a reminder, if you shift+double-click on the search terms (predicates) in your current search, you can now edit them! I also added an right-click 'edit' menu entry, if you do not like the shift+double-click.

OR and system:rating predicates are now editable. Also, 'invertible' predicates like inbox/archive or tag/-tag will now be stacked in the editable panel as buttons you can click to flip.

Message too long. Click here to view full text.

3 posts omitted.


Anonymous 11/28/2020 (Sat) 07:19:56 [Preview] No.847 del
Not sure if this is the correct place to mention it, but I've found that Hydrus creates really fragmented write-ahead log files since sqlite is demanding regular flushes to the drive every time even a little bit more data is processed. This flushing is normal behavior for sqlite, since it's intended to ensure writes go through. It's also fine on flash-based media because there's essentially no seek time, but for platter media it's absolute hell. A bandaid is to defrag the wal file while it's very large, or to grow a separate file, defrag it, shrink it, and then rename it to the wal file and copy over the content which gets the new wal file its own little section of the drive to play with. Sadly Hydrus resets this on a regular basis.

The root cause of the resetting fragmentation is that Hydrus' copy of sqlite doesn't have the "sqlitefcntlpersistwal" file option set, which means the wal file is deleted and recreated every time Hyrus starts. This means that the new wal file is shoved into whatever tiny 2kb hole the OS can find to start with, and since the wal is for file integrity, the file is flushed to disk without caching, resulting in several hundred 2kb-sized blocks scattered around the drive. It slowly fragments again until it's hitting hundreds of fragments, and the end result is incredibly high seek times on platter media.

Defragging a wal and the datbase and reserving space for growth allows for approximately 20k rows/s if it's the only thing running, whereas once the wal is deleted and recreated, it fragments and demands lots of seeks for sync writes (and for reading the wal to apply to the database) and drops to low single digit rows/s. I know the instructions say to use an SSD if syncing with external databases because HDD performance is so bad, but if it's reasonably easy to do, setting the wal to be persistent would significantly improve the performance for people who don't have an SSD.

Thanks for your time.

Reference:

https://www.sqlite.org/draft/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlpersistwal


Anonymous Board owner 11/28/2020 (Sat) 22:20:29 Id: 042b72 [Preview] No.848 del
>>847
Thank you, this is very useful information. I am not sure if I have the ability to set these opcodes from my python layer, but I will investigate this. If I can set a permanent WAL of a certain healthy size, it sounds like that would be a great improvement for HDD users. Can you say more how you achieved this to get 20k rows/s? Did you use a custom sqlite.dll, or were you able to set an environment variable to change my code to do this?

Another potential option here, although it is a bit crazy, and probably stupid if the user is on an older system with an HDD, may be to tell SQLite to use a memory journal, or memory temp file. I wonder if there is a similar problem as the fraggy WAL with fraggy temp files (SQLite creates files to store temp table data when it does larger transactions).

I have discovered that some PTR syncing is creating absolutely gigantic temporary files, sometimes >20GB. I want to investigate shrinking this significantly, although I worry that making more smaller commits will just result in more HDD work overall.

Also ultimately, I will be adding filters to the PTR in the coming months so users can just sync with a subset of PTR data (e.g. just creator/series/character tags), which I hope to produce several speed and storage improvements for users on slower drives.


Anonymous 11/29/2020 (Sun) 02:40:40 [Preview] No.850 del
>>848
I don't think setting the wal to a static size will be helpful or feasible, though I'm not familiar with the way sqlite handles the internals of the wal file. I suspect the wal file has a very specific internal format, and padding it out will either misalign it with the shared memory wal index, or cause other issues. Furthermore, the problem is that the wal file has to shrink and grow repeatedly, so having a base size won't help that much (besides prevent the original chunks of the wal from being shoved in the MFT). What the process would have to be for HDD users is let the program run for a while until it's built up a large wal file, but before it's been committed, and then kill the program and defrag the wal file. Alternatively, Hydrus could create a large dummy file with the same name as the wal files on first run (or pad the wal file with say, 5GiB of zeroes from the maintenance menu) and prompt the user to defrag the files, and then trim the contents of the dummy files back to the length of the original wal file. If that's too much of a pain for a small group of users, though, just setting the wal file to be persistent still allows a manual fix like I did to stick.
I didn't try to force the sqlite database to preserve the wal file since I was just testing the impact of fragmentation.
The way I accomplished the WAL-resizing was the following: I killed hydrus partway through an update run, which leaves the wal files, shared memory wal indexes, etc. (the design goal of wal is to allow the wal file to keep the "committed" rows, so working as intended)
I used a third party tool to generate a large file. (in my case I made the new file 10GiB large) After defragging the file with Contig, the contents of the file were replaced with the contents of the master database wal file and trimmed to the length of the original wal file. The original wal file was deleted, and the generated/defragged/altered/trimmed file was renamed to the original wal file's name. This process was repeated for the cache wal file and mappings wal file. The end result was that the new wal files had plenty of free disk space to expand into and were free to expand or shrink without fragmenting, while retaining the contents of the original fragmented wal files. The benefits last until Hydrus deletes and recreates the wal file, which I observed happening sometimes after a single update was finished and the DB unlocked, sometimes seemingly after a random number of passes after the DB unlocked, and sometimes not until Hydrus was shut down and restarted. I'm not sure what exactly is affecting that, and haven't tried any tests on that since it won't matter if the wal is set to persist.


Anonymous Board owner 12/01/2020 (Tue) 22:17:17 Id: 7d8d9d [Preview] No.857 del
>>850
Thanks. I was wondering yeah if I could somehow say 'hey SQLite, make sure the WAL doesn't shrink below 1GB' or something, so we could guarantee in some way there would be a big chunk that could get nicely defragged.

Atm, I close and re-open the connection to SQLite every 30 minutes, precisely because I know this clears the WAL file, which under heavy PTR processing can grow to multiple GB and does not always seem to want to truncate in a hurry. I commit transaction changes every 30 seconds. Because the WAL does not instantly shrink after a commit, I have figured it has a similar internal structure to the main SQLite database, with a bunch of empty pages that could be re-used later. Maybe there is a vacuum or truncate command--

EDIT: I looked up the PRAGMAs I can call, and while I don't think I can activate that opcode, I can just switch some users to a PERSISTENT journal mode, which I think generally does what we want here (have a persistent defraggable file for journalling), just without some of the file system optimisations that WAL offers:

https://sqlite.org/pragma.html#pragma_journal_mode
https://sqlite.org/pragma.html#pragma_journal_size_limit

So if I said 'ok, keep a <=1GB journal file around', that'll give us something to play with here. I have two launch switches atm, --no_wal, which turns on TRUNCATE, and --db_memory_journaling, which switches on MEMORY, but I think I should obsolete those and move to a unified --db_journal_mode="PERSISTENT", and also have some pre-boot .ini file support or similar so you can arrange it (and so I can save it in options) without the command line switch.

Also, it seems I can force a WAL truncate to 0 bytes when I want with this:

https://sqlite.org/pragma.html#pragma_wal_checkpoint

I'll play with this as well, give our recent giganto journals.

Message too long. Click here to view full text.



Release Tomorrow (later today)! Anonymous Board owner 12/02/2020 (Wed) 10:11:06 Id: 7d8d9d [Preview] No.858 del
I had a great week. I fixed bugs in drag-select, an undisplayable tag, some autocomplete result weirdness, and logical problems that were causing some grandparents and siblings not to appear in the new virtual system; I fleshed out some maintenance modes and added better autocomplete lookups in manage tags; and I sped up several autocomplete, sibling, parent, and general tag routines.

The release should be as normal tomorrow, maybe a little late.



Version 418 Anonymous Board owner 11/18/2020 (Wed) 23:18:22 Id: 57c62a [Preview] No. 839 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=lkk6m14htzw [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v418/Hydrus.Network.418.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v418/Hydrus.Network.418.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v418/Hydrus.Network.418.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v418/Hydrus.Network.418.-.Linux.-.Executable.tar.gz

I was ill this week, so I didn't have as much work time as I wanted. I did get some basic system predicate editing done though, if you want to try that out!

editing system predicates

This works, but it is basic, not super polished.

If you shift+double-click on a system predicate that is in your active search (like 'system:filesize>200KB'), it should now launch a dialog to edit it in place. You can also edit several at once. This should make doing several 'time imported < 7 days ago' ... '6 days ago' ... '5 days ago' searches in a row to chase something down a bit easier!

Message too long. Click here to view full text.



Anonymous Board owner 11/18/2020 (Wed) 23:19:06 Id: 57c62a [Preview] No.840 del
full list

- almost all system predicates are now editable if you shift+double-click them! you can also edit several at once in the same dialog
- if you double-click on any predicate type that is not editable but does have an inverse version (e.g. archive/inbox, has audio/no audio, and tag/-tag), the inverse version(s) will be swapped in
- all legacy custom system predicate defaults are eliminated this week. the panels now show a fixed default on launch, and will get a flexible favourites system in future, along with 'recently entered' quick buttons
- restored the 'show system:everything' and 'hide archive/inbox' options, which were inadvertantly hidden when file system predicate defaults were hidden, to the new _options->search_ panel
- fixed the borked list height for the file viewing statistics system pred panel checkbox lists
- fixed an issue where namespace:anything predicates would not propagate to new pages on 'open page with these tags' commands
- .
- boring code specifics:
- updated almost all the system predicate panels to take arbitrary initialisation values, and wrote a 'can I edit this' test for all predicate types to help some finnicky which-panel-and-pred-to-use issues
- wrote some new filtering code and a little UI to handle editing of system preds
- cleaned up some of the taglist item activation code

next week

I am feeling better already, so I think I'll be good for a normal work week. I'll try to polish these editable predicates off, and then I want to deal with some reported bugs in the parents and siblings logic.


Release Tomorrow! Anonymous Board owner 11/25/2020 (Wed) 07:12:16 Id: ecbf89 [Preview] No.842 del
I had a great week. I expanded the editable search predicates, making it easier to edit, and expanding what you can edit, including rating and OR predicates, and I was able to add custom defaults for all editable system predicate types. There are also misc fixes and additions, and significant speed/cancelability improvements to tag searches and wildcard-based tag and file lookups.

The release should be as normal tomorrow.



Version 417 Anonymous Board owner 11/11/2020 (Wed) 23:30:51 Id: ac1d92 [Preview] No. 829 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=Vim61JtH3fA [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v417/Hydrus.Network.417.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v417/Hydrus.Network.417.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v417/Hydrus.Network.417.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v417/Hydrus.Network.417.-.Linux.-.Executable.tar.gz

I had a great week working on a variety of small jobs.

The hydrus network version updates this week, so any clients that currently sync with the PTR will need to update to continue syncing. There is no rush for this.

all misc this week

First of all, the hydrus network version increments from 18 to 19 this week. Clients and servers can only talk to each other if they are running on the same version, and the PTR will update today, so if you want to keep syncing with the PTR, you will have to update. This version update was to ensure that all clients syncing with tag repositories are now on the new virtual sibling and parents systems. There is no rush to get this done, and no penalty if you delay--you will just get a polite popup saying there is a version mismatch, and your PTR service will be paused.

Message too long. Click here to view full text.



Anonymous Board owner 11/11/2020 (Wed) 23:31:12 Id: ac1d92 [Preview] No.830 del
I also fixed some macOS shortcuts. Control and Command should now both be recognised and labelled correctly, and trackpad scrolling through the media viewer should be at a more appropriate pace.

The Newgrounds downloader is updated. It should now be able to get all the content for artists with more than 60 or 70 items in their galleries. Also rolled in is an updated parser for realbooru, who apparently recently changed their site format.

I messed up the UPnP error parsing last week--sorry if you got any of this! I cleaned it up, wrote some overdue tests so I won't make this mistake again, and gave the network->data->manage upnp dialog a full pass. It should do all its long-delay jobs without blocking the UI.

Advanced: URL Classes now have 'header override' settings. Any time an URL is hit, those headers will be inserted into the request! I needed this to get the updated Newgrounds gallery parser to work, if you want to see an example. Be careful with this, but it may just solve several of our more tricky problems. Also, sorry, the URL Class edit UI is increasingly a hellstack, I will rework it at some point into tabbed pages or something.

full list

- the hydrus network version is increased this week from 18 to 19. clients and servers can only talk to each other when they are on the same version, so please update your clients if you wish to keep talking to the PTR, and your own servers if you have a home network setup or similar. if a server and client are on different versions, you will get a polite error when they next try to talk, and sync will be paused
- added 'run all export folders now' shortcut command to 'main window' shortcut set
- added shortcuts to the 'main gui' shortcut set for navigating the currently selected page. you can move left, right, to the leftmost on the current row, or to the rightmost. the left and right will cycle up a page of pages layer when at left/rightmost boundaries, letting you iterate through all pages in a depth-first manner
- updated the default newgrounds parser to deal with artists with more than 60/70 items in one art gallery (essentially, some clever 'next page' fetching now occurs to get older info that in your browser is drawn in as you scroll down). if you have some subscriptions for artists where you know this is true, try doing a full reset on them
- added realbooru to the hydrus defaults. they also apparently just switched away from a gelbooru 0.2.x site, so if you have a gelbooru parser with a realbooru example URL, I remove that example URL
- updated the page initial media load routine to my new async job
- updated the imported file presentation page-publish routine to my new async job

Message too long. Click here to view full text.



Anonymous Board owner 11/11/2020 (Wed) 23:33:00 Id: ac1d92 [Preview] No.831 del
- macOS shortcuts:
- the client's shortcut system now detects macOS-specific 'scroll start/end' states, and will not spam scrolls or errors when these states are held
- the client's shortcut system now attempts to detect artificial trackpad scroll/wheel events, and adapts the relative speed of scroll event generation according to the respective trackpad velocity. let me know how the hell this works for you in media viewer etc... (issue #710)
- the client's shortcut system now detects Control and Command as separate and reliable modifiers in macOS, with correct shortcut string rendering (issue #717)
- .
- upnp:
- fixed the awful typos in the upnp add-mapping error handling I changed last week. I am sorry for this!
- improved the async mappings and external ip fetch routines in upnp dialog. closing the dialog while a job is going on should now be completely ok
- upnp dialog add, edit, and delete actions are now async (they won't hang the UI while they work)!
- all the upnp async jobs should now disable the main list controls while they work
- fixed the 'edit' action on upnp dialog to correctly remove old and existing mappings depending on what was edited
- when adding a mapping for an (external_port,protocol) that is already mapped, the upnp dialog now asks if you want to overwrite, rather than just failing with a notification
- after an async action in upnp dialog, and a mappings refresh triggered, the cached external IP should now be properly restored to the status area
- pulled parsing code out of upnp code and wrote some proper unit tests for this so stupid typo errors should not happen again
- .
- parsing:
- subsidiary page parser separation formulae that throw an exception will now be ignored, as if they parsed nothing. in the weird case that you might receive json or html, you can now create subsidiary parsers for both types, and the one that fails will do so gracefully and silently

Message too long. Click here to view full text.



Anonymous Board owner 11/18/2020 (Wed) 04:53:18 Id: bc8108 [Preview] No.838 del
I unfortunately had a bit of a mixed week. I was short on work time, so I could only get basic editable system predicates done. This means you can change a system:filesize (and so on) after you add it to a search.

The release should be as normal tomorrow.



Version 416 Anonymous Board owner 11/05/2020 (Thu) 00:07:11 Id: fc4d64 [Preview] No. 824 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=IPjvDE-rKo0 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v416/Hydrus.Network.416.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v416/Hydrus.Network.416.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v416/Hydrus.Network.416.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v416/Hydrus.Network.416.-.Linux.-.Executable.tar.gz
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v416.tar.gz

I had a great week catching up on a mix of smaller work.

highlights

The manage tags dialog is now more efficient at making changes for very large numbers of files. My test application of 6 tags to 10,000 files went from 52 seconds to 4.8! A variety of other tag presentation updates should also benefit from the optimisations here.

Message too long. Click here to view full text.

Edited last time by hydrus_dev on 11/05/2020 (Thu) 00:20:57.


Anonymous Board owner 11/05/2020 (Thu) 00:07:48 Id: fc4d64 [Preview] No.825 del
- multi-column lists:
- spent a bunch of time cleaning out how I calculate multi-column list preferred initial width/preferred current width/minimum width, and made the final column more flexible in its resizing. instances of dialog suddenly getting gigantic because of a final column that wants to size itself at 1,000px should be completely gone, and lists that are shrunk due to non-last-column resizing will now adapt to this situation and not try to flex back to total initial width.
- multi-column lists now have horizontal scrollbars again for those situations where the parent window is thinner than their (now better calculated) minimum size
- improved the multi-column list num_rows height calculation, it should have less empty space at the bottom for lists that grow as items are entered into them (such as in the download pages)
- .
- manage tags megajob speedup:
- sped up manage tags final application step when entering many tags for many thousands of files at once
- optimised UI-side per-file tag cache (re)generation, reducing overhead and surplus work
- granularised UI-side per-file tag cache (re)generation based on the four current tag display contexts--now, if a system (e.g. manage tags dialog) only needs storage tags, the different display tags do not need to be regenerated
- optimised all tag filtering, which is also used in UI-side tag cache regen
- overall, giganto manage tag dialog jobs should now be faster in several ways. on my dev machine, adding 6 tags to 10k reasonably tagged files went down from 52s to 4.8. even larger jobs will still need a lump of CPU time, but they should scale more efficiently (what was previously O( num_tag_changes x num_total_mappings ) is now O( num_total_mappings ), and better at that)
- when a huge number of tags is added at once in the manage tags dialog, 'recent tags' is now populated more carefully

next week

I want to do more of this work, catching up on older items and things that have piled up in the past few weeks. There are eight weeks left in the year. I want to do one large round of network updates, but otherwise just more like this!

Message too long. Click here to view full text.



Anonymous Board owner 11/05/2020 (Thu) 00:27:54 Id: fc4d64 [Preview] No.826 del
I accidentally put the 415 release links in here when I first uploaded, fixed now!


Release Tomorrow! Anonymous Board owner 11/11/2020 (Wed) 07:36:54 Id: 9bac1d [Preview] No.828 del
I had a great week. I did a variety of small work, fixing bugs, adding new shortcuts, updating some libraries and downloaders, cleaning up old code and UI, and adding a couple of advanced features to the downloader system.

The network version will go up tomorrow, which means clients will have to update if they want to keep syncing with the PTR. There is no need to rush, and no penalty if you delay--syncing will just pause until you update.

The release should be as normal tomorrow.