/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.

Boards | Catalog | Bottom

Check to confirm you're not a robot
Drawing x size canvas

Remember to follow the rules

Max file size: 350.00 MB

Max files: 5

Max message length: 4096

Version 415 Anonymous Board owner 10/28/2020 (Wed) 22:39:32 Id: ba0ea0 [Preview] No. 821 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=SW9aJkqrUKw [Embed]
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v415/Hydrus.Network.415.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v415/Hydrus.Network.415.-.Windows.-.Installer.exe
app: https://github.com/hydrusnetwork/hydrus/releases/download/v415/Hydrus.Network.415.-.macOS.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v415/Hydrus.Network.415.-.Linux.-.Executable.tar.gz
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v415.tar.gz

I had a mixed week. I greatly reduced the lag that 414 could get after large sibling/parent changes, but unfortunately I could not get much else done.

all misc this week

I had wanted to do a bunch of small work this week, but my schedule fell apart a bit, and then the 'one last thing' for the new parents cache, which was rejiggering the background maintenance code to do smaller jobs at once, ended up being very difficult. I worked hard on figuring out a method, and then when I was ready to move on, I realised there was a design flaw and had to re-do a large part of it! In any case, it now works much better--even if you have thousands of tags with one common parent, they can now calculate one at a time, nice and quietly in the background. Some million-strong siblings and parents might still be a bit too laggy for an HDD to calculate in normal time, but an SSD should be generally ok. If it still lags for you, you can make it only work in idle time under tags->siblings/parents sync.

Message too long. Click here to view full text.

Anonymous Board owner 10/28/2020 (Wed) 22:39:53 Id: ba0ea0 [Preview] No.822 del
full list

- in _options->gui pages_ you can now set the main window's page tab alignment to top/left/right/bottom (previously it was just top/left). this property now updates for all page of pages on options ok, it no longer needs client restart (issue #642)
- the maintenance task that migrates tag display from the current values to the ideal application now works in significantly smaller steps. big lag from adding hundreds of childen to one parent (or similar siblings) should now be radically reduced
- rejiggered some layout in the new tag display dialogs
- added green/red texts to the new tag display dialogs to talk about when sync can work atm and how fast to expect changes to apply
- reordered the new tag 'siblings/parents info' right-click menu so the dynamic 'has x siblings/parents' submenus are on the bottom
- added basic client api calls for /add_files/..., delete_files, undelete_files, archive_files, and unarchive_files. they take 'hash' and 'hashes' parameters. I am throwing these out at the end of the week, so they don't have documentation or proper unit tests, but feel free to play with them (issue #393)
- sped up some UI refresh on content update for very large sessions
- sped up right-click tag/file menu any/all select actions on very large file sessions

next week

I'll try to get some more small things done, ha ha. I have IRL election day stuff to do, so it'll unfortunately be a slightly light week again. Release should be on Wednesday as normal, maybe a little late.

Release Tomorrow! Anonymous Board owner 11/03/2020 (Tue) 21:10:08 Id: c6ae04 [Preview] No.823 del
I had a great week. I caught up on smaller work, added some new default watchers, fixed some of the dodgy multi-column list sizing, and greatly sped up the manage tags dialog when dealing with thousands of files at once.

The release should be as normal tomorrow, maybe a little late.

Version 414 Anonymous Board owner 10/21/2020 (Wed) 22:48:14 Id: 194989 [Preview] No. 815 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=u3gfgq5UnNc [Embed]
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v414/Hydrus.Network.414.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v414/Hydrus.Network.414.-.Windows.-.Installer.exe
app: https://github.com/hydrusnetwork/hydrus/releases/download/v414/Hydrus.Network.414.-.macOS.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v414/Hydrus.Network.414.-.Linux.-.Executable.tar.gz
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v414.tar.gz

This is the first release of a complicated new system. If you are a cautious user, feel free to wait a week for me to iron out any surprise bugs.

I had a good four weeks converting tag parents to the new 'virtual' database cache.

The release does not need to do a big update this week. It will ask you if you have an SSD or an HDD (EDIT: with a special caveat for PTR users, see below), but that is it. All the mandatory heavy work that came in with 407 now happens in the background.

Message too long. Click here to view full text.

Anonymous Board owner 10/21/2020 (Wed) 22:48:47 Id: 194989 [Preview] No.816 del
smoother sibling/parent calculation

LATE EDIT: It seems the PTR has one very large parent structure that causes a one-time lag of a few minutes even for SSD users. This is happening because we are catching up on all the parents at once. You will get a popup when you update asking if you are on SSD or HDD. If you select SSD, be aware the client is likely to lock up for a few minutes soon after boot. Just let it do its work, or select 'HDD' on that update dialog and it will happen when you are not looking.

I was not happy with the large lag introduced when I added siblings. Ten minutes to see a new PTR sibling application is too long, and HDD users were almost completely nuked. Rolling parents into the same system would have made it worse. So, another aim of this big work was to reduce that calculation time and smooth it out in the background.

This is now done. Applying changes to siblings or parents should be very quick in all cases, but those changes may not instantly appear in your actual tags. The client now keeps track of what it currently has and what it should ideally have, and migrates silently in the background, in pieces. Most of the time, changes should appear within a second or two, but if a whole bunch just changed, it will take longer. You can check what is going on under tags->sibling/parent sync.

I also improved the recalculation code. The sibling and parent systems are now unified and expressed in a more granular 'tag implication' algebra, and rather than clearing bad numbers and regenerating them from source, the client now defines the difference between the current implication structure and the ideal, and only applies those differences. Reducing all structure changes to 'add implication' and 'remove implication' allowed me to implement this migration tech and optimise the queries significantly.

This is an entirely new system. If it causes lag for certain situations (e.g. if you have a 100k+ file session open), let me know. You can always turn it off under the sibling/parent sync menu. If you select 'hdd' during update, the migration will only happen in idle time, but I'd be interested to know how it performs in active time as well, if you are feeling adventurous.

If you are running a pre-407 client, you should be able to update straight to this week and skip almost all that work. Should™.

Anonymous Board owner 10/21/2020 (Wed) 22:49:52 Id: 194989 [Preview] No.817 del
full list

- tl;dr: you don't have to do anything. if you haven't heard of a tag parent before, no worries. the database should work better now
- .
- top level:
- parents are now completely virtual! this means that when you add a tag parent, the tags that 'fill in' to make it show do not really exist in storage, only in a computed cache. if you decide to undo the parent, the implications are recalculated and the virtual tags disappear, with no permanent changes made. also, petitioning a parent will 'preview' the delete, just as siblings now does
- siblings and parents are now unified, and the logic is improved. all parents apply to all siblings, so no more worries about retro-active filling-in. the siblings and parents code is now basically 'nice'. this was a lot of quite complicated work, and it solves a number of lingering issues from the original prototypes I made several years ago. I will still do some smaller work and little fixes I am sure in the near future, but the 'big' siblings and parents work is done!
- like with the recent siblings change, the client no longer needs to do the 'loading parent tags' step when booting--everything is now handled at the db level
- like with the recent siblings change, you can now edit which services apply their parents to which service, now under _services->manage where siblings and parents apply_
- in the _manage tags_ dialog (and some other places), tags with parent implications now show a '(x parents)' after their label, much like the 'will display as' sibling suffix. I do not like this, but I ran out of time. I hope to add a more advanced actual listing of virtual tags with a nice 'ghostly' colour or similar in future
- right-clicking on a tag in a specific tag domain now shows a 'siblings and parents' submenu with detailed info on all known siblings and parents in that domain
- 'tag' menu entries are moved from the top 'services' menu to a new 'tags' menu. 'pending', when available, is also moved right
- the process of changing siblings or parents, or which services apply where, is no longer a CPU-laggy process! actual changes, however, may not appear immediately. a maintenance task now tracks what is currently applied and what is 'ideal', and slowly migrates to the ideal in the background in little chunks. in most situations, the changes are very quick, but if you are behind due to big recent changes, they may be delayed. you can manage when this maintenance runs and see the current status under _tags->siblings/parents sync_. this is an entirely new thing, so feedback on IRL work would be appreciated--there may be some kinds of siblings or parents that cause a whole bunch of annoying lag
- the PTR has a lot of non-virtual parents that were hard-added in older versions over the years. most are fine, but some are like the 'shadow'->'shadow the hedgehog' debacle. now the source of the problem is fundamentally solved, this problem will reduce over time. with luck, before the end of the year, no more will be added at all, and thanks to the janitors, the worst offenders should be chipped away
- during all this work, a bug with tag siblings and parents repository processing has been revealed (some users do not 'get' all siblings/parents for some reason). now the system is nice and undoable, this will be more easily addressed in coming weeks, with automatic retroactive fixes rolling out to all clients

Anonymous Board owner 10/21/2020 (Wed) 22:50:52 Id: 194989 [Preview] No.818 del
- boring details:
- like with siblings, wrote a parents structure object that constructs the parents tree without loops more simply and reliably. it populates a new parents quick-lookup table in the database, for which a full suite of lookup and maintenance methods are written
- parents and siblings virtual tag presentation is now unified into a single 'display' (i.e. vs 'storage') system with a more granular tag implication algebra (essentially 0-n rows of 'if A is in storage, show B in display' for every tag) that can calculate new and updated display tags and counts without having to do the expensive 'clear-and-regen' that 408-413 used
- wrote functions to quickly add or remove a display implication to the 'all known files' or specific file service tag display cache
- migrated all the combined and specific tag display cache update code (add/remove files, add/remove mappings, add/remove sibling/parent, add/remove sibling/parent application, and misc regen maintenance calls) to use the implication system instead of the sibling 'ideal' system (basically moving from 1->1 to 1->n)
- completely rewrote the complicated 'all known files' cache 'with tags' and 'with and without tags' lookup routines to use much less overhead in general and to use a single, albeit complicated, count-based query that carefully chooses whether to select the 'with tags' and 'without tags' portions using tags or files where available as the primary selector based on existing autocomplete count data
- replaced all usage of the old ui-side 'tag parents manager' object. as parents pop in virtually and do not need to be bundled intentionally to various content updates, this was mostly just clearing now-surplus code, but for instance in 'write' autocomplete searches, the parents that appear below search results are now generated at the db level on first search, rather than looked-up live in UI time
- the parents and siblings lookup tables are now split into two views: what the display cache currently holds, and what it ideally should hold. when adding new sibling or parent data, only the fast ideal table is changed
- a new complicated maintenance function now takes actual and ideal data for a particular unsynced tag, hashes out the implication changes needed to effect a migration, and performs it
- a new maintenance manager and accompanying db code now track and manage calls to migrate actual to ideal display presentation, and to update UI afterwards
- as tag display changes are now more frequent, I have made the routine that refreshes tag UI after sibling/parent changes more efficient. tag display now only refreshes for files that have the affected tags in a particular change
- wrote the UI panel and dialog to show and hurry up current sync status, and all the background hooks for that
- added 'tag parents lookup' entry to the database 'regenerate' menu. this routine and the 'siblings' variant are now very quick thanks to the new actual/ideal maintenance system
- updated my sibling unit tests to deal with the new actual->ideal syncing
- improved the speed of mappings cache updates when deleting files
- deleted all the old combined/specific 'regen chain' code and the sibling-based 'get sole/any tagged files' search code
- optimised and generally cleaned a bunch of the new cache code, particularly cutting out overhead for unusual/small situations

Message too long. Click here to view full text.

Anonymous Board owner 10/21/2020 (Wed) 22:51:12 Id: 194989 [Preview] No.819 del
- the rest:
- fixed an issue where long-running 'similar files' search was not cleaning its memory use properly as the job was going on, resulting in out-of-mem errors on very large clients (issue #669)
- thanks to user submission, rolling in a fix for the default pixiv tag search downloader
- cloudscraper updated to 1.2.48
- removed surplus executables from linux and macOS builds (win32 upnpc exe was causing anti-virus false positive on mac lmao)

next week

Back to small work. There are a thousand things to catch up on, so I'll mostly grind away at that. This work was big and difficult, and some IRL stuff hit me for the first couple of weeks, so the year is running out. My overall plan is still to have some network upgrades done and then a 'what to work on next' poll up for Christmas, but I will take a few weeks of simple work first.

Release Tomorrow! Anonymous Board owner 10/28/2020 (Wed) 07:16:40 Id: 8e4a93 [Preview] No.820 del
I had a mixed week. While 414's parents cache seems to have gone well, making the new maintenance code less laggy proved difficult, and it monopolised my time this week. I did a small amount of other work, but not much, so 415 will mostly be a 'polish' release that makes 414 nicer for all users.

The release should be as normal tomorrow.

Version 413 Anonymous Board owner 09/23/2020 (Wed) 21:41:54 Id: c8c9d8 [Preview] No. 803 [Reply] [Last 50 Posts]

Message too long. Click here to view full text.

1 post omitted.

Anonymous Board owner 09/23/2020 (Wed) 22:15:23 Id: c8c9d8 [Preview] No.805 del
The 'no_proxy' setting may not work quite right, btw. Seems to be some odd IP/domain behaviour. Me and a user are looking at it. If you use a proxy and have any trouble, disable it in settings to go back to old behaviour. If you don't use a proxy in options->connection, nothing has changed, you don't have to do anything.

Parents Cache Update 1 Anonymous Board owner 09/29/2020 (Tue) 22:31:29 Id: 8c4acc [Preview] No.806 del
I had an ok week. I got stuck into the parents cache and related maintenance updates. I have a plan and I am grinding away at it. There is still a lot to do, but current expectation is 414 will be out on October the 14th.

Parents Cache Update 2 Anonymous Board owner 10/07/2020 (Wed) 00:58:35 Id: 8c4acc [Preview] No.811 del
I had another ok week. I am not working as fast as hoped, but the new maintenance system works. The very long wait times to recalculate siblings is gone--it now all happens in the background over time. I am now working on converting parents to the new system.

I am less confident I can get 414 ready and good for October the 14th, but we'll see how it goes.

Parents Cache Update 3 Anonymous Board owner 10/14/2020 (Wed) 00:06:20 Id: f7a563 [Preview] No.812 del
I had a great week. The siblings and parents database code is mostly in place, and I was successful in unifying their application method to a simpler algebra that should result in faster siblings/parents work across the board.

I now have to tie up many database loose ends, cleanse the old UI-side parents system, and polish. About 150 small things and 3 big things left. I think I can be ready for October 21st!

Release Tomorrow! Anonymous Board owner 10/21/2020 (Wed) 06:54:07 Id: b9fb04 [Preview] No.814 del
I had a good four weeks migrating tag parents to the new database cache. Like siblings, parents are now applicable across services and completely undo-able. Furthermore, the large CPU requirements we experienced after the siblings cache are gone--a new maintenance routine cuts all parents and siblings work into pieces and figures it all out quietly in the background.

The release should be as normal tomorrow. It should only take a few seconds to update!

Version 412 Anonymous Board owner 09/16/2020 (Wed) 22:26:27 Id: cc303e [Preview] No. 796 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=3-K94kdbTS8 [Embed]
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v412/Hydrus.Network.412.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v412/Hydrus.Network.412.-.Windows.-.Installer.exe
app: https://github.com/hydrusnetwork/hydrus/releases/download/v412/Hydrus.Network.412.-.macOS.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v412/Hydrus.Network.412.-.Linux.-.Executable.tar.gz
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v412.tar.gz

I had a great week catching up on smaller jobs, improving search speeds, and adding a 'lite' 407->408 update mode for HDD users who sync with the PTR. There are also a couple of new applications for the Client API.

Update this week will take a few seconds to a few minutes as new database indices are created.

sibling and search speeds

Message too long. Click here to view full text.

Anonymous Board owner 09/16/2020 (Wed) 22:29:36 Id: cc303e [Preview] No.797 del
new client api applications

A user has been working hard at making a web browser for the client via the Client API, called Hydrus Web. It is now ready at https://github.com/floogulinc/hydrus-web ! If you have a bit of networking experience, please check it out - it allows you to browse your client on your phone!

Also Anime Boxes, a booru-browsing application, is adding Hydrus as an experimental browseable 'server' this week, also through the Client API. Check it out at https://www.animebox.es/ !

I also updated the Client API help to talk more about HTTPS and connections across the internet, here: https://hydrusnetwork.github.io/hydrus/help/client_api.html

full list

- client api:
- added Hydrus Web, https://github.com/floogulinc/hydrus-web, to the Client API page. It allows you to access your client from any web browser
- added Anime Boxes, https://www.animebox.es/, to the Client API page. This booru-browsing application can now browse hydrus!
- the /add_urls/add_url command's 'service_names_to_tags' parameter now correctly acts like 'additional' tags, and is no longer filtered by any tag import options that may apply. that old name still works, but the more specific synonym 'service_names_to_additional_tags' is now supported and recommended (issue #456)
- the /add_urls/add_url command now takes a 'filterable_tags' parameter, which will be merged with any parsed tags and will be filtered in the same per-service way according to the current tag import options.
- the client api help is updated to talk about this, and the client api version is now 14
- updated client api help to talk about http/https

Anonymous Board owner 09/16/2020 (Wed) 22:30:58 Id: cc303e [Preview] No.798 del
- the rest:
- the 407->408 update step now opens a yes/no dialog before it happens to talk about the big amount of CPU and HDD work coming up. it offers the previous 'full' version that takes all the work, and a 'lite' version that applies no siblings and is much cheaper. if you have been waiting on a PTR-syncing HDD client, this should let you update in significantly less time. there is still some copy work in lite mode, but it should not be such a killer
- the 'manage where tag siblings apply' dialog now has big red warning text talking about the current large CPU/HDD involved in very big changes
- a bunch of file-location loading and searching across the program has the opportunity to run very slightly faster, particularly on large systems. update will take a few seconds to make these new indices
- namespace and subtag tag searches and other cross-references now have the opportunity to run faster. update will take another couple of minutes to drop and remake new indices
- gave tag and wildcard search a complete pass, fixing and bettering my recent optimisations, and compressing the core tag search optimisation code to one location. thank you for the feedback everyone, and sorry for the recent trouble as we have migrated to the new sibling and optimisation systems
- gave untagged/has_tags/has_count searches a similar pass, mostly fixing up namespace filtering
- gave the new siblings code a similar pass, ensuring a couple of fetches always run the fast way
- gave url search and fetch code a similar pass, accounting better for domain cross-referencing and file cross-referencing
- fixed a typo bug when approving/denying repository file and mapping petitions
- fixed a bug when right-clicking a selection of multiple tags that shares a single subtag (e.g. 'samus aran' and 'character:samus aran')
- thanks to some nice examples of unusual videos that were reported as 1,000fps, I improved my fallback ffmpeg metadata parsing to deal with weird situations more cleverly. some ~1,000fps files now reparse correctly to sensible values, but some either really produce 1000 updates a second due to malformation or bad creation, or are just handled that way due to a bug in ffmpeg that we will have to wait for a fix for
- the hydrus jpeg mime type is now the correct image/jpeg, not image/jpg, thanks to users for noticing this (issue #646)
- searching for similar files now requires up to 10,000x less sqlite query initiation overhead for large queries. the replacement system has overhead of its own, but it should be faster overall
- improved error handling when a database cannot connect due to file system issues
- the edit subscription(s) panels should be better about disabling the ui while heavy jobs, like large subscription resets, are running
- the edit subscription(s) panels now do not allow an 'apply' if a big job is currently disabling the ui

Message too long. Click here to view full text.

Anonymous Board owner 09/16/2020 (Wed) 22:31:20 Id: cc303e [Preview] No.799 del
- boring details:
- improved a pre-optimisation decision tool for tag search that consults the autocomplete cache for expected end counts in order to make a better decision. it now handles subtag searches and multiple namespace/subtag searches such as for wildcards
- wrote fast tag lookup tools for subtag and multiple namespace/subtag
- fixed some bad simple tag search optimisation code, which was doing things in the wrong order!
- optimised simple tag search optimisations when doing subtag searches
- polished simple tag search code a bit more
- added brief comments to all the new cross joins to reinforce their intention
- greatly simplified the multiple namespace/subtag search used by wildcards
- fixed and extended tag unit tests for blacklist, filterable, additional, service application, overwrite deleted filterable, and overwrite deleted additional
- added a unit test for tag whitelist
- extended the whole 'external tags' pipeline to discriminate between filterable and additional external tags, and cleaned up several parts of the related code
- moved the edit subscription panel asynchronous info fetch code to my new async job object
- cleaned up one last ugly 'fetch query log containers' async call in edit subscriptions panel
- moved the edit subscription(s) panels asynchronous log container code to my new async job object
- misc code cleanup

next week

Message too long. Click here to view full text.

Release Tomorrow! Anonymous Board owner 09/23/2020 (Wed) 01:48:02 Id: f14974 [Preview] No.802 del
I had an ok week. I mostly fixed bugs, aiming for a nice 'clean' release before I go for the parents database cache.

The release should be as normal tomorrow.

Version 411 Anonymous Board owner 09/09/2020 (Wed) 22:15:23 Id: 98d815 [Preview] No. 791 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=NgcYLIvlp_k [Embed]
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v411/Hydrus.Network.411.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v411/Hydrus.Network.411.-.Windows.-.Installer.exe
app: https://github.com/hydrusnetwork/hydrus/releases/download/v411/Hydrus.Network.411.-.macOS.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v411/Hydrus.Network.411.-.Linux.-.Executable.tar.gz
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v411.tar.gz

I had a good week. The siblings work is complete.


The last areas that used the old siblings systems are now migrated to the new fast cache. The 'loading tag siblings' step of client boot, which for PTR users could be several seconds of work, is no longer needed!

Message too long. Click here to view full text.

Anonymous Board owner 09/09/2020 (Wed) 22:17:39 Id: 98d815 [Preview] No.792 del
full list

- misc:
- fixed the 'system:(like/dislike) rating = x' search predicate string, which was saying 'unknown' rather than 'like/dislike' in several cases
- fixed a 'current_count' error in the new file search optimisation code for tag searches where the tag did not exist for any files in the domain (i.e. autocomplete count=0). thank you to users for helpful reports here
- fixed the recent file search optimisation code to handle 'system:time imported' when it was mixed with tags or search predicates that would pre-populate the query file pool with file domain cross-referenced files. sorry for the trouble!
- the forced delay overhead for table analysis is reduced from 0.1s to 0.02s. whenever many mostly empty tables need to be analyzed (like on first boot shutdown, when it is usually 100+ tables), it now zips by
- .
- siblings/tag improvements:
- typing a shorthand sibling like 'lotr' into an 'all known tags' 'read' autocomplete - like on a default search page - now reliably discovers and matches text entry to ideal sibling results like 'series:lord of the rings'. this was previously buggy and unreliable--it now allows the match using better db knowledge, even when the merged 'all known tags' services involved disagree on siblings
- when typing tags into a 'searching immediately' page that has media, the autocomplete count results that only refer to that media will now match shorthand sibling inputs to the ideal result. media-populated tag search now takes a little bump of extra CPU to fetch results (they are now passed through the db to get nice siblings info), so it is also cached for the duration of your typing (previously, the counts were re-computed on every new keystroke, so this should be significantly smoother to work with on large pages even if that first keystroke takes a moment to give results)
- when typing into a 'write' autocomplete, like in manage tags, the process that promotes the entry text and known siblings to that entry and a potential ideal sibling result to the top of the list is now more sane. it now also only uses results with nonzero count. we'll see how this last change works out IRL
- when typing into a 'read' or 'write' autocomplete, the pre-search tag insert no longer has sibling insertion/swapping. it was unreliable before, with weird sibling-swapping in the short moment before real results returned. if you have slow results and often quick-add tags into search pages or manage tags, let me know how this works for you
- the 'additional tags' tag input dialog off the tag import options edit panel now shows the 'will display as' label
- the 'favourite', 'file lookup', and 'recent' tag suggestion panels now show the 'will display as' label
- the 'related' suggestion panel, which works on a slightly different system, now shows the 'will display as' label
- the 'tag suggestions' options panel's 'favourite tags' edit lookup and list now displays 'will display as' labels and correctly finds service-specific siblings in its results (e.g. you type 'lotr', it also finds 'series:lord of the rings')

Message too long. Click here to view full text.

Anonymous Board owner 09/09/2020 (Wed) 22:18:18 Id: 98d815 [Preview] No.793 del
- the last UI-side siblings work is cleared:
- the UI-side tag siblings cache is no longer used. the sometimes multi-second 'loading tag siblings' step of boot no longer happens!
- media autocomplete fetches are now asynchronously populated with siblings data via the db
- the exact-match and sibling 'insert' predicates at the top of pre-load and post-load read and write autocompletes now rely exclusively on db data for sibling matching
- taglists now present 'will display as' labels asynchronously and are better about updating those labels when the list's underlying tag service changes
- the taglist right-click menu that shows siblings to copy now fetches that submenu's contents asynchronously from the database
- the test panel on a blacklisting tag filter now asynchronously fetches tag siblings to test against from the database
- the actual blacklist tag filter test now fetches tag siblings to test against from the database
- reworked my custom tag listbox to handle asynchronous text decoration, and unified sibling decoration for media taglists and string taglists
- updated my old async updater class to be more flexible for different job types, and cleaned the code that already used it
- wrote a simple class for one-shot async jobs
- wrote a simple db lookup for UI-side tag sibling chain members
- wrote a simple db lookup for UI-side tag ideal siblings
- bunch of misc sibling, db, and ui work and cleanup to make all this work

next week

Message too long. Click here to view full text.

Release Tomorrow! Anonymous Board owner 09/15/2020 (Tue) 23:38:26 Id: 8d4d4c [Preview] No.795 del
I had a great week. I was able to catch up on smaller work, optimised a variety of db functions, including the new sometimes-slow tag and wildcard queries, and added a 'lite' mode for the big 407->408 update so PTR-syncing users on HDDs can update in less time.

The release should be as normal tomorrow.

Version 410 Anonymous Board owner 09/02/2020 (Wed) 22:41:28 Id: a6ea8b [Preview] No. 787 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=N7GaWASGcgs [Embed]
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v410/Hydrus.Network.410.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v410/Hydrus.Network.410.-.Windows.-.Installer.exe
app: https://github.com/hydrusnetwork/hydrus/releases/download/v410/Hydrus.Network.410.-.macOS.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v410/Hydrus.Network.410.-.Linux.-.Executable.tar.gz
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v410.tar.gz

I had an ok week. I wasn't as productive as I hoped, but I am happy with the mostly optimisation work.


After some more profiling in IRL situations, and with more helpful info from users, I have done another round of profiling for the new sibling cache, and more besides. A database technique I use for many purposes is now more reliable (fewer lag spikes), and has less CPU overhead. If you found some systems (like the 'related tags' suggestions in manage tags dialog) sometimes took a few seconds to work in the past couple of weeks, they should now be fast again. And you should find many types of file search, particularly those with multiple search predicates, and general tag processing, should be faster than before.

Message too long. Click here to view full text.

Anonymous Board owner 09/02/2020 (Wed) 22:41:53 Id: a6ea8b [Preview] No.788 del
full list

- general work:
- fixed a bug in the new file service filtering code that was stopping file upload commands to file repositories or ipfs services from sticking
- fixed an issue with the export files dialog auto-close-when-done function
- I think I fixed a possible bug in the boot file location repair/recovery dialog sometimes not saving corrected paths on unusual file systems
- file migration cancel button and shut off timer should work a bit more reliably, more to come here
- copying subscription quality csv info to clipboard no longer does nice human numbers (you now get 1234, not csv-breaking 1,234)!
- may have fixed a very rare 'or predicate' error when opening a dialog with a 'read' autocomplete input, like export folder or file maintenance jobs dialogs
- all pages are better about dealing with missing (i.e. recently deleted) services on load, and autocompletes also
- error handling from servers with strange character encodings should be better about dealing with null characters
- cleaned up the combined display regen chain code
- deleted some obselete db code
- .
- optimisation review:
- after more profiling, and thanks to additional input from users, I have done another round of optimisation for the new caches. using a new technique, more than just mappings are sped up this week - a number of queries that were prone to lag spikes should now have much more reliable speed and also be faster when hammered often
- .

Message too long. Click here to view full text.

Anonymous Board owner 09/02/2020 (Wed) 22:42:17 Id: a6ea8b [Preview] No.789 del
- other optimisations:
- mappings processing
- sibling processing
- wildcard tag searches, with and without namespaces, particularly when mixed with other search terms
- 'tag as number' searches, with and without namespaces, particularly when mixed with other search terms
- searching for tags when mixed with other search terms
- has notes/no notes
- searching files on 'all known files' with general file metadata system predicates (like size, filetype)
- url class, url domain, and url regex file searches, particularly when mixed with other search terms
- num tag file searches when mixed with other search terms
- has/not has tags file searches when mixed with other search terms
- sped up specific display chain regen significantly, with similar separate current/pending optimisations as last week's for combined
- converted specific display cache overall regen to use a copy followed by the new chain regen rather than additive file import
- sped up combined display chain regen a little bit
- the splash window now updates itself with less UI overhead, so spammy updates (like the new tag regen code) use a little less CPU and fewer UI context switches

next week

Message too long. Click here to view full text.

Release Tomorrow! Anonymous Board owner 09/08/2020 (Tue) 22:49:25 Id: b43780 [Preview] No.790 del
I had a good week. I fixed some recent search bugs and I capped off the new siblings work. Everything siblings is now running off the new cache, so the slow 'loading tag siblings' step of boot no longer occurs!

The release should be as normal tomorrow.

Version 409 Anonymous Board owner 08/27/2020 (Thu) 01:30:28 Id: deeceb [Preview] No. 783 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=puU4EvBTPxA [Embed]
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v409/Hydrus.Network.409.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v409/Hydrus.Network.409.-.Windows.-.Installer.exe
app: https://github.com/hydrusnetwork/hydrus/releases/download/v409/Hydrus.Network.409.-.macOS.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v409/Hydrus.Network.409.-.Linux.-.Executable.tar.gz
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v409.tar.gz

I had a great week fixing some bugs and optimising the new tag siblings cache. The new code works much faster now.


I am very happy that there do not seem to have been any obvious errors with the new sibling database cache. Unfortunately, a couple of areas were working inefficiently, which IRL testing helped to diagnose. I put a lot of time into this this week and was very successful - some sections take 10% less time, some 90%, and one critical query now takes 99% less time. It depends on many factors, but many things are faster overall. In particular, tag processing speed, which took a real hit, is back up to good speed, and setting new tag sibling application rules now only needs to regenerate for changed siblings, so if you add (or remove) your own five 'my tags' siblings onto the PTR, the client now only has to do two seconds of work, not ten minutes.

Message too long. Click here to view full text.

Anonymous Board owner 08/27/2020 (Thu) 01:32:06 Id: deeceb [Preview] No.784 del
full list

- siblings:
- the slowest of the new sibling regen & update code has received a full optimisation pass. some sections take 10% less time, some 90%, and one critical query takes 99% less time. overall, several big jobs work much faster, and ptr processing, which slowed significantly for many users, should be back up to a good speed. uploading pending tags (which tend to be for local files) should be much faster in particular. let's do another round of IRL observation and profiling this week, and I'll keep at it
- the various 'display' regeneration routines now provide more progress status text, drilling down to the x/y siblings being collapse-counted, or number of files added to a cache, and generally all tag sibling regen got a status update polish pass
- optimised the way tag sibling application is set--now, only the tag siblings that are changed need to have their counts regenerated. hence, if you just apply (or remove) your own five 'my tags' siblings onto the PTR, the client now only has to do two seconds of work, not ten minutes
- .
- the rest:
- fixed the annoying issue with media viewer mouseovers stealing focus/activation from the manage tags dialog. this can now only happen if current focus is on a hover window. sorry for the delay!
- updated manage tag parents dialog to state the pairs being petitioned on the 'petition reason entry' dialog
- updated manage tag parents and siblings dialogs to have appropriate 'reason' suggestions for petitions (previously, they were inheriting the same suggestions as for add)
- ipfs network jobs now have a minimum 'reply' connection timeout of two hours (so giganto directory pushes won't throw an error). connection timeout remains the same, so if the server is hanging on that, it'll still notice
- fixed the 'test address' button on the IPFS manage services panel
- petitioning an IPFS file when there is no IPFS multihash entry in the db no longer causes an error. now, in this case, the file entry is removed with no change made.
- when pending to or petitioning from a file service, a quick filter is now applied to discard invalid files (i.e. (not) already in the service). any weird logical holes where this might occur should now be fixed
- export folders now catch and report missing file errors more nicely
- export folders now remember the last error they encountered and report that in the edit export folders dialog

Anonymous Board owner 08/27/2020 (Thu) 01:32:35 Id: deeceb [Preview] No.785 del
- boring tag siblings optimisations:
- optimised the tag manager generation routine to use any common file domains for fast cache lookup for any subset of the files available, rather than falling back to 'all known files' domain when there is no single common file domain
- optimised the new 'all known files' display autocomplete cache to use similar faster specific files cache lookups when available
- optimised how the 'all known files' display cache regenerates tag sibling chains. it now takes a shortcut when given non-sibling tags and tags where all but one sibling member have zero count, and it can count current and pending counts separately according to the most efficient counting method (e.g. most pre-display pending counts are 0 across the board, so even if current count is a million, the pending count can often be assumed without lookup overhead). furthermore, the 'clever' count has better query planning and less non-sqlite data overhead, and with experimental data is now chosen more carefully. what was previously a 22s job on a test database now takes 5s
- deduplicated how new mappings are filtered to all the specific cache domains, significantly reducing overhead
- massively optimised a critical - and the slowest - part of the new 'combined' cache that handles add/pend mappings pre-insert presence testing, speeding up the core query about 100x!
- reduced some overhead when doing file service_id normalisation in repository processing
- split up specific chain regen into groups to reduce memory usage
- optimised specific display tag cache 'add file' updates, and thereby basic cache regeneration, to be just a little faster for files that have multiple sibling tags
- all predicates made in the database are now populated with ideal and chain sibling information, and this is used for '(will display as xxx)' labels and autocomplete tag search filtering (e.g. you type in 'lotr', it matches an autocomplete result of 'lord of the rings'). there are still some ui-made predicates to figure out, so the old system remains as a fallback
- related tags lookup is a tiny bit faster and now populates its predicates with ideal and chain sibling info at the db level
- cleaned up some 'fetch related tags' code, might make it a bit faster for large tag counts
- cleaned up the way some mapping tables are fetched
- unified table/table_name nomenclature in the db code
- updated an old data->ui status presentation method (it typically does stuff like "regenning some stuff: 500/10,000"), to not hog so much UI time and not yield worker threads so often when new statuses are coming in real fast
- several late optimisations based on IRL testing

Message too long. Click here to view full text.

Release Tomorrow! Anonymous Board owner 09/02/2020 (Wed) 05:40:55 Id: deeceb [Preview] No.786 del
I had an ok week. I wasn't as productive as I hoped, but I fixed some bugs and made great progress in optimising the new sibling code, and older code too. The various lag spikes in systems like 'related tags' should be smoothed out, many file searches are snappier, and tag processing should be faster than it was in v407.

The release should be as normal tomorrow.

Version 408 Anonymous Board owner 08/19/2020 (Wed) 23:03:41 Id: 39340f [Preview] No. 776 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=X8zLP-TBLcA [Embed]
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v408/Hydrus.Network.408.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v408/Hydrus.Network.408.-.Windows.-.Installer.exe
app: https://github.com/hydrusnetwork/hydrus/releases/download/v408/Hydrus.Network.408.-.macOS.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v408/Hydrus.Network.408.-.Linux.-.Executable.tar.gz
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v408.tar.gz

I had a great couple of weeks moving tag siblings forward. The update this week will take some time as a new cache is generated. On an SSD with the PTR synced, about 10 to 20 minutes, depending on how many files you have. Some of the new tag sibling application rules may have 5-10 minute delays on edit as well. If you have a large/PTR client and are on an HDD, or you would rather just wait for the kinks to be ironed out, you might like to wait a week or two for me to further optimise this new code.


tl;dr: Siblings are faster and better now, you don't have to do anything. Some parents will not appear with new downloads - don't worry about it, they will all fill back in nicely soon.

Message too long. Click here to view full text.

1 post omitted.

Anonymous Board owner 08/19/2020 (Wed) 23:07:24 Id: 39340f [Preview] No.778 del
full list

- tag siblings cache:
- tl;dr: siblings are faster and better now, you don't have to do anything. some parents will not appear with new downloads - don't worry about it, they will all fill back in nicely soon
- wrote the first version of a 'tag display' cache, which stores not your tags as they are, but how they appear after display rules such as siblings, parents, and filtering are applied, meaning this data need not be calculated every time on thumbnail load. this week marks the first concrete step forward in an improvement of siblings and parents storage, and begins with just siblings. all siblings and front-end tags work should be generally faster and more accurate
- part one is for tag domains cross-referenced with file domains. it maintains virtual sibling-collapsed mappings and autocomplete counts through mappings added, deleted, pended or pend-rescinded, files added/deleted, and siblings added/removed
- part two is for tag domains on the 'all known files' domain (i.e. no file domain). it maintains virtual sibling-collapsed autocomplete counts through mappings added, deleted, pended or pend-rescinded, and siblings added/removed
- both parts also support full table drop/regen (under the new database->regenerate->tag display mappings cache) for when my logic inevitably miscounts something. the existing regen 'tag mappings cache'/'tag siblings lookup' commands also regen the display mappings cache, since it relies on them
- when tag siblings on a repository are petitioned to be deleted, they are now instantly discounted from tag sibling application (previously, they had to be uploaded and committed to count, now both pending and petitioned offer a quick preview of outcome)
- the display cache supports the tag sibling service application rules under the 'services menu', and regen when that changes, so you can now turn siblings on and off, and apply them across services. as a result, the old 'apply all siblings to all services' option is now gone! as parents will undergo a similar change soon, and the siblings changes this week may lead to some undesired parents in the interim, the 'apply all parents to all services' option is also gone
- tag autocomplete counts in the form (x-y) due to siblings are eliminated. it will still do it when combining the same merged tag across different services, or when an unnamespaced tag includes how many potential namespaced will also be found
- the following search types now obey tag sibling application rules accurately: number of tags search, namespace:anything search, wildcard search, tag search (on a per-tag-domain basis, previously it was globally hacked to all siblings), tag-as-number search. for instance, if you search series:anything, a file that has 'metroid' tag-siblinged to 'series:metroid' will now correctly appear.
- the above search types are now exact to how the tag displays. if you have for files that are tagged 'samus' on either tag service A or B, and service B has a sibling for that to 'character:samus aran', searching for 'samus' gets the results in A, 'character:samus aran' gets the results in B. previously it was an expensive logical mish-mash of 'sure, try and get everything behind the scenes'. now it searches what you see
- searching for files in the advanced 'all known files' domain currently has no sibling support for the above search types. autocomplete counts should be good, and the results that come up should have the correct tag display, but the actual results are calculated based on storage tags. getting this to work without doubling the size of the db is tricky, so it will have to be ongoing work

Anonymous Board owner 08/19/2020 (Wed) 23:07:56 Id: 39340f [Preview] No.779 del
- all tag siblings are now completely virtual. this means that when a tag comes in via a downloader or other means, it will not be automatically coerced to its ideal sibling in the actual db storage tables (the true tags you see in manage tags dialog), but remain as it is. there is no change in sibling appearance in normal operation--it still _displays_ and searches as its ideal sibling. the same will happen to parents in the coming months, and in the interim period, parents no longer apply across siblings. as siblings can come and go from anywhere, they are now divorced from actual stored tag mappings. in a similar way, the manage tags dialog no longer supports the 'hard-replace siblings and parents' command, nor the 'auto-replace with sibling' command. this may be jarring to workflows and preferences, so please bear with me and let me know what feels particularly bad. and please don't worry too much about parents not always being added in the meantime--I hope to do the same transition for them in four of five weeks, and all gaps will be filled in. also, in the coming weeks, I expect to improve manual tagging workflow by indent-grouping edit-view siblings together (ditching the old 'will display as' text) for easier review and selection, a bit like parents. actual 'hard' siblings and parents that do always get irreversibly renamed/added in storage will come in the future as a separate system
- the 'add_siblings_and_parents' client api parameter no longer adds siblings, and soon will be retired completely
- I had wanted to completely eliminate the old UI level siblings manager this week, but there are still some systems, mostly tag autocomplete work, that need it and are tricky to swap. I stripped it down, at least, and reduced its update delay to 2 seconds. therefore, the 'loading tag siblings' step of boot still occurs, albiet a _little_ faster. I hope to have it gone soon
- this is some complicated code affecting core systems. almost everything 'siblings' is now different. there are likely to be laggy parts, awkward new workflow, and possibly some update or miscount bugs as I iron it out. the good news, now they are all virtual, is that problems are undoable. please report any issues, and I will work on it as I keep pushing on this and on parents
- please expect your client.caches.db file to expand in size about 10-30% or so this week. the update itself will take a few minutes as the improved tag lookups and new caches are regenerated from empty
- .
- boring mostly db optimisation list:
- after some thought, moved those new options for tag sibling application down to the db. previously, they were stored in an UI object for convenience, but since everything is going down to the db, it is worth doing it properly down there. thus they reset this week to the default
- I also removed that complicated 'all known tags' page in the tag sibling application options--it wasn't doing enough to justify itself

Anonymous Board owner 08/19/2020 (Wed) 23:09:05 Id: 39340f [Preview] No.780 del
- tag siblings lookup cache now obeys the tag sibling application rules and regenerates the appropriate cache when the options change
- tightened up the db tag siblings lookup cache and wrote more tools for it. it had a couple of blind spots for getting all siblings in a chain. also optimised the lookup for en-masse tag operations
- tightened up my tag sibling structure builder object, which was not eliminating loops but collapsing them to (generally harmless, but not desireable) (A,A) pairs
- extended mappings and siblings lookup caches to perform various sorts of tag sibling collapse filtering, to determine files that do or do not have another tag mapping on a tag sibling chain
- optimised the existing mappings cache in several ways
- optimised cross-domain file cache mappings filtering, and cleaned the code
- optimised autocomplete count fetching from the mappings cache, particularly for large result sets
- optimised how the combined autocomplete count generates from nothing
- optimised how tags are loaded for search results (thumbnails)
- optimesed basic tag search
- greatly optimised how the mappings caches request cross-domain file cache filtering
- broke up the rescind_pending/add mappings job into simpler separate parts, which was needed for accurate display cache counting. this may end up fixing the other weird pending miscount bug we had
- the cached 'display' tags are now loaded with regular media results, not generated on the fly on first request (unless in the advanced 'all known files' domain, where it is done quickly on first load at the db level)
- converted the db over to using its local sibling lookup cache for all sibling jobs
- all data-level content updates to media result objects now occur in the database loop, reducing lag and eliminating a single UI event loop gap when the objects the UI relies on were desynchronised
- optimised how the tag and hash id-to-definition cache maintains itself
- cleaned up cache code generally

Message too long. Click here to view full text.

Anonymous Board owner 08/19/2020 (Wed) 23:09:50 Id: 39340f [Preview] No.781 del
- the rest:
- fixed the stupid manage tag siblings dialog input/ok bug I introduced last week
- fixed the pair preview label in manage tag siblings dialog when it asks to enter a reason for a remove petition
- I believe I fixed the annoying recent bug where the top-right hover window would sometimes not position itself correctly on a window size/move until the top hover was shown once
- fixed a bug where the 'do you want to do shutdown work?' dialog was not abandoning shutdown if cancelled (rather than yes/no)
- updated the 'has free space to do db transaction?' checker, which needs to test device partitions, to do two sweeps--first only fast local devices, then potentially mega laggy network discovery if the mount point is not found (hydev was wondering why it was suddenly taking nine seconds to close his test client!)
- fixed another issue with double-clicking some addremove/queue listboxes when no edit button is set--now in this case they all delete on a double-click
- fixed a little bad error handling on pending content upload. an error with petitioning certain IPFS uploads is not yet fixed

next week

I was unable to completely replace the UI-side siblings manager. It is a bit smaller and faster, but you will still get the slow 'loading tag siblings' on boot. There are about four or five things it still does that I still need to move down to the new cache, so I will keep hammering away. I'll also improve the speed of these new caches' regen. Other than that, I have two weeks of other jobs to catch up on, so just some misc work.

I just realised I forgot to fix the annoying manage tags/media viewer focus stealing! I'll fix it for 409!

Release Tomorrow! Anonymous Board owner 08/26/2020 (Wed) 03:10:42 Id: 9a3e25 [Preview] No.782 del
I had a great week fixing some bugs and making the slowest parts of the new sibling cache code work much faster and more efficiently. The lag that was introduced last week should be reduced, I hope, by about 4-10 times, making it much less noticeable. The UI feedback on longer jobs is also improved.

The release should be as normal tomorrow.

Version 407 Anonymous Board owner 08/05/2020 (Wed) 21:37:04 Id: 99495e [Preview] No. 765 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=3Rup3EdA0kw [Embed]
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v407/Hydrus.Network.407.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v407/Hydrus.Network.407.-.Windows.-.Installer.exe
app: https://github.com/hydrusnetwork/hydrus/releases/download/v407/Hydrus.Network.407.-.macOS.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v407/Hydrus.Network.407.-.Linux.-.Executable.tar.gz
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v407.tar.gz

I had an ok week. I did some prep work for siblings, and fixed some bugs.


The right-click menu off a normal taglist now provides a whole bunch of different copy options, depending on whether you want all or selected, tags or subtags, and with or without counts.

Message too long. Click here to view full text.

1 post omitted.

Anonymous Board owner 08/05/2020 (Wed) 21:41:03 Id: 99495e [Preview] No.767 del
full list

- sibling prep:
- I am preparing for a new siblings database cache for v408. this will ultimately make siblings (and parents) faster, more accurate, more powerful, and simple to undo. I have decided, as part of it, to make siblings and parents completely virtual (i.e. the tags won't exist for real, they'll be implied). better tools to manage hard-replace siblings and parents will come later, as trying to support both situations at once has not been excellent
- .
- created options to hold per-service sibling and parent preferences, so you'll be able to set up '"my tags" siblings and then "ptr" siblings apply to "my tags"' or 'no parents apply to this service'
- wrote UI for the sibling options under 'services->manage where tag siblings apply'. you can play with it if you like, and it saves values, but it is not plugged in yet and makes no changes
- siblings logic is a little tighter. the db and gui side of siblings structure calculation is more unified, petitioned siblings are discounted properly on all generation, and the db side now resolves conflict decisions the same on every regen. the gui-side still runs on an older structure, but will be updated to exactly mirror the db
- updated and unified how large numbers of raw tag siblings are fetched in the database. it also supports fast tag slicing, speeding up sibling cache maintenance. the siblings lookup cache now uses this method for regeneration and update calls
- .
- the rest:
- tag right-click menu copying now supports all combinations of selected/all, tags/subtags, and no_count/with_counts where appropriate (issue #325)
- if the media viewer is too thin for the top hover window to fit into its space, the top-right hover now drops down below it. I don't really like how this looks, and will probably instead figure out a flow layout so the toolbar buttons always fit, but at least they are now accessible (issue #388)
- altered the above fix--if the top-right hover window can be shrunk to fit in the available space, it will now squeeze in, only bumping down if it can't
- moving the mouse off an activated (e.g. clicked) hover window now instantly activates the main canvas. this should fix up some fast swallowed clicks and annoying click-to-activate issues with the center-right duplicates hover window, which does not hide (issue #384)
- the duplicates hover window now positions correctly if its min size is too wide to fit in a thin media window
- if you make changes to a parser or content parser, there is now a yes/no confirmation when trying to cancel the dialog

Message too long. Click here to view full text.

Anonymous Board owner 08/05/2020 (Wed) 21:41:34 Id: 99495e [Preview] No.768 del
next week

So, I will try to get this cache working. I need to essentially mirror the existing tag and autocomplete caches, but after siblings collapse, ensure that is maintained through tag or sibling changes, and then plug it in to everything. There is a bunch of ancient code to update around here, and other auxiliary system to improve, so this may end up being a two week job. I'll update on Tuesday night as normal.

Anonymous Board owner 08/08/2020 (Sat) 17:57:13 Id: 99495e [Preview] No.770 del
Hey, I messed up the siblings dialog in 407, it won't ok correctly. I'll fix it all up in the bigger siblings work in 408.

No Release Tomorrow! Anonymous Board owner 08/11/2020 (Tue) 23:01:10 Id: 99495e [Preview] No.772 del
I had a good week getting into the meat of the tag siblings database work. There is a lot to do, but I have a plan I am happy with, it is going well, and I am confident I will have a good first version ready for next week, Wednesday the 19th.

So, no release tomorrow!

Release Tomorrow! Anonymous Board owner 08/19/2020 (Wed) 07:08:20 Id: 0798bc [Preview] No.775 del
I had a great couple of weeks moving tag siblings forward. The client now handles siblings more quickly and accurately using a new database cache. They are easier to undo going forward, and you can also choose where siblings apply on a per-service basis (e.g. turning off the PTR's default siblings and using your own instead).

The release should be as normal tomorrow.

Version 406 Anonymous Board owner 07/29/2020 (Wed) 21:24:33 Id: a9d519 [Preview] No. 760 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=UZr6b4mBoxI [Embed]
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v406/Hydrus.Network.406.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v406/Hydrus.Network.406.-.Windows.-.Installer.exe
app: https://github.com/hydrusnetwork/hydrus/releases/download/v406/Hydrus.Network.406.-.macOS.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v406/Hydrus.Network.406.-.Linux.-.Executable.tar.gz
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v406.tar.gz

I had a good week. It is now easy to deduplicate queries in subscriptions.


Managing multiple subscriptions can get complicated. Figuring out which queries are in which sub, whether and where you have duplicates, is tedious. A recent bug with query pasting also could have introduced some duplicates within the same sub. This week should make it simple to manage.

Message too long. Click here to view full text.

Anonymous Board owner 07/29/2020 (Wed) 21:24:57 Id: a9d519 [Preview] No.761 del
full list

- subscription management:
- the manage subscriptions dialog now has a 'deduplicate' button. it is enabled whenever your subs of a particular downloader contain duplicate queries. it launches a semi-bananas but thorough 4-step process to ask if you want to do upper/lower-case deduplication, then which downloader, then which queries, then which master sub(s) to retain the queries
- subscription dedupe within the same sub keeps the query with the most files
- the manage subscriptions dialog also now has a 'lowercase' button that coerces all queries of the selected subs to lowercase
- when pasting a list of queries into a subscription, the 'already in sub' test is now caseless. pasting "Samus_Aran" into a sub already with "samus_aran" will not add anything
- .
- misc:
- url classes now have a checkbox to keep fragment data (e.g. "#kwGFb3xhA3k8B") during URL normalisation. this data is not sent to the server and is not useful for almost all sites, but for sites like Mega, it contains useful clientside javascript navigation or access info if you open the URL in your browser
- fixed video resolution parsing for some unusual SAR files. this stretches a video slightly, usually when the vid was created or converted with older analog tech (e.g. NTSC)
- fixed rating system predicate label for 'rated/not rated'
- the issue where miscounts in pending upload data would persist, sometimes leading to an annoying 'pending (13)' style menu that would not clear without debug action, is now fixed in a cheap way. on any upload action, this cached count is reset. a fix for the actual unusual miscount situation will have to come later
- the different in-memory manager objects now save changes at different time intervals--lightweight things like favourite searches still save not long after any change, but column widths, network sessions, and bandwidth use now save only every ten minutes
- I _may_ have fixed an issue with favourite tags not sticking correctly or resetting when added en masse via the tag right-click menu
- I believe I fixed a rare but permanent ui hang on highlighting a gallery or watcher when that same downloader was spamming through a largely 'already in db/previously deleted' list
- copying tags 'with counts' now works correctly for simple tag views (previously, it only worked for 'predicate' views)

Message too long. Click here to view full text.

Simple Release Tomorrow! Anonymous Board owner 08/05/2020 (Wed) 00:54:39 Id: f168ce [Preview] No.763 del
I had an ok week. Some of the work I did (siblings improvements) is not ready to be turned on yet, so tomorrow's release mostly just has bug fixes, including several related to the media viewer's hover windows positioning and behaviour.

The release should be as normal tomorrow.