/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.

Boards | Catalog | Bottom

Check to confirm you're not a robot
Drawing x size canvas

Remember to follow the rules

Max file size: 350.00 MB

Max files: 5

Max message length: 4096

Version 455 Anonymous Board owner 09/22/2021 (Wed) 22:03:37 Id: 9764e1 [Preview] No. 1138 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=Gs069dndIYk [Embed]
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v455/Hydrus.Network.455.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v455/Hydrus.Network.455.-.Windows.-.Installer.exe
app: https://github.com/hydrusnetwork/hydrus/releases/download/v455/Hydrus.Network.455.-.macOS.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v455/Hydrus.Network.455.-.Linux.-.Executable.tar.gz

I had an ok couple of weeks. Some quality of life is improved, the UI should save sessions a bit quicker, and the database repairs itself more of itself.

Your client may warn you about a missing index on first boot. If it does, that's great--some new repair code is working!

session save

Some users with large sessions (typically around 2-3+ million 'weight') are experiencing severe UI lockups and general lag. This week I deal with part of that problem by greatly reducing the amount of CPU needed to save most sessions. Now pages know if they have been changed since being loaded, and if they have no changes, they can skip a bunch of session save work.

Message too long. Click here to view full text.

Anonymous Board owner 09/22/2021 (Wed) 22:04:44 Id: 9764e1 [Preview] No.1139 del
database repair

tl;dr: Boot checks are better, you don't have to do anything.

I have spent most of my 'cleanup' time this year breaking the client database into smaller modules. These modules are now responsible for their own repair (which happens on boot), and they do much of it in a neater automatic way, using the same set of definitions it uses to initialise on first start to check whether an existing database is missing anything now. Most of my old hardcoded 'oh, you are missing this table' code is now simplified and modularised, works more reliably, and will be much easier to maintain in future.

You may get a popup when you boot saying you are missing one or more indices. There are several legacy reasons why this might be the case, but most often it will be because you once recovered from former hard drive damage and a clone/recover was unable to rescue an index. Whatever the case, these can now be regenerated automatically (they store duplicated data, so can be regenerated 100%), so please let it run. You might even notice PTR processing working faster afterwards.

Many tables can also be regenerated, and some can be completely repopulated. It is now possible (though not a great idea!) to start a client without a client.caches.db or client.mappings.db file and have it recover to a bootable state. Error presentation around this is improved, with instructions on what to do next in the case of critical problems.

I will continue modularising more of the database and fleshing out this repair code to cover more of the schema!

full list

- misc:
- many of the simple system predicates (width, size, duration, etc..) now support the '≠' (not equal) operator! searches only support one of these at once for now (e.g. you can't say height != 640 AND height != 1080--it'll pick one of these pseudorandomly)
- the watcher page list right-click menu now has 'copy subjects', which copies the selected watchers' 'subject' texts to clipboard

Message too long. Click here to view full text.

Anonymous Board owner 09/22/2021 (Wed) 22:08:42 Id: 9764e1 [Preview] No.1140 del
- ui freezes:
- session pages can now detect if they have had no saveable changes since a certain time. they use this ability to skip redundant session save CPU time for pages with no changes since the last session save
- for now, since the smallest atom of the session system is a whole page, gallery and watcher pages can only save time if _every_ downloader in the page has had no changes, so in their case this optimisation mostly only applies to completely finished/paused pages. it is still better to have several medium size downloader pages than one gigantic one
- a new database maintenance task ensures that optimisation cannot accidentally lose a page (from something like an unfortunate timing of a session save after many manual session deletes)
- the existing optimisation that skips 'last session' save on no changes now initialises its data as the 'last session' is first loaded (rather than on first save), meaning that if there are no changes while the client is open, no new 'last session's will be saved at all
- misc session save code cleanup
- .
- database repair, mostly boring:
- a client can now boot with client.caches.db missing and will rebuild that file. almost all of its tables are now able to automatically repopulate (issue #975)
- all the new modules I have been working on are now responsible for their own repair. this includes missing indices, missing tables, and table repopulation where possible. modules now know which of their tables are critical to a boot, what version each table and index was added, and now manage both initial and later-created service tables and indices
- essentially, all newer database repair code is now modularised rather than hardcoded. the repair and creation code actually now share the same table and index definitions. the code is more reliable, checkpoints its good work in case of later failure, and will be much easier to maintain and expand in future
- lots of module repair and initialisation code is refactored and generally given a full pass
- the core mappings cache regeneration routine now takes transaction checkpoints throughout its job to save progress and reduce journal size
- master definition critical error detection code is no longer hardcoded!
- mapping storage repair code is no longer hardcoded!
- similar files repair code is no longer hardcoded!
- parent or sibling cache repair repopulation is no longer hardcoded!

Message too long. Click here to view full text.

Version 454 Anonymous Board owner 09/08/2021 (Wed) 22:27:35 Id: bd99d7 [Preview] No. 1134 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=Ch24PpzMKog [Embed]
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v454/Hydrus.Network.454.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v454/Hydrus.Network.454.-.Windows.-.Installer.exe
app: https://github.com/hydrusnetwork/hydrus/releases/download/v454/Hydrus.Network.454.-.macOS.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v454/Hydrus.Network.454.-.Linux.-.Executable.tar.gz

I had an ok week mostly working on making images look better.

warped tiles

tl;dr: Images look better now.

I put a lot of time into the tiled renderer this week, mostly fixing 'stretch-warped' tiles. These are most obvious in the duplicate viewer when you are flipping between two very similar images that have slightly different resolution--you would see one part of one of the images stretched one pixel horizontally or vertically compared to the other.

Message too long. Click here to view full text.

Anonymous Board owner 09/08/2021 (Wed) 22:27:54 Id: bd99d7 [Preview] No.1135 del
full list

- misc:
- when a downloader page fetch gives a 500 Server Error, it is now handled better, status numbers are updated to 'failed' quickly, and I believe the post-500 downloader deadlock issue (downloaders staying as 'pending', not 'working', after several of these 500s) should also be fixed (issue #898 maybe, but many other reports also)
- when finishing a very large archive/delete filter, the UI should not hang so much to commit the changes. the changes may be delayed a second or more, if your client is currently chugging, so if you are a user who hits F5 real quick after committing archive/delete, let me know how you get on. I've tried to mitigate for your situation, but it may not be perfect.(issue #845)
- the Edit URL Class dialog will now refuse to OK if the API Converter fails.
- the Edit URL Class dialog will now put up an 'are you sure?' messagebox if the URL Class matches its own example API URL
- fixed some gallery error handling when a gallery cannot be parsed
- improved 'cannot parse' error reporting text to include more information
- the client api /manage_pages/add_files command should now always preserve the sort of both the file_ids or hashes parameter
- fixed an instance where image bitmaps were being handled incorrectly (issue #876)
- the client will no longer report shutdown work due for a repository when the work is on a currently paused content type
- fixed some logic related to the advanced tag option of 'fetch page even if url recognised and already in db'
- improved the error message when an unlucky duplicate boot causes a locked database error
- I rearranged and clarified a couple of links in the 'advanced' area of the help index page
- added some help texts and tooltips to the edit checker options dialog and fixed some borked layout
- did a quick hack to fix some db repair code that wasn't dealing with some missing module information. I will brush the repair code up in the near future so that modules can repair themselves

Message too long. Click here to view full text.

Anonymous Board owner 09/08/2021 (Wed) 22:28:50 Id: bd99d7 [Preview] No.1136 del
- improved tile coordinate safety checks for extremely small images (e.g. 1x1) when blown up to ridiculous zooms so there are more than one tile per pixel
- this took a bunch of work, and I am happy that I figured out some solutions, but I believe it may be impossible to get perfect answers here. please try out this new version and let me know how it goes, particularly in the duplicate viewer where warping is obvious. I think ultimately I may replace this with a single tile system that goes over the borders of your screen, eliminating the stitching problems entirely, although this will eat more memory and CPU

next week

I still have plenty of old bug reports to clear from the issue tracker, so I will keep plugging away at them. I want to focus on the UI freezes an increasing number of users are seeing. I also want to do some database cleanup and database repair code improvements

No Release Tomorrow! Anonymous Board owner 09/15/2021 (Wed) 05:03:22 Id: 4d5701 [Preview] No.1137 del
I had a not great week. Some IRL landed on me, so I was not as productive as I wanted. I did manage to upgrade the database repair code in a pretty neat way, but it is not an exciting thing that most users will benefit from immediately, so rather than do a release tomorrow, I will catch up with regular work.

So, 455 will be next week, on the 22nd! I'll try and fix the UI freezes and clear out more issue tracker bugs!

Version 453 Anonymous Board owner 09/01/2021 (Wed) 21:52:24 Id: 4213b1 [Preview] No. 1130 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=YA6A8xohpxc [Embed]
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v453/Hydrus.Network.453.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v453/Hydrus.Network.453.-.Windows.-.Installer.exe
app: https://github.com/hydrusnetwork/hydrus/releases/download/v453/Hydrus.Network.453.-.macOS.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v453/Hydrus.Network.453.-.Linux.-.Executable.tar.gz

I had a great week. I ended up mostly fixing bugs. If you have a large client, the update may take a few minutes this week.

bug fixes

I discovered a semi-important tag processing bug last week--for some users, files that were imported before they added the PTR were not getting all the correct tag info and would not appear in some tag searches. I fixed the underlying file-tracking bug and have added a retroactive 'gap-filler' to this week's update. If you used the client for a long time but added/reset the PTR recently, you'll have more gaps so your update will take longer.

Another persistent issue has been the 'tiling artifacts' with my new tile-based image renderer, where at certain zooms an image would have one or more horizontal or vertical blurred/janked line of pixels. You'd notice it on cleaner vector images where a smooth curved black line would suddenly jag at one zoom. I had trouble reproducing this at first, but some users got it quite often, and with their help I was able to figure it out. I have rewritten the part of my renderer that was failing, and I think I now have the artifacts completely fixed for all 'normal' zooms. Let me know how you get on!

Message too long. Click here to view full text.

Anonymous Board owner 09/01/2021 (Wed) 21:52:49 Id: 4213b1 [Preview] No.1131 del
full list

- qol and misc:
- the network job status labels around waiting for 'subscription'/'download page'/'watcher' forced wait slots are reworded. now they just say a more plain 'waiting to work' with a time estimate, and if a job does not get a chance to work this check cycle, it says 'a different xxx got the chance to work' for a few seconds.
- if a network job does not get bandwidth on a check cycle, it now says 'a different network job got the bandwidth' for a few seconds
- when waiting on bandwidth or gallery work, network jobs should count down more smoothly, one second at a time, not skip a second so often
- network job widgets are now better about updating the layout of their two text labels. the status text on the left should take all the available pixels much better, sharing with the '64KB/s' speed text as it changes width and disappear
- added a new user-made darkmode QSS stylesheet called 'OledBlack' to default hydrus, try it out under _options->style_
- if the tag domain in a search page is other than 'all known tags', the 'selection tags' box, which limits itself to the current domain's tags, now explicitly labels itself with that domain
- consolidated and optimised the pre-work checks on all importers/downloaders in pages. pages with idling/finished/paused downloaders will consume just a little less CPU and need to talk to fewer important objects
- renamed the shortcut sets for viewer/preview media windows and clarified that they are mouse only for now. the new seek command works with these, but you'd have to map ctrl+right-click or something
- improved the system predicate unit tests to catch datatype problems like with last week's hotfix and system:time imported
- advanced archive/delete stuff: wrote up a neat idea I had about using local parents applied to the PTR to make fast multi-tag processing workflows here: https://hydrusnetwork.github.io/hydrus/help/advanced_parents.html#parent_favourites
- .
- bug fixes:
- an important tag search bug is fixed. for some users, files that were imported before a service was added were not appearing in some of that service's search results, or their tag counts were not added in certain tag autocomplete results. this file miscount is fixed, and holes will be filled on database update. it should not take too long to fix, although different users will have different situations
- this bug was leading to artificially fast PTR processing speeds on some clients as their older files were being skipped. if you have used the client for a long time but only added the PTR recently, sorry if you notice it slow down! it is now working correct!

Message too long. Click here to view full text.

Anonymous Board owner 09/01/2021 (Wed) 21:54:05 Id: 4213b1 [Preview] No.1132 del
- client api:
- OR predicates are now supported in the client api! Just nest within the tag list, and it'll bundle the nested list into an OR. there's an example in the client api help
- some permissions testing in file search is tightened up--now we have OR and system predicates, if you do not submit any regular positive tags, the search permissions have to be 'allow anything'
- fixed an issue where the client api would let you ask about sha256 hashes of incorrect length (and would ultimately make a master database id for these borked hashes, even the empty string!!). now the client api throws a 400
- fixed a bug in /manage_pages/get_pages where all pages were marked as 'selected'=true (issue #841)
- in the client api, if you use missing file_id(s) on a request for a file, thumbnail, metadata about a file, or when trying to add a files to a page, it now gives 404 correctly (rather than 500) (issue #961)
- added a section to the client api help on variable encoding, including an example of how to convert a python tag list to JSON+URL encoded string
- added new unit tests for OR pred parsing and the hash length check
- client api version is now 20

next week

Unfortunately, I did not spend time on notes parsing as I had planned. The people helping to run the github issue tracker pointed me to old bug reports that I had not been keeping track of, so I am going to spend a little time prioritising clearing out that queue (and generally trying to integrate important github issues into my weekly routine more, as I often have trouble naturally focusing on priorities). So, next week will likely be more like this!

Release Tomorrow! Anonymous Board owner 09/08/2021 (Wed) 02:56:53 Id: 6821b1 [Preview] No.1133 del
I had an ok week, mostly more bug fixes. Another issue with the image viewer--slightly warped tiles that were obvious in the duplicate viewer--is now much better, and I think I fixed an issue of downloaders sometimes getting stuck on 'pending' forever after serious network problems.

The release should be as normal tomorrow.

Version 452 Anonymous Board owner 08/25/2021 (Wed) 22:49:32 Id: 5a5d15 [Preview] No. 1127 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=iePWOFOSl_U [Embed]
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v452/Hydrus.Network.452.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v452/Hydrus.Network.452.-.Windows.-.Installer.exe
app: https://github.com/hydrusnetwork/hydrus/releases/download/v452/Hydrus.Network.452.-.macOS.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v452/Hydrus.Network.452.-.Linux.-.Executable.tar.gz

I had an ok week. I ended up doing a lot of boring behind the scenes prep, but there is also some nice new quality of life. If you sync with the PTR, update will take a few minutes this week.

all misc this week

After a very long delay, I finally have shortcuts for seeking video back/forwards in the media viewer. New users will get ctrl+left/right to seek back 2.5 seconds or forwards 5 seconds. Existing users need to add their own--please check it under the 'media viewers - all' shortcut set. You can set whatever seek distance you like, and even set multiple with different distance jumps. Sorry for how long it took to get this in--I had to update my shortcut system first!

I did some behind the scenes tag work this week. The database can now handle larger 'tag as number' searches. Update will take a few minutes to load these bigger numbers into the fast search cache.

Message too long. Click here to view full text.

Anonymous Board owner 08/25/2021 (Wed) 22:50:26 Id: 5a5d15 [Preview] No.1128 del
- vastly improved the cancel speed for searches in the realm of 'get the files that have any xxx tag', be that all tags or a namespace wildcard, and also some important search setup for various 'all known files' search pages. if you have ever tried to search the PTR raw and run into a three minute uncancellable initial setup lag, it should be gone now!
- when you right-click on files in a specific local file domain (e.g. trash or my files), the 'view this file's dupes' number check is now run on 'all local files' as well, and if the numbers differ, a second menu is shown for all local files. this should make it easier to chase dupes of trashed files that are still untrashed while also allowing a trash-only search
- fixed a critical bug in repository mapping processing that was not adding mappings to certain caches for files imported before the repo was added, and/or the new repository 'per content type' processing reset. this mostly manifested as these files not showing up in search results despite the tag being there. there is more work to do here, so it is top priority next week, and likely some maintenance to regen the bad caches
- .
- boring rewrites for multiple local file services:
- many users have asked for the ability, when multiple local file domains are available, to search multiple file domains at once. I spent time this week doing background work to support this, and a related concept of searching 'deleted' files, which is a current gap in the program and not always covered by 'all known files'. nothing significant changes, but almost all the file search code now works on n file domains rather than 1, but for now n=1 lmao
- made a new 'database search context' object to handle a virtualised but still simple and fast file 'location view' at the database level
- the primary file search call is converted to use this object. references to a single file service are replaced with the view or its components
- all duplicate file search code is moved to the new location search context
- searching by 'system:import time' now works over multiple domains, although it is a little muddled. in future, import time predicate will have an optional specific file service and do 'import time' vs 'deleted time'
- 'system:local' and 'system:not local' is adapted so it can still work fast with multiple file domains, sped up worst case 'local' time, and a new optimisation lets it run fast for 'deleted from local files' too
- sorting search files by import time is now only supported if the search domain is just one domain

next week

While I was working on database stuff this week, I discovered a problem with repository processing that meant files imported before a repository was added (or mappings reset, with the new 'reset by content type' function) would not get fully processed and may not show up in tag searches. I fixed half of that bug this morning, but there is more to do, so that's top priority.

Message too long. Click here to view full text.

Release Tomorrow! Anonymous Board owner 09/01/2021 (Wed) 02:36:16 Id: dd23a6 [Preview] No.1129 del
I had a great week, ending up focusing mostly on bug fixes. The 'jank line' tile artifact in the image viewer is fixed for all normal zooms, and I also fixed the long-time tag search issue I discovered last week. There's also some quality of life improvements, and OR searching is added to the Client API.

The release should be as normal tomorrow.

Version 451 Anonymous Board owner 08/18/2021 (Wed) 22:42:38 Id: dc7d79 [Preview] No. 1122 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=Cv87NJ2xX0k [Embed]
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v451/Hydrus.Network.451.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v451/Hydrus.Network.451.-.Windows.-.Installer.exe
app: https://github.com/hydrusnetwork/hydrus/releases/download/v451/Hydrus.Network.451.-.macOS.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v451/Hydrus.Network.451.-.Linux.-.Executable.tar.gz

I had a great week cleaning code and fixing bugs. If you have a big database, it will take a minute to update this week.

all misc this week, most bug fixes

I fixed a critical bug in tag siblings. It was causing some tag siblings to be forgotten, particularly on local tag services, and particularly when a tag had a sibling removed and added (e.g. 'delete A->B, then add A->C') in the same transaction. I will keep working here and will trigger a new sibling reprocess in the future just as 450 did so we can fix more PTR issues.

The new content-based processing tracking had a couple more issues. Some Linux users, in particular, it just broke for, due to a SQLite version issue. I have fixed that, and I have fixed some issues it caused for IPFS. There are new unit tests to make sure this won't happen again.

Message too long. Click here to view full text.

Anonymous Board owner 08/18/2021 (Wed) 22:43:04 Id: dc7d79 [Preview] No.1123 del
- misc:
- I fixed a critical bug in tag sibling storage when a 'bad' tag's mapping is removed (e.g. 'delete A->B') and added ('add A->C') in the same transaction, and in a heap of fun other situations besides, that mostly resulted in the newly added sibling being forgotten. the bug was worse when this was on a local tag service via the manage siblings dialog. this problem is likely the cause of some of our weird sibling issues on clients that processed certain repository updates extremely quickly. I will keep investigating here for more issues and trigger another sibling reset for everyone in the future
- the 'show some random pairs' button on the duplicates page is nicer--the 'did not find any pairs' notification is a popup rather than an annoying error dialog, and when there is nothing found, it also clears the page of thumbs. it also tries to guess if you are at the end of the current search, and if so, it will not do an auto re-fetch and will clear the page without producing the popup message
- fixed a bug that meant file order was not being saved correctly in sessions! sorry for the trouble!
- import of videos is now a little faster as the ffmpeg call to check resolution and duration is now retained to check for presence of an audio channel
- when files are imported, the status messages are now much more granular. large and CPU-heavy files should move noticeably from hash generation to filetype calculation to metadata to actual file copying
- fixed a database query bug in the new processing progress tracking code that was affecting some (perhaps older) versions of sqlite
- when you trash/untrash/etc... a file in the media viewer, the top hover text now updates to show the file location change
- fixed a typo bug in the new content type tracking that broke ipfs pinning yet again, sorry for the trouble! (issue #955)
- I fleshed out my database pending and num_pending tests significantly. now all uploadable content types are tested, so ipfs should not break at the _db_ level again
- the page tab menu now clumps the 'close x pages' into a dynamic submenu when there are several options and excludes duplicates (e.g. 'close others' and 'close to the left' when you right-click the rightmost page)
- the page tab menu also puts the 'move' actions under a submenu
- the page tab menu now has 'select' submenu for navigating home/left/right/end like the shortcuts
- fixed some repository content type checking problems: showing petition pages when the user has moderation privileges on a repository, permission check when fetching number of petitions, and permissions check when uploading files
- fixed a typo in the 'running in wine' html that made the whole document big and bold
- across the program, a 'year' for most date calculations like 'system:time imported: more than a year ago' is now 365 days (up from 12 x 30-day months). these will likely be calendar calculated correctly in future, but for now we'll just stick with simple but just a bit more accurate
- fixed a bug in mpv loop-seek when the system lags for a moment just when the user closes the media viewer and the video loops back to start

Message too long. Click here to view full text.

Anonymous Board owner 08/18/2021 (Wed) 22:45:00 Id: dc7d79 [Preview] No.1124 del
- boring database cleanup:
- to cut down on redundant spam, the new query planner profile mode only plans each unique query text once per run of the mode
- also fixed an issue in the query planner with multiple-row queries with an empty list argument
- refactored the tag sibling and parent database storage and lookup code out to separate db modules
- untangled and optimised a couple of sibling/parent lookup chain regeneration calls
- moved more sibling and parent responsibility to the new modules, clearing some inline hardcoding out of the main class
- cleaned up a bunch of sibling, parent, and display code generally, and improved communication between these modules, particularly in regards to update interactions and display sync
- the similar files data tables are migrated to more appropriate locations. previously, they were all in client.caches.db, now the phash definition and file mapping tables are in master, and the similar files search record is now in main

next week

I've managed to catch up on some critical issues, and some IRL stuff also eased up, so I have some breathing room. I want to put some more time into multiple local file services, which has been delayed, likely some way to search over previously deleted files.

Release Tomorrow! Anonymous Board owner 08/25/2021 (Wed) 02:11:16 Id: d4040a [Preview] No.1126 del
I had an ok week. I improved some quality of life and finally got around to shortcuts to seek video.

The release should be as normal tomorrow.

Bug Reports Anonymous Board owner 08/08/2019 (Thu) 00:20:53 Id: 58c04a [Preview] No. 9 [Reply] [Last 50 Posts]
Please feel free to submit any bug reports here.
64 posts and 12 images omitted.

Anonymous 03/25/2021 (Thu) 03:22:26 Id: ad3466 [Preview] No.1038 del
hey im trying to use the url downloader and it wont find parse this url e621.net/posts/2662369

where do I go or what do I do? who knows how many downloads were affected

Anonymous 03/25/2021 (Thu) 04:03:43 Id: ad3466 [Preview] No.1039 del
same with this e621 posts/2659888

Anonymous 04/27/2021 (Tue) 10:30:08 Id: 15cfd9 [Preview] No.1052 del
Not sure if this is a bug or if I'm missing something, but I screwed up and ran out of space on the disk I keep my DB on. It's not the first time this has happened, but this time it seems to have broken Hydrus somewhat.

It takes a long time to start now, and then it works slowly but normally for a while before it decides it's out of space again, at which point I can't make any changes, like deleting files to free up space.

I've used the moments where it does work to delete some of the bigger files to free up 25 GB. This is 5% of the entire disk, and although I think it's made it a little better, meaning it takes a bit longer for it to think it's run out of space, it hasn't actually fixed the problem.

Is there a way to fix this other than replacing the database file entirely and reimporting the nearly 500 GB worth of client files?

Anonymous 05/04/2021 (Tue) 23:04:07 Id: 69d0a8 [Preview] No.1057 del
(9.15 KB 574x116 Capture.PNG)
I'm following what this message is telling me to do. I don't know how or when this popped up but I just noticed it when I looked at Hyrdus today. I'm on v434, Win10 19042.

Anonymous 08/19/2021 (Thu) 03:01:22 [Preview] No.1125 del
I'd like to start this post off by saying that I'm using the AUR version of Hydrus and I know there isn't any real support for Linux builds in general, but I'm still posting this here in case someone else had the same problem, or possibly even knows what fucked up and how to fix it. Alright -

Dark mode seemed to just stop working for me one day. It looks exactly like the default white color scheme except the backdrop for image preview areas is the dark gray color that dark mode typically uses. I checked the "colours" in options and everything seems to be fine too (although I don't think you can change the color of the hydrus window itself through there anyway). This just kind of suddenly started happening after booting up Hydrus one day. It wasn't after updating or anything like that either.

Version 450 Anonymous Board owner 08/11/2021 (Wed) 23:05:34 Id: 612109 [Preview] No. 1117 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=fLLKwIfs1NM [Embed]
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v450/Hydrus.Network.450.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v450/Hydrus.Network.450.-.Windows.-.Installer.exe
app: https://github.com/hydrusnetwork/hydrus/releases/download/v450/Hydrus.Network.450.-.macOS.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v450/Hydrus.Network.450.-.Linux.-.Executable.tar.gz

🎉🎉 MERRY 450! 🎉🎉

I had an ok week. Last week's experimental release went well, so I have polished that, and I otherwise caught up with a variety of small work.

If you sync with the PTR, update will take a couple of minutes.

all misc this week

Message too long. Click here to view full text.

Anonymous Board owner 08/11/2021 (Wed) 23:15:15 Id: 612109 [Preview] No.1118 del
full list

- misc:
- when exporting files from the file export window, a cancellable popup job with progress updates is also created. if you close the window, you can still cancel the job from the popup
- fixed a crash bug in file export window
- system:num file relationships (duplicates) now correctly only returns files in the current file search domain (previously, it returned all files, including those previously deleted etc...)
- I rearranged some of the thumbnail menu file relationships actions menu. I'm not really happy with this, but a shuffle is easier than a full rework
- fixed the '4k' resolution label replacer, which was looking at 2060 height not 2160 by mistake
- the phash generation routine (part of the duplicates system, happens on image imports) now uses less memory and CPU for images with an alpha channel (pngs and still gifs), and if those images are taller or wider than 1:30 or 30:1, the phashes are also better quality
- the 'fill in subscription gap' popup button now correctly boots its created downloader when the action also opens a new downloader page. previously, due to overactive safety code, it would hang on 'pending' until a client restart. related similar 'start downloader after creating page' actions off drag and drop or client api should also be more reliable
- .
- repositories (also the various improvements in 449-experimental are folded in):
- fixed an issue with some 'force repository account refresh' code not kicking in immediately
- when a client sees repository update period change, it now recalculates the metadata next check time
- fixed a bug with the new repo sync where updates just added from additive sync were not being processed until client restart. related long-term buggy 'do we have this hash in updates?' and 'how many updates are there?' tests for update metadata are also fixed
- the experimental by-content-type repository reset from last week now leaves pending content in place
- the reset also now clears cached service info counts for files, tags, and mappings

Message too long. Click here to view full text.

Anonymous Board owner 08/11/2021 (Wed) 23:15:45 Id: 612109 [Preview] No.1119 del
next week

Next week is cleanup. The long term database breakup job has been going well, making code simpler and easier to expand, so I think more of that. I also started a new database profiling system this week, and I want to experiment with it a bit.

Release Tomorrow! Anonymous Board owner 08/18/2021 (Wed) 03:58:03 Id: d3a181 [Preview] No.1121 del
I had a great work week. I fixed a heap of bugs!

The release should be normal time tomorrow.

Q&A Thread Anonymous Board owner 08/08/2019 (Thu) 00:24:05 Id: 348093 [Preview] No. 10 [Reply] [Last 50 Posts]
Please feel free to ask questions about hydrus here.

As a reminder, the help and getting started guide is here:

184 posts and 12 images omitted.

Anonymous 07/03/2021 (Sat) 22:00:14 Id: d7fc6b [Preview] No.1097 del
How do I construct an API call that would search the files by tag as number? What about by date?

Anonymous 07/08/2021 (Thu) 19:54:19 Id: ca5173 [Preview] No.1102 del
Is it possible to just grab the tags used in the PTR without syncing to it? I have an HDD and can't sync the entire thing because of it. If I can just grab the tags I don't need to worry about fucking up tags later when I do sync.

Anonymous 07/29/2021 (Thu) 00:56:05 Id: a0e000 [Preview] No.1111 del
can the program import tags from famous booru sites (sankaku, gelbooru, rule34, danbooru...)? i have been using an addon for firefox that downloads images from booru sites into /folder/copywrite/artist+character+website+MD5 hash. and now i have over 40k files divided over 2k folders with very niche copywrite names and artists. all handpicked and downloaded over like 7 years period. its getting way too big for me to consider organising these files into a respectable gallery manually. and i barely open that folder because it makes me depressed and im more used to the booru tag system by now because it does the job and there is more content being added there. but at the same time i dont wanna snap and shift+del the folder because i know i have some good shit there.
i haven't tried this software yet but i'd really love if it can grab the file md5 hash and import all of its tags from other booru sites. such function would be a huge help. then i can go on and manually decide what pics and other files are worth keeping and what is not.

Anonymous 08/03/2021 (Tue) 16:37:39 Id: 987563 [Preview] No.1112 del
noob question: how do I cancel the backups ?

Anonymous 08/16/2021 (Mon) 11:32:23 Id: 9a34ee [Preview] No.1120 del
Is there an undocumented length limit on the system:tag as number comparison queries? The GUI won't let me enter more than 8 digits for the ~ < > operations, while I need 9 for my custom ids and 10 for dates as numbers. The client api also finds nothing for those queries.

Version 448 Anonymous Board owner 07/28/2021 (Wed) 22:02:45 Id: ae2aa4 [Preview] No. 1108 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=VEwVAV3VPw4 [Embed]
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v448/Hydrus.Network.448.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v448/Hydrus.Network.448.-.Windows.-.Installer.exe
app: https://github.com/hydrusnetwork/hydrus/releases/download/v448/Hydrus.Network.448.-.macOS.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v448/Hydrus.Network.448.-.Linux.-.Executable.tar.gz

Hey, I am sorry, Endchan has been down or not allowed login/posting the past couple weeks when I needed to get release out, so there was a gap.

I had an ok couple of weeks. I was pretty ill in the middle, but I got some good work done overall. .wav files are now supported, PSD files get thumbnails, vacuum returns, and the Client API allows much cleverer search.

client api

I have added some features to the Client API. It was more complicated than I expected, so I couldn't get everything I wanted done, but I think this is a decent step forward.

Message too long. Click here to view full text.

1 post omitted.

Anonymous Board owner 07/28/2021 (Wed) 22:04:41 Id: ae2aa4 [Preview] No.1110 del
- in the downloader system, if a download object has any hashes, it now no longer consults urls for pre-import predictions. this saves a little time looking up urls and ensures that the logically stronger hashes take precedence over urls in all cases (previously, they only took precedence when a non-'looks new' status was found)
- fixed an ugly bug in manage tag siblings/parents where tags imported from clipboard or .txt were not being cleaned, so all sorts of garbage with capital letters or leading spaces could be entered. all pairs are now cleaned, and anything invalid skipped over
- the manage tag filter dialog now cleans all imported tag rules when using the 'import' button (issue #768)
- the manage tag filter dialog now allows you to export the current tag filter with the export button
- fixed the 'edit json parse rule' dialog layout so if you transition from a short display to a string match that has complicated controls, it should now expand properly to show them all
- I think I fixed an odd bug where when uploading pending mappings while more mappings were being added, the x/y progress could accurately but unhelpfully continually reset to 0/y, with an ever-decreasing y until it was equal to the value it had at start. y should now always grow
- hydrus servers now put their server header on a second header 'Hydrus-Server', which should allow them to be properly detectable through a proxy that overrides 'Server'
- optimised a critical call in the tag mappings update database routine. for a service with many siblings and parents, I estimate repository processing is 2-7% faster
- optimised the 'add/delete file' database routines in multiple ways, particularly when the file(s) have many deleted tags, and for the local file services, and when the client has multiple tag services
- brushed up a couple of system predicate texts--things like num_pixels to 'number of pixels'
- .
- boring database refactoring:
- repository update file tracking and service id normalisation is now pulled out to a new 'repositories' database module
- file maintenance tracking and database-level file info updates is now pulled out to a new 'files maintenance' database module
- analyse and vacuum tracking and information generation is now pulled out to a new 'db maintenance' database module
- moved more commands to the 'similar files' module
- the 'metadata regeneration' file maintenance job is now a little faster to save back to the database

Message too long. Click here to view full text.

Experimental Release Tomorrow! Anonymous Board owner 08/04/2021 (Wed) 04:02:06 Id: cc1d47 [Preview] No.1113 del
I had a mixed week. I completed a long delayed maintenance routine for repositories, letting them track tags, siblings, and parents processing separately, but it proved much more complicated than I expected, and while I am happy with the work, I have nothing else to show. Since the change also touches core areas of repository processing, I want to do a limited beta test before I roll it out to everyone.

The release should be normal time tomorrow, but it will be an experimental release, only recommended for advanced users.

Version 449 (Experimental) Anonymous Board owner 08/04/2021 (Wed) 22:31:22 Id: e0032d [Preview] No.1114 del
https://youtube.com/watch?v=boFZ3cAws20 [Embed]
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v449/Hydrus.Network.449.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v449/Hydrus.Network.449.-.Windows.-.Installer.exe
app: https://github.com/hydrusnetwork/hydrus/releases/download/v449/Hydrus.Network.449.-.macOS.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v449/Hydrus.Network.449.-.Linux.-.Executable.tar.gz

I had a mixed week. I was able to get a long-planned maintenance routine completed, but that's all I have to show. This is an experimental release, only for advanced users who want to help me test it.

repository processing tracking

Since I haven't got anything really fun this week, and this changes something delicate, I only want advanced users to check it out for now. If you have experience with the program, run a regular backup, sync with the PTR or another repository, and want to help me out, then please update this week and use your repository normally. Let me know if you run into any trouble. One thing I noticed just now is my IRL client didn't want to catch up to some final processing until I restarted it.

Note this update will delete your pending siblings and parents, so commit before you update! I'll make it so it doesn't do this next week.

Message too long. Click here to view full text.

Anonymous Board owner 08/04/2021 (Wed) 22:31:47 Id: e0032d [Preview] No.1115 del
- in review services, you now see definition updates and all a repository's content types processing progress independently (just files for a file repo, but mappings, siblings, and parents for a tag repo). most of the time they will all be the same, but each can be paused separately. it is now possible (though not yet super efficient, since definitions still run 100%) to sync with the PTR and only grab siblings and parents by simply pausing mappings in review services
- I have also split the 'network' and 'processing' sync progress gauges and their buttons into separate boxes for clarity
- the 'fill in content gaps' maintenance job now lets you choose which content types to do it for
- also, a new 'reset content' maintenance job lets you choose to delete and reprocess by content type. the nuclear 'complete' reset is now really just for critical situations where definitions or database tables are irrevocably messed up
- all users have their siblings and parents processing reset this week. next time you have update processing, they'll come back in over about fifteen minutes, and with luck we'll wipe out some years-old legacy bugs and hopefully discover some info about the remaining bugs. most importantly, we can re-trigger this reprocess in just a few seconds to quickly test future fixes
- a variety of tests such as 'service is mostly caught up' are now careful just to test for the currently unpaused content types
- if you try to commit some content that is currently processing-paused, the client now says 'hey, sorry this is paused, I won't upload that stuff right now' but still upload everything else that isn't paused. this is a ' service is caught up' issue
- tag display sync, which does background work to make sure siblings and parents appear as they should, will now not run for a service if any of the services it relies on for siblings or parents is not process synced. when this happens, it is also shown on the tag display sync review panel. this stops big changes like the new sibling/parent reset from causing display sync to do a whole bunch of work before the service is ready and happy with what it has. with luck it will also smooth out new users' first PTR sync too
- clients now process the sub-updates of a repository update step in the order they were generated on the server, which _may_ fix some non-determinant update bugs we are trying to pin down
- all update processing tracking is overhauled. all related code and database access techniques have been brushed up and should use less CPU and fail more gracefully

next week

This work knocked me out. I had half hoped it would be a simple little thing, just splitting one x/y into multiple, but instead it spiralled out into ten different 'ah, but what about that?' and 'man, that's actually been running bad for ages'. Rather than kick out garbage on a core system, I decided to give it some proper time and do extra IRL testing. However, I am behind on messages, recent bug reports, other small work, and the Client API, so I'll now get to that.

Release Tomorrow! Anonymous Board owner 08/11/2021 (Wed) 03:32:22 Id: e3465e [Preview] No.1116 del
I had an ok week. The update storage change last week went well, so that is polished and ready for everyone. I also caught up on some small fixes and quality of life and extended the Client API a little further.

The release should be normal time tomorrow.

Version 446 Anonymous Board owner 07/07/2021 (Wed) 21:26:32 Id: 76b34f [Preview] No. 1099 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=asojeparbK0 [Embed]
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v446/Hydrus.Network.446.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v446/Hydrus.Network.446.-.Windows.-.Installer.exe
app: https://github.com/hydrusnetwork/hydrus/releases/download/v446/Hydrus.Network.446.-.macOS.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v446/Hydrus.Network.446.-.Linux.-.Executable.tar.gz

I had an ok week. The client does not have a huge changelog this week, but the server has a neat privacy improvement.

The PTR is going to be doing a heap of maintenance this week. It will be 'busy' a lot, approximately one hour of busy and then three hours free. Please bear with it, and if you have a million mappings to upload, I recommend you just give it a break and come back later. I am not totally sure how long it will take. Best case it is a day, worst case it might take four or five.

null account

tl;dr: The hydrus server is now even more anon. You don't have to do anything.

Message too long. Click here to view full text.

2 posts omitted.

Anonymous 07/12/2021 (Mon) 03:47:44 Id: 181271 [Preview] No.1103 del
Thanks for the reply. Great program you've made. It's really revealed that the biggest obstacle to organisation was always me. Even with hydrus I can't be fucked to tag most of my shit.

Anonymous Board owner 07/14/2021 (Wed) 06:41:10 Id: 5c598a [Preview] No.1104 del
Yeah, I find it difficult too. The PTR is great, if you can devote the SSD space for it. But you also need to set up workflows and so on just to get through all your inbox. I really haven't figured out the answer there, and for a decade I guess I've slowly seen all my other queues turn infinite too--I add more films every month to my queue than I watch, more vidya than I can play, more music than I can listen to, and more little files through hydrus than I can process.

If everything goes tits up but I end up in a functioning bunker, I'll probably be set for twenty years and finally catch up, ha ha ha.

Release Tomorrow! Anonymous Board owner 07/14/2021 (Wed) 06:41:26 Id: 5c598a [Preview] No.1105 del
I had a great week. I reworked how file status is stored in the database, greatly accelerating many functions all over the program, particularly on very large clients. I also fixed some annoying bugs and added some quality of life.

The release should be as normal tomorrow.

Anonymous 07/15/2021 (Thu) 13:09:06 Id: 7059bb [Preview] No.1106 del
the key to tagging is not to tag fully, but work in waves.

mine is
safe / suggestive / explicit - this is a hard gate so if it doesn't have this tag, its not going to the next tag section
real / drawn - another hard gate but its FAR to useful for me to not get this in on the first parse go
tag further - this implies there is something special there, usually accompanied by favorite, that I need to pay attention to
and a few other 'series' gates

from here it gets more specialized. this method lets me tag 1 file every second or so and makes sure all files are 'new' files, so im not wondering about duplicates and shit like that. that said my method relies on me being done with duplicate parsing which is currently at 1.7 million files, so fuck me.

my next method for mass culling rather then tagging is a 10 star system.

with one of the updates I can progressively add ratings with a hotkey, so they start out at 0 star

Message too long. Click here to view full text.

Anonymous 07/18/2021 (Sun) 01:57:50 [Preview] No.1107 del
I've been meaning to help out with the PTR. I guess this is a reminder. Is it ok to use tor as a proxy for the ptr? I haven't tried yet but is it blocked or are there any security concerns? I really want to start petitioning these random images that are coming up for some of my tag searches.