/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.

Boards | Catalog | Bottom

Check to confirm you're not a robot
Name
Email
Subject
Comment
Password
Drawing x size canvas
File(s)

Remember to follow the rules

Max file size: 350.00 MB

Max files: 5

Max message length: 4096


Q&A Thread Anonymous Board owner 08/08/2019 (Thu) 00:24:05 Id: 348093 [Preview] No. 10 [Reply] [Last 50 Posts]
Please feel free to ask questions about hydrus here.

As a reminder, the help and getting started guide is here:

https://hydrusnetwork.github.io/hydrus/help/index.html
184 posts and 12 images omitted.


Anonymous 07/03/2021 (Sat) 22:00:14 Id: d7fc6b [Preview] No.1097 del
How do I construct an API call that would search the files by tag as number? What about by date?


Anonymous 07/08/2021 (Thu) 19:54:19 Id: ca5173 [Preview] No.1102 del
Is it possible to just grab the tags used in the PTR without syncing to it? I have an HDD and can't sync the entire thing because of it. If I can just grab the tags I don't need to worry about fucking up tags later when I do sync.


Anonymous 07/29/2021 (Thu) 00:56:05 Id: a0e000 [Preview] No.1111 del
can the program import tags from famous booru sites (sankaku, gelbooru, rule34, danbooru...)? i have been using an addon for firefox that downloads images from booru sites into /folder/copywrite/artist+character+website+MD5 hash. and now i have over 40k files divided over 2k folders with very niche copywrite names and artists. all handpicked and downloaded over like 7 years period. its getting way too big for me to consider organising these files into a respectable gallery manually. and i barely open that folder because it makes me depressed and im more used to the booru tag system by now because it does the job and there is more content being added there. but at the same time i dont wanna snap and shift+del the folder because i know i have some good shit there.
i haven't tried this software yet but i'd really love if it can grab the file md5 hash and import all of its tags from other booru sites. such function would be a huge help. then i can go on and manually decide what pics and other files are worth keeping and what is not.


Anonymous 08/03/2021 (Tue) 16:37:39 Id: 987563 [Preview] No.1112 del
noob question: how do I cancel the backups ?


Anonymous 08/16/2021 (Mon) 11:32:23 Id: 9a34ee [Preview] No.1120 del
Is there an undocumented length limit on the system:tag as number comparison queries? The GUI won't let me enter more than 8 digits for the ~ < > operations, while I need 9 for my custom ids and 10 for dates as numbers. The client api also finds nothing for those queries.



Version 448 Anonymous Board owner 07/28/2021 (Wed) 22:02:45 Id: ae2aa4 [Preview] No. 1108 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=VEwVAV3VPw4 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v448/Hydrus.Network.448.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v448/Hydrus.Network.448.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v448/Hydrus.Network.448.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v448/Hydrus.Network.448.-.Linux.-.Executable.tar.gz

Hey, I am sorry, Endchan has been down or not allowed login/posting the past couple weeks when I needed to get release out, so there was a gap.

I had an ok couple of weeks. I was pretty ill in the middle, but I got some good work done overall. .wav files are now supported, PSD files get thumbnails, vacuum returns, and the Client API allows much cleverer search.

client api

I have added some features to the Client API. It was more complicated than I expected, so I couldn't get everything I wanted done, but I think this is a decent step forward.

Message too long. Click here to view full text.

1 post omitted.


Anonymous Board owner 07/28/2021 (Wed) 22:04:41 Id: ae2aa4 [Preview] No.1110 del
- in the downloader system, if a download object has any hashes, it now no longer consults urls for pre-import predictions. this saves a little time looking up urls and ensures that the logically stronger hashes take precedence over urls in all cases (previously, they only took precedence when a non-'looks new' status was found)
- fixed an ugly bug in manage tag siblings/parents where tags imported from clipboard or .txt were not being cleaned, so all sorts of garbage with capital letters or leading spaces could be entered. all pairs are now cleaned, and anything invalid skipped over
- the manage tag filter dialog now cleans all imported tag rules when using the 'import' button (issue #768)
- the manage tag filter dialog now allows you to export the current tag filter with the export button
- fixed the 'edit json parse rule' dialog layout so if you transition from a short display to a string match that has complicated controls, it should now expand properly to show them all
- I think I fixed an odd bug where when uploading pending mappings while more mappings were being added, the x/y progress could accurately but unhelpfully continually reset to 0/y, with an ever-decreasing y until it was equal to the value it had at start. y should now always grow
- hydrus servers now put their server header on a second header 'Hydrus-Server', which should allow them to be properly detectable through a proxy that overrides 'Server'
- optimised a critical call in the tag mappings update database routine. for a service with many siblings and parents, I estimate repository processing is 2-7% faster
- optimised the 'add/delete file' database routines in multiple ways, particularly when the file(s) have many deleted tags, and for the local file services, and when the client has multiple tag services
- brushed up a couple of system predicate texts--things like num_pixels to 'number of pixels'
- .
- boring database refactoring:
- repository update file tracking and service id normalisation is now pulled out to a new 'repositories' database module
- file maintenance tracking and database-level file info updates is now pulled out to a new 'files maintenance' database module
- analyse and vacuum tracking and information generation is now pulled out to a new 'db maintenance' database module
- moved more commands to the 'similar files' module
- the 'metadata regeneration' file maintenance job is now a little faster to save back to the database

Message too long. Click here to view full text.



Experimental Release Tomorrow! Anonymous Board owner 08/04/2021 (Wed) 04:02:06 Id: cc1d47 [Preview] No.1113 del
I had a mixed week. I completed a long delayed maintenance routine for repositories, letting them track tags, siblings, and parents processing separately, but it proved much more complicated than I expected, and while I am happy with the work, I have nothing else to show. Since the change also touches core areas of repository processing, I want to do a limited beta test before I roll it out to everyone.

The release should be normal time tomorrow, but it will be an experimental release, only recommended for advanced users.


Version 449 (Experimental) Anonymous Board owner 08/04/2021 (Wed) 22:31:22 Id: e0032d [Preview] No.1114 del
https://youtube.com/watch?v=boFZ3cAws20 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v449/Hydrus.Network.449.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v449/Hydrus.Network.449.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v449/Hydrus.Network.449.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v449/Hydrus.Network.449.-.Linux.-.Executable.tar.gz

I had a mixed week. I was able to get a long-planned maintenance routine completed, but that's all I have to show. This is an experimental release, only for advanced users who want to help me test it.

repository processing tracking

Since I haven't got anything really fun this week, and this changes something delicate, I only want advanced users to check it out for now. If you have experience with the program, run a regular backup, sync with the PTR or another repository, and want to help me out, then please update this week and use your repository normally. Let me know if you run into any trouble. One thing I noticed just now is my IRL client didn't want to catch up to some final processing until I restarted it.

Note this update will delete your pending siblings and parents, so commit before you update! I'll make it so it doesn't do this next week.

Message too long. Click here to view full text.



Anonymous Board owner 08/04/2021 (Wed) 22:31:47 Id: e0032d [Preview] No.1115 del
- in review services, you now see definition updates and all a repository's content types processing progress independently (just files for a file repo, but mappings, siblings, and parents for a tag repo). most of the time they will all be the same, but each can be paused separately. it is now possible (though not yet super efficient, since definitions still run 100%) to sync with the PTR and only grab siblings and parents by simply pausing mappings in review services
- I have also split the 'network' and 'processing' sync progress gauges and their buttons into separate boxes for clarity
- the 'fill in content gaps' maintenance job now lets you choose which content types to do it for
- also, a new 'reset content' maintenance job lets you choose to delete and reprocess by content type. the nuclear 'complete' reset is now really just for critical situations where definitions or database tables are irrevocably messed up
- all users have their siblings and parents processing reset this week. next time you have update processing, they'll come back in over about fifteen minutes, and with luck we'll wipe out some years-old legacy bugs and hopefully discover some info about the remaining bugs. most importantly, we can re-trigger this reprocess in just a few seconds to quickly test future fixes
- a variety of tests such as 'service is mostly caught up' are now careful just to test for the currently unpaused content types
- if you try to commit some content that is currently processing-paused, the client now says 'hey, sorry this is paused, I won't upload that stuff right now' but still upload everything else that isn't paused. this is a ' service is caught up' issue
- tag display sync, which does background work to make sure siblings and parents appear as they should, will now not run for a service if any of the services it relies on for siblings or parents is not process synced. when this happens, it is also shown on the tag display sync review panel. this stops big changes like the new sibling/parent reset from causing display sync to do a whole bunch of work before the service is ready and happy with what it has. with luck it will also smooth out new users' first PTR sync too
- clients now process the sub-updates of a repository update step in the order they were generated on the server, which _may_ fix some non-determinant update bugs we are trying to pin down
- all update processing tracking is overhauled. all related code and database access techniques have been brushed up and should use less CPU and fail more gracefully

next week

This work knocked me out. I had half hoped it would be a simple little thing, just splitting one x/y into multiple, but instead it spiralled out into ten different 'ah, but what about that?' and 'man, that's actually been running bad for ages'. Rather than kick out garbage on a core system, I decided to give it some proper time and do extra IRL testing. However, I am behind on messages, recent bug reports, other small work, and the Client API, so I'll now get to that.


Release Tomorrow! Anonymous Board owner 08/11/2021 (Wed) 03:32:22 Id: e3465e [Preview] No.1116 del
I had an ok week. The update storage change last week went well, so that is polished and ready for everyone. I also caught up on some small fixes and quality of life and extended the Client API a little further.

The release should be normal time tomorrow.



Version 446 Anonymous Board owner 07/07/2021 (Wed) 21:26:32 Id: 76b34f [Preview] No. 1099 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=asojeparbK0 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v446/Hydrus.Network.446.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v446/Hydrus.Network.446.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v446/Hydrus.Network.446.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v446/Hydrus.Network.446.-.Linux.-.Executable.tar.gz

I had an ok week. The client does not have a huge changelog this week, but the server has a neat privacy improvement.

The PTR is going to be doing a heap of maintenance this week. It will be 'busy' a lot, approximately one hour of busy and then three hours free. Please bear with it, and if you have a million mappings to upload, I recommend you just give it a break and come back later. I am not totally sure how long it will take. Best case it is a day, worst case it might take four or five.

null account

tl;dr: The hydrus server is now even more anon. You don't have to do anything.

Message too long. Click here to view full text.

2 posts omitted.


Anonymous 07/12/2021 (Mon) 03:47:44 Id: 181271 [Preview] No.1103 del
>>1092
Thanks for the reply. Great program you've made. It's really revealed that the biggest obstacle to organisation was always me. Even with hydrus I can't be fucked to tag most of my shit.


Anonymous Board owner 07/14/2021 (Wed) 06:41:10 Id: 5c598a [Preview] No.1104 del
>>1103
Yeah, I find it difficult too. The PTR is great, if you can devote the SSD space for it. But you also need to set up workflows and so on just to get through all your inbox. I really haven't figured out the answer there, and for a decade I guess I've slowly seen all my other queues turn infinite too--I add more films every month to my queue than I watch, more vidya than I can play, more music than I can listen to, and more little files through hydrus than I can process.

If everything goes tits up but I end up in a functioning bunker, I'll probably be set for twenty years and finally catch up, ha ha ha.


Release Tomorrow! Anonymous Board owner 07/14/2021 (Wed) 06:41:26 Id: 5c598a [Preview] No.1105 del
I had a great week. I reworked how file status is stored in the database, greatly accelerating many functions all over the program, particularly on very large clients. I also fixed some annoying bugs and added some quality of life.

The release should be as normal tomorrow.


Anonymous 07/15/2021 (Thu) 13:09:06 Id: 7059bb [Preview] No.1106 del
>>1103
the key to tagging is not to tag fully, but work in waves.

mine is
safe / suggestive / explicit - this is a hard gate so if it doesn't have this tag, its not going to the next tag section
real / drawn - another hard gate but its FAR to useful for me to not get this in on the first parse go
funny
favorite
tag further - this implies there is something special there, usually accompanied by favorite, that I need to pay attention to
art
and a few other 'series' gates

from here it gets more specialized. this method lets me tag 1 file every second or so and makes sure all files are 'new' files, so im not wondering about duplicates and shit like that. that said my method relies on me being done with duplicate parsing which is currently at 1.7 million files, so fuck me.

my next method for mass culling rather then tagging is a 10 star system.

with one of the updates I can progressively add ratings with a hotkey, so they start out at 0 star

Message too long. Click here to view full text.



Anonymous 07/18/2021 (Sun) 01:57:50 [Preview] No.1107 del
>>1099
I've been meaning to help out with the PTR. I guess this is a reminder. Is it ok to use tor as a proxy for the ptr? I haven't tried yet but is it blocked or are there any security concerns? I really want to start petitioning these random images that are coming up for some of my tag searches.



Version 445 Anonymous Board owner 06/30/2021 (Wed) 22:38:46 Id: a5f35a [Preview] No. 1094 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=ulvvQHD4184 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v445/Hydrus.Network.445.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v445/Hydrus.Network.445.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v445/Hydrus.Network.445.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v445/Hydrus.Network.445.-.Linux.-.Executable.tar.gz

I had a great week mostly working on optimisations and cleanup. A big busy client running a lot of importers should be a little snappier today.

optimisations

Several users have had bad UI hangs recently, sometimes for several seconds. It is correlated with running many downloaders at once, so with their help I gathered some profiles of what was going on and trimmed and rearranged some of the ways downloaders and file imports work this week. There is now less stress on the database when lots of things are going on at once, and all the code here is a little more sensible for future improvements. I do not think I have fixed the hangs, but they may be less bad overall, or the hang may have been pushed to a specific trigger like file loads or similar.

So there is still more to do. The main problem I believe is that I designed the latest version of the downloader engine before we even had multiple downloaders per page. An assumed max of about twenty download queues is baked into the system, whereas many users may have a hundred or more sitting around, sometimes finished/paused, but in the current system each still taking up a little overhead CPU on certain update calls. A complete overhaul of this system is long overdue but will be a large job, so I'm going to focus on chipping away at the worst offenders in the meantime.

Message too long. Click here to view full text.



Anonymous Board owner 06/30/2021 (Wed) 22:39:17 Id: a5f35a [Preview] No.1095 del
full list

- misc:
- fixed some weird bugs on the pathname tagging dialog related to removal and re-adding of tags with its 'tags just for selected files' list. previously, in some circumstances, all selected paths could accidentally share the same list of tags, so further edits on a subset selection could affect the entire former selection
- furthermore, removing a tag from that list when the current path selection has differing tags should now successfully just remove that tag and not accidentally add anything
- if your client has a pending menu with 'sticky' small tag count that does not seem to clear, the client now tries to recognise a specific miscount cause for this situation and gives you a little popup with instructions on the correct maintenance routine to fix it
- when pending upload ends, it is now more careful about when it clears the pending count. this is a safety routine, but it not always needed
- when pending count is recalculated from source, it now uses the older method of counting table rows again. the new 'optimised' count, which works great for current mappings, was working relatively very slow for pending count for large services like the PTR
- fixed rendering images at >76800% zoom (usually 1x1 pixels in the media viewer), which had broke with the tile renderer
- improved the serialised png load fix from last week--it now covers more situations
- added a link, https://github.com/GoAwayNow/Iwara-Hydrus, to Iwara-Hydrus, a userscript to simplify sending Iwara videos to Hydrus Network, to the client api help
- it should now again be possible to run the client on Windows when the exe is in a network location. it was a build issue related to modern versions of pyinstaller and shiboken2
- thanks to a user's help, the UPnPc executable discoverer now searches your PATH, and also searches for 'upnpc' executable name as a possible alternative on linux and macOS
- also thanks to a user, the test script process now exits with code 1 if the test is not OK
- .
- optimisations:
- when a db job is reading data, if that db job happens to fall on a transaction boundary, the result is now returned before the transaction is committed. this should reduce random job lag when the client is busy

Message too long. Click here to view full text.



Anonymous Board owner 06/30/2021 (Wed) 22:40:07 Id: a5f35a [Preview] No.1096 del
- the 'review threads' debug UI now has two new tabs for the job schedulers. I will be working with UI-lag-experiencing users in future to see where the biggest problems are here. I suspect part of it will overhead from downloader thread spam, which I have more plans for
- all jobs that threads schedule on main UI time are now profiled in 'callto' profile mode
- .
- site encoding fixes:
- fixed a problem with webpages that report an encoding for which there is no available decoder. This error is now caught properly, and if 'chardet' is available to provide a supported encoding, it now steps in fixes things automatically. for most users, this fixes japanese sites that report their encoding as "Windows-31J", which seems to be a synonym for Shift-JIS. the 'non-failing unicode decode' function here is also now better at not failing, ha ha, and it delivers richer error descriptions when all attempts to decode are non-successful
- fixed a problem detecting and decoding webpages with no specified encoding (which defaults to windows-1252 and/or ISO-8859-1 in some weird internet standards thing) using chardet
- if chardet is not available and all else fails, windows-1252 is now attempted as a last resort
- added chardet presence to help->about. requests needs it atm so you likely definitely have it, but I'll make it specific in requirements.txt and expand info about it in future
- .
- boring code cleanup:
- refactored the base file import job to its own file
- client import options are moved to a new submodule, and file, tag, and the future note import options are refactored to their own files
- wrote a new object to handle current import file status in a better way than the old 'toss a tuple around' method
- implemented this file import status across most of the import pipeline and cleaned up a heap of import status, hash, mime, and note handling. rarely do downloaders now inspect raw file import status directly--they just ask the import and status object what they think should happen next based on current file import options etc...
- a url file import's pre-import status urls are now tested main url first, file url second, then associable urls (previously it was pseudorandom)
- a file import's pre-import status hashes are now tested sha256 first if that is available (previously it was pseudorandom). this probably doesn't matter 99.998% of the time, but maybe hitting 'try again' on a watcher import that failed on a previous boot and also had a dodgy hash parser, it might
- misc pre-import status prediction logic cleanup, particularly when multiple urls disagree on status and 'exclude previously deleted' is _unchecked_

Message too long. Click here to view full text.



Simple Release Tomorrow! Anonymous Board owner 07/07/2021 (Wed) 07:16:02 Id: 51afa1 [Preview] No.1098 del
I had an ok week. I mostly focused on a new privacy improvement for repositories like the PTR, but otherwise I fixed a few bugs and improved some downloader UI.

The release should be as normal tomorrow.



Version 444 Anonymous Board owner 06/23/2021 (Wed) 22:25:58 Id: 8287cf [Preview] No. 1089 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=0OmOQs0ziSU [Embed]
This whole vid is great. Watching a ton of this stuff was a decent whack of my vacation.
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v444/Hydrus.Network.444.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v444/Hydrus.Network.444.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v444/Hydrus.Network.444.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v444/Hydrus.Network.444.-.Linux.-.Executable.tar.gz

I had an ok week getting back to speed. There's some new help regarding the new PTR accounts and I fixed some bugs.

accounts and help

If you use the PTR, you may have seen a note appear on the public account that it is now 'read only' and can now only download updates. If you would like to upload tags, siblings, or parents, please generate your own account either by clicking the popup button you get when trying to upload, or just hit up the PTR on services->manage services and click the easy button there. This pretty much completes the long project me and the guys running the PTR have been working on: moving all uploading users to their own still-anon-but-seperate accounts so the janitors can work through groups of petitions and undo future mistakes without it all being merged together in one account. I have given both of these pages a complete pass to represent the changes and updated screenshots etc... to be more modern:

https://hydrusnetwork.github.io/hydrus/help/access_keys.html

Message too long. Click here to view full text.



Anonymous Board owner 06/23/2021 (Wed) 22:26:37 Id: 8287cf [Preview] No.1090 del
full list

- gave the 'access keys' and 'privacy' help pages a complete pass. the access keys section talks about the read-only shared key, and how to generate you own account, and the privacy section now compiles, as comprehensively as I could, our various discussions about multiple accounts, what you shouldn't upload to the PTR (basically your own name lol), self-signed https certificates, and what information is actually stored on an account
- expanded the 'getting started - installing' help page with a 'how to run the client' section, including bundling the excellent Linux virtual memory guide written by a user
- fixed the new 'fill in subscription with gap downloader' button, which was initialising with the wrong downloader at times (usually on the first gap downloader opened, when it opened a new page with it)
- you can now set 'all known files' for the tag autocomplete in 'write' contexts (e.g. manage tags dialog) when not in advanced mode
- cleaned up how a variety of delayed UI calls are registered and present information about themselves. every UI job now has a nice human name for debug purposes. this should improve program stability and clear some odd rare errors when closing some dialogs (this mostly affected certain linux users)
- when an asynchronous UI job fails with a dead window, or if fails to publish to its window for a non-dead reason and then the window dies before that failure returns, the error handling code now catches and silences the error. an example of this would be clicking 'refresh account' on review services, then closing the window before the lagging job raises 'connection failure'
- when windows are rescued from off screen, their frame key is now stated in the popup note
- if your version of OpenCV is unable to load PNG files, your client should now be able to load serialised object PNGs (like those in the downloader system) correctly (the same PIL fallback for regular media files now works for deserialisation too)
- the hydrus log path is finally month-zero-padded, ha ha ha
- misc cleanup and label fixes

next week

I got caught up in some behind the scenes admin work this week (mostly fun updating my IDE to python 3.8), and the help docs, which I really wanted to fill out, so I regret my actual changelog is so light. Should be back to full steam next week. I'll hammer away at the reports and profiles I have of lagging UI and get back to grinding at multiple local file services.


Anonymous 06/24/2021 (Thu) 21:27:34 Id: a315a8 [Preview] No.1091 del
Does the watcher support watching twitter profiles? If not what do you recommend for scraping twitter.


Anonymous Board owner 06/30/2021 (Wed) 07:30:20 Id: a85949 [Preview] No.1092 del
>>1091
Try the 'subscriptions' system:

https://hydrusnetwork.github.io/hydrus/help/getting_started_subscriptions.html

For twitter, which can be pretty quick, I recommend setting the 'checker options' on the subscription so it checks at least once a day.


Release Tomorrow! Anonymous Board owner 06/30/2021 (Wed) 07:30:38 Id: a85949 [Preview] No.1093 del
I had a great week mostly working on optimisation and cleanup. Busy clients should be a bit snappier. I also fixed some bugs!

The release should be as normal tomorrow.



Version 443 Anonymous Board owner 06/09/2021 (Wed) 22:26:11 Id: 8246a0 [Preview] No. 1085 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=NgYIIPszZjA [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v443/Hydrus.Network.443.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v443/Hydrus.Network.443.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v443-macos/Hydrus.Network.443-macos.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v443/Hydrus.Network.443.-.Linux.-.Executable.tar.gz

I had a great week doing nice cleanup and quality of life work.

Hey, we had a problem getting the macOS release to build this week. The macOS link above goes to a build using a simpler and faster method. It should work fine, but please let me know if you have any trouble. As always, back up before you update!

highlights

Popup messages can now launch complex jobs from a button. The first I've added is when a subscription hits its 'periodic' file limit. The situation itself is now better explained, and a button on the popup will create a new downloader page with the specific query set up with an appropriate file limit to fill in the gap. The second is if you try to upload some content to a repository that your account does not have permission for (this is affecting sibling- and parent-uploading PTR users as the shared public account is changing), the popup message that talks about the issue now has a button that takes you straight to the manage services panel for the service and starts automatic account creation.

Message too long. Click here to view full text.



Anonymous Board owner 06/09/2021 (Wed) 22:26:53 Id: 8246a0 [Preview] No.1086 del
full list

- quality of life:
- when subscriptions hit their 'periodic file limit', which has always been an overly technical term, the popup message now explains the situation in better language. it also now provides a button to automatically fill in the gap via a new gallery downloader page called 'subscription gap downloaders' that gets the query with a file limit five times the size of the sub's periodic download limit
- I rewrote the logic behind the 'small initial sync, larger periodic sync' detection in subscription sync, improving url counting and reliability through the third, fourth, fifth etc... sync, and then generalised the test to also work without fixed file limits and for large-gallery sites like pixiv, and any site that has URLs that often produce multiple files per URL. essentially, subs now have a nice test for appropriate times to stop url-adding part way through a page (typically, a sub will otherwise always add everything up to the end of a page, in order to catch late-tagged files that have appeared out of order, but if this is done too eagerly, some types of subs perform inefficiently)
- this matters for PTR accounts: if your repository account does not have permissions to upload something you have pending, the popup message talking about this now hangs around for longer (120 seconds), explains the issue better, and has a button that will take you directly to the _manage services_ panel for the service and will hit up 'check for auto-account creation'
- in _manage services_, whenever you change the credentials (host, port, or access key) on a restricted service, that service now resets its account to unknown and flags for a swift account re-fetch. this should solve some annoying 'sorry, please hit refresh account in _review services_ to fix that manually' problems
- a new option in maintenance and processing allows you to disable idle mode if the client api has had a request in the past x minutes. it defaults disabled
- an important improvement to the main JobScheduler object, which farms out a variety of small fast jobs, now massively reduces Add-Job latency when the queue is very busy. when you have a bunch of downloaders working in the background, the UI should have much less lag now
- the _options->speed and memory_ page has a full pass. the thumbnail, image, and image tile caches now have their own sections, there is some more help text, and the new but previously hardcoded 10%/25% cache and prefetch limits are now settable and have dynamic guidance text that says 'about a 7,245x4,075 image' as image cache options change
- all the cache options on this page now apply instantly on dialog ok. no more client restart required!
- .
- other stuff, mostly specific niche work:
- last week's v441->442 update now has a pre-run check for free disk space. users with large sessions may need 10GB or more of free space to do the conversion, and this was not being checked. I will now try to integrate similar checks into all future large updates
- fixed last week's yandere post parser link update--the post url class should move from legacy moebooru to the new yandere parser correctly
- the big maintenance tasks of duplicate file potentials search and repository processing will now take longer breaks if the database is busy or their work is otherwise taking a long time. if the client is cluttered with work, they shouldn't accidentally lag out other areas of the program so much
- label update on ipfs service management panel: the server now reports 'nocopy is available' rather than 'nocopy is enabled'

Message too long. Click here to view full text.



Anonymous Board owner 06/09/2021 (Wed) 22:27:21 Id: 8246a0 [Preview] No.1087 del
- 'database is complicated' menu label is updated to 'database is stored in multiple locations'
- _options->gui pages->controls_ now has a little explanatory text about autocomplete dropdowns and some tooltips
- migrate database dialog has some red warning text up top and a small layout and label text pass. the 'portable?' is now 'beneath db?'
- the repositery hash_id and tag_id normalisation routines have two improvements: the error now shows specific service_ids that failed to lookup, and the mass-service_hash_id lookup now handles the situation where a hash_id is mapped by more than one service_id
- repository definition reprocessing now corrects bad service_id rows, which will better heal clients that previously processed bad data
- the client api and server in general should be better about giving 404s on certain sorts of missing files (it could dump out with 500 in some cases before)
- it isn't perfect by any means, but the autocomplete dropdown should be a _little_ better about hiding itself in float mode if the parent text input box is scrolled off screen
- reduced some lag in image neighbour precache when the client is very busy
- .
- boring code cleanup:
- removed old job status 'begin' handling, as it was never really used. jobs now start at creation
- job titles, tracebacks, and network jobs are now get/set in a nicer way
- jobs can now store arbitrary labelled callable commands, which in a popup message becomes a labelled button
- added some user callable button tests to the 'make some popups' debug job
- file import queues now have the ability to discern 'master' Post URLs from those that were created in multi-file parsing
- wrote the behind the scenes guts to create a new downloader page programmatically and start a subscription 'gap' query download
- cleaned up how different timestamps are tracked in the main controller

Message too long. Click here to view full text.



Release Tomorrow! Anonymous Board owner 06/23/2021 (Wed) 06:12:49 Id: 9e853b [Preview] No.1088 del
I had an ok week getting back to speed. I got caught up in some admin and help doc work, so my changelog is a little light, but I was able to clear out some bugs and other annoyances.

The release should be as normal tomorrow.



Version 442 Anonymous Board owner 06/03/2021 (Thu) 00:01:47 Id: 69c54e [Preview] No. 1078 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=bpEFn3MFyfA [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v442/Hydrus.Network.442.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v442/Hydrus.Network.442.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v442/Hydrus.Network.442.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v442/Hydrus.Network.442.-.Linux.-.Executable.tar.gz

I had a great week. An important part of GUI Sessions is overhauled, which should save a lot of hard drive time for larger clients.

gui sessions

I always encourage a backup before you update, but this week it matters more than normal. If you have a client with large sessions with many important things set up, make sure you have a backup done before you update! I feel good about the code, and I try to save data on various failures, but if your situation gives errors for an unforeseen reason, having the backup ready reduces headaches all around!

Like the subscriptions and network objects breakups I've done in the past year, I 'broke up' the monolithic GUI Session object this week. Now, when your session has changes, only those pages that have changed will be saved, saving a ton of CPU and HDD write I/O. Furthermore, sessions that share duplicate pages (this happens all the time with session backups), can now share that stored page, saving a bunch of hard drive space too. Like with subscriptions, some users are pushing multiple gigabytes of session storage total, so there is a good amount of work to save here.

Message too long. Click here to view full text.



Anonymous Board owner 06/03/2021 (Thu) 00:02:19 Id: 69c54e [Preview] No.1079 del
the rest

I did a bunch of little stuff--check the changelog if you are interested.

I have also turned off the interval VACUUM maintenance and hidden the manual task for now. This was proving less and less useful in these days of huge database files, so I will bring it back in future on a per-file basis with some UI and more specific database metadata.

EDIT: Thanks to a user submission, yande.re post parser is updated to pull tags correctly if you are logged in. I hoped my update code would move the link over from the old parser correct, but it did not. I'll fix this for next week, but if you download from yande.re while logged in, please hit network->downloader components->manage url class links and move 'yande.re file page' from moebooru to 'yande.re post page parser'.

We fixed a couple more problems with the new builds--the Linux and Windows extract builds have their surplus 'ubuntu'/'windows' directories removed, and the Linux executables should have correct permissions again. Sorry for the trouble!

And after some tests, we removed the .py files and the source from the builds. I long-believed it was possible to run the program from source beside the executables, but it seems I was mistaken. Unless you are running the build-adjacent source pretty much on the same machine you built on (as my tests years ago were), you get dll conflicts all over the place. If you want to run from source, just extract the source proper in its own fresh directory. I've also fleshed out the 'running from source' help beyond setting up the environment to talk more about the actual downloading and running of the program. I'll continue work here and hope to roll out some easy one-and-done setup scripts to automate the whole thing.

full list

- gui sessions:
- gui sessions are no longer a monolithic object! now, each page is stored in the database separately, and when a session saves, only those pages that have had changes since the last save are written to db. this will massively reduce long-term HDD writes for clients with large sessions and generally reduce lag during session save intervals
- the new gui sessions are resilient against database damage--if a page fails to load, or is missing from the new store, its information will be recorded and saved, but the rest of the session will load

Message too long. Click here to view full text.



Anonymous Board owner 06/03/2021 (Thu) 00:03:24 Id: 69c54e [Preview] No.1080 del
- the page and session objects are now more healthily plugged into my serialisation system, so it should be much easier to update them in future (e.g. adding memory for tag sort or current file selection)
- .
- the rest:
- when subscriptions die, the little reporting popup now includes the death file velocity ('it found fewer than 1 files in the last 90 days' etc...)
- the client no longer does vacuums automatically in idle time, and the soft/full maintenance action is removed. as average database size has grown, this old maintenance function has increasingly proved more trouble than it is worth. it will return in future as a per-file thing, with better information to the user on past vacuums and empty pages and estimates on duration to completion, and perhaps some database interrupt tech so it can be cancelled. if you really want to do a vacuum for now, do it outside the program through a SQLite intepreter on the files separately
- thanks to a user submission, a yande.re post parser is added that should grab tags correct if you are logged in. the existing moebooru post parser default has its yande.re example url removed, so the url_class-parser link should move over on update
- for file repositories, the client will not try to sync thumbnails until the repository store counts as 'caught up' (on a busy repo, it was trying to pull thumbs that had been deleted 'in the future'). furthermore, a 404 error due a thumb being pulled out of sync will no longer print a load of error info to the log. more work will be needed here in future
- I fixed another stupid IPFS pin-commit bug, sorry for the trouble! (issue #894)
- some maintenance-triggered file delete actions are now better about saving a good attached file delition reason
- when the file maintenance manager does a popup with a lot of thumbnail or file integrity checks, the 'num thumbs regenned/files missing or invalid' number is now preserved through the batches of 256 jobs
- thoroughly tested and brushed up the 'check for missing/invalid files' maintenance code, particularly in relation to its automatic triggering after a repository processing problem, but I still could not figure out specifically why it is not working for some users. we will have to investigate and try some more things
- fixed a typo in client api help regarding the 'service_names_to_statuses_to_display_tags' variable name (I had 'displayed' before, which is incorrect)
- .
- build fixes:
- fixed the new Linux and Windows extract builds being tucked into a little 'ubuntu'/'windows' subfolder, sorry for the trouble! They should both now have the same (note Caps) 'Hydrus Network' as their first directory
- fixed the new Linux build having borked permissions on the executables, sorry for the trouble!
- since I fixed the urllib3 problem we had with serialised sessions and Retry objects, I removed it from the requirements.txts. now 'requests' can pull what it likes

Message too long. Click here to view full text.



Anonymous Board owner 06/03/2021 (Thu) 00:05:43 Id: 69c54e [Preview] No.1081 del
- misc boring refactoring and db updates:
- created a new 'pages' gui module and moved Pages, Thumbs, Sort/Collect widgets, Management panel, and the new split Session code into it
- wrote new container objects for sessions, notebook pages, and media pages, and wrote a new hash-based data object for a media page's management info and file list
- added a table to the database for storing serialised objects by their hash, and updated the load/save code to work with the new session objects and manage shared page data in the hashed storage
- a new maintenance routine checks which hashed serialisables are still needed by master containers and deletes the orphans. it can be manually fired from the _database->maintenance_ menu. this routine otherwise runs just after boot and then every 24 hours or every 512MB of new hashed serialisables added, whichever comes first
- management controllers now discard the random per-session 'page key' from their serialised key lookup, meaning they serialise the same across sessions (making the above hash-page stuff work better!)
- improved a bunch of access and error code around serialised object load/save
- improved a heap of session code all over
- improved serialised object hashing code

next week

I have one more week of work before my vacation. There's a ton of little jobs I have been putting off--checking new downloaders users sent in, some more help docs to work on, and magically growing multi-column list dialogs--as well as emails and other messages I haven't got to. I'll try to tidy up those loose ends as best I can before I take my break. I'll also deal with any problems with these new GUI Sessions.


Anonymous 06/05/2021 (Sat) 04:57:10 Id: 5b4011 [Preview] No.1083 del
excellent update hydev, all of my previous stutters are gone! thank you very much!


Release Tomorrow! Anonymous Board owner 06/09/2021 (Wed) 06:31:03 Id: 1dfb0b [Preview] No.1084 del
I had a great week working on small quality of life issues. A couple of bugs are fixed, some UI lag is reduced, and I worked on some layout too. Just a mix of cleanup before my vacation next week.

I have some unavoidable IRL tomorrow, so the release may be a bit later than usual.

>>1083
Great, thanks for letting me know!



Version 441 Anonymous Board owner 05/27/2021 (Thu) 01:22:04 Id: d94f94 [Preview] No. 1074 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=EJLNLWv-nmM [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v441/Hydrus.Network.441.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v441/Hydrus.Network.441.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v441/Hydrus.Network.441.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v441/Hydrus.Network.441.-.Linux.-.Executable.tar.gz

I had an ok week. Not as much as I wanted, but there are some nice Client API improvements.

all misc this week

The test builds from last week seem to work ok, so they are now master. The built clients now use Python 3.8, and the security libraries (like OpenSSL) are all much newer--and will reliably stay up to date in future--so a whole bunch of things across the client should have slightly better performance. There are no special install instructions, they seem to work on an existing install just as normal. Let me know if you do run into any problems!

I fixed some more bad tiles calculations for the new tiled image renderer. Some files that seem to have little black lines on an edge at some zooms, or previews that just turn up black, should be fixed! Error reporting is also nicer.

Message too long. Click here to view full text.



Anonymous Board owner 05/27/2021 (Thu) 01:22:30 Id: d94f94 [Preview] No.1075 del
- client api additions:
- the client api has a new /manage_headers/set_user_agent call, which is a simple hack for now for external programs to set the 'Global' User-Agent. it should allow for some CloudFlare solutions when just copying cookies is not enough
- the client api has a new /get_services call, which talks about more services and also exposes service_keys for the first time, which are likely to be useful in future. check out the help for an example. the old /add_tags/get_tag_services call is now deprecated, please move to the new call
- the client api /version call now responds with 'hydrus_version' as well, which this week will be 441
- the client api now has a semi-experimental /manage_database/lock system, just like the server's. a new 'manage database' permission is added for this. don't play around with this system idly.
- the client api should now support sha256 hash parameters if they start with a type prefix like 'sha256:0123789abcdef...'
- the client and server's database lock commands now wait up to five seconds for the database to finish disconnecting to respond
- expanded client api unit tests to cover the above
- the client api version is now 17
- .
- boring multiple local file services work:
- the main search object now stores the file domain using a new 'location context' object that will in future hold multiple file services and can say whether we should search files currently in a domain, or those once deleted from it. a variety of back-end search code has been updated to deal with this more flexible situation
- removed more static references to the single 'my files' domain in db and related code. in a couple places, like mr. bones, it now fetches 'all local files', but this will likely be updated in future to a new umbrella 'all non-trash, non-repo-update-files local files' service

next week

I've had some real trouble keeping up recently, but that's ok. A bunch of it is out of my control, so I'll keep pushing anyway. Next week is due to be a 'medium' job week, and I would like to break up the gui session object into smaller pieces. Instead of saving the whole thing, it'll track and save and share individual pages. This will greatly reduce the random CPU lag and HDD use on any client with a large session, let crazy users to store more than 500,000 files in a session at once, and allow us to save changes more often. Basically the same improvement I made to subscriptions and the network objects in the last year, but for gui sessions.

Message too long. Click here to view full text.



Release Tomorrow! Anonymous Board owner 06/02/2021 (Wed) 05:06:07 Id: 6f2170 [Preview] No.1077 del
I had a great week. I succeeded in overhauling the client's GUI sessions, greatly reducing the storage and write I/O required for sessions. This particularly benefits clients that have sessions storing many files or URLs.

The release should be as normal tomorrow.



unable to keep my tags visible and overall finicky xPinkyDash 05/26/2021 (Wed) 07:31:28 Id: 1c9594 [Preview] No. 1073 [Reply] [Last 50 Posts]
whattup,

I've been fiddling with hydrus for a bit but cannot figure out how to keep my tags visible and usable at all times. I have two tags so far, but expect to use dozens, if not hundreds in total in the future.

The whole thing isn't easy for me introductory tutorial that I've gone over so far has basically been "select image, press f3, do ur tag stuff".

Can I see all my own created tags without selecting an image and pressing f3 and be able to edit/delete them? Also, how do I keep tags in the "selection tags" section at all times?

Thanks in advance.


Anonymous 05/31/2021 (Mon) 08:10:43 Id: d5fc20 [Preview] No.1076 del
I don't think it's possible. "Selection tags" is for tags applied to the images you have currently selected. Standard intended usage involves literally thousands of tags (I have 7k and I tag somewhat sparsely), which would be taxing on the database and kind of useless to have visible at all times. You can set the suggested tags options to show you more suggestions faster though by using "manage tag display and search" under tags on the top bar.



Version 440 Anonymous Board owner 05/19/2021 (Wed) 23:11:21 Id: df61fc [Preview] No. 1069 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=mO1j6xx2HhQ [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v440/Hydrus.Network.440.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v440/Hydrus.Network.440.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v440/Hydrus.Network.440.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v440/Hydrus.Network.440.-.Linux.-.Executable.tar.gz

I had an unfortunately short week, but I did some good work. The tiled renderer has nice fixes.

tiled renderer

I regret the tiled renderer, while good most of the time, had crashes when it did go wrong. To stop this with any new errors that might pop up, the whole block now has an additional layer of error catching around it. If a tile fails to render for any reason, you now get a black square, and if some coordinate space cannot be calculated or a portion of the image is truncated, the system will now print some errors but otherwise ignore it.

A particular problem several users encountered was legacy images that have EXIF rotation metadata but were imported years ago when the client did not understand this. Therefore, hydrus thought some old image was (600x900) when it then loaded (900x600). In the old system, you would have had a weird stretch, maybe a borked rotation, but in my new tiled system it would try to draw tiles that didn't exist, causing our errors-then-crashes. The client now recognises this situation, gives you a popup, and automatically schedules metadata regeneration maintenance for the file.

Message too long. Click here to view full text.



Anonymous Board owner 05/19/2021 (Wed) 23:12:10 Id: df61fc [Preview] No.1070 del
new builds to test

This is probably just for advanced users. If you would like to help test, make sure you have a backup before you test anything on a real database!

A user has been working hard on replicating the recent macOS build work for the other releases, cribbing my private build scripts together into a unified file that builds on github itself from the source, as well as rolling out a Docker package. I have had a look over everything and we agree it is ready for wider testing, so if you would like to help out, please check out the test v440 builds here:

https://github.com/hydrusnetwork/hydrus/releases/tag/v440-test-build

These should work just like my normal builds above--the scripts are using PyInstaller and InnoSetup as I do on my machines, so it all assembles the same way--but we are interested in any errors you nonetheless encounter. We may need to hammer out some stricter library version requirements for older machines, since until now we've basically been relying on my home dev environments staying static until I next remember to run pip update.

Once we have these working well, I'd like to use this system for the main build. It makes things easier and more reliable on my end, and should improve security since the builds are assembled automatically in clean environments with publicly viewable scripts rather than my random-ass dev machines using my own dlls, batch files, and prayers. Who knows, we may even defeat the anti-virus false positives.

Also, if you would like to try the Docker package, check it out here:

https://github.com/users/hydrusnetwork/packages/container/package/hydrus

I don't know much about Docker, so while I can't help much, I'll still be interested in any feedback. If and when we are ready to switch over here, I'll be updating my help with any appropriate new backup instructions and links and so on.

Message too long. Click here to view full text.



Anonymous Board owner 05/19/2021 (Wed) 23:13:17 Id: df61fc [Preview] No.1071 del
- the rest:
- added a prototype 'delete lock' for archived files to _options->files and trash_ (issue #846). this will be expanded in future when the metadata conditional object is made to lock various other file states, and there will be some better UI feedback, a padlock icon or similar, and some improved dialog texts. if you use this, let me know how you get on!
- you can now set a custom namespace sort in the file sort menu. you have to type it manually, like when setting defaults in the options, but it will save with the page and should load up again nicely in the dialog if you edit it. this is an experiment in prep for better namespace sort edit UI
- fixed an issue sorting by namespaces when one of those namespaces was hidden in the 'single media' tag context. now all 'display' tags are used for sort comparison groups. if users desire the old behaviour, we'll have to add an option, so let me know
- the various service-level processing errors when update files are missing or janked out now report the actual hash of the bad update file. I am chasing down one of these errors with a couple of users and cannot quite figure out why the repair code is not auto-fixing things
- fixed a problem when the system tray gets an activate event at unlucky moments
- the default media viewer zoom centerpoint is now the mouse
- fixed a typo in the client api with wildcard/namespace tag search--sorry for the trouble!
- .
- some boring multiple local file services cleanup:
- if you have a mixture of trash and normal thumbnails selected, the right-click menu now has separate choices for 'delete trash' and 'delete selected' 'physically now'
- if you have a mixture of trash and normal thumbnails selected, the advanced delete dialog now similarly provides separate 'physical delete' options for the trashed vs all
- media viewer, preview viewer, and thumbnail view delete menu service actions are now populated dynamically. it should say 'delete from my files' instead of just 'delete'
- in some file selection contexts, the 'remote' filter is renamed to 'not local'

next week

Message too long. Click here to view full text.



Release Tomorrow! Anonymous Board owner 05/26/2021 (Wed) 05:28:12 Id: d45e3b [Preview] No.1072 del
I had an ok week. I did a mix of different small work and added some new commands to the Client API, including something that should help some difficult login situations.

The release should be as normal tomorrow.