/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.

Boards | Catalog | Bottom

Check to confirm you're not a robot
Drawing x size canvas

Remember to follow the rules

Max file size: 350.00 MB

Max files: 5

Max message length: 4096

Version 439 Anonymous Board owner 05/12/2021 (Wed) 22:16:52 Id: 036f18 [Preview] No. 1065 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=CyPKxkH3vB8 [Embed]
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v439/Hydrus.Network.439.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v439/Hydrus.Network.439.-.Windows.-.Installer.exe
app: https://github.com/hydrusnetwork/hydrus/releases/download/v439/Hydrus.Network.439.-.macOS.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v439/Hydrus.Network.439.-.Linux.-.Executable.tar.gz

I had an ok week. The new tiled renderer is improved.

tiled renderer

The new image drawing system generally worked well! There were a couple of bugs, and it still has some limitations, but in general it really improved zoom and precache performance.

For the bugs, first of all, there was a rare crash, I think triggered by loading a very unlucky coincidence of tile and image size. Then clipboard bitmap copy threw an error, tiny images could not deal with extremely small zoom, and clients under heavy load could sometimes have trouble initialising the viewer. I have fixed them all, but let me know if you have any more trouble!

Message too long. Click here to view full text.

Anonymous Board owner 05/12/2021 (Wed) 22:17:16 Id: 036f18 [Preview] No.1066 del
full list

- tiled image renderer improvements:
- I believe I fixed the 'non c-contiguous' crash issue with the new tile renderer. I had encountered this while developing, but it was still happening in rare situations--I _think_ in an unlucky edge case where a zoomed tile had the same resolution as the full image rotated by ninety degrees! there is now an additional catch for this situation, as well, to catch any future logical holes.
- fixed a bug in the new renderer when copying an image to clipboard
- I greatly mitigated the tiling artifacts with two changes:
- - zoomed in tiles are now resized with a padding area of up to 4 pixels, with the actual tile cropped afterwards, which allows bilinear and lancsoz interpolation to get accurate neighbour data and have gradient math line up with neighbouring tiles more accurately
- - on resize and zoom, media canvases now dynamically change tile size to 'neater' float/integer conversion dimensions to reduce sub-pixel panning alignment artifacts (e.g. if your zoom is 300%, the tile is now going to have a dimension that is a multiple of 3)
- I hacked in a 'rescue offscreen media' calculation after any zoom event. now, if the window is completely out of view after a zoom, it'll snap to the nearest borders, lining against them or overlapping into a buffer zone depending on the zoom. let me know what you think!
- I fixed a PyQt5 specific object tracking bug, I think the new renderer now works ok for PyQt5!
- cleaned up some ugly code in the resize section that may have been resulting in incorrect interpolation algorithm choice in some situations
- fixed a divide by zero issue when zooming out tiny images hugely (e.g. 32x32 at 1%)
- media windows now try to have at least 1x1 size, just to catch some other weird error situations
- similarly, tile and native sample sizes will have a minimum of size 1x1, which should fix issues during a delayed startup (issue #872)
- cleaned up some misc media viewer and tile renderer code
- .
- the rest:

Message too long. Click here to view full text.

Anonymous Board owner 05/12/2021 (Wed) 22:17:42 Id: 036f18 [Preview] No.1067 del
- as the "Retry has no attribute..." network errors have appeared in new forms, I gave the core of the problem another look. we could never really figure this out, but it seemed to be a network version thread safety issue. I think I have ruled this out, and I now believe these may have been occuring during faulty pickling during network session save/load. I fixed the problem here, so with luck this issue will not reappear--if you have had this a lot, let me know how you get on!
- I broke the requirements.txt into several variants based on platform. we are going to try to pin down good fixed versions of python-mpv and requests/urllib3 for each platform
- I also updated the 'running from source' help significantly, moving everything to the requirements.txt and making sections for things like FFMPEG and libmpv
- Also updated the source and contact help around my work style and contact preferences
- the test.py file now only does the final input() confirmation if there is an interactive stdin to respond

next week

Next week is code cleanup and some little jobs that have slipped through the cracks. Nothing too clever, but I want to fit in some misc boring work.

Thanks everyone!

Small Bugfix Release Tomorrow! Anonymous Board owner 05/19/2021 (Wed) 06:10:44 Id: ed884a [Preview] No.1068 del
I had an ok week, but my work time was unfortunately cut short by IRL. I might not normally do a release, but I'd like to get some neat fixes out for the new tiled renderer, which has additional protections in 440 and should no longer crash. There's also some misc cleanup and quality of life.

The release should be as normal tomorrow.

Version 438 Anonymous Board owner 05/05/2021 (Wed) 21:20:01 Id: 0734e0 [Preview] No. 1059 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=LE8QKcriHH4 [Embed]
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v438/Hydrus.Network.438.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v438/Hydrus.Network.438.-.Windows.-.Installer.exe
app: https://github.com/hydrusnetwork/hydrus/releases/download/v438/Hydrus.Network.438.-.macOS.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v438/Hydrus.Network.438.-.Linux.-.Executable.tar.gz

Hey, this causes errors if you are running from source and using PyQt5 (PySide2 is fine)! All the releases above are PySide2, so they are ok! I will fix this for next week, so if you are source+QtPy5, please hold off for now.

I had a great week overhauling the media viewer's image rendering. Zooming and navigation should be a lot smoother now!

image tiles

tl;dr: the media viewer now zooms and navigates with less lag and flicker

Message too long. Click here to view full text.

Anonymous Board owner 05/05/2021 (Wed) 21:20:31 Id: 0734e0 [Preview] No.1060 del
I am really pleased with this week's work, but there are some drawbacks: I did it quick, so I cannot promise it is good. The most obvious bug already is that at around 200-500% zoom you start to see tiling artifacts. I know what causes this (interpolation algorithms not getting full pixel neighbour data from my simple tesselating tiles) and have a plan to fix it (adding a tile border pre-resize, and then cropping). There is also an issue when the 'virtual' image exceeds about 32,000x32,000, so I hacked a zoom block for that. There may be some weird files that render with other stitching artifacts or bad tile data. Note also that hydrus's 'Animation' renderer (the soundless fallback if you do not have mpv support) does NOT use tiling yet, so it still sucks at zooming! Please let me know how you get on!

If you have a steam-powered GPU or a machine with only 4GB of ram, you might like to wait for 439 so I can address any surprise bugs or performance issues.

PTR and account permissions

The PTR is changing how its accounts work. The shared public account is transforming to a 'read-only' account that can only download, so if you want to upload, you'll be going to manage services to auto-create your own privileged account. This is being done to improve janitor workflow for the various petitions, which were all being merged together because of the shared account. With the recent network updates, it will soon be easier for janitors to send simple messages back to these individual accounts, like 'that proposed sibling was not approved because...'.

Unfortunately, various permission and account-management code has not been tested much until now, so as the PTR guys have been trying this stuff out, I have been working to improve bad notifications and workflows. This week I rounded out account permissions testing with uploading. Hydrus no longer tries to upload content the current account does not have permission for, and if you end up in that situation, popup messages now tell you what is going on. It also catches if your account is currently 'unsynced', with instructions to fix.

Similarly, under 'manage siblings/parents', you can now see and edit all tag repositories (previously, they were hidden if you currently had no permission), but you get a label telling you if you don't have permission.

full list

- media viewer:
- I have hacked in tile-based image rendering for the media viewer. this has always been planned as a larger, longer-term job, but the problem of large images is only getting worse, so I decided to just slam out a prototype in a week. if you have a steam-powered GPU or 4GB ram, you might like to wait until next week to update so I can iron out any surprise bugs or performance problems
- images are now cut into tiles that are rendered on demand, so whenever the image is zoomed larger than the media viewer window, only those tiles currently in view have CPU and memory spent on resizing and storage. as you pan around, new tiles are rendered as needed, and old discarded. this makes zooming in super fast and low memory, even for large images!

Message too long. Click here to view full text.

Anonymous Board owner 05/05/2021 (Wed) 21:21:13 Id: 0734e0 [Preview] No.1061 del
- futhermore, a new 'image tile' cache is added. you can customise size and timeout under _options->speed and memory_ like for images and thumbnails. this is a dedicated cache for remembering image resize computation across images and zooms. once you have seen both situations once, flicking back and forth between two images or zoom levels is now generally always instant! this new cache starts at a healthy default of 256MB. let's see how that amount works out IRL--I think it will be plenty
- I tuned the image renderer cache--it no longer caches huge images that eat more than 25% its total size--meaning these images only hang around as long as you are looking at them--and the prefetch call that pre-renders several files previous/next to the current image no longer occurs on images that would eat more than 10% the cache size. this should greatly reduce weird flicker and other lag when browsing through a series of mega-images (which before would stomp through the cache in quick succession, barging each other out of the way and wasting a bunch of CPU). in real world terms, this basically means that with an image cache of 200MB, you should have slower individual image performance but much better overall performance looking at images with more than about 5k resolution. the dreaded 14,000x12,000 png will still bonk you on the head to do the first render, but it won't try to uselessly prefetch or flush the whole cache any more
- if you are currently looking at a static image, neighbour prefetch now only starts once the image is rendered, giving the task in front of you a bit more CPU time
- new options for prefetch delay and previous/next distance are added to 'speed and memory'
- note this does not yet apply to the old hydrus animation renderer. that still sucks at high zoom!
- another future step here is to expand prefetch to tiles so the first view of the 'next' media is instant, but let's let all this breathe for a bit. if you get bugs, let me know!
- due to a Qt issue, I am stopping zoom-in events that would make the 'virtual' size of the image greater than 32,000x32,000
- .
- account permission improvements:
- to group sibling and parent petitions by uploader (and thus help janitor workflow), the PTR is moving to a system where the public account is download-only and accounts that can upload content are auto-generated in manage services. this code has not been tested much before, and it revealed some very bad reporting and handling of current permissions. I move this forward this week:
- if your repository account is currently unsynced from a serious previous error, any attempt to upload pending data will result in a little popup and the upload being abandoned
- manage tag siblings and parents will now show service tabs even if the account for those services does not seem currently able to upload tags or siblngs
- if your repository account is currently unsynced from a serious previous error, this is now noted in red text in manage siblings and manage parents
- if your repository account does not have sibling/parent upload permission, this is now noted in red text in manage siblings and manage parents. you will be able to pend and petition siblings and parents ok
- if your repository account does not have mapping/sibling/parent upload permission of the right kind, your client will no longer attempt to upload these content types, and if there is pending count for one of these types, a popup will note this on an upload attempt
- .
- the rest:

Message too long. Click here to view full text.

Anonymous Board owner 05/05/2021 (Wed) 21:22:28 Id: 0734e0 [Preview] No.1062 del
- fixed a secondary error when dropping non-list, non-downloader pngs on Lain's easy downloader import window, and fixed a 'no interesting objects' reporting test when dropping multiple pngs
- added a 'cache report mode' to help debug image and thumb caching issues
- refactored the media viewer code to a new 'canvas' submodule
- improved the error reporting when a thumbnail cannot be generated for a file being imported
- fixed an error in zoom center calculation when a change zoom event was sent in the split-second during media viewer initialisation
- I think I fixed an issue where pages could sometimes not automatically move on from 'loading initial files' statusbar text when initialising the session
- the requirements.txt now specifies 'requests' 2.23.0 exactly, as newer versions seemed to be giving odd urllib3 attribute binding errors (seems maybe a session thread safety thing) when recovering from connection failures. this should update the macOS build as well as anyone running from source who wants to re-run the requirements.txt. I hacked in a catch for this error case anyway, just a manual retry like a normal connection error, we'll see how it goes (issue #665)
- patched an unusual file import bug for a flash file with an inverted bounding box that resulted in negative reported resolution. flash now takes absolute values for width and height

next week

Back to multiple local file services. Mostly more backend cleanup and prepping File Import Options and the Client API for talking to multiple locations.

Release Tomorrow! Anonymous Board owner 05/12/2021 (Wed) 02:47:46 Id: 2c0ed2 [Preview] No.1064 del
I had an ok week. I improved the new image rendering system, cleaning up instability and errors and mitigating the tiling artifacts. I also fixed some other issues and optimised some database queries.

The release should be as normal tomorrow.

Version 437 Anonymous Board owner 04/28/2021 (Wed) 22:50:11 Id: 081b2e [Preview] No. 1054 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=FfBdUvVQpNQ [Embed]
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v437/Hydrus.Network.437.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v437/Hydrus.Network.437.-.Windows.-.Installer.exe
app: https://github.com/hydrusnetwork/hydrus/releases/download/v437/Hydrus.Network.437.-.macOS.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v437/Hydrus.Network.437.-.Linux.-.Executable.tar.gz

I had a good week mostly fixing bugs and optimising. It will take a couple of seconds to update this week.

all misc this week

I reduced a heap of UI lag on clients that have pages with a lot of collected (like 'collect by creator') media. If you often have five or ten thousand files collected and you noticed your client was getting choppy just when running some downloaders, I hope this improves things. Let me know how you get on!

I started real work on multiple local file services. Most of it is boring behind the scenes stuff, but as part of it, I overhauled the trash system this week. A heap of logic is improved, it is ready for more than one 'my files' service, and now hydrus remembers when files are deleted. Delete timestamps have never been recorded clientside, and unfortunately we cannot recover old information retroactively, but it is stored for all deletes from now on. Whenever the client wants to say 'this file was deleted', it should now have 'at an unknown time' or a nicer '3 days ago' suffix.

Message too long. Click here to view full text.

Anonymous Board owner 04/28/2021 (Wed) 22:50:43 Id: 081b2e [Preview] No.1055 del
full list

- misc:
- hydrus now keeps a track of when files were deleted! this information has never been recorded clientside, and it is sadly not retroactively recoverable, but it is stored for all deletes from now on. on occasion, when hydrus says 'this was deleted from xxx', it will now have 'at an unknown time' or a nice '3 days ago' string attached. it will take a few seconds to update this week as the new table data is created
- the 'trash' panel on review services now has an 'undelete all' button
- fixed a typo error in manage services when auto-creating a service account when more than one type of account can be created
- the thread watcher page now sorts the status column secondarily by next check time (previously, equal status would sort alphabetically by subject as a fallback secondary sort)
- I have renamed some network concepts across the program. before we had access keys, account keys, and registration keys--now we have access keys (secret password for account), account ids (identifier for account that jannies may need), and registration tokens (one-time token used to create a new account). I hope this reduces some confusion
- reduced some overhead when fetching media results for a search, and when refreshing their tags on major content updates
- fixed a 'no such table: mem.temp_int_hash_id_1'-style database error state that could persist for 30 seconds or more after certain rare rollbacks
- fixed the FlipFlip link html in the client api help
- fingers crossed, I fixed that bad Applications shortcut in the new macOS release
- fixed a couple more instances of 'pulsing' progress gauges. now they should be blank
- .
- more efficient updates in sessions with collected media:
- several updates this week should reduce client UI lag when the session contains any pages with a lot of collected media, particularly when you are also running several downloaders (which spam all sorts of content updates across the client):
- the content update pipeline now tests collections for their files before content processing, and now filters down to process just the updates in a group that apply

Message too long. Click here to view full text.

Anonymous Board owner 04/28/2021 (Wed) 22:51:12 Id: 081b2e [Preview] No.1056 del
- boring multiple local file services stuff:
- I cleaned a bunch of old hardcoded references to 'my files' and related code. it is not very interesting, but there are a few hundred references to clean up and convert to a system that supports 1-to-n local services, and this week I started hacking away, mostly presentation stuff, labels on menus and so on
- your 'my files' now has a separate deletion record to the 'all local files' domain. its count shows in 'review services', and for the moment will just be 'all local files' plus the count in trash, but this will become more important when you can have multiple 'my files'
- behind the scenes re-jiggering means that the deletion record now records deletion time and original import time. delete and undelete transitions are neater as a result
- logically, files are now generally no longer moved to the trash nor undeleted from there, they instead fall there when they are in 'all local files' but no longer in any local domain, and are undeleted back to a specific service. a bunch of awkwardness is cleaned up, and import/delete/undelete content updates are regeared and ready for multiple local file services
- a whole bunch of little things have been fixed and changed behind the scenes. I cleaned file service code in general as I went. examples of little things fixed:
- - a 'delete and do not keep a deletion record' action now correctly does not change the cached number of deleted files as reported in review services
- - the 'clear deletion record and try again' 'remove from trash' component now uses a unified and improved and UI-updating 'untrash' database action, with correct service count changes and UI-side status changes
- - the 'clear deletion record and try again' action on downloader import queues now handles mixes of actually deleted files and files just in trash more neatly
- - in the very odd situation that you are looking at a non-local file on 'all known files' and it is then imported using 'archive on import', its thumbnail and metadata now fade in correctly as archived
- added some unit tests to test the new file delete/undelete transitions
- cleaned up a bunch of hacky old db SELECT code

next week

Next week is a 'medium size' job week. I would like to try putting some time into the ancient image rendering pipeline and related systems like previous/next file prefetch. The basic media viewer has been jank and bad at zooming for too long. I am not sure I can make it beautiful, but I will try to clean some things up.

Message too long. Click here to view full text.

Release Tomorrow! Anonymous Board owner 05/05/2021 (Wed) 05:45:03 Id: e45e91 [Preview] No.1058 del
I had a great week focusing almost entirely on improving media viewer performance. I have overhauled the way images are zoomed and drawn to screen and completely eliminated the additional lag and memory bloat from zooming big images. Furthermore, the caching of rendered image data is greatly improved, so flicking back and forth between neighbouring images or different zooms is, for the most part, now instant.

The release should be as normal tomorrow.

Version 436 Anonymous Board owner 04/20/2021 (Tue) 22:45:41 Id: 34aa23 [Preview] No. 1049 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=i_u3hpYMySk [Embed]
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v436/Hydrus.Network.436.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v436/Hydrus.Network.436.-.Windows.-.Installer.exe
app: https://github.com/hydrusnetwork/hydrus/releases/download/v436/Hydrus.Network.436.-.macOS.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v436/Hydrus.Network.436.-.Linux.-.Executable.tar.gz

I had a great few days mostly cleaning and fixing things. If you sync with the PTR, update will take a minute this week.

macos release polish

I cleaned up the new macOS release. It seems to have launched and otherwise generally worked last week, but there was a bug in finding the specific database location macOS users are used to. Without the '--d' launch parameter, it was creating an empty new db inside the app, in the 'db' dir hydrus would normally use (and the really old App used to use, if you remember that), and hence would say 'hey, this looks like the first time you are running the program...' on boot. I have fixed the 'I am running in an app' detection and the ~/Library/Hydrus database path calculation routine, so everything should be back to normal.

It also has the old readme and Applications shortcut in the dmg, and the filename should be fixed too. I expect this to be the only macOS release I put out from now on. Let me know if you have any more trouble!

Message too long. Click here to view full text.

Anonymous Board owner 04/20/2021 (Tue) 22:46:10 Id: 34aa23 [Preview] No.1050 del
full list

- macOS:
- I fixed an issue with last week's Big Sur compatible release where it wasn't finding your old database correctly--it was defaulting to a different location, so without a specific launch command otherwise, it started a fresh db and said 'hey, looks like first time you ran the program'. if you are a long-time user of hydrus, please install and run 436 as usual, it should figure out your old db location correctly as ~/Library/Hydrus without any launch command override needed
- If you never ran any of the old macOS builds, and you started using hydrus for the first time on macOS last week with the experimental Big Sur compatible build, your brand new database is in a funky location! don't update yet, or you will delete it! You will want to copy your .db files and the client_files folder from inside_the_435_app/Contents/MacOS/db to ~/Library/Hydrus, which should for most people be /Users/(YOU)/Library/Hydrus. feel free to ask for help if you can't figure this out
- fixed a 'this is macOS' platform check for newer macOS releases, which ensures the 'userpath' fallback is correctly initialised to ~/Library/Hydrus
- fixed the new macOS github workflow build script to tell hydrus that it is running from inside an App, so it knows to default to the userpath fallback correctly
- the macOS build now has the old filename
- it also has the ReadMeFirst.rtf file and Applications shortcut
- collected the new build-related files in static/build_files, which will likely see more files in future
- .
- pending tag cache regen:
- two new maintenance tasks are added to the database->regenerate menu--one that forces a recalc of your total 'pending' count as used in the pending menu, and one that recalculates the cached pending tag mappings for storage tags (just like the display one added some time ago, but one layer deeper). the menu entries are relabelled appropriately
- these routines will be run on database update, and should correct the bad pending menu counts many users discovered last week (the new efficient way that the pending count is calculated exposed some legacy bad cached pending storage mappings entries. we'll see if they come back, or if this is just clearing up bad counts hanging around from ages ago)
- the quick pending mapping cache regen routines take a little longer to initialise now, but they now clear out surplus tag data, rather than just regenerating the 'correct' tags
- .
- misc:

Message too long. Click here to view full text.

Anonymous Board owner 04/20/2021 (Tue) 22:46:42 Id: 34aa23 [Preview] No.1051 del
- the 'refresh account' button in _review services_ now works on the new async system. it presents errors nicely
- a repository's current update period is now stated in its review services panel
- review services now says 'checking for updates in...' rather than 'next update due...', which is more accurate and will matter more with small update times
- fixed some false positive instances of 'this server was not a tag repo' error in the network engine.
- the hydrus server now also outputs hydrus specific 'Server' header (rather than some twisted default) on 'unsupported request' 404s and any other unusual 'infrastructure' 4XX or 5XX
- if the repository updates in the filesystem are lacking some required file information when calculating what to process, the client now queues those files for a metadata regen maintenance job and raises a cleaner error
- just as a safety measure, if a repository ever happens to deliver a metadata update slice with a 'next update due' time that has already passed, the client now adds a buffer and checks tomorrow instead
- a new program launch argument, db_transaction_commit_time, lets you change how often the database's changes are committed to disk. default is 30 (seconds) for client, 120 for server
- altering the repository update period now prints a summary of the change to the log
- updated the ipfs links in the help
- updated the main help index.html and the github readme.md with the user-run repo and wiki at https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts

next week

I may or may not be tied up with IRL stuff for a bit. Once I am back to things, I will keep working on smaller issues and get started on the pre-work for multiple local file services. There are several hundred locations where the 'my files' service is hardcoded as the local file reference, so a decent part of this work, before I get to file service migration and new location import options, will just be putting some time into ancient code.

Release Tomorrow! Anonymous Board owner 04/28/2021 (Wed) 04:30:10 Id: f4c7c6 [Preview] No.1053 del
I had a good week fixing bugs and cleaning old code. Client performance is improved for large sessions, particularly those that have 'collected' page media, and I got started on multiple local file services by overhauling the core file delete/trash/undelete system. As a neat side-effect, the client now remembers when you delete files.

The release should be normal time tomorrow.

Version 435 Anonymous Board owner 04/14/2021 (Wed) 22:50:51 Id: 7106ee [Preview] No. 1045 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=-ZdLbiF9gCo [Embed]
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v435/Hydrus.Network.435.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v435/Hydrus.Network.435.-.Windows.-.Installer.exe
NEW app: https://github.com/hydrusnetwork/hydrus/releases/download/v435/HydrusNetwork-v435.dmg
OLD app: https://github.com/hydrusnetwork/hydrus/releases/download/v435/Hydrus.Network.435.-.macOS.old.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v435/Hydrus.Network.435.-.Linux.-.Executable.tar.gz

I had a good week. I continued last week's server work and have a macOS Big Sur build available for testing.

new macOS build

This was delayed far longer than I wanted, but I think I have it figured out now. Thanks to work from Suika and ReAnzu on github figuring out the script, the macOS build is now put together on github itself, on Catalina. This new build thus works on Big Sur, and should generally have less compatibility jank on any newer macOS machine. I have included my old 10.12-built release as well today, just in case I messed something up, but if you are a macOS person, please give the new one a try and let me know how you get on.

Assuming this works out ok for the majority of people, I'll be moving to it as the official macOS build. Anyone who is still on an old macOS will have to run from source to stay updated with hydrus. The new release is just out of testing, so I'll clean up the filename and add the readme.rtf and so on so it looks prettier. I will also explore reducing the size (300MB vs 137MB wew), but I am told this may be unavoidable due to some new macOS thing that bundles both old and new versions of libraries in the App.

Message too long. Click here to view full text.

Anonymous Board owner 04/14/2021 (Wed) 22:51:54 Id: 7106ee [Preview] No.1046 del
the rest

I had another go at multi column list 'last column' resizing. Dialogs shouldn't judder around so much when you resize them, and more instances of dialogs growing or shrinking slightly on the every re-open should be mitigated.

Tag upload to a repository has a new safety check for a particular error state. I know one user got stuck with some pending mappings that wouldn't clear--in fact it lead to an endless cycle of uploading!--which is now caught, and a new maintenance routine can fix it.

The Client API's file search now converts tags with asterisks into wildcard or 'namespace' search terms. Its version is now 16. Also added to the Client API help page is a link to https://ififfy.github.io/flipflip/#/ , an advanced slideshow engine that now supports hydrus as a source.

full list

- misc:
- a new macOS build that should run on Big Sur is now ready, it should be attached to this release. it is built on github automatically, and is thanks to hard work from Suika and ReAnzu. I am attaching my old release as well, just in case I messed up somewhere on my end. if you are a macOS user, please try the new App! it will not work on very old macOS like 10.12, but if this works out today for the majority of macOS users, I will be moving to just putting this new build out going forward. I'll add some polish like the readme.rtf and harmonise the filename etc.. too. I'd love to cut the filesize down, but this may not be possible (it is apparently some modern macOS thing where it bundles old and new versions of libraries in the same App so you basically get it twice)
- the bottom-right corner of the regular media viewer canvas now also shows media zoom
- the StringSorter object now has a simple 'reverse' sort type
- the infamous multi-column list 'last column' width calculations are improved: first, dialogs with multi-column lists should no longer judder back and forth a single character's width as you expand the parent window. also, the last column saved size (which is used in dialog relaunch width initialisation) is now snapped to rounded 5-character intervals, which should mitigate various 'fuzzy' reasons for some dialogs to remember a larger or smaller size and grow or shrink one or more characters' width on the next launch
- the _help->debug->gui actions_ menu has a new entry to reset all multi-column list saved widths back to default
- the 'edit OR predicate' panel when you shift+double-click an OR predicate now expands horizontally and vertically with the window

Message too long. Click here to view full text.

Anonymous Board owner 04/14/2021 (Wed) 22:53:13 Id: 7106ee [Preview] No.1047 del
- new server stuff:
- a new permission is added to hydrus service accounts--'manage options'. any account with 'manage account types' will get this by default on update
- any account on a repository with 'manage options' permission will now see 'change update period' in the admin services menu! it launches a time delta control with the current update period and will send the new one up to the server. the client will resync account, options, and metadata immediately, and the server will generate any now-due updates immediately, so you should be able to watch changes occur in 'review services' and the server terminal live. other users will catch up to the new time when they next hit an update. various hardcoded check periods (like how often due updates are checked for and delay-buffered clientside and serverside) are shrunk significantly. the whole system should react to changes better
- the minimum settable update time is now 10 minutes (the default value remains 100,000 seconds), but I recommend you try larger, say an hour minimum, at least to start. the network generally works more efficiently with higher numbers, and be warned, if you are adding 144 updates a day, there may be bloat problems after a year
- let me know how this goes, whether you are running a server on a LAN or just a regular user running on one who gets a new update time!
- the new 'full metadata resync' routine now triggers an immediate metadata update sync and wakes the daemon involved, so it should now happen as you watch
- fixed the new pause/play buttons on review services to use neutral pause/play icons, not the downloader pause/play
- brushed up metadata sync status string on review services
- cleaned misc server and network code
- cleaned up some old clientside service code
- the client api now supports wildcard and namespace tags in the file search call
- client api version is now 16
- added https://ififfy.github.io/flipflip/#/ , a slideshow engine that now supports hydrus as a source, to the client api page

next week

I have unavoidable IRL next week (jury duty), so I can't do the release on Wednesday. I will put a smaller one out on Tuesday!

Message too long. Click here to view full text.

Release Tomorrow! Anonymous Board owner 04/20/2021 (Tue) 03:22:06 Id: 965772 [Preview] No.1048 del
I had a great few days mostly fixing bugs and cleaning things. The new macOS build is more polished, and I fixed an issue where it could have trouble finding your default database location. I believe I also fixed the issue some users had with 'sticky' pending menu counts, then added some UI quality of life, and wrote an experimental tag display mode that replaces underscores with spaces.

The release should be normal time tomorrow. I have unavoidable IRL on Wed, so it is moved up a day.

Version 434 Anonymous Board owner 04/07/2021 (Wed) 22:51:31 Id: 2275fd [Preview] No. 1042 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=f1nvZ74OY_I [Embed]
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v434/Hydrus.Network.434.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v434/Hydrus.Network.434.-.Windows.-.Installer.exe
app: https://github.com/hydrusnetwork/hydrus/releases/download/v434/Hydrus.Network.434.-.macOS.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v434/Hydrus.Network.434.-.Linux.-.Executable.tar.gz

I had a great three weeks working on long overdue network updates. Most of the changes this week restore old account management UI for server admins, but there are a couple of fixes and improvements for regular users as well.

The network version goes up to 20 today. Clients and servers can only talk to each other if they are on the same version, so if you want to talk to the PTR, you will want to update at some point. No rush, but if you try to talk to the server on an older client, you will get a polite error message.

regular user stuff

On review services, you can now pause repository account sync, update downloading, and update processing separately. If you don't want to process right now but still want the client to be grabbing update files for later, or you want to pause all service network traffice for a while but still process your backlog, it should now be easy to set up.

Message too long. Click here to view full text.

Anonymous Board owner 04/07/2021 (Wed) 22:54:16 Id: 2275fd [Preview] No.1043 del
admin improvements

After a very long delay, account management is finally back. If you have permission to modify accounts, the 'modify an account' service admin menu item works again and launches a fully revamped window. You can also launch account modification from manage tags for a particular tag selection, the thumbnail right-click for file repository files, or the petition processing page.

The new window shows better data about accounts, and lets you zoom in on individual ones to see current account type, expiration, banned status, or message. It also shows if you are editing yourself!

The account modifications are: change account type, set/extend/clear account expiration, ban, unban, and the new 'set message'. All modifications now print janny and subject account key to the server log. I removed the old 'superban' (which deleted all that user's content contributions), but it will likely return in future.

The service admin menu also has a new 'review all accounts' entry that lets you see a simple summary list of all accounts, which you can then select for modification.

The manage account types dialog now lets you set up auto-account creation. It works a bit like subscriptions, with a 'account creation velocity' limit of 'x accounts per y time delta', like '10 per day'. This defaults to off obviously, but if you set some account types to auto-create, users will be able to see that in manage services.

future admin improvements

As always, there was much more I wanted to do than could get done. I wasn't able to finish improving petition processing workflow or add custom update duration, a serverside tag filter, or a server message, but I was able to get the important network stuff done I needed for these things. Since incrementing the network version is always such an obstacle, I am confident I will now be able to work on these things in normal weekly work. Please let me know how they work out for you as I roll them out.

full list

Message too long. Click here to view full text.

Release Tomorrow! Anonymous Board owner 04/14/2021 (Wed) 06:47:07 Id: 7cca52 [Preview] No.1044 del
I had a good week. I fixed several bugs and added some quality of life, got a Big Sur build working for macOS users to try out, and for users who run smaller servers on a LAN or between friends, hydrus repositories now support changing their 'update periods'.

The release should be as normal tomorrow.

Version 433 Anonymous Board owner 03/17/2021 (Wed) 22:34:52 Id: c0812f [Preview] No. 1029 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=twOFs6zJq8Q [Embed]
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v433/Hydrus.Network.433.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v433/Hydrus.Network.433.-.Windows.-.Installer.exe
app: https://github.com/hydrusnetwork/hydrus/releases/download/v433/Hydrus.Network.433.-.macOS.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v433/Hydrus.Network.433.-.Linux.-.Executable.tar.gz

I had a good week doing simple cleanup and fixes. This is intended to be a 'clean' release. There will be no release for about three weeks.

all misc this week

Thanks to the work of a user (who was also very patient as I slowly added the features he needed), we now have a new native twitter downloader! It gets video now! (Including gif, which on twitter is just a small mp4!) It is called 'twitter syndication' and uses a new access method. Its main limitation seems to be that its search can only reach about 500 tweets into the past. Please check it out, and if it works well for you, please move your subscriptions over from nitter to save them some bandwidth. This is the first version, so it may well have future updates.

I have written a new 'scrollable' menu button, which works like the file sort selector--instead of a dropdown, you get a menu, and you can scroll through it in either direction with your mouse. I have converted several old small choice dropdowns to this new button: file sort ascending/descending, collect-by 'collect/leave unmatched', and all the new tag sort dropdowns I added last week. Try putting your mouse over and scrolling them! This is a little thing, but I hope it smooths out the workflow here. I'll likely use it on more things.

Message too long. Click here to view full text.

Anonymous Board owner 03/17/2021 (Wed) 22:35:09 Id: c0812f [Preview] No.1030 del
full list

- misc:
- thanks to the effort of a user, this week reintroduces a native twitter downloader! it now gets video! (and 'gifs', which on twitter are just mp4s) please experiment with it and move your nitter subs back to this as you find success. it is called 'twitter syndication' and uses a different access method to get tweet info. it should get the highest resolution videos and images. the search has limited lookup distance, perhaps 500 tweets, but should work for most subscription purposes. this is a first version and may have future updates
- on the main gui, middle-clicking and left-double-clicking to open the 'new page' dialog, and right-clicking to open the page menu, should now only work on the page tabs or page greyspace. middle-clicking on some random downloader page greyspace should no longer spawn these commands. also, tiny change--middle-clicking now activates here on click _release_ rather than _press_
- on the file sort widget, the asc/desc sort dropdown is now a 'scrollable' menu button. since sorts are just one of two values, you can now scroll either direction to flip it
- the 'collect/leave unmatched' in the collect control is also now a scrollable menu button
- the new tag sort dropdowns are now all scrollable menu buttons. go ham with them
- middle-clicking the collect-by dropdown now clears it
- namespace file sort now supports a-z and z-a sorting. files with none of the matching namespaces still count as 'less than a' in a-z terms, but since I am updating all this code, perhaps this could get more attention. I don't use this much, so if you do, let me know what you would like
- network job controls now show their jobs' current bandwidth limits on their cog menu, split up by network context. you can edit the bandwidth rules directly from this menu, and if it is using defaults, set a specific ruleset
- userpath generation routines used for database location fallback and default export directory determination now recover from failure in the case of undefined user directory. the client will now not boot if the userpath is needed but undefined
- the client api no longer prints empty lists for any tag statuses on file metadata calls, nor service entries that have no tags at all
- fixed the new subscription 'caught up to small initial sync' calculation, which last week was only firing properly after one page of results
- subscriptions are now better about saving interesting status notes on their gallery logs, rather than overwriting with boring 'no new urls found' messages
- fixed a typo bug in the new range header implementation for unended ranges
- .

Message too long. Click here to view full text.

Anonymous 03/19/2021 (Fri) 08:08:39 Id: 70edb1 [Preview] No.1032 del
God bless you Hydrusdev

Progress Update 1 Anonymous Board owner 03/24/2021 (Wed) 02:48:14 Id: 2a160e [Preview] No.1037 del
I had a good week getting started on the network updates. I cleaned out ancient code and improved some admin-side workflow. There is plenty more to do.

The release is still expected to be another two weeks, for April 7th.

Thanks m8, keep pushing.

Progress Update 2 Anonymous Board owner 03/31/2021 (Wed) 03:32:00 Id: 83acae [Preview] No.1040 del
I had another good week grinding on the network code. I fixed and cleaned old admin functions and started on future extensions for these systems. The main tasks remaining are improvements to the janitor petition processing UI and new features for core repository storage like a tag filter and dynamic update times.

Unless I really fall into a hole, the release is still expected for next week, for April 7th.

Release Tomorrow! Anonymous Board owner 04/07/2021 (Wed) 05:37:28 Id: 8eccc7 [Preview] No.1041 del
I had a great three weeks working on network functions and serverside admin controls I had neglected. The old account management controls are working again with many improvements, account types can now be set up as user-creatable straight from the client (no need for registration keys!), and much of the code is cleaner and handles errors better. In the rewrites, I also laid the groundwork for custom update periods, serverside tag filtering, and faster petition processing, which I hope to get done in the coming weeks.

Regular users who do not admin a server do not have so many changes, although I have fixed bugs and added more controls to repositories, so you can now pause account sync, update downloading, and update processing separately, and I fixed a logical problem with processing that was sticking some users with deleted siblings.

The 'network version' goes up tomorrow, and the PTR should be updating, so if you sync with the PTR, you will want to update some time to keep talking to it! There's no rush, but you'll get a little error popup if you try to talk to an updated server on an older client.

The release should be as normal tomorrow.

Version 432 Anonymous Board owner 03/10/2021 (Wed) 23:41:48 Id: 9c7aff [Preview] No. 1020 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=_ZDUHbIVaz8 [Embed]
^ After years of seeing all sorts of smaller edits using this footage, I finally discovered this source vid this week. If you like fine art, please watch it in full, it is eight minutes of unbroken kino. It gets a bit heavy at the end, not appropriate for kids. If you don't like old art, no worries, I was just chuffed this week to have finally found this thing.

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v432/Hydrus.Network.432.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v432/Hydrus.Network.432.-.Windows.-.Installer.exe
app: https://github.com/hydrusnetwork/hydrus/releases/download/v432/Hydrus.Network.432.-.macOS.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v432/Hydrus.Network.432.-.Linux.-.Executable.tar.gz

I had a great week. There are a bunch of different little fixes and improvements this week, and the tag sort dropdown is now easier to use.

tag sort

Rather than a big list, the tag sort dropdown is now a dynamic control that splits the different variables (sort type, asc/desc, group by) into different mini-dropdowns that show and hide when available. Confusing labels like 'lexicographic' and 'ignore namespace' are replaced with clean 'sort by tag/subtag/count'. I hope it is a bit easier to work with now. Give it a go and let me know what you think.

Message too long. Click here to view full text.

Anonymous Board owner 03/10/2021 (Wed) 23:42:17 Id: 9c7aff [Preview] No.1021 del
advanced users

I added a String Selector/Slicer object to the parsing system. You can select the nth item or the mth to nth. m can be 'start', n can be 'end', and either can be a negative index. Try mixing it with the new String Sorter! The String Processing UI has also had an update pass to better support multiple strings--it now shows everything its parent formula had, so you can better test sorting or how processing is changing the third string in the list or similar.

The Server/Client API now support single 'range' requests for files (basically anything not html/json). Apparently Safari was having trouble getting videos from the Client API, let's see if this fixes it. If not, I can try supporting the (more complicated) multiple range requests. The server's error handling is also improved--on certain 'emergency' errors, it was sometimes hanging without closing the connection properly.

full list

- tag sorting:
- the tag sort dropdown has been replaced with a dynamic control. rather than one big list with all possible permutations, you now work on each variable (sort type, asc/desc, group by) separately. what you are actually sorting is easier to understand and select
- my stupid "lexicographic/incidence" labelling is replaced with the simpler and neater 'tag', 'subtag', and 'count'
- when in the manage tags dialog and sorting by tag or subtag, you can now turn off the 'use sibling' sort.
- I'd like to further neaten the workflow here, making the individual dropdowns flip back and forth with a mouse scroll in either directior rather than being just up/down allowed. let me know overall how you find this new control
- the 'tag sort' object is updated behind the scenes as well. your old value should be converted automatically
- fixed an issue with count tag sorting where deleted tag counts were being counted even when not displayed
- if you try to search tags on a page of thumbnails that holds an invalid tag, this is now caught gracefully and you get a little popup saying 'please run the repair invalid tags routine'
- .

Message too long. Click here to view full text.

Anonymous Board owner 03/10/2021 (Wed) 23:42:57 Id: 9c7aff [Preview] No.1022 del
- advanced string processing:
- added a String Selector/Slicer object to the parsing system. this object allows you to select the nth item in a list of parsed strings or the mth to nth items. m can be 'start' and n can be 'end', and negative indices are allowed for both. pair it with the new Sorter for some neat new tricks!
- the string processing edit UI is now _more_ multi-string-aware. the test panel has had a code cleanup pass and now has a list of all the starting strings in the test data (e.g. all the urls parsed by the formula that launched the UI) rather than just the first, and a list of all results from that list. selecting any of the starting strings populates the 'single string' area, so you can now zoom in on one particular string to see what is happening to it
- the String Sorter edit UI now gets all the strings at that stage of processing, so you can review the sort properly
- the new String Slicer edit UI similarly gets all the strings at that stage of processing
- future updates will expand multi-string presentation and testing. I'd like to show the whole list at each stage
- .
- server/client api core improvements:
- I had a go at supporting the Range header for file (basically this means anything non-html/json) requests. I added tests and it seems to work. as I understand this mostly applies to browsers pulling video from the Client API. to start, I am supporting single range requests. if it is needed, I'll try to get Multi Range requests right, but for now they'll 416
- the client now understands 416 ("can't do that requested range m8") errors
- I reworked the serverside error handling chain. this has been borked for a long time due to my own lack of understanding of twisted's deferred system, and certain late-stage errors were just not being handled right. the server should no longer hang on these and now should print error info correctly, including a rough 500 in true late emergencies, and terminate the connection correctly
- .
- boring:
- fixed up a handful of typo-borked unit tests
- fixed my ordinal (xst, xnd, xrd, xth) text generator to deal with 11, 12, and 13 correctly lmao
- started some db maintenance routines and logistics to recover definitions and remove orphans in future, I feel great about it so far, but it'll have to wait for more of my db 'modules' refactoring to be more useful
- updated the mpv dll on the Windows release to 2021-02-28, it may improve some video support/performance

Message too long. Click here to view full text.

Release Tomorrow! Anonymous Board owner 03/17/2021 (Wed) 07:14:50 Id: fa5034 [Preview] No.1028 del
I had a good week fixing bugs and adding a little quality of life.

The release should be as normal tomorrow. I expect it to be the last release for a few weeks so I can work on a larger network overhaul.

Version 431 Anonymous Board owner 03/03/2021 (Wed) 23:19:53 Id: 8ea1ca [Preview] No. 1009 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=CU5NKzKiAGU [Embed]
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v431/Hydrus.Network.431.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v431/Hydrus.Network.431.-.Windows.-.Installer.exe
app: https://github.com/hydrusnetwork/hydrus/releases/download/v431/Hydrus.Network.431.-.macOS.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v431/Hydrus.Network.431.-.Linux.-.Executable.tar.gz

I had a challenging week, but I got some decent fixes in.

all misc this week

Pretty much everything was little fixes this week. Nothing hugely significant, but check the changelog if you have been waiting on something.

The e621 parser now pulls 'lore' tags, which seems to be some new 'canonical' tag. I have put them in 'lore' namespace, so you can exclude them easily if you want. One user suggested they are useful for certain blacklists.

Message too long. Click here to view full text.

Anonymous Board owner 03/03/2021 (Wed) 23:20:20 Id: 8ea1ca [Preview] No.1010 del
- the manage parser test panel now catches all network errors. error data back from the server is presented better, and the traceback is now viewable in a special new button on the network job control
- the edit shortcut set dialog now gives a veto text popup if you try to ok with a shortcut set twice (previously, it would ok and merge down to one command randomly). support for multiple commands per shortcut will come in future
- entering alt+number in the shortcut entry dialog on windows will no longer spam some errors about 'null character'
- fixed a 'this object is too huge' check in the database, which mostly affects gui sessions with millions of objects, to check against 1 billion bytes max size rather than 1GB, as here https://sqlite.org/limits.html (issue #816)
- fixed the 8chan.moe parser, which was pulling hash incorrectly. it should now save more bandwidth
- updated the e621 parser to pull their (new?) 'lore' tags, which all end in _(lore) and refer to canonical gender and some spice. they come with the 'lore' namespace for now, we'll see how it works out. a user reports these are useful for blacklists
- .
- new string processing sort step (for advanced users):
- string processing objects have a new processing step: String Sorter
- this sorter can sort the whole list of strings, either strict lexicographic or 'human sort' that does numbers properly, asc/desc
- it can also take a regex for the sort 'key', so you can sample just the number or name you want for sort purposes
- content parsers no longer have the 'sort formulae results' controls. any content parser with existing sort has been converted so its string processing object has a String Sort step appended
- the string processing UI is still built around single string processing, so the test UI here is essentially non-functional, but you can see the sort happen in the formula test parse panel
- I will add a String Slicer in future to sample the list of strings, so you'll be able to grab the top item etc...
- .
- boring code cleanup:
- refactored the refresh call in filename tagging dialog to nicer Qt signals

Message too long. Click here to view full text.

Anonymous 03/09/2021 (Tue) 15:06:14 Id: 259937 [Preview] No.1018 del
I am trying to contribute to making the PTR better by adding siblings/parent and removing garbage, but this doesn't help. In manage tags window, I sibling a tag like this:

tag1 (displays as tag2)

How do I now add parents to tag2? I can't. Also the sibling change does not appear until I commit the pending tags, which means I'll have to close the manage tags window and lose my place in the tag list (I often have thousands of files open at once so the list will be long). Annoying.

Release Tomorrow! Anonymous Board owner 03/10/2021 (Wed) 07:39:40 Id: bf064a [Preview] No.1019 del
I had a great week. I caught up on some small fixes and improvements, finished off a new parsing tool for advanced users, improved behind the scenes server code, and reworked the tag sorting dropdown to be easier to use.

The release should be as normal tomorrow.

Thank you for your feedback. I will extend the tag menu when there is a sibling displayed like that to allow for commands on the sibling.

I would also like to work on the sibling/parent and general tag mapping edit logic, pulling it out of their specific dialogs, so I can better support quicker 'just do a quick change' actions for all these things. I don't like having to boot a big dialog just to add one thing either.

e621 Anonymous 03/04/2021 (Thu) 05:56:33 Id: 2af9d4 [Preview] No. 1011 [Reply] [Last 50 Posts]
tried to login after changing password and I get this message about remember cookie missing in post login step
it says in logged in when i look at the manage logins