- client api additions:
- the client api has a new /manage_headers/set_user_agent call, which is a simple hack for now for external programs to set the 'Global' User-Agent. it should allow for some CloudFlare solutions when just copying cookies is not enough
- the client api has a new /get_services call, which talks about more services and also exposes service_keys for the first time, which are likely to be useful in future. check out the help for an example. the old /add_tags/get_tag_services call is now deprecated, please move to the new call
- the client api /version call now responds with 'hydrus_version' as well, which this week will be 441
- the client api now has a semi-experimental /manage_database/lock system, just like the server's. a new 'manage database' permission is added for this. don't play around with this system idly.
- the client api should now support sha256 hash parameters if they start with a type prefix like 'sha256:0123789abcdef...'
- the client and server's database lock commands now wait up to five seconds for the database to finish disconnecting to respond
- expanded client api unit tests to cover the above
- the client api version is now 17
- .
- boring multiple local file services work:
- the main search object now stores the file domain using a new 'location context' object that will in future hold multiple file services and can say whether we should search files currently in a domain, or those once deleted from it. a variety of back-end search code has been updated to deal with this more flexible situation
- removed more static references to the single 'my files' domain in db and related code. in a couple places, like mr. bones, it now fetches 'all local files', but this will likely be updated in future to a new umbrella 'all non-trash, non-repo-update-files local files' service

next week

I've had some real trouble keeping up recently, but that's ok. A bunch of it is out of my control, so I'll keep pushing anyway. Next week is due to be a 'medium' job week, and I would like to break up the gui session object into smaller pieces. Instead of saving the whole thing, it'll track and save and share individual pages. This will greatly reduce the random CPU lag and HDD use on any client with a large session, let crazy users to store more than 500,000 files in a session at once, and allow us to save changes more often. Basically the same improvement I made to subscriptions and the network objects in the last year, but for gui sessions.

I had a great week. I succeeded in overhauling the client's GUI sessions, greatly reducing the storage and write I/O required for sessions. This particularly benefits clients that have sessions storing many files or URLs.

The release should be as normal tomorrow.

new builds to test

This is probably just for advanced users. If you would like to help test, make sure you have a backup before you test anything on a real database!

A user has been working hard on replicating the recent macOS build work for the other releases, cribbing my private build scripts together into a unified file that builds on github itself from the source, as well as rolling out a Docker package. I have had a look over everything and we agree it is ready for wider testing, so if you would like to help out, please check out the test v440 builds here:

https://github.com/hydrusnetwork/hydrus/releases/tag/v440-test-build

These should work just like my normal builds above--the scripts are using PyInstaller and InnoSetup as I do on my machines, so it all assembles the same way--but we are interested in any errors you nonetheless encounter. We may need to hammer out some stricter library version requirements for older machines, since until now we've basically been relying on my home dev environments staying static until I next remember to run pip update.

Once we have these working well, I'd like to use this system for the main build. It makes things easier and more reliable on my end, and should improve security since the builds are assembled automatically in clean environments with publicly viewable scripts rather than my random-ass dev machines using my own dlls, batch files, and prayers. Who knows, we may even defeat the anti-virus false positives.

Also, if you would like to try the Docker package, check it out here:

https://github.com/users/hydrusnetwork/packages/container/package/hydrus

I don't know much about Docker, so while I can't help much, I'll still be interested in any feedback. If and when we are ready to switch over here, I'll be updating my help with any appropriate new backup instructions and links and so on.

- the rest:
- added a prototype 'delete lock' for archived files to _options->files and trash_ (issue #846). this will be expanded in future when the metadata conditional object is made to lock various other file states, and there will be some better UI feedback, a padlock icon or similar, and some improved dialog texts. if you use this, let me know how you get on!
- you can now set a custom namespace sort in the file sort menu. you have to type it manually, like when setting defaults in the options, but it will save with the page and should load up again nicely in the dialog if you edit it. this is an experiment in prep for better namespace sort edit UI
- fixed an issue sorting by namespaces when one of those namespaces was hidden in the 'single media' tag context. now all 'display' tags are used for sort comparison groups. if users desire the old behaviour, we'll have to add an option, so let me know
- the various service-level processing errors when update files are missing or janked out now report the actual hash of the bad update file. I am chasing down one of these errors with a couple of users and cannot quite figure out why the repair code is not auto-fixing things
- fixed a problem when the system tray gets an activate event at unlucky moments
- the default media viewer zoom centerpoint is now the mouse
- fixed a typo in the client api with wildcard/namespace tag search--sorry for the trouble!
- .
- some boring multiple local file services cleanup:
- if you have a mixture of trash and normal thumbnails selected, the right-click menu now has separate choices for 'delete trash' and 'delete selected' 'physically now'
- if you have a mixture of trash and normal thumbnails selected, the advanced delete dialog now similarly provides separate 'physical delete' options for the trashed vs all
- media viewer, preview viewer, and thumbnail view delete menu service actions are now populated dynamically. it should say 'delete from my files' instead of just 'delete'
- in some file selection contexts, the 'remote' filter is renamed to 'not local'

next week

I had an ok week. I did a mix of different small work and added some new commands to the Client API, including something that should help some difficult login situations.

The release should be as normal tomorrow.

full list

- tiled image renderer improvements:
- I believe I fixed the 'non c-contiguous' crash issue with the new tile renderer. I had encountered this while developing, but it was still happening in rare situations--I _think_ in an unlucky edge case where a zoomed tile had the same resolution as the full image rotated by ninety degrees! there is now an additional catch for this situation, as well, to catch any future logical holes.
- fixed a bug in the new renderer when copying an image to clipboard
- I greatly mitigated the tiling artifacts with two changes:
- - zoomed in tiles are now resized with a padding area of up to 4 pixels, with the actual tile cropped afterwards, which allows bilinear and lancsoz interpolation to get accurate neighbour data and have gradient math line up with neighbouring tiles more accurately
- - on resize and zoom, media canvases now dynamically change tile size to 'neater' float/integer conversion dimensions to reduce sub-pixel panning alignment artifacts (e.g. if your zoom is 300%, the tile is now going to have a dimension that is a multiple of 3)
- I hacked in a 'rescue offscreen media' calculation after any zoom event. now, if the window is completely out of view after a zoom, it'll snap to the nearest borders, lining against them or overlapping into a buffer zone depending on the zoom. let me know what you think!
- I fixed a PyQt5 specific object tracking bug, I think the new renderer now works ok for PyQt5!
- cleaned up some ugly code in the resize section that may have been resulting in incorrect interpolation algorithm choice in some situations
- fixed a divide by zero issue when zooming out tiny images hugely (e.g. 32x32 at 1%)
- media windows now try to have at least 1x1 size, just to catch some other weird error situations
- similarly, tile and native sample sizes will have a minimum of size 1x1, which should fix issues during a delayed startup (issue #872)
- cleaned up some misc media viewer and tile renderer code
- .
- the rest:

- as the "Retry has no attribute..." network errors have appeared in new forms, I gave the core of the problem another look. we could never really figure this out, but it seemed to be a network version thread safety issue. I think I have ruled this out, and I now believe these may have been occuring during faulty pickling during network session save/load. I fixed the problem here, so with luck this issue will not reappear--if you have had this a lot, let me know how you get on!
- I broke the requirements.txt into several variants based on platform. we are going to try to pin down good fixed versions of python-mpv and requests/urllib3 for each platform
- I also updated the 'running from source' help significantly, moving everything to the requirements.txt and making sections for things like FFMPEG and libmpv
- Also updated the source and contact help around my work style and contact preferences
- the test.py file now only does the final input() confirmation if there is an interactive stdin to respond

next week

Next week is code cleanup and some little jobs that have slipped through the cracks. Nothing too clever, but I want to fit in some misc boring work.

Thanks everyone!

I had an ok week, but my work time was unfortunately cut short by IRL. I might not normally do a release, but I'd like to get some neat fixes out for the new tiled renderer, which has additional protections in 440 and should no longer crash. There's also some misc cleanup and quality of life.

The release should be as normal tomorrow.

I am really pleased with this week's work, but there are some drawbacks: I did it quick, so I cannot promise it is good. The most obvious bug already is that at around 200-500% zoom you start to see tiling artifacts. I know what causes this (interpolation algorithms not getting full pixel neighbour data from my simple tesselating tiles) and have a plan to fix it (adding a tile border pre-resize, and then cropping). There is also an issue when the 'virtual' image exceeds about 32,000x32,000, so I hacked a zoom block for that. There may be some weird files that render with other stitching artifacts or bad tile data. Note also that hydrus's 'Animation' renderer (the soundless fallback if you do not have mpv support) does NOT use tiling yet, so it still sucks at zooming! Please let me know how you get on!

If you have a steam-powered GPU or a machine with only 4GB of ram, you might like to wait for 439 so I can address any surprise bugs or performance issues.

PTR and account permissions

The PTR is changing how its accounts work. The shared public account is transforming to a 'read-only' account that can only download, so if you want to upload, you'll be going to manage services to auto-create your own privileged account. This is being done to improve janitor workflow for the various petitions, which were all being merged together because of the shared account. With the recent network updates, it will soon be easier for janitors to send simple messages back to these individual accounts, like 'that proposed sibling was not approved because...'.

Unfortunately, various permission and account-management code has not been tested much until now, so as the PTR guys have been trying this stuff out, I have been working to improve bad notifications and workflows. This week I rounded out account permissions testing with uploading. Hydrus no longer tries to upload content the current account does not have permission for, and if you end up in that situation, popup messages now tell you what is going on. It also catches if your account is currently 'unsynced', with instructions to fix.

Similarly, under 'manage siblings/parents', you can now see and edit all tag repositories (previously, they were hidden if you currently had no permission), but you get a label telling you if you don't have permission.

full list

- media viewer:
- I have hacked in tile-based image rendering for the media viewer. this has always been planned as a larger, longer-term job, but the problem of large images is only getting worse, so I decided to just slam out a prototype in a week. if you have a steam-powered GPU or 4GB ram, you might like to wait until next week to update so I can iron out any surprise bugs or performance problems
- images are now cut into tiles that are rendered on demand, so whenever the image is zoomed larger than the media viewer window, only those tiles currently in view have CPU and memory spent on resizing and storage. as you pan around, new tiles are rendered as needed, and old discarded. this makes zooming in super fast and low memory, even for large images!

- futhermore, a new 'image tile' cache is added. you can customise size and timeout under _options->speed and memory_ like for images and thumbnails. this is a dedicated cache for remembering image resize computation across images and zooms. once you have seen both situations once, flicking back and forth between two images or zoom levels is now generally always instant! this new cache starts at a healthy default of 256MB. let's see how that amount works out IRL--I think it will be plenty
- I tuned the image renderer cache--it no longer caches huge images that eat more than 25% its total size--meaning these images only hang around as long as you are looking at them--and the prefetch call that pre-renders several files previous/next to the current image no longer occurs on images that would eat more than 10% the cache size. this should greatly reduce weird flicker and other lag when browsing through a series of mega-images (which before would stomp through the cache in quick succession, barging each other out of the way and wasting a bunch of CPU). in real world terms, this basically means that with an image cache of 200MB, you should have slower individual image performance but much better overall performance looking at images with more than about 5k resolution. the dreaded 14,000x12,000 png will still bonk you on the head to do the first render, but it won't try to uselessly prefetch or flush the whole cache any more
- if you are currently looking at a static image, neighbour prefetch now only starts once the image is rendered, giving the task in front of you a bit more CPU time
- new options for prefetch delay and previous/next distance are added to 'speed and memory'
- note this does not yet apply to the old hydrus animation renderer. that still sucks at high zoom!
- another future step here is to expand prefetch to tiles so the first view of the 'next' media is instant, but let's let all this breathe for a bit. if you get bugs, let me know!
- due to a Qt issue, I am stopping zoom-in events that would make the 'virtual' size of the image greater than 32,000x32,000
- .
- account permission improvements:
- to group sibling and parent petitions by uploader (and thus help janitor workflow), the PTR is moving to a system where the public account is download-only and accounts that can upload content are auto-generated in manage services. this code has not been tested much before, and it revealed some very bad reporting and handling of current permissions. I move this forward this week:
- if your repository account is currently unsynced from a serious previous error, any attempt to upload pending data will result in a little popup and the upload being abandoned
- manage tag siblings and parents will now show service tabs even if the account for those services does not seem currently able to upload tags or siblngs
- if your repository account is currently unsynced from a serious previous error, this is now noted in red text in manage siblings and manage parents
- if your repository account does not have sibling/parent upload permission, this is now noted in red text in manage siblings and manage parents. you will be able to pend and petition siblings and parents ok
- if your repository account does not have mapping/sibling/parent upload permission of the right kind, your client will no longer attempt to upload these content types, and if there is pending count for one of these types, a popup will note this on an upload attempt
- .
- the rest:

- fixed a secondary error when dropping non-list, non-downloader pngs on Lain's easy downloader import window, and fixed a 'no interesting objects' reporting test when dropping multiple pngs
- added a 'cache report mode' to help debug image and thumb caching issues
- refactored the media viewer code to a new 'canvas' submodule
- improved the error reporting when a thumbnail cannot be generated for a file being imported
- fixed an error in zoom center calculation when a change zoom event was sent in the split-second during media viewer initialisation
- I think I fixed an issue where pages could sometimes not automatically move on from 'loading initial files' statusbar text when initialising the session
- the requirements.txt now specifies 'requests' 2.23.0 exactly, as newer versions seemed to be giving odd urllib3 attribute binding errors (seems maybe a session thread safety thing) when recovering from connection failures. this should update the macOS build as well as anyone running from source who wants to re-run the requirements.txt. I hacked in a catch for this error case anyway, just a manual retry like a normal connection error, we'll see how it goes (issue #665)
- patched an unusual file import bug for a flash file with an inverted bounding box that resulted in negative reported resolution. flash now takes absolute values for width and height

next week

Back to multiple local file services. Mostly more backend cleanup and prepping File Import Options and the Client API for talking to multiple locations.

I had an ok week. I improved the new image rendering system, cleaning up instability and errors and mitigating the tiling artifacts. I also fixed some other issues and optimised some database queries.

The release should be as normal tomorrow.

full list

- misc:
- hydrus now keeps a track of when files were deleted! this information has never been recorded clientside, and it is sadly not retroactively recoverable, but it is stored for all deletes from now on. on occasion, when hydrus says 'this was deleted from xxx', it will now have 'at an unknown time' or a nice '3 days ago' string attached. it will take a few seconds to update this week as the new table data is created
- the 'trash' panel on review services now has an 'undelete all' button
- fixed a typo error in manage services when auto-creating a service account when more than one type of account can be created
- the thread watcher page now sorts the status column secondarily by next check time (previously, equal status would sort alphabetically by subject as a fallback secondary sort)
- I have renamed some network concepts across the program. before we had access keys, account keys, and registration keys--now we have access keys (secret password for account), account ids (identifier for account that jannies may need), and registration tokens (one-time token used to create a new account). I hope this reduces some confusion
- reduced some overhead when fetching media results for a search, and when refreshing their tags on major content updates
- fixed a 'no such table: mem.temp_int_hash_id_1'-style database error state that could persist for 30 seconds or more after certain rare rollbacks
- fixed the FlipFlip link html in the client api help
- fingers crossed, I fixed that bad Applications shortcut in the new macOS release
- fixed a couple more instances of 'pulsing' progress gauges. now they should be blank
- .
- more efficient updates in sessions with collected media:
- several updates this week should reduce client UI lag when the session contains any pages with a lot of collected media, particularly when you are also running several downloaders (which spam all sorts of content updates across the client):
- the content update pipeline now tests collections for their files before content processing, and now filters down to process just the updates in a group that apply

- boring multiple local file services stuff:
- I cleaned a bunch of old hardcoded references to 'my files' and related code. it is not very interesting, but there are a few hundred references to clean up and convert to a system that supports 1-to-n local services, and this week I started hacking away, mostly presentation stuff, labels on menus and so on
- your 'my files' now has a separate deletion record to the 'all local files' domain. its count shows in 'review services', and for the moment will just be 'all local files' plus the count in trash, but this will become more important when you can have multiple 'my files'
- behind the scenes re-jiggering means that the deletion record now records deletion time and original import time. delete and undelete transitions are neater as a result
- logically, files are now generally no longer moved to the trash nor undeleted from there, they instead fall there when they are in 'all local files' but no longer in any local domain, and are undeleted back to a specific service. a bunch of awkwardness is cleaned up, and import/delete/undelete content updates are regeared and ready for multiple local file services
- a whole bunch of little things have been fixed and changed behind the scenes. I cleaned file service code in general as I went. examples of little things fixed:
- - a 'delete and do not keep a deletion record' action now correctly does not change the cached number of deleted files as reported in review services
- - the 'clear deletion record and try again' 'remove from trash' component now uses a unified and improved and UI-updating 'untrash' database action, with correct service count changes and UI-side status changes
- - the 'clear deletion record and try again' action on downloader import queues now handles mixes of actually deleted files and files just in trash more neatly
- - in the very odd situation that you are looking at a non-local file on 'all known files' and it is then imported using 'archive on import', its thumbnail and metadata now fade in correctly as archived
- added some unit tests to test the new file delete/undelete transitions
- cleaned up a bunch of hacky old db SELECT code

next week

Next week is a 'medium size' job week. I would like to try putting some time into the ancient image rendering pipeline and related systems like previous/next file prefetch. The basic media viewer has been jank and bad at zooming for too long. I am not sure I can make it beautiful, but I will try to clean some things up.

I had a great week focusing almost entirely on improving media viewer performance. I have overhauled the way images are zoomed and drawn to screen and completely eliminated the additional lag and memory bloat from zooming big images. Furthermore, the caching of rendered image data is greatly improved, so flicking back and forth between neighbouring images or different zooms is, for the most part, now instant.

The release should be as normal tomorrow.

full list

- macOS:
- I fixed an issue with last week's Big Sur compatible release where it wasn't finding your old database correctly--it was defaulting to a different location, so without a specific launch command otherwise, it started a fresh db and said 'hey, looks like first time you ran the program'. if you are a long-time user of hydrus, please install and run 436 as usual, it should figure out your old db location correctly as ~/Library/Hydrus without any launch command override needed
- If you never ran any of the old macOS builds, and you started using hydrus for the first time on macOS last week with the experimental Big Sur compatible build, your brand new database is in a funky location! don't update yet, or you will delete it! You will want to copy your .db files and the client_files folder from inside_the_435_app/Contents/MacOS/db to ~/Library/Hydrus, which should for most people be /Users/(YOU)/Library/Hydrus. feel free to ask for help if you can't figure this out
- fixed a 'this is macOS' platform check for newer macOS releases, which ensures the 'userpath' fallback is correctly initialised to ~/Library/Hydrus
- fixed the new macOS github workflow build script to tell hydrus that it is running from inside an App, so it knows to default to the userpath fallback correctly
- the macOS build now has the old filename
- it also has the ReadMeFirst.rtf file and Applications shortcut
- collected the new build-related files in static/build_files, which will likely see more files in future
- .
- pending tag cache regen:
- two new maintenance tasks are added to the database->regenerate menu--one that forces a recalc of your total 'pending' count as used in the pending menu, and one that recalculates the cached pending tag mappings for storage tags (just like the display one added some time ago, but one layer deeper). the menu entries are relabelled appropriately
- these routines will be run on database update, and should correct the bad pending menu counts many users discovered last week (the new efficient way that the pending count is calculated exposed some legacy bad cached pending storage mappings entries. we'll see if they come back, or if this is just clearing up bad counts hanging around from ages ago)
- the quick pending mapping cache regen routines take a little longer to initialise now, but they now clear out surplus tag data, rather than just regenerating the 'correct' tags
- .
- misc:

- the 'refresh account' button in _review services_ now works on the new async system. it presents errors nicely
- a repository's current update period is now stated in its review services panel
- review services now says 'checking for updates in...' rather than 'next update due...', which is more accurate and will matter more with small update times
- fixed some false positive instances of 'this server was not a tag repo' error in the network engine.
- the hydrus server now also outputs hydrus specific 'Server' header (rather than some twisted default) on 'unsupported request' 404s and any other unusual 'infrastructure' 4XX or 5XX
- if the repository updates in the filesystem are lacking some required file information when calculating what to process, the client now queues those files for a metadata regen maintenance job and raises a cleaner error
- just as a safety measure, if a repository ever happens to deliver a metadata update slice with a 'next update due' time that has already passed, the client now adds a buffer and checks tomorrow instead
- a new program launch argument, db_transaction_commit_time, lets you change how often the database's changes are committed to disk. default is 30 (seconds) for client, 120 for server
- altering the repository update period now prints a summary of the change to the log
- updated the ipfs links in the help
- updated the main help index.html and the github readme.md with the user-run repo and wiki at https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts

next week

I may or may not be tied up with IRL stuff for a bit. Once I am back to things, I will keep working on smaller issues and get started on the pre-work for multiple local file services. There are several hundred locations where the 'my files' service is hardcoded as the local file reference, so a decent part of this work, before I get to file service migration and new location import options, will just be putting some time into ancient code.

I had a good week fixing bugs and cleaning old code. Client performance is improved for large sessions, particularly those that have 'collected' page media, and I got started on multiple local file services by overhauling the core file delete/trash/undelete system. As a neat side-effect, the client now remembers when you delete files.

The release should be normal time tomorrow.

the rest

I had another go at multi column list 'last column' resizing. Dialogs shouldn't judder around so much when you resize them, and more instances of dialogs growing or shrinking slightly on the every re-open should be mitigated.

Tag upload to a repository has a new safety check for a particular error state. I know one user got stuck with some pending mappings that wouldn't clear--in fact it lead to an endless cycle of uploading!--which is now caught, and a new maintenance routine can fix it.

The Client API's file search now converts tags with asterisks into wildcard or 'namespace' search terms. Its version is now 16. Also added to the Client API help page is a link to https://ififfy.github.io/flipflip/#/ , an advanced slideshow engine that now supports hydrus as a source.

full list

- misc:
- a new macOS build that should run on Big Sur is now ready, it should be attached to this release. it is built on github automatically, and is thanks to hard work from Suika and ReAnzu. I am attaching my old release as well, just in case I messed up somewhere on my end. if you are a macOS user, please try the new App! it will not work on very old macOS like 10.12, but if this works out today for the majority of macOS users, I will be moving to just putting this new build out going forward. I'll add some polish like the readme.rtf and harmonise the filename etc.. too. I'd love to cut the filesize down, but this may not be possible (it is apparently some modern macOS thing where it bundles old and new versions of libraries in the same App so you basically get it twice)
- the bottom-right corner of the regular media viewer canvas now also shows media zoom
- the StringSorter object now has a simple 'reverse' sort type
- the infamous multi-column list 'last column' width calculations are improved: first, dialogs with multi-column lists should no longer judder back and forth a single character's width as you expand the parent window. also, the last column saved size (which is used in dialog relaunch width initialisation) is now snapped to rounded 5-character intervals, which should mitigate various 'fuzzy' reasons for some dialogs to remember a larger or smaller size and grow or shrink one or more characters' width on the next launch
- the _help->debug->gui actions_ menu has a new entry to reset all multi-column list saved widths back to default
- the 'edit OR predicate' panel when you shift+double-click an OR predicate now expands horizontally and vertically with the window

- new server stuff:
- a new permission is added to hydrus service accounts--'manage options'. any account with 'manage account types' will get this by default on update
- any account on a repository with 'manage options' permission will now see 'change update period' in the admin services menu! it launches a time delta control with the current update period and will send the new one up to the server. the client will resync account, options, and metadata immediately, and the server will generate any now-due updates immediately, so you should be able to watch changes occur in 'review services' and the server terminal live. other users will catch up to the new time when they next hit an update. various hardcoded check periods (like how often due updates are checked for and delay-buffered clientside and serverside) are shrunk significantly. the whole system should react to changes better
- the minimum settable update time is now 10 minutes (the default value remains 100,000 seconds), but I recommend you try larger, say an hour minimum, at least to start. the network generally works more efficiently with higher numbers, and be warned, if you are adding 144 updates a day, there may be bloat problems after a year
- let me know how this goes, whether you are running a server on a LAN or just a regular user running on one who gets a new update time!
- the new 'full metadata resync' routine now triggers an immediate metadata update sync and wakes the daemon involved, so it should now happen as you watch
- fixed the new pause/play buttons on review services to use neutral pause/play icons, not the downloader pause/play
- brushed up metadata sync status string on review services
- cleaned misc server and network code
- cleaned up some old clientside service code
- the client api now supports wildcard and namespace tags in the file search call
- client api version is now 16
- added https://ififfy.github.io/flipflip/#/ , a slideshow engine that now supports hydrus as a source, to the client api page

next week

I have unavoidable IRL next week (jury duty), so I can't do the release on Wednesday. I will put a smaller one out on Tuesday!

I had a great few days mostly fixing bugs and cleaning things. The new macOS build is more polished, and I fixed an issue where it could have trouble finding your default database location. I believe I also fixed the issue some users had with 'sticky' pending menu counts, then added some UI quality of life, and wrote an experimental tag display mode that replaces underscores with spaces.

The release should be normal time tomorrow. I have unavoidable IRL on Wed, so it is moved up a day.

admin improvements

After a very long delay, account management is finally back. If you have permission to modify accounts, the 'modify an account' service admin menu item works again and launches a fully revamped window. You can also launch account modification from manage tags for a particular tag selection, the thumbnail right-click for file repository files, or the petition processing page.

The new window shows better data about accounts, and lets you zoom in on individual ones to see current account type, expiration, banned status, or message. It also shows if you are editing yourself!

The account modifications are: change account type, set/extend/clear account expiration, ban, unban, and the new 'set message'. All modifications now print janny and subject account key to the server log. I removed the old 'superban' (which deleted all that user's content contributions), but it will likely return in future.

The service admin menu also has a new 'review all accounts' entry that lets you see a simple summary list of all accounts, which you can then select for modification.

The manage account types dialog now lets you set up auto-account creation. It works a bit like subscriptions, with a 'account creation velocity' limit of 'x accounts per y time delta', like '10 per day'. This defaults to off obviously, but if you set some account types to auto-create, users will be able to see that in manage services.

future admin improvements

As always, there was much more I wanted to do than could get done. I wasn't able to finish improving petition processing workflow or add custom update duration, a serverside tag filter, or a server message, but I was able to get the important network stuff done I needed for these things. Since incrementing the network version is always such an obstacle, I am confident I will now be able to work on these things in normal weekly work. Please let me know how they work out for you as I roll them out.

full list

I had a good week. I fixed several bugs and added some quality of life, got a Big Sur build working for macOS users to try out, and for users who run smaller servers on a LAN or between friends, hydrus repositories now support changing their 'update periods'.

The release should be as normal tomorrow.

full list

- misc:
- thanks to the effort of a user, this week reintroduces a native twitter downloader! it now gets video! (and 'gifs', which on twitter are just mp4s) please experiment with it and move your nitter subs back to this as you find success. it is called 'twitter syndication' and uses a different access method to get tweet info. it should get the highest resolution videos and images. the search has limited lookup distance, perhaps 500 tweets, but should work for most subscription purposes. this is a first version and may have future updates
- on the main gui, middle-clicking and left-double-clicking to open the 'new page' dialog, and right-clicking to open the page menu, should now only work on the page tabs or page greyspace. middle-clicking on some random downloader page greyspace should no longer spawn these commands. also, tiny change--middle-clicking now activates here on click _release_ rather than _press_
- on the file sort widget, the asc/desc sort dropdown is now a 'scrollable' menu button. since sorts are just one of two values, you can now scroll either direction to flip it
- the 'collect/leave unmatched' in the collect control is also now a scrollable menu button
- the new tag sort dropdowns are now all scrollable menu buttons. go ham with them
- middle-clicking the collect-by dropdown now clears it
- namespace file sort now supports a-z and z-a sorting. files with none of the matching namespaces still count as 'less than a' in a-z terms, but since I am updating all this code, perhaps this could get more attention. I don't use this much, so if you do, let me know what you would like
- network job controls now show their jobs' current bandwidth limits on their cog menu, split up by network context. you can edit the bandwidth rules directly from this menu, and if it is using defaults, set a specific ruleset
- userpath generation routines used for database location fallback and default export directory determination now recover from failure in the case of undefined user directory. the client will now not boot if the userpath is needed but undefined
- the client api no longer prints empty lists for any tag statuses on file metadata calls, nor service entries that have no tags at all
- fixed the new subscription 'caught up to small initial sync' calculation, which last week was only firing properly after one page of results
- subscriptions are now better about saving interesting status notes on their gallery logs, rather than overwriting with boring 'no new urls found' messages
- fixed a typo bug in the new range header implementation for unended ranges
- .

God bless you Hydrusdev

I had a good week getting started on the network updates. I cleaned out ancient code and improved some admin-side workflow. There is plenty more to do.

The release is still expected to be another two weeks, for April 7th.

>>1032
Thanks m8, keep pushing.

I had another good week grinding on the network code. I fixed and cleaned old admin functions and started on future extensions for these systems. The main tasks remaining are improvements to the janitor petition processing UI and new features for core repository storage like a tag filter and dynamic update times.

Unless I really fall into a hole, the release is still expected for next week, for April 7th.

I had a great three weeks working on network functions and serverside admin controls I had neglected. The old account management controls are working again with many improvements, account types can now be set up as user-creatable straight from the client (no need for registration keys!), and much of the code is cleaner and handles errors better. In the rewrites, I also laid the groundwork for custom update periods, serverside tag filtering, and faster petition processing, which I hope to get done in the coming weeks.

Regular users who do not admin a server do not have so many changes, although I have fixed bugs and added more controls to repositories, so you can now pause account sync, update downloading, and update processing separately, and I fixed a logical problem with processing that was sticking some users with deleted siblings.

The 'network version' goes up tomorrow, and the PTR should be updating, so if you sync with the PTR, you will want to update some time to keep talking to it! There's no rush, but you'll get a little error popup if you try to talk to an updated server on an older client.

The release should be as normal tomorrow.

advanced users

I added a String Selector/Slicer object to the parsing system. You can select the nth item or the mth to nth. m can be 'start', n can be 'end', and either can be a negative index. Try mixing it with the new String Sorter! The String Processing UI has also had an update pass to better support multiple strings--it now shows everything its parent formula had, so you can better test sorting or how processing is changing the third string in the list or similar.

The Server/Client API now support single 'range' requests for files (basically anything not html/json). Apparently Safari was having trouble getting videos from the Client API, let's see if this fixes it. If not, I can try supporting the (more complicated) multiple range requests. The server's error handling is also improved--on certain 'emergency' errors, it was sometimes hanging without closing the connection properly.

full list

- tag sorting:
- the tag sort dropdown has been replaced with a dynamic control. rather than one big list with all possible permutations, you now work on each variable (sort type, asc/desc, group by) separately. what you are actually sorting is easier to understand and select
- my stupid "lexicographic/incidence" labelling is replaced with the simpler and neater 'tag', 'subtag', and 'count'
- when in the manage tags dialog and sorting by tag or subtag, you can now turn off the 'use sibling' sort.
- I'd like to further neaten the workflow here, making the individual dropdowns flip back and forth with a mouse scroll in either directior rather than being just up/down allowed. let me know overall how you find this new control
- the 'tag sort' object is updated behind the scenes as well. your old value should be converted automatically
- fixed an issue with count tag sorting where deleted tag counts were being counted even when not displayed
- if you try to search tags on a page of thumbnails that holds an invalid tag, this is now caught gracefully and you get a little popup saying 'please run the repair invalid tags routine'
- .

- advanced string processing:
- added a String Selector/Slicer object to the parsing system. this object allows you to select the nth item in a list of parsed strings or the mth to nth items. m can be 'start' and n can be 'end', and negative indices are allowed for both. pair it with the new Sorter for some neat new tricks!
- the string processing edit UI is now _more_ multi-string-aware. the test panel has had a code cleanup pass and now has a list of all the starting strings in the test data (e.g. all the urls parsed by the formula that launched the UI) rather than just the first, and a list of all results from that list. selecting any of the starting strings populates the 'single string' area, so you can now zoom in on one particular string to see what is happening to it
- the String Sorter edit UI now gets all the strings at that stage of processing, so you can review the sort properly
- the new String Slicer edit UI similarly gets all the strings at that stage of processing
- future updates will expand multi-string presentation and testing. I'd like to show the whole list at each stage
- .
- server/client api core improvements:
- I had a go at supporting the Range header for file (basically this means anything non-html/json) requests. I added tests and it seems to work. as I understand this mostly applies to browsers pulling video from the Client API. to start, I am supporting single range requests. if it is needed, I'll try to get Multi Range requests right, but for now they'll 416
- the client now understands 416 ("can't do that requested range m8") errors
- I reworked the serverside error handling chain. this has been borked for a long time due to my own lack of understanding of twisted's deferred system, and certain late-stage errors were just not being handled right. the server should no longer hang on these and now should print error info correctly, including a rough 500 in true late emergencies, and terminate the connection correctly
- .
- boring:
- fixed up a handful of typo-borked unit tests
- fixed my ordinal (xst, xnd, xrd, xth) text generator to deal with 11, 12, and 13 correctly lmao
- started some db maintenance routines and logistics to recover definitions and remove orphans in future, I feel great about it so far, but it'll have to wait for more of my db 'modules' refactoring to be more useful
- updated the mpv dll on the Windows release to 2021-02-28, it may improve some video support/performance

I had a good week fixing bugs and adding a little quality of life.

The release should be as normal tomorrow. I expect it to be the last release for a few weeks so I can work on a larger network overhaul.