I had another light week. I added some tuning options for the new 'related tags' search and cleared a handful of misc small work.

The release should be as normal tomorrow.

related tags

EDIT: In final testing, this stuff was working slow IRL for me, so I reduced its search space for today. You are going to see a lot of 'search domain is too large' on the PTR. I'll keep working!

Thanks to a user who sent in a great proof of concept, the 'related tags' column (which shows in manage tags and you can turn on under options->tag suggestions) has some new buttons to test. I called them 'new 1' and 'new 2'. The first works fast but may have limited results on repositories, the second--which only shows on repositories--can only search small-count tags but gives deep results.

Secondly, if you select some tags in the main tags list, the 'related tags' search will now only search those tags! If you can't remember a character name, then select a 'series' tag and click for a related tags search, and you'll get a bunch of them!

Thirdly, these new search buttons now work on manage tags launched from multiple thumbnails!

Please give this a go and let me know how you get on with IRL data. The quality of the new search results is better than my old routines, but I need to do a bunch more tuning and optimise their search. Are there too many recommendations (is the tolerance too high?)? Is the sorting good (is the stuff at the top relevant or often just noise?)? Which source tags (e.g. 'page:x') give the worst suggestions?

I'm sufficiently happy with the new performance, particularly with the tag selection, that once this is working nice, I think I'll make it default-on for everyone. Lots of people missed this system, and I always thought it was a bit jank.

other highlights

I wrote some sidecar import/export help here! https://hydrusnetwork.github.io/hydrus/advanced_sidecars.html

>>1453
twitter gay, thanks for fix
I suppose this means my nitter subs are broken too?

On a different topic, does Hydrus do anything with DeepBooru or are there any extensions for it you know of that I can maybe use to try some autotagging?

>>1455
Is DeepBooru the same as Deep Danbooru, or the same basic model, so you could switch it in place? Hydrus-dd offers several ways to plug into Deep Danbooru, here:

https://gitgud.io/koto/hydrus-dd

I was talking with someone else today about how I'd like to do more with this sort of tech in future, just making it easier to work with the sidecars you need or the Client API whatever. I'm sure image-to-text is going to benefit from all this text-to-image tech that's now floating around, especially now it is becoming reasonable to train your own models to do stuff. Hydrus as a big metadata-rich archive seems like it plugs into that tech pretty well!

I unfortunately had a light week. The new 'related tags' search is polished and turned on by default for all users, and I fixed some small bugs.

The release should be as normal tomorrow.

>>1448
>Minis Forum
I recently picked up an Minis Forum HM80, and I like it a lot. It's my first computer from this company, and it's my first desktop in about a decade. I've been using laptops for so long before this, but this is a nice change of pace. It's a lot faster than I thought it would be too.

I had a low productivity week, so I am going to skip the release and get more done on Wednesday.

v514 should be on the 25th of January. Thanks everyone!

----

For Client API developers, I am clearing out some obsolete calls and data structures in v514! Since we have a spare week, please review the pending breaking changes:

* '/add_tags/get_tag_services' is removed! use '/get_services' instead!
* 'hide_service_names_tags', previously made default true, is removed and its data structure 'service_names_to_statuses_to_display_tags' is also gone! move to the new 'tags' structure.
* 'hide_service_keys_tags' is now default true. it will be removed in 4 weeks or so. same deal as with 'service_names_to_statuses_to_display_tags'--move to 'tags'
* 'system_inbox' and 'system_archive' are removed from '/get_files/search_files'! just use 'system:inbox/archive' in the tags list

Full documentation preview, including some new commands and parameters, here:

https://hydrusnetwork.github.io/hydrus/developer_api_future.html

>>1450
>Yeah I almost feel like some part of them fell off the back of a truck,
The price is great. It's very affordable, and you get a lot of bang for your buck.

I had a good couple of weeks working on a lot of different things. There's updated downloaders, a heap of bug fixes, optimisation and quality of life, a significant Client API update, and a new related tags method to test out.

The release should be as normal tomorrow. (actually later today)

I had a good week doing one thing. The Client API now supports the main duplicates commands, fetching current file relationships and the main filter workflows, and then setting basic file relationships back again. This is a complicated feature for users who play around with the latest Client API tech, so tomorrow's release is for those advanced users only!

The release should be as normal tomorrow.

Is there a way to show the total duration of all selected video files? It shows the total file size (rounded), but I can't find an option to show the total duration.

Is there some way to apply to all selected images tags like page:1, page:2, etc in same order as it visible now?

>>1442
I had this exact same thought over my vacation when I was looking at some vidya captures I had made. I'll add this!

>>1443
Not yet, but I would like to add this. I've never really been happy with how hydrus handles 'page:1' style tags, but we'll never get better at it as long as it is difficult to easily add them to files after they are imported.

I had a great week back at work. I cleared a variety of small jobs and spent some extra time on the duplicates system. Several duplicates search bugs are fixed, and you can now set a potential pairs search for 'one file is in search A, the other is in search B'.

The release should be as normal tomorrow.

I had a good week mostly fixing bugs, including the recent annoying-but-harmless shutdown bug some users had and the macOS app boot bug. There's also some UI quality of life.

The release should be as normal tomorrow.

>>1425

Thanks for your continuing work on the project man, much appreciated. I have some feedback/wishes:

1. 4chan thread watcher via Tor gives 403 Cloudflare error - simple downloader works. Is there something one can do about this beyond switching circuits/proxy/clearnet? I mentioned this before and the browser header shouldn't make a difference here, as downloads as such work, just not the watcher.

2. 4chan simple downloader fails at .webms in same setup, these have to be manually added.

3. Add a second/third bar for tabs, so that the titles are not squeezed when having many tabs open.

4. PDF/ebook compatbility to use Hydrus for more informative tasks (this might get you a lot of support from librarian types).

5. Grouping of tags - as with the namespace/creator group by function, have colored bars or something that groups all items belonging to that category - or some similar function to have a "folder like" grouping in the main window. Tags are awesome for precise searches/classifications, but sometimes that "all in one" feeling that folders offer might be useful; I may have overlooked something though.

As always, I really love the work you do and you simplify a lot of time intensive stuff. I'd love to have a setup working that scrapes threads automatically by topic, I was thinking of a hydrus + something scripted setup that identifies threads by title/keyword and then automatically adds them to the watcher. This would save so much time and erase blind spots when covering news topics etc. Any ideas as to the feasibility?

>>1426
Thanks, I'm glad you like it!

1) The watcher hits up the 4chan API (in this form https://a.4cdn.org/m/thread/16086187.json), whereas the simple downloader hits the main html. My guess is that's what CF is blocking. If you had to set up headers or User-Agent to get the html to work, might want to replicate it for a.4cdn.org. Hydrus Companion may help here to copy cookies, not sure. If CF just blocks API hits from TOR, you may be out of luck.

2) Yeah, I bet the simple downloader looks for img tags or something, whereas webms are under 'video' or an iframe or something. Simple downloader can't do two things at once very well, I'm afraid! If you are feeling brave, you could try editing the simple parsing formula to try to get both img and (video), but I'm not sure if it is that clever.

3) Check out the 'page of pages'. Hit F9->special->page of pages. That'll let you nest your page tabs.

4) Thanks. Yeah, I'd like some basic support for more text formats, and 'any file' support, which will take some extra work for some internal technical reasons. But I'd like it. However, I do not think I can support nice text reading any time soon. I'm terrible at drawing lines, but I know I'll never have the time to make something anywhere near as good as Calibre, so the support will stay as an 'open externally' button, like PDFs now, for a long time.

5) Yeah, I absolutely want [+] expand/tree format for the tag list. That list is my own custom widget, so I have to code everything in it. But this is something I really want for myself too, and options for things like 'start with title tags collapsed'. And custom namespace sorting so you can put creator up top, that sort of stuff.

Having auto-scraping has been a long-time thought. I'm hesitant to make too many automatic searchers, since I like human eyeballs behind big decisions. I know some users, however, who have had big success writing their own scripts that fetch the current thread catalog, scan it for threads they want, and then pass them on to Hydrus using the Client API. If you are ok with scripting, you might like to play around with it: https://hydrusnetwork.github.io/hydrus/client_api.html

As always, I'm short on work time. When I eventually roll some of 4 + 5 out, let me know how it goes for you, and if you have any other problems or ideas.

>>1428
1) Will try that out, once I find the time. Thanks for the answer!

2) I'll experiment on that as well and will report back with how it goes.

3) Nice, awesome!

4) I'd be enough to merely have the works indexed inside Hydrus. You can always CTRL+E to open in your preferred reader/program with all the capabilities you need on the media itself. It's just so you can collect and organize the files, which Hydrus really blows anything else I know out of the water. There's some academic software that lets you organize your sources and stuff, but it's more of a list thing. Since you can add tons of metadata... well let's just say Hydrus and zlib would go hand in hand ;)

5) Yeah, collapsible tag trees would make stuff more easily readable. I know you can organize by custom namespaces and that is already really powerful, but it'd be cool to have tag:one tag:two tag:three below each other in the tag list and in the folder list as well, in "blocks" so to speak.

I feel you on automation. One misstep and you practically DDoS a site or flood your hard drive. I also didn't listen to your warning in the documentation haha. "and then THEY DO SO only to work through thousands of files", I felt that so hard and it brought a smile to my face. But once you have figured out what you want to parse daily etc. you then really want the automation for these specifica.

I really do appreciate your time man. Your project is one I always look into the documentation again and check the forums here. That's a rare exception for a really solid project. If you go the academic/library route, you might catch some licensing stuff to capitalize on your outstanding work, like for official use and stuff.

birthday

The first non-experimental beta of hydrus was released on December 14th, 2011. We are now going on eleven years.

I had a challenging 2022. Several IRL problems appeared from nowhere, but I decided to keep on trudging, and we got some work done regardless. I'm satisfied with the result. We started the year with the run up to multiple local file services, a long-planned expansion to the sheer shape of the database, and with that, beyond the new file services, came an array of new database search tech, from the ability to search deleted files and search unions of different file domains to enhanced cancel tech for snappier UI and more accurate tag counts and the nice user-made file history chart. We also saw new time search tech, with nicer UI, the ability to search the last view time of a file, the recording of archive times, and the broad 'modified time' parsing expanson, which combines post times from many sources to give you a more precise 'source time' of the things you download. Thanks to user help, we also figured out the new help, which looks great, is easily searchable, and is simple to edit. We hammered out more stable versions of mpv and other spicy libraries for the different builds and user environments. The UI had numerous quality of life improvements, from dejanked layouts and sizing calculations to fixed custom widgets and overhauled hover windows to cleaner selection logic and quick-action buttons for easy pastes and mass-application actions. File notes are finally parsable and now display on the media viewer. Sidecar importing and exporting became more powerful. EXIF and other human-readable file metadata is now parsed and viewable. The serverside janitor workflow finally received some improvement. The Client API grew to support autocomplete tag searching, note editing, cleverer file reference and file delete, nicer thumbnail fetching, better error/disconnect handling, tag display type support, more hash fetchings and lookups, and the new timestamps and file metadata statuses like EXIF. And the program became significantly easier for any user to run from source.

There is still no end of work to do, and I am thankfully able to continue, so I plan to keep going into 2023. I deeply appreciate the feedback, help, and support over the years. Thank you!

If you would like to further support my work and are in a position to do so, my simple no-reward Patreon is here: https://www.patreon.com/hydrus_dev

Congrats on turning 11, dev. Hope you're doing well.
I've been using Hydrus off and on since 2015, but I reimported a few times early on so my oldest files are from 6 years ago. I've been going through some of them, finally getting around to sorting a bunch of stuff I scraped from tumblr back then.
I just transferred my "active" copy of Hydrus to a new drive, figured the original one might be getting worn out, best to keep it as the backup. In the process, it seems like it forgot to keep sound off, that or sound in-client is just a recent addition.
Is there a way I can toggle sound on and off globally for the program?

I had a good week. I fixed some bugs, added some new import and tag search options for advanced users, and made thumbnails look crisp at high UI scale.

The release should be as normal tomorrow.

>>1437
Thanks. Yeah, I am doing good overall. Looking forward to my vacation next week though!

We added 'mpv' support I think two or three years ago--that's where the sound is coming from. You should notice video plays a lot smoother too. Any time you are looking at a noisy video, move your mouse to the bottom of the video frame so the scanbar pops up. On the right should be a 'speaker' symbol. Click it and you'll get global mute. There's another speaker symbol in a popup that just mutes preview/media viewer.

There's some shortcuts for this too, under file->shortcuts->global.

>>1438
Thanks!

I had a good week. I focused on improving file notes, making it easier to copy them from one file to another and adding them to the duplicate merge settings. I also made taglist drag-selection feel nicer.

The release should be as normal tomorrow.

>>1429
Unfortunately this is not supported yet. I'd love it though, and other users. It will take a lot of work, so I cannot predict when it will happen.

>>1430
Probably this is naive and I miss something, but can I do it via client.master DB editing?
For example id of artist namespace is 1 and creator is 2. Whats happen if in table tags i replace all 1 in namespace_id column by 2?

>>1431
These are my local tags and I want to adjust them to the hydrus standard. So it will be fully ok if there is no sibling for it or something else. Simple replace, that's all that I want.

>>1431
>>1432
Unfortunately I would say it is not reasonably possible. If you want to poke around client.master.db, feel free, but you must make a backup beforehand.

The stuff in client.master.db is replicated in client.caches.db, and the 'master' mappings records are in client.mappings.db, which are again replicated in the caches db. You'd likely have to change ids or definitions in multiple locations to make this work without errors, and all your tag counts might need to be regenerated since we are talking about a merge rather than a rename here. With few exceptions, the database just isn't simple enough to edit a namespace by hand.

>>1433
Oh, I see, better I do not touch it. Thank you for your help.

everything is broke again
venv won't start because there is no activate.bat
why do you hate me so much, why do you make me go through all this pain and torture? just because i won't submit to windows10?

I had a great week fixing some bugs, adding a hash lookup to the Client API, and attaching a Tag Filter to the PTR, which will make it easier to clear out some less useful and misparsed tags right as they are uploaded, no petition needed.

The release should be as normal tomorrow.

>>1420
Sorry to hear your trouble. Can you talk more about what happened? Did you change the venv, or could an anti-virus or other program have interfered with it? If the activate.bat has disappeared, try running the setup-venv.bat again--that'll clear out your old venv folder and reinstall everything.

Since you have had problems with this, could you tell me what has been most difficult to get going? Has my 'running from source' help been confusing at any point? Did the setup bat fail? How did things break before, and what was the solution? How could I make it easier? I have been adjusting the help and the scripts in recent weeks to be more clear on each step, so I regret it hasn't worked for you.

>>1421
I managed to make it work by building from scratch in a new folder.
Before I had database failure, due to my mistake of not backing it up properly.
Venv failed before until I found working mix of qt5, old python, new mpv.
It's painful because I don't even know the difference and simply seeking blindly for something that will run somehow.

>>1423
Sorry again for the problems. I am glad you are working now. Which old version of python worked for you? 3.8 or earlier? To make it less of a 'blind' situation in future, how could I improve my help? Are there specific version recommendations I could make for people on Windows 7?

You can back up the venv folder, btw, so now you have something that works, you can save it all--it is completely portable.

I will try and ensure that hydrus keeps working for Qt5 and older python as long as I reasonably can, so let me know if a future update breaks something.

I had an ok week fixing some bugs, expanding the Client API a little, and improving some server admin tools.

The release should be as normal tomorrow.