other highlights

I also added a 'tag in reverse' checkbox to the new 'manage tags' 'incremental tagger' thing. It adds tags like 5, 4, 3, 2, 1 instead of 1, 2, 3, 4, 5.

And all new system:url predicates have new labels. They are all a bit simpler, and they should copy/paste into the system predicate parsing system. All existing system:url predicates still have their old labels, so if this is a big deal, you'll want to recreate them and re-save your session/search.

Thanks to a user, the new docx, xlsx, and pptx file formats get some nicer thumbnails and a little metadata. It should all recalculate soon after update.

The Client API is now more careful about which files you can undelete, and it also now lets you clear file deletion records.

next week

I want to put proper time into getting a 'future build' working. Last time I tried, I ran into some technical problems related to the newer libraries I wanted to bundle, so I'll see if I can fix it all and have a test release for people to try. Otherwise, I just want to clear out some small jobs that this URL work boshed.

Anyone knows if I can download files (through url downloader, non-booru site) whilst keeping their original filenames somehow (as tags or whatever)? And in a way that I can have the files renamed when exporting? I found plenty of places saying it can be done but I've tried messing with the tag import options among others and nothing seems to work.

>>1623
Servers typically give that filename in the 'http header' of a file download. Hydrus generally does not parse this in the same way that your browser would and does not push it into the general metadata pipeline of a file import, so I'm afraid the only way you can figure this out is to parse it from the html file or similar, as you would for other metadata like tags or related URLs.

In general, also, most downloader makers do not parse the filename. The imageboard watchers like the 4chan parser does parse for the filename, and gives it a 'filename:' namespace. By default, I have these set not to add in the 'tag import options' for watchers, since most users don't want them.

If you feel clever and brave, you can try to add a 'content parser' to a downloader you are interested in to try and grab filename from the boorus you like, but beyond that, I'm sorry to say I don't have a good answer for you. I've thought a few times about making a 'filename' tag service that remembers hard drive import filenames, and could potentially get server-set filenames too, but every time I return to the idea, I realise it'll probably just get overwhelmed by 'Image.jpg' garbage that isn't typically useful.

Although I can't give you a nice answer, you might like to read my FAQs on this question, just so you see better where I am coming from and why I don't want to put much time into this:

https://hydrusnetwork.github.io/hydrus/faq.html#filenames
https://hydrusnetwork.github.io/hydrus/faq.html#external_files (somewhat related)

And if you would like to dabble with making/editing your own downloaders, check out the help here:

https://hydrusnetwork.github.io/hydrus/downloader_intro.html

I'd seen those pages already, thank you. I understand about filenames but this time I really needed them. Over 90,000 work-related images that were numbered in a specific order, and that ordering had to be kept. I've found another solution but thank you for your work and your reply.

I had a good week. I fixed a bunch of small issues, and I figured out the problems with the 'future' build, so I'll also have a special version for more advanced users to test out.

The release should be as normal tomorrow.

>>1625
Great, I am glad you found a solution!

I had an ok week. I reworked some URL handling in ways that are mostly important behind the scenes and cleared some small jobs. It'll just be a simple release.

The release should be as normal tomorrow.

The v567 release is cancelled! We found a problem and I need to do more work on it. I will move it on to next Wednesday, sorry for the inconvenience!

I had a great week. I tightened up the URL storage/handling improvements that I was not confident in last week, so I am very happy to put out the release tomorrow. There are also advanced new tools for downloader makers for handling 'ephemeral token' parameters along with new quality of life UI in the manage URL Classes dialog. For normal users, there are also several bug fixes, file handling improvements, and a couple little things in system predicates, emoji tag presentation, and reversed tags in the new incremental tagger.

The release should be as normal tomorrow.

I had a good week. The 'incremental tagger' system is working, allowing you to tag page:3, page:4, page:5 ... page:17 and similar to a selection of files, and .docx files are now importable.

The release should be as normal tomorrow.

I had a simple week, mostly fixing a bunch of bugs.

The release should be as normal tomorrow.

Yeah, this is tricky. I still think your best answer is to wait for me to implement a proper xml/html parser for sidecars, or do the parsing yourself in an external script (e.g. with python) and then convert your html into nicer .txt files that hydrus can suck up easier. You could also filter your undesired tags better there.

If the 'description' string is always in the same location, and that location is always the first or last index of your list of strings, then the 'string selector' might help. It does list slicing like in programming, if you are familiar, like "my_list[4:6]". If you aren't familiar with that, or the description is in the middle of the list here, you are correct in trying a string match, which is basically a filter.

If the description line has a classname in the html, you might be able to exclude it with a string match before you Split/Convert all the html garbage away.

I am not totally sure what you mean by being unable to import by pngs, but if I click 'import->from pngs' (pic related is from the 'sidecars' tab of the 'add tags/urls with the import' dialog after you drop some files on the client), I get a file dialog that allows me to import a png like this. Do you get different?

Update:

(i am using a crucial mx500 500gb with luks and ext4, if that helps)

I was looking at launch options to fix the storage issue, do you think that --no_db_temp_files would help? I have 16gb ram, is that enough?

>>1610
>>1611
>>1612
Yeah, the write is most likely temp storage. I've heard that in some situations, the actual amount written to disk is less since some temp-file magic means some of the shortest-lived data is purged from the write cache before it can actually be committed, but I don't have good numbers on the topic.

The PTR does a lot of database work. The no_db_temp_files parameter will reduce it by a good amount. Not sure how much, but it would reduce it.

Increasing db_transaction_commit_period may also reduce it, I think, since it will reduce the commit frequency (at the cost of increasing per-transaction size, which will strain your temp folder and slow processing time).

A user also wrote this document with a lot of related info, if it helps: https://hydrusnetwork.github.io/hydrus/Fixing_Hydrus_Random_Crashes_Under_Linux.html

I'm not a Linux expert myself, so I can't talk super confidently here. Let me know how you get on!

I had a good week. A bunch of system predicates now support customisable +/- ranges, both by absolute and percentage values.

The release should be as normal tomorrow.

I had a good week. I fixed some bugs and made some shortcuts a bit nicer to deal with, and there are a bunch of macOS improvements.

The release should be as normal tomorrow.

In future I hope to have an xml/html parser for sidecars, but I can't promise when it will be ready. So, if you have time, I think save these for later. Otherwise, parse them yourself and make JSON/txt files, or just run the download again if possible.

alright, thanks for your help!

I had a tepid week. I fixed some bugs and improved some quality of life, that's all!

The release should be as normal tomorrow.

misc

In v559, it turns out I did make a '54 years ago' mistake, on moving files from one local file domain to another. This is fixed, and on database update, any borked timestamps that were saved should be corrected too. Let me know if you encounter any more issues.

Mr Bones and the File History Chart are now under the database menu.

There's a new checkbox under 'files and trash', 'Remove files from view when they are moved to another local file domain'. This actually re-introduces the unintended behaviour we had a couple weeks ago, which was happening under 'remove when trashed', but targeted to the correct logical trigger.

The gallery downloader cog button now has a 'don't add new dupes' option, which discards the query_text/source pairs you add if they are already in the download list. If you need to paste a hundred queries from a big mishmash list, I think this will help.

As a very brief test, I've added 'sort files by pixel hash'. If you have some pixel hashes in your current query, this will make them stand out, but the rest of the sort data is meaningless. I had wanted to try out some similar hacky new tech with 'sort by perceptual hash', but thumbnails don't have quick access to that data yet, so it'll have to wait.

I wasn't sure when my internet was going to come back up, so I dumped a bunch of time into some longer cleanup jobs. There's a ton of updated code behind the scenes, particularly around the way files get informed about new metadata, but nothing you need to care about. I have some new plans here and can see the path to getting all sorts of new metadata commands integrated into the shortcut and undo systems. I changed a couple thousand lines, so there may be a typo bug somewhere--let me know if you try to set any unusual content changes and it throws an error!

next week

Now we have tech that relies on file order, I think it is time we get the ability to conveniently reorder thumbnails. This will also help when I figure out cascading tagging in 'manage tags'. I'd love to have reordering by simple drag and drop, but that may be too complicated to slip into one week. I'll see what I can do.

Hey, I screwed up a line in all the rewrites and one error in the duplicate filter is particularly bad, sometimes causing crashes. If this has hit you, there's a hotfix here:

https://github.com/hydrusnetwork/hydrus/releases/tag/v560a

I had a mixed week. I added thumbnail rearrangement, and I cleared some bugs and quality of life stuff, but I didn't get as much as I wanted done.

The release should be as normal tomorrow.