/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.

Posting mode: Reply

Check to confirm you're not a robot
Drawing x size canvas

Remember to follow the rules

Max file size: 350.00 MB

Max files: 5

Max message length: 4096

Manage Board | Moderate Thread

Return | Catalog | Bottom

Expand All Images

Version 370 Anonymous Board owner 10/03/2019 (Thu) 00:10:55 Id: 48e809 [Preview] No. 148
https://youtube.com/watch?v=h27wMuv_--s [Embed]
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v370/Hydrus.Network.370.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v370/Hydrus.Network.370.-.Windows.-.Installer.exe
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v370/Hydrus.Network.370.-.OS.X.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v370/Hydrus.Network.370.-.Linux.-.Executable.tar.gz
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v370.tar.gz

I had a great week. There's a big improvement to tag display management and a bunch of smaller stuff.

tag display

Part of this ongoing tag overhaul work I am doing is to catch up some ancient systems that aren't able to deal with the size of the dbs we have now. The old 'tag censorship' is probably the biggest example of this. I have rewritten this to work in a more sensible way using newer objects.

The main difference is services->manage tag censorship is now service->manage tag display. You can now hide tags from the front UI using the new Tag Filter interface, with separate filters for 'single' views, like thumbnail banners and the taglist on the media viewer, and 'multiple' views, which for now means the 'selection tags' list on normal search pages. For instance, you might like to hide page: tags from the selection tags list, since they usually clutter things, or you might like to only show creator/series/character. You can set any normal whitelist or blacklist, and filtered for each tag service (e.g. 'only show creator/series/character tags from the PTR') and/or blanket rules for all services.

Right-clicking on a tag in one of the lists will provide options to hide it or its namespace from that list type. These menu actions now take effect immediately!

Anonymous Board owner 10/03/2019 (Thu) 00:13:45 Id: 48e809 [Preview] No.149 del
The behind-the-scenes of how these display tags are generated and served is also much improved from the tag censorship system. The lists are accessed in a unified way and are cached (previously, they were manually filtered on the fly in each instance), so a variety of thumbnail selection and navigation tasks should be a little bit faster, and changes to the filters should cause reliable refresh updates across the program. Tag siblings are now also cached using the same system, so this layer of presentation is also faster.

Unfortunately, these caches do have to be computed the first time you load a file. It slows initial load by a small amount, but in my tests not a very noticeable one. I would like to move at least some of these caches down to a persistent location in the database at some point, so the load is faster (this will also help things like accurate autocomplete result file counts for tags with siblings, a long-term goal).

Another next step is to add filters for storage, so you will be able to take the next step of saying 'do not allow x, y, z tags on this service', which stop parsers from pending certain tags and will save db space for users who process a big repo like the PTR but only want a subset of what it offers (e.g. 'no title: tags').

If you used tag censorship previously, your rules should be converted to the new system, but make sure to check them out. Previously, your 'censored' tags were hidden even from the manage tags dialog, but this is no longer true--this week's work adds reliable display rules, and making 'true' storage rules (and perhaps a filter for autocomplete search results?) will come later.

the rest

I have added system:modified date. It works just like system:time imported but plugs into the new modified date metadata. It is fun to put in 'find files modified more than ten years ago'. Remember that if you have many files in your db, the new file maintenance manager may still be retroactively generating this data for them in the background. You can check your current maintenance queues under database->maintain->file maintenance->review scheduled jobs.

Anonymous Board owner 10/03/2019 (Thu) 00:16:39 Id: 48e809 [Preview] No.150 del
Files with duration (video, audio, animated images, swf) will now get a ' ▶ ' label on their thumbnail, just like the recent speaker icon for 'has audio'. The 'has audio' one overrides the 'duration' one, but this is a nice simple way to see videos vs still images. You can edit the label used under options->audio and duration. I do not really like how the ' ▶ ' looks, so maybe we'll ultimately go to icons here, or if someone finds a nicer default unicode string, let me know.

Thanks to a user's submission, I am folding in two new pixiv url classes to recognise their new format, which had broken the default pixiv downloaders.

'Separating' large subscriptions is now easier--if a sub has more than 100 queries, it will now be quickly and easily halved, so you don't have to pick through a gigantic and laggy list of 300 query checkboxes. A note on the subs edit panel now recommends breaking up subs with >200k items.

The 'quality info' button on advanced users' edit subscription panels now let you copy the data to your clipboard in CSV format. The way this works is still pretty hacky, but if you want to analyze this data more closely, let me know how it goes!

There was a bug in the new non-interrupting tag repository processing system that meant in certain situations (usually on slower machines), some tags may not have been processed. This bug is fixed, and I have set the last eight weeks' of updates to be reprocessed to fill in gaps. Since the vast majority of the data is already there, this reprocessing should be very fast.

The various '110/250' import status columns in the downloader UI now sorts better, by num_total first, then num_done. It previously did by percentage done, but it wasn't really useful.

The new 'tag migration' panel is now available straight off the services menu.

full list

- tag display updates:
- the old tag censorship system is now replaced by a broader tag display manager that will deal with tag storage and presentation settings. this is the first step, and some of it is not yet completely functional or as efficient as intended
- management for the new tag display manager can be found under services->manage tag display. you can set per-service and all-services filters for 'multiple' display like the 'tag selection' boxes and for 'single' file views, like thumbnails and the media viewer
- existing censorship rules will be added to the new manager and will apply to 'selection list' and 'single media' display rules
- censorship/display rules no longer apply to underlying storage views, which are unfilterable for now, so the manage tags dialog and the autocomplete lookup will now show all tags
- page and media viewer taglists now have new right-click menu options for hiding--they will provide hide options for the specific tag clicked, and its namespace more broadly, and will apply immediately to that kind of taglist (previously, this was just the tag, and launched the tag censorship edit panel)
- all 'tag manager' objects behind every media object in the client now pre-compute cache layers for different tag presentation contexts. operations such as sibling collapse are now only done on file load or new siblings

Anonymous Board owner 10/03/2019 (Thu) 00:17:02 Id: 48e809 [Preview] No.151 del
- for now, initial media load will take slightly longer, but various tag display updates and autocomplete tag fetches on media will be faster
- changes to siblings and the new tag display rules will now trigger a reliable (although, for siblings, delayed by a few seconds) and complete tag list and thumbnail and media viewer refresh
- changes to tag presentation will now correctly update collection thumbnails
- some complicated sibling display and counts are now more precise
- cleaned up some tag/siblings/thumb refresh notification code
- cleaned up all tag manager access code
- cleaned up a variety of related tag fetching, counting, and display code
- .
- the rest:
- added 'system:modified date'. it works just like system:time imported
- files with duration but no audio will now have a ' ▶ ' label in the top-left of their thumbnails, like the 'has audio' one. you can edit this label under options->audio and duration. I don't really like how this looks, so maybe we'll go to icons. let me know what you think
- fixed an iteration timing bug in the new asynchronous repository mappings processing that meant large lists of mappings within an update object may be occasionally truncated, leaving some mappings unprocessed. this would more affect users on slower machines running 'process now'
- any tag repository content updates issued in the last eight weeks will be scheduled for reprocessing to cover the above issue and fill in gaps for most user situations. since the vast majority of the data was added as intended, they should catch up very fast
- added two pixiv url classes for their new url format
- the edit subs panel now recommends users break up subs with >200k urls
- the 'separate' subscriptions button now has a new 'break in half' option. subscriptions that have more than 100 queries will auto-choose this to separate
- the 'quality info' button for advanced users' edit subscription panels now gives the option to additionally copy the info to your clipboard in CSV format
- the lists on the gallery downloader, thread watcher, subscription, and subscriptions panels now sort their progress column by ( y, x ) (given a total status of x/y). previously, this was preceded by a percent-done sort
- the hydrus network engine now recognises 429 bandwidth responses
- on 429 or 509 bandwidth responses, network jobs will now go through the regular reconnection delay loop and try again later (previously, they just failed)
- added 'tag migration' to 'services' menu for quick-launch
- tag migration's 'go' action now skips the second confirmation if you are in advanced mode
- expanded the 'reset' review services button (only visible to advanced users) to allow 'softer' resets that simply reprocess definition/content without deletion (a 'filling in the gaps' command)
- fixed the 'process now' review services button disable check, which was being overzealous
- cleaned up some of the new repo 'caught up' checking code
- improved stability of review services 'refresh account' call
- the client api /manage_pages/get_page_info call now returns a list of hash_ids beside the list of hashes, in simple or not simple mode
- fixed a bug where tag import options that still had a secret deleted service reference were causing tag-parse errors on import jobs
- fixed some other places that were not handling service disappearance neatly
- added a note to the install/backup help to mention not to use continuous cloud-sync backups on your live db directory
- misc unit test refactoring

next week

Tag filters are used in plenty of places now, so I'll set up a 'favourites' system into the edit panel so you can save and load common presets. If that goes well, I'll get started on 'storage/processing' tag filters and some related db tag cleanup/recycling maintenance.

Anonymous 10/05/2019 (Sat) 15:33:43 Id: 0be909 [Preview] No.154 del
Thanks for your continuous work.
I switched to GNU/Linux recently and am trying to get it running.
You're awesome man!

Anonymous Board owner 10/06/2019 (Sun) 22:17:47 Id: c05b20 [Preview] No.162 del
Thanks, I am really glad you like it.

The Linux version is pretty unstable for some flavours of Linux/Window Manager, but with luck this will clear up significantly in six weeks, when we move from wx to Qt. Please let me know how that change works for you.

Anonymous 10/07/2019 (Mon) 00:59:43 Id: d29c24 [Preview] No.169 del
Two odd files I found when importing my old files to a new Hydrus db.

This file has a mkv header not a webm header. I'm not sure if Hydrus is actually wrong considering browsers treat any mkv with the right codecs as essentially a webm.

ParseFFMPEGDuration accidently grabs the line from metadata title instead of the actual Duration line in FFMPEG causing an exception.

On a side note, it would be super handy to be able to add the title metadata from webms as a tag during import.

Anonymous 10/07/2019 (Mon) 05:43:46 Id: c312d6 [Preview] No.172 del
Have a bit of shit to do so likely wont upgrade this week, however, this is an error I get every time I change the ratings of multiple images when its I want to say more then 20

Error binding parameter 0 - probably unsupported type.
Traceback (most recent call last):
File "include\HydrusDB.py", line 581, in _ProcessJob
result = self._Write( action, *args, **kwargs )
File "include\ClientDB.py", line 13929, in _Write
elif action == 'save_options': self._SaveOptions( *args, **kwargs )
File "include\ClientDB.py", line 11009, in _SaveOptions
self._c.execute( 'UPDATE options SET options = ?;', ( options, ) )
sqlite3.InterfaceError: Error binding parameter 0 - probably unsupported type.

Anonymous 10/07/2019 (Mon) 13:33:01 Id: ddf1bb [Preview] No.173 del
Hydrus dev, I only have 4gb ram, and my computer is pretty shit. Until now when using Hydrus I was able to use it, but it would be frozen for maybe 80% of the time I was using it. If I wanted to import a batch of images, I would realistically have to sit watching the client for 20 minutes until it eventually unfroze, then if the import is large enough it'd freeze for another 20 minutes, then if I wanted to sit there individually tagging images it would largely be hell. I didn't really consider myself as having access to my images in Hydrus, since unless I remember the specific tags it has, I just lose too much time waiting for the client to unfreeze.

But a few days ago I updated from version 268 to 270, and I don't know what happened but it's literally unusable for me now. I would say it's now frozen over 99% of the time, wih no practical uptime. For example, I have subscriptions, but I am unable to click them, because it takes many minutes for the client to highlight it blue. Then when I do see it turn blue, I click, but there is a possibility it isn't registered, and instead the cursor turns to the loading circle, and another click turns the window the "not responding" shade of white. If it does click, it takes many minutes (20-30) to open in a new tab. It will probably only partially load; the actual images won't load until 30 minutes more. Tagging is impossible: I need to type the part of the tag I remember into a text document, then paste it into Hydrus, but of course being able to input text in the tagging window, clicking an image, anything... 20-30 minutes to do any action.

I haven't successfully done anything meaningful in Hydrus since updating a few days ago. I don't know how to revert client versions but if possible it would be the only option for me to be able to use Hydrus. Hydrus is the only program I keep open (not even a torrent client otherwise, anything). I'm on Windows 7 with 72% ram usage, 14% CPU usage, and Hydrus uses 2m- 2.3m ram it seems. I don't think the ram usage changed between versions. Is there any chance you can diagnose why this happened and revert the new strain on my hardware maybe.

Anonymous 10/07/2019 (Mon) 22:01:22 Id: d12b35 [Preview] No.174 del
I had the same situation you have.
My remedy is turn off the feature that do maintainance when you're using.
This feature was introduced two or three week ago so that's the culprit I think.
btw my spec is 10gb memory and A10APU so my pc isn't a complete toaster.

Anonymous 10/07/2019 (Mon) 23:38:13 Id: ddf1bb [Preview] No.175 del
Dude, you instantly fixed it. Thank you so much. That was fucked, honestly.

It was easier to find the setting than I expected since it was under File > maintenance and processing > Run file maintenance during normal time.

Maybe I'm the only one using Hydrus from a toaster, but if I hadn't complained here I would've just hoped for it to revert back when installing the latest version. But my latest subscriptions (just the notifications) would be lost to a force restart of the client since I've shutdown before checking them before, and they were lost on client restart. Let alone that if this started two weeks ago and wasnt reverted, it wouldn't be reverted the following release either.

If this setting is preferable to have on for computer that can handle it then sure, but for me it was the difference between slugging my way through Hydrus (before), or a completely unusable Hydrus (after).

I think that obviously I wouldn't really put the burden of Hydrus with this performance on anyone else, but I figured if I ever did have a computer that could do 1080p playback, flawless video playback in browser, had more than 4gb ram in 2019, basic shit, it would be nice to have Hydrus and it wouldn't be a compromise. I don't even think my computer is that unrealistic for someone to be using. But I do have to close my browser if I'm not actively using it, because no ram, alongside Hydrus, it's to much.

Anonymous Board owner 10/07/2019 (Mon) 23:57:37 Id: c05b20 [Preview] No.176 del
Thank you for this report and the files.

I have fixed the broken one for 371. It was exactly as you saw.

For the second one, as I understand it, the interesting thing about webms is that they all are mkvs, just with certain video/audio codecs. They are not their own file format. If a mkv has vp8 or vp9, and optionally vorbis or opus, it is a webm (which also still being an mkv). I do not know of any finer test, but I would be interested to know if there were.

Parsing title metadata is an interesting thought. I have a long-term plan to harvest EXIF data from jpegs--perhaps I could plug the similar data for videos when I get around to setting up this data store and parsing options and so on.

Anonymous Board owner 10/08/2019 (Tue) 00:02:15 Id: c05b20 [Preview] No.177 del
Thank you for this report. I think I see how this could be happening. I will check it this week.

Anonymous Board owner 10/08/2019 (Tue) 00:15:54 Id: c05b20 [Preview] No.178 del
Thank you for this report and information. I am sorry you have had a hard time. Some of hydrus is built with the assumption of being able to eat a decent whack of ram/cpu/hdd at times, but I do not want it to be even 80% unusable for anyone. My ideal is that the client adapts its background maintenance based on available resources.

The new file maintenance system is all its own unified system. I will write in some sort of CPU throttle to ensure it notices a slower system and scales back its work significantly.

When it is convenient and if you are comfortable sharing, can either of you try two different 'profile' runs? Hit help->debug->profile modes->db profile mode/pubsub profile mode and then do some regular work. Do not do both modes at the same time, and be warned that pubsub profiles can be pretty spammy, so the client may be very juddery for you and you'll have to wait or force-kill. Turn the modes off after they gather a bunch of info, or restart the client to reset. They will dump giant profile logs to your install_dir/db directory with timing information on db/UI calls. They may have sensitive info in them, so please email them to me or DM on discord rather than making a public pastebin. Or, if you like, you can scroll through and curate the slower job info and just share that publicly.

If I can find what is eating up your CPU, I'll know what to target.

By the way, the biggest CPU hit on a client is the size of the UI session. If your total thumbnails+download urls in your session is above 10k, you should free up your client significantly by closing pages and clearing out completed download queues. My normal number I recommend is 50k, but lets see how 10k works for your slower machines. I have an ancilliary job to measure the 'weight' of the current session so I have something to get better debug info and hang user warnings off of.

Anonymous 10/08/2019 (Tue) 01:02:24 Id: d29c24 [Preview] No.180 del
you are correct webms are mkvs.

looking at webm files with a hex editor, normal webms will have webm in the header, but the one i shared has matroska instead.

the only reason i noticed was because instead of upgrading from v304 to v370, i dumped tags and reimported on a new database and that file type was the only one different and reading through the changelog it looks like you changed how webms were detected. thought you may be interested.

btw thank you for the quick response. but more importantly thank you for the well written help files.

Anonymous 10/08/2019 (Tue) 01:51:43 Id: ddf1bb [Preview] No.181 del
I will try to do what you ask of. I don't know what conceivable sensitive info it could fetch (besides my pixiv logic details, arbitrary tags), but I'll email you it privately anyway.

When you say do some "regular work", my use of Hydrus is pretty basic: I only search for images I randomly remember of sometimes, and import rips of Twitter usernames under the generated filename of the chrome addon that preserves a lot of information. I paste pixiv ids into a "gallery" tab to rip them. I add pixiv subscriptions. I add tag parents/ siblings. I arbitrarily tag. Sometimes I rip a porn video and preserve its case-insensitive link via a tag if possible.

Also this is random but a few months ago I made a thread on your 8ch board saying I invented the proof to the big bang. I believed I really did at the time, and the person I showed it to was my doctor who dishonestly tried to downplay it, say I only have a normal path in life... it was fucked. I was actually visiting him for an entire year prior because I needed a doctor's evaluation to be eligible for autismbux, but I kept making progress on my mindset (not realizing at the time that I was inventing the proof to the big bang), and I would show him these papers I would sometimes spend hours sketching, which was about the linear progression of geometry and similar perverted shit. He seemed excited the entire time until the latest visit when I said I want to show it to someone for the sake of my future. He used abuse tactics on me... I guess it's not important to elaborate...

Well I've made more progress since then... without showing him... to be honest I'm unironically surprised the fbi hasn't gone forward and killed me + claimed my work as their own, because I know they fucking have it- the doctor scanned everything to his work computer, it's probably as accessible as my dental records.

Anyway this is fucked. I was taking notes to make a speech, and I wasn't going to shout out Hydrus honestly and I still won't probably. Image tagging today in general is pretty much entirely social posturing and is arbitrarily limited depending on which hugbox is in charge of which image tagging platform, so I was going to borderline make my own.

Life sucks man. I'm not stupid... even if I fail to protect myself, do well for myself, for virtually all my life... I realize too late anything was even abuse, always...

The society humans create for themselves... the level of discourse humans offer or entertain the thought of in any context, even formal contexts... there is no accident anywhere, so will I really be enlightening anyone by making an invented the proof to the big bang speech... No. I expect to be censored, interrupted, and killed. If any facet of the proof to the big bang were articulated and proposed as an argument for anything, were it not titled "proof to the big bang", humans would reject it, use abuse tactics to undermine discourse even reaching it, rather than even attempt to disprove it... I've experienced this my whole life...

So I didn't really have anything to say except that you realistically saw my thread, so fuck you and fuck the illiterate social posturing discourse-undermining yes men you surround yourself with. When you're too emotional toddler illiterate to even propose any logic whatsoever to defend your "I have no argument and I must social posture" excuse of a philosophy, then you just have a small penis. Sorry about your small penis. I'm sure the social validation you receive compensates for it heavily.

Anonymous 10/08/2019 (Tue) 06:58:09 Id: d12b35 [Preview] No.182 del
Hi dev.
>When it is convenient and if you are >comfortable sharing, can either of you try >two different 'profile' runs?
Of course!
I'm pleasure to do it if it will help you.
As fast as possible I'll send you a profile.
Thanks always dev!I'm a big fan of you!

>By the way, the biggest CPU hit on a client >is the size of the UI session. If your total >thumbnails+download urls in your session is >above 10k, you should free up your client >significantly by closing pages and clearing >out completed download queues. My normal >number I recommend is 50k, but lets see how >10k works for your slower machines. I have >an ancilliary job to measure the 'weight' of >the current session so I have something to >get >better debug info and hang user >warnings off of.
Well in my case this might be the cause.
I was opening pages containing 200k files while importing gigantic huge number of files which is 1.4 millions.

What I meant by the btw line was the fault is not low spec pc but there is something else.
So I think everything's gonna alright.

Anonymous 10/08/2019 (Tue) 07:44:49 Id: c312d6 [Preview] No.183 del
Ran into an issue, so im on the edge of where the program is stable enough to do shit but locks up for a bit every minute or so, right now it's not too bad so I don't bother making it run better

so, I found an image that was an edit, I wanted to keep the image but im not keeping a retardely large edited file, so I went to see if I can find a speculative match, but there was no file relationships or regenerate options under manage... I dont use the other pop out menu options enough to know if there is a difference.

Thinking that this may have been an issue with searching a file tag, I moved the file to its own window to do the search and it was there, and it was there in the window the image came from... I think I ran into this issue before but I was never paying close enough attention to notice

Anonymous 10/08/2019 (Tue) 11:56:05 Id: 9d26bb [Preview] No.184 del
Log sent!
Tell me if something is missing.
Sorry for late.
I was away from my PC when posting previous one.

Anonymous 10/09/2019 (Wed) 06:49:35 Id: c312d6 [Preview] No.191 del
Ok going though more fucking gifs and webm's jesus I hate doing this but it needs to get done.

so, I know that hotkeys are at least on the table/in the works, this will for me make this complaint moot, but would it be possible to increase the size of the play bar? I want to scrub though things to get to the point, but due to it being 20 pixels high, I end up missing it quite a bit after a few thousand files have been parsed. would it be possible to make that section user definable big? if not, an option to have it be 40-60 pixels big? at least for me this would make misclicking a bit harder.

Anonymous Board owner 10/10/2019 (Thu) 00:14:53 Id: c05b20 [Preview] No.201 del
Thanks m8, I will read through those emails and respond there properly on Saturday, when I am back working at my dev machine. I would like to put some optimisation into 372, so I appreciate having your data.

Anonymous Board owner 10/10/2019 (Thu) 00:21:54 Id: c05b20 [Preview] No.202 del
Thank you for this report. Is it possible that when you first tried it, your database was locked? When the thumbnail right-click generates its menu, it needs to do a real fast db hit to fetch similar files metadata. If the db is locked, it skips that to ensure the menu gets made fast and I think should replace the similar files stuff with a label like 'db is locked, could not fetch info'. I have a longer-term plan to load duplicate file info with the initial media load and no longer need this menu db hit, but until then, that's the compromise. Since your client has some lag, could it be in part db-based?

I will have a look to make sure the menu properly shows a nice message when the db is locked, and to make sure the 'search for similar' still appears even if so. It is possible it got hidden somehow when I recently reorganised things.

If you are confident the db was not locked, or it seems to be hidden anyway, is there any chance you could take a screenshot or just list what your manage menu does look like? Maybe one menu entry is stomping on the file relationships menu out of order or something.

Anonymous Board owner 10/10/2019 (Thu) 00:24:08 Id: c05b20 [Preview] No.203 del
Sure, thank you for this suggestion. That is a good idea.

Relatedly, when we move to Qt, which should be done about five weeks from now, it looks like the client will be able to offer the option of a full embedded MPV window instead of my native renderer. This will handle video a hell of a lot better. I'll be keen to see if I can hook hydrus shortcuts into this embedded window as well.

Top | Return | Catalog | Post a reply