/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.

Posting mode: Reply

Check to confirm you're not a robot
Drawing x size canvas

Remember to follow the rules

Max file size: 350.00 MB

Max files: 5

Max message length: 4096

Manage Board | Moderate Thread

Return | Catalog | Bottom

Expand All Images

Version 369 Anonymous Board owner 09/25/2019 (Wed) 22:08:17 Id: 6988a8 [Preview] No. 115
https://youtube.com/watch?v=ACAvBBMb6BM [Embed]
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v369/Hydrus.Network.369.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v369/Hydrus.Network.369.-.Windows.-.Installer.exe
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v369/Hydrus.Network.369.-.OS.X.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v369/Hydrus.Network.369.-.Linux.-.Executable.tar.gz
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v369.tar.gz

I had a good week. The file maintenance system works better, and files now know their 'modified' dates.

file maintenance

The new file maintenance system now works smarter and more productively. Previously, it did rare bursts of hard work in idle time, and now it does small amounts of work all the time. You do not have to do anything--it will now keep up with its behind-the-scenes queues more quickly and quietly, hopefully without giving you any hassle. The only significant difference is file maintenance no longer occurs on shutdown. If you are not interested in any details, you can skip the following completely.

Its settings under options->maintenance and processing have changed a little--rather than the old '200 files per day' default, it now has faster '1 work unit per 2 seconds/20 seconds' for idle/normal time. These values should not tax your system or interfere with other programs or interrupt your browsing. If you have a faster or slower computer, feel free to change them. If the work does make your browsing juddery, please do turn off work in normal time, which you can do in the options or now more quickly at database->maintain->file maintenance. The 'work unit' is part of a new weighting system that lets fast jobs--like checking for file presence on the hard drive--count for less work in the throttle and process faster than bigger jobs like regenerating thumbnails

When I added the 'go through all video files and scan for has_audio property' job some weeks ago, it added 100,000+ jobs for many users. The 200/day default was not cutting it, so this new 'lots of light work all the time' mode should get these big jobs done in reasonable time without going nuts. It also updates thumbnails in real-time, so there is the smallest chance you'll see one of the new 'has audio' icons fade into one of your thumbnails right as you look at it! The new processing also helps for another large retroactive job being added this week--figuring out file modified dates:

Anonymous Board owner 09/25/2019 (Wed) 22:08:55 Id: 6988a8 [Preview] No.116 del
file modified dates

Many users have asked if the client can let you see and search by files' filesystem modified dates. This week does the first half of this. Files imported to hydrus will now know their modified date, display it in the information lines on a file's right-click menu, and let you sort thumbnails by it.

Files downloaded in hydrus will have the same modified date as their import time (this information is not preserved on websites), but any files you import from your hard drive with dates from 2003 or something should get that date.

Your existing files will not know their modified dates immediately, but a large maintenance job has been added to the file maintenance system to retroactively figure them out. This work will catch up in a few days to a few weeks, depending on the size of your collection. Modified dates are preserved through a hydrus import, so it should all work out correct. The only caveat to this is hydrus did not preserve modified date as of, I think, about four years ago. If you are a long time user, any files that were imported before then will likely have modified dates equal to their hydrus import dates.

I hope to add system:modified date next week, to work exactly like system:time imported.

the rest

You can now sort your tags lexicographically while ignoring namespace. This makes 'series:metroid' sort before 'character:samus aran' because 'metroid' < 'samus aran'.

I split up the repository download/process code components a bit this week. It doesn't make a huge difference, but now there is a 'download now' button on the review services panel that lets you get a metadata sync immediately rather than having to wait for the background daemon to do it. The split should improve some locking/delaying issues some users had, particularly with the initial sync.

The Linux build has a much newer version of OpenCV. Let me know if your images load quicker/more reliably!

The 'export files' panel has a saner button for selecting and editing the tag services that export to neighbouring .txt files. It remembers the last selection of services better than before.

Anonymous Board owner 09/25/2019 (Wed) 22:09:25 Id: 6988a8 [Preview] No.117 del
full list

- file maintenance:
- the file maintenance manager now works continuously in the background, optionally in both idle and active time, with two different throttles, which are now always active
- as usual, the default throttles are low-load (1 heavy job every 2 (idle) or 20 (normal) seconds), so as not to interfere with your browsing or other programs--feel free to speed them up as you wish
- the options for file maintenance under 'maintenance and processing' are updated, and quick-pause actions are now available under database->maintain->file maintenance
- the file maintenance manager no longer works on shutdown
- the file maintenance manager will now only make a popup if it is started by the user--it otherwise now works silently in the background
- the file maintenance manager now weights its jobs, so quick jobs will run faster and heavy jobs will space out more. exact weights, if you are interested, are now under the 'see description' button on the maintenance panel
- file maintenance jobs now report to the debug file report mode
- improved some misc file maintenance code, particularly with how the panel talks to the manager
- media with new metadata will now refresh their thumbnails (for now, this means updating the has_audio icon)
- .
- modified timestamps:
- the client now records file modified timestamps of all file imports!
- on update, the retroactive population of this data for all existing local files will be scheduled on the file maintenance system, which has a new job type for this
- the modified time now appears on a file's information lines that present on a right-click
- the modified time can be sorted with the new 'file: modified time' sort
- .
- the rest:
- added lexicographic sort by subtag (ignoring namespace) to the normal taglist sort selection
- reworded the sort by lexicogrphic (grouped by namespace), to be (group unnamespaced)
- the export files panel now has an explicit button to change the neighbouring .txt file tag services
- on duplicate merge action options panel, 'sync archive' is no longer disabled for advanced users' 'alternates' duplicate action
- split the download and process sync components of repositories a little
- added a 'download now' button to repositories' review services panels, to hurry up metadata/update download when possible
- the 'process now' button's enable/disable states should now be more reliable
- the 'refresh account' button now disables when a repository is paused
- improved stability of 'process now' button post-job updating
- added a subscription option to the downloading option panel to change how many file-fails in a run will cause a sync to stop working early
- re-added the truncated image loading mode to the debug->data actions menu. this has hung indefinitely with some bad files, so it not on by default
- fixed an issue with copying an external local booru url with a upnp port
- fixed an unrecoverable ui hang when a modal popup wants to self-terminate while a child yesno is open
- if on a hydrus request the session key is invalid (due, for instance, to a recent serverside session clearing :^)), the session key cookie will now correctly be cleared clientside so a new one can be generated automatically on the next request
- hydrus services can now take the access key as their credential using the 'Hydrus-Key' header. more options will come here, basically the same as the client api
- network jobs waiting on a login process now continue faster once the login is complete (5s sleep cycle down to 1s)
- perhaps fixed some linux problems with tag migration panel, perhaps not
- caught and silenced a rare unimportant services shutdown error
- updated to opencv 4.1.1 on the linux build
- updated windows ffmpeg to 4.2.1

next week

Finishing up the modified date system search predicate, doing a bit of client api work, and getting into the meat of the new unified tag display system, which I briefly started this week.

Anonymous 09/26/2019 (Thu) 02:16:47 Id: b1e302 [Preview] No.118 del
I would always recommend in program conversion as you could then send both over to duplicate queue that is just filled with conversions to easily see if the job was done right. as for conversion persets, I never understood why the front ends who know the people using it are to dumb for scripting/command line use, dont have some good, well optimized settings.

Anonymous 09/27/2019 (Fri) 00:11:57 Id: 0b80e8 [Preview] No.119 del
>I never understood why the front ends who know the people using it are to dumb for scripting/command line use
I don't understand why anything uses command line anymore. Or at least that's all they have. I'm shit at command lines, but there's not much excuse to not have even a basic GUI in the current year. Unless you're a loonix user.

Anonymous 09/27/2019 (Fri) 03:30:46 Id: b1e302 [Preview] No.120 del
Ok, I have 2 issues right now that i'm not really sure of.

so I had 800mb of space to work with a few days ago on my hdd, I look at it today, and it had 50, I cleared 400mb quick and did some shit and that number is down to 200mb...
apparently over the last 2 days only 300mb of files are found

Not sure what the hell is going on there,

now the more pressing one, I did the repository sync after having it paused for so long because it just took forever to do. my db is now 11-12 gb larger than it was before. its still not done syncing.

the size of the db isn't to big of a problem for me, its a bit large, but it's not unmanageable. but taking up less space is better.

Anonymous 09/27/2019 (Fri) 03:34:09 Id: b1e302 [Preview] No.121 del
the way its explained to me, and it's a shit explanation, is there is a lot of power you have in command line, and making a ui would need to be updated every single time what you do gets expanded.

the people doing the stuff to make the program work don't need the gui to test, the command line is overall faster as less overhead, and if you know what you want its all right there.

the big problem I have with this is now im forced to read a 20+ page manual to have a hope of making an attempt, and the options I may find useful are now things I need to know exist rather then i stumbled on it.

Anonymous 09/27/2019 (Fri) 08:03:54 Id: b1e302 [Preview] No.122 del
just compared folders from my backup to the new one

Anonymous 09/27/2019 (Fri) 17:03:43 Id: b1e302 [Preview] No.123 del
just had 2 system hangs, both of these occurred while doing repository syncs and forced a hard reset each time... not 100% sure if its hydrus or not, as one happened after 81 days of uptime, so I am not 100% sure what the cause was. just know that everything hard froze with no blue screen, while I heavily doubt hydrus, its one of the common links between the two crashes

Anonymous 09/27/2019 (Fri) 19:35:53 Id: 3237f0 [Preview] No.124 del

You're lucky if they wrote you 20 pages of documentation. That's the mark of an extremely considerate developer. It's rare to come across small projects with anything even close to a being called a manual.

Also GUI frameworks are a massive pain in the ass to work with. They're unintuitive to learn, difficult to debug, and worse, they're extremely tedious. If someone is developing a project for you, for free, you should cut them a little slack for focusing on the parts of programming they actually find satisfying.

Anonymous 09/28/2019 (Sat) 06:17:52 Id: b1e302 [Preview] No.126 del
almost any project that is worth using has documentation of some sort, as for gui and dupshit level presets that are good, I honestly say suck it up, if the program is command line, and will never have a gui, the least people can do is make a batch file that will have the 'presets' or examples loaded.

Its little things that can guide someone to do what they want that would be nice. hell, due to manga being such a bitch to deal with I had to make a batch file for creating folders, fun thing to do with 0 help or knowing what i'm even looking for, but getting the help, even if the help fails, I am able to look into it and figure out what went wrong and may be able to fix it. and because this is so useful here

@echo off
for %%a in (*.*) do (
md "%%~na" 2>nul
cd ..
del %%~na

save that as a batch file, .bat and put that in a folder, from there put files in the folder, it will make folders based on files.

this one

for f in *\_*; do mv "$f" "${f//_/ }"; done

was suppose to remove _ from folders but it failed I believe.

the main thing with needing to sift through documentation is that if you don't know exactly what something is called, or what the program you are trying to use calls it, you are kind of screwed. at that point said program is functionally useless unless you find a community that is happy to help, which they typically are not and go on to call you retarded for not being able to do 'simple' things.

Anonymous 09/28/2019 (Sat) 09:29:54 Id: b1e302 [Preview] No.127 del
>>122 Finished sync,

so... 63gb
any suggestions... really I would like this to be as small as possible but jesus christ 63gb... something seems really wrong

Anonymous 09/28/2019 (Sat) 11:47:32 Id: b1e302 [Preview] No.128 del
got 2 errors after a gpu fuck up due to scrubbing video in mpc-hc

C++ assertion "(int)m_pages.Count() == (int)::SendMessageW((((HWND)GetHWND())), (0x1300 + 4), 0, 0L)" failed at ..\..\src\msw\notebook.cpp(343) in wxNotebook::GetPageCount():
File "include\ClientGUICommon.py", line 319, in RecalcSizes
sizer_item.SetDimension( pos, size )

C++ assertion "(int)m_pages.Count() == (int)::SendMessageW((((HWND)GetHWND())), (0x1300 + 4), 0, 0L)" failed at ..\..\src\msw\notebook.cpp(343) in wxNotebook::GetPageCount():
File "include\ClientGUICommon.py", line 319, in RecalcSizes
sizer_item.SetDimension( pos, size )

Anonymous 09/28/2019 (Sat) 18:48:06 Id: c44e81 [Preview] No.131 del
(26.80 KB 879x550 firefox.png)
>Files downloaded in hydrus will have the same modified date as their import time (this information is not preserved on websites)

Maybe I'm wrong but I think websites do serve the files original modified date. In Firefox if you right click any image, View Image Info->General tab you can see the original time of the file stored in the server.


Would it be possible to have Hydrus save this date as the modified date when importing from a booru or other websites?

Anonymous Board owner 09/28/2019 (Sat) 22:13:18 Id: 6988a8 [Preview] No.132 del
My best guess is the guys who are good at writing codecs are generally thinking more about bytes than user interface. They put out a very specific tool, and it is someone else's job to make a pretty wrapper. I guess for whatever reason, doing that second step isn't popular. I'm happy for it to be me for hydrus purposes.

I agree that it is easier to check a checkbox with a nice long label/tooltip than have to dive into some man page, and a decent UI gives you options to save and load common profiles much easier.

Anonymous Board owner 09/28/2019 (Sat) 22:14:54 Id: 6988a8 [Preview] No.133 del
I think we talked elsewhere (and eliminated that dupe repo), but final db size for a syncing client should be about 18GB for client.mappings.db and 5GB for client.master.db.

Anonymous Board owner 09/28/2019 (Sat) 22:18:48 Id: 6988a8 [Preview] No.134 del
That sounds pretty bad! I write hydrus in python, so I normally cannot cause a blue screen even if I wanted to, but the only exception I have seen to that was when some graphics drivers were causing it, and Hydrus's calls to 'OpenCL', which is GPU acceleration for image loading, were triggering that bug. it was Nvidia drivers, and the fix was to update to the new version.

Have you had any GPU issues recently? If you go into your driver options and disable OpenCL for client.exe, does that help?

It may not have been repo processing, but some file maintenance like video reparse or thumbnail regen happening in the background.

I recommend pausing repo processing under services->pause and file maintenance under database->maintain->file maintenance if you want to test more.

Anonymous Board owner 09/28/2019 (Sat) 22:24:01 Id: 6988a8 [Preview] No.135 del
Thank you for this report. I am not sure what is going on here. Perhaps the OS-level UI stuff got screwed up and it ran out of new window/message ids or something to give? That's a pretty low-level error, deeper than I normally deal with. If the GPU got fucked, my best guess is client/system restart for a solution. If you tried that, did the error come back?

Anonymous Board owner 09/28/2019 (Sat) 22:33:10 Id: 6988a8 [Preview] No.136 del
Thank you for pointing this out! I hadn't realised the boorus provided this for files themselves. Even if a date is slightly off due to mirror-caching or something, that is still better than what we currently have.

I have been talking with some other users about perhaps using the parsed source time as well. I'll try to put some time into this this week.

Anonymous 10/01/2019 (Tue) 09:44:49 Id: a48b9c [Preview] No.141 del
If you put in limit:2000 and sort by media views, while having default order in options both set to random, it returns images in the same general order rather than randomizing the sort of images with the same number of media views, so you keep seeing the same few images at the top.

Anonymous 10/02/2019 (Wed) 12:26:46 Id: 605a00 [Preview] No.146 del
I'm on an old version Hydrus and was testing out this version, specifically: Hydrus.Network.369.-.Windows.-.Extract.only.zip

Got an error where I can't run FFMPEG when I set the database to anywhere else.

About box -> FFMPEG: 4.2.1
No problems.

client.exe --db_dir="C:\test_db"
About box -> FFMPEG: unable to execute ffmpeg

[WinError 6] The handle is invalid
Traceback (most recent call last):
File "include\HydrusVideoHandling.py", line 58, in GetFFMPEGVersion
process = subprocess.Popen( cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, **sbp_kwargs )
File "subprocess.py", line 667, in init
File "subprocess.py", line 905, in _get_handles
File "subprocess.py", line 955, in _make_inheritable
OSError: [WinError 6] The handle is invalid

Anonymous 10/02/2019 (Wed) 20:23:25 Id: b1e302 [Preview] No.147 del
For the most part, the issue I had only happens after I have 50 days of uptime so I lump it into bullshit just building up till windows cant take it, and its not a blue screen, everything just freezes up and stops responding at all when this happens. the only common link between the two was hydrus being open and doing the repository so I figured mentioning it was better then not.

as for drivers... my nvidia ones are months/year or so old, with nvidia drivers i'm on a 'if its not broke don't fix it' due to how horribly that crap handled my monitor, kicking to a black screen 5-6 times a day, now don't get me wrong, scrubbing a video will throw system error more than most things I do amd or nvidia, but 6 times a day was just not acceptable so the first driver that solved the issue I go it and till something breaks in a way I can confirm its gpu related, i'll stay on it.

yea saw ya in hydrus discord so I figured it would be easier to get some help there,

image is my final numbers, those seem to be in line with what you think they should be or is there still some area to cut fat?

scrubbing video on my computer will sometimes cause the monitor to stop working, a monitor restart will fix the issue, but hydrus tossed these errors along with it so though I should report them.

Anonymous Board owner 10/06/2019 (Sun) 22:03:55 Id: 49f621 [Preview] No.159 del
Thank you for this report. I will check this out. I wonder if I am already inserting a secondary sort of media_viewtime or similar that is not causing conflicts and falling back to the 'random' secondary you have set.

Anonymous Board owner 10/06/2019 (Sun) 22:12:50 Id: 49f621 [Preview] No.160 del
Thank you for this report. This is unusual. Can you try turning on help->debug->report modes->subprocess report mode and trying help->about again? This should dump a whole bunch of environment info. If you compare this between installs, are there any obvious differences?

Normally, it should not matter where the db folder is, since it is targeting install_dir/bin/ffmpeg, and I haven't seen problems here for Windows before, but obviously something odd is going on here.

Is there any chance that your client/OS has lots of threads/processes open? This specific WinError 6 can be due to too many processes being open and Windows not being able to open a new one. Could it be, say, that hydrus had reason to created hundreds of new video thumbnails recently? Or do a lot of background file maintenance?

I am updating my 371 FFMPEG about window error reporting to name the FFMPEG path, just so we can check this is pointing to the right location.

Anonymous Board owner 10/06/2019 (Sun) 22:16:25 Id: 49f621 [Preview] No.161 del
Yeah, that db size looks great. A vacuum may shrink client.master.db a couple gigs, but that looks normal for a heavy db with millions of files.

We'll be on Qt in six weeks, so with luck the whole UI will be drawing pixels to screen in a more reliable way. Please let me know if your freezing and other weird behaviour improves/changes when that happens.

Anonymous 10/07/2019 (Mon) 00:44:31 Id: 605a00 [Preview] No.168 del
I'm running Win7x64 and testing with empty dbs so I don't think there's any reason for Hydrus to be making any calls.

The only difference in the env vars was that the separate db instance had an extra line:

The odd thing is that I ended up getting Hydrus to run from source with Python 3.7.3 and the problem doesn't occur at all. Other than extra pipenv env vars, there weren't any obvious difference in env vars between my source instances and the compiled instances.

I'll test 371 when it drops and see what ffmpeg path is generated and go from there... maybe try my own builds with pyinstaller and track down the problem with my machine.

Top | Return | Catalog | Post a reply