/hydrus/ - Hydrus Network

Hydrus Bunker

Posting mode: Reply

Check to confirm you're not a robot
Drawing x size canvas

Remember to follow the rules

Max file size: 350.00 MB

Max files: 5

Max message length: 4096

Manage Board | Moderate Thread

Return | Catalog | Bottom

Expand All Images

Version 371 Anonymous Board owner 10/09/2019 (Wed) 22:27:01 Id: 83b1de [Preview] No. 194
https://youtube.com/watch?v=aZClzeSIVNA [Embed]
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v371/Hydrus.Network.371.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v371/Hydrus.Network.371.-.Windows.-.Installer.exe
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v371/Hydrus.Network.371.-.OS.X.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v371/Hydrus.Network.371.-.Linux.-.Executable.tar.gz
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v371.tar.gz

I had a good week. There are several fixes, and I have added 'favourites' management for tag filters.

favourite tag filters

The Tag Filter is the newish whitelist/blacklist panel that manages which tags will be parsed, blacklisted, migrated, or, more recently, hidden on the new tag display system. It is being used in more and more places, so I have added the ability to save and load certain filters for quicker re-access.

So, now every tag filter edit panel has save/load/delete buttons in the top-right. They all share the same list of favourites. I also put in some example defaults. Let me know if you run into any trouble with it!

the rest

Thumbnails now sort by 'number of tags' and namespace sorts using the 'single' tag display context. So, if you have your thumbs set to hide page tags from the PTR, these page tags will no longer be included in a 'creator-series-volume-chapter-page' sort, for instance.

I fixed several smaller issues--a kind of video that would not import correctly, an unusual 'options' saving error after setting a number of ratings on several thumbnails, seeing duplicate thumbnails on some downloader highlights, gallery downloads that stopped early due to malformed post URLs, download pages sometimes secretly performing a collect and sort when a default collection is set, the media viewer being unable to show certain malformed files, and remote thumbnails not refreshing correctly when their respective file is imported.

Advanced users only: URL Classes now let you set whether requests for URLs matching the class will send a 'referer' http header. Normally, hydrus will send any 'previous' URL as the 'referer' (typically the gallery page when visiting a post page, or the post page when downloading a file), just as if you were visiting it in your web browser, but you can set it to never send any referrer, always use an auto-generated referrer, or use the auto-generated one when no natural referrer is available (which can happen on a user drag-and-drop URL import). The auto-generated referrer is a standard StringConverter operation on the URL being visited itself--if you can figure out a way to infer it from the original URL, or just replace it with a static value, I think some more perculiar downloaders should be possible.

Anonymous Board owner 10/09/2019 (Wed) 22:27:37 Id: 83b1de [Preview] No.195 del
For server administrators: The server no longer tries to slip in a quick vacuum when it is doing a backup. The vacuum command is now a forced, full vacuum command off the same admin menu. Also added are forced 'lock on', 'lock off', and 'test lock' commands that turn the 'server busy' state on and off and disconnect the db, letting you run a third-party backup if you wish.

For server janitors: You can now right-click petition rows to copy their tags. Multiple selections work, and siblings/parents offer left/right copy actions.

full list

- the edit tag filter panel now has load/save/delete buttons at the top to manage tag filter favourites. it starts with a handful of examples
- sorting thumbnails by num tags or namespaces now uses the 'single' tag display context
- the 'sort by media views/viewtime' sorts now do not put the other (viewtime/views) as an implicit secondary sort, so as to better let the user's secondary sort be used
- highlighting a downloader should now not be able to create a page with duplicate thumbnails
- all thumbnail pages now do an addition de-dupe check when they are created with media
- when a gallery page parser now adds new urls to a file import list, urls that are invalid will now be skipped (previously, they threw an error and failed the parse
- fixed a bug where if a default collect is set, pages without a collect (e.g. download pages) would nonetheless initialise with collected+sorted initial media on session load
- file imports now publish the same 'refresh existing media metadata' call as the file maintenance system, meaning if the import already exists in the gui session as an 'unknown thumb', it should now refresh itself correctly
- if the media canvas is called to display an invalid media (due to mime mixup or a faulty parse that slips through), it should now better recognise that and skip/dump out
- fixed import of videos that have 'Duration:' in their title metadata
- improved the error reporting when the old options object fails to save
- removed some old ratings dialog position options storage that was causing errors on certain ratings dialog ok events
- url classes now support options regarding the 'referer' http header they send (their referral url). you can set an optional converter to generate a referral url based on the url class's url and choose to always use the given referrer if available, never use a referrer, use the converter if no referrer is available, or always use the converter
- the network report mode now reports on referral urls used in requests
- the 'quoted' referral url (a unicode workaround) is now only applied if the referral url cannot be encoded to latin-1
- the janitorial petitions processing page now lets you copy tags and left/right tags of pairs with a right-click on selected checkbox rows
- cleaned a little server code
- improved how the server sets and releases its 'currently busy' mode
- the server no longer does <5min vacuums in a backup command
- added a specific 'vacuum' server POST command that forces a full vacuum
- added 'lock_on' and 'lock_off' server POST commands to lock the server and shut down the db, and restart
- the new vacuum, lock_on, lock_off, and a 'is server busy?' check commands are added to the services->admin menu
- added 'pause and disconnect' ability to the database mainloop
- added some unit tests for url classes and the new referral url conversions and server commands
- cleaned some of the thumbnail banner/icon drawing code
- some misc label fixes

Anonymous Board owner 10/09/2019 (Wed) 22:28:33 Id: 83b1de [Preview] No.196 del
next week

I have one more week before the large wx->Qt migration occurs, which I expect to take about four weeks. Since there will be long break, I will try to mostly work on small things so that last wx release is a 'clean' one without any earth-shattering bugs. I'd like to have some 'session weight' counting so the client will notice and notify you if your session gets too large.

If you have missed the news, 8chan is planned to relaunch in the near future under 8kun.net. Assuming the launch goes well and they are able to stay up, I am interested in moving the main /hydrus/ board back there. They are planning to migrate board content over to the new site for BOs who ask, and I have--thankfully, I still had my BO 'shared secret' lying around in my password manager--so with luck, the old 8chan /hydrus/ should be migrated and we'll be somewhat back to normal.

Anonymous 10/10/2019 (Thu) 04:29:11 Id: 2682d0 [Preview] No.205 del
I assume that mpv being embedded will also allow for sound, and if this makes sound possible in the program, would it be possible to have a drop down on the program for sound? something like an enable toggle, and if it's possible a 'normalize' option. going through webms, and needing to leave it up to sound to tell me if it's worth keeping or not, I have noticed quite a bit of audio that won't go above 5% and some audio that's right after that quite clip won't back off 100%

This is more a down the road when it happens thing, but would it be possible to in program have some form of a compressor (bring lows up and highs down) and have volume based off that when enabled? this is a qol thing that I assume most people never knew they wanted because when we watch videos, adjusting the volume once every 10+ minutes isn't a problem, but going though 3-20 second long gif replacements... this is where it could come in and bee a greatly appreciated feature.

Anonymous 10/10/2019 (Thu) 09:16:43 Id: 2682d0 [Preview] No.207 del
something just occurred to me which would make backup a bit less large. so with the tag repositories, would it be possible to have that file outside of the hydrus folder, and symbolically link its location, effectively cutting 18gb out of what needs to be backed up if I just move the folder once every time I update?

Anonymous 10/10/2019 (Thu) 12:27:53 Id: 7d082b [Preview] No.208 del
I managed to fix this problem with ffmpeg not running with dbs in a separate directory. v371 showed me that the ffmpeg path was correctly generated so I went and got pyinstaller working and built my own compiled version.

After doing some testing and googling I found this page: https://stackoverflow.com/questions/40108816/python-running-as-windows-service-oserror-winerror-6-the-handle-is-invalid

Adding stdin=subprocess.PIPE to the subprocess.Popen for ffmpeg on the about box and it worked first go.
After Hydrus wouldn't import a webm, I added it to all ffmpeg calls in HydrusVideoHandling.py and HydrusAudioHandling.py and everything seems fine.

Not sure what side effects there might be, but considering all the other subprocess' using PIPE (such as the ones in NAT punch) have stdin=subprocess.PIPE I feel confident.

Anonymous Board owner 10/12/2019 (Sat) 21:23:35 Id: 83b1de [Preview] No.213 del
Yeah, as I undersand it, it'll be a fully functional embedded window with full player support. I think I can interact with it and change its volume etc... programatically.

I have strong feelings about global volume. I will make sure the client has a 'silence all' mode with shortcuts and all that. I'll see if I can get some global volume controls and options as well. Normalise is probably a bit complicated to start with, unless mpv offers that natively. We'll see what it can do once we have it going.

Anonymous Board owner 10/12/2019 (Sat) 21:32:21 Id: 83b1de [Preview] No.214 del
Probably not. The db does not work well if its four files are out of sync, and I would not recommend any backup scheme that does not keep them together. I can't easily pull the tag repo 'table' out of the db, as it is 'processed' and closely coupled with everything else.

Can you talk a bit more about your backup/update routine? I am not sure I completely understand, and perhaps we can make that more efficient.

Anonymous Board owner 10/12/2019 (Sat) 21:36:28 Id: 83b1de [Preview] No.215 del
Damn, thank you for putting the effort in here. I will look into this and update the code as you have done. I don't send anything on stdin to ffmpeg, but I guess your OS wants that link.

Anonymous 10/14/2019 (Mon) 06:30:37 Id: 2682d0 [Preview] No.216 del
more or less goes like this, when i update the program, move the old database folder to a hdd as a mirror and install the new version, that way im never really going backwards to much.

however what i'm thinking is, if one of those files are dedicated specifically to the repository, and if shit hit the fan could be redownloaded, the need to have a running backup of that is far lesser, so instead of leaving it in the db, I symbolically link it to the db, this should have the program see the file and use it like it normally would, but on backup it would not transfer the file over, just the symbolic link, meaning that 18(nearly 19)gb can be cut from the backup cutting what gets backed up from 42gb down to 25

if client.mappings is just repository and not keeping a solid backup of this would be detrimental, i think symbolically linking it would be a good way to save on the versioned backup

Anonymous 10/14/2019 (Mon) 06:32:27 Id: 2682d0 [Preview] No.217 del
oh, I should point out, the backup I have here is the most recent one made when I updated, but due to having shit to do, I didn't go with 370 and went to 371

Anonymous Board owner 10/19/2019 (Sat) 17:24:14 Id: d63590 [Preview] No.228 del
Thank you for the update. If you aren't already, I recommend a program like FreeFileSync, which will work much faster than a manual folder copy for backing up. It will let you mirror to the same folder and only update files with new modified dates, so you aren't always re-copying media files.

You can try doing some symlink stuff, but I think SQLite may have a problem with it. The automatic recovery options for client.mappings.db are not excellent yet (for if, say, it was corrupted/lost, and you needed the client to auto-generate a fresh empty one), so I recommend not playing around with this too much for now.

Top | Return | Catalog | Post a reply