/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.

Posting mode: Reply

Check to confirm you're not a robot
Drawing x size canvas

Remember to follow the rules

Max file size: 350.00 MB

Max files: 5

Max message length: 4096

Manage Board | Moderate Thread

Return | Catalog | Bottom

Expand All Images

Version 368 Anonymous Board owner 09/18/2019 (Wed) 23:54:49 Id: 42a464 [Preview] No. 96
https://youtube.com/watch?v=KhWArBhPWu0 [Embed]
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v368/Hydrus.Network.368.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v368/Hydrus.Network.368.-.Windows.-.Installer.exe
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v368/Hydrus.Network.368.-.OS.X.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v368/Hydrus.Network.368.-.Linux.-.Executable.tar.gz
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v368.tar.gz

I had a great week. The PTR has moved successfully, and I got multiple local tag services working.

PTR has moved

The Public Tag Repository has changed management. I am no longer running it or involved in working as a janitor. There is more information here:


As a result, the PTR no longer has bandwidth limits! The user now running it is also putting together a janitorial team to catch up on petitions as well. About ten million delete mapping petitions and six thousand add sibling petitions had piled up! If you would like to talk to the new management, they are available on the discord. The current plan is to keep running the PTR with the same loose rules as I did--the main concern was overcoming my bandwidth limits.

If you currently sync with the PTR, you will be given a yes/no dialog when you update asking if you would like to keep using the PTR at the new location. If you select yes, the client will automatically update your service (the only credentials difference is that instead of it being at hydrus.no-ip.org, it is now at ptr.hydrus.network), and you will keep syncing and be able to continue uploading without skipping a beat. If you select no, your PTR stay as-is but will pause. If you still sync with my read-only test file repository, this will be paused automatically.

While I am very thankful for the 650 million submissions I have had to the PTR over the past seven years, it is a load off my mind to no longer be responsible for it. I much prefer being a developer than an administrator and hope to use the extra time and upcoming feedback to work on improving the admin side of hydrus repositories, which, due to me being the primary user, have always had debug-tier UI and half-broken features.

The various areas in the help have been updated to reflect that the PTR is no longer mine, and the quick setup under the help menu is now just help->add the public tag repository.

Anonymous Board owner 09/18/2019 (Wed) 23:55:20 Id: 42a464 [Preview] No.97 del
multiple local tag services

I planned to only start work on multiple local tag services (being able to have more than one 'local tags', just like you can have more than one tag repository) this week, but I accidentally finished it! it turned out not to be as huge a job as I thought, so I just piled some time into it and got it done.

So you can now add new 'local tag' services under services->manage services. You can add (and delete) as many as you want, but you have to have at least one. It is now possible, for instance, to create a new separate local tag service just for your subjective 'favourite'-style tags, or one that only pulls tags from a certain booru, or from filename imports. You can also use the new tag migration system to move tags between your local tag services. I know some users have wanted a separate local tag service as a prep area for what they will submit to the PTR--I think all the tools are now in place for this.

As a side note, just to emphasise that it is not the only local service you can have, if your local tags service still has the default name 'local tags', it will be renamed to the new default 'my tags' on update. Feel free to rename it to whatever you like, again under manage services.

Once I have some better tag show/hide tech working, I'll likely add additional default tag services that do not show their content in the main UI but do pull all tags from downloaders and filenames from hard drive imports, so you will be able to retroactively 'mine' this information store for your real tag services if you miss it the first time around.

I am really pleased with this feature. If you have been interested in it yourself, let me know how you get on. Adding multiple local file services is another long-planned feature, unfortunately significantly more complicated (I think 8-12 times at least), but I would like to hear how tags go for now.

Anonymous Board owner 09/18/2019 (Wed) 23:56:27 Id: 42a464 [Preview] No.98 del
whoops, forgot to paste this as second post:

running your own continuance of the PTR

If you are an advanced user and would like to run your own version of the PTR from where I left off, be that a public or private thing, I have uploaded the same sanitized and 'frozen' version of the server db we used in the transfer here:


If you do run your own and would like it to be public, let me know and I'll happily add its info to my help files and the auto-setup links.

Hydrus repositories are anonymous. No IPs or other identifying info is logged. There was not much to sanitize, but out of an abundance of caution I deleted the various petition 'reasons' people have submitted over the years, as some had some personal jokes to me, and I collapsed the content submission timestamp record across the db to be no more detailed than what a syncing client already knows. Since this timestamp collapse reduces server-specific knowledge about uploads and does not affect server operation, it is now standard practise for repositories going forward. If you run a repository, updating this week will take a few minutes (or, ha ha, if you have 650 million mappings, about five hours) to update its existing data.

I also uploaded Hydrus Tag Archives of the PTR's tag/sibling/parent content, which I simply made with the new tag migration system and am making available for convenience. If you know python or SQLite and would like to play with this data, check them out.

Be warned that these archives unpack to large files that work best on an SSD. The server db is a 5.5GB .7z that will ultimately grow (after following an internal readme guide) to 42GB or so. As I write this document, the namespaced/unnamespaced mappings Hydrus Tag Archives are not yet uploaded (even zipped, they are 5GB each), so please check back later if you do not yet see them.

Anonymous Board owner 09/18/2019 (Wed) 23:58:43 Id: 42a464 [Preview] No.99 del
full list

- multiple local tag services:
- you can now add additional local tag services under services->manage services!
- new local tag services will appear in manage tags and tag import options and so on, just like when you add a tag repository
- you can also delete local tag services, but you must have at least one
- the default local tag service created for a new client is now renamed from 'local tags' to 'my tags'. any existing user with their local tag service called 'local tags' will be renamed on update to 'my tags'
- .
- ptr migration:
- the ptr has been successfully migrated to user management! hydrus dev is no longer involved in running or administering it. the old bandwidth limits are removed! it has the same port and access key, but instead of hydrus.no-ip.org, it is now at ptr.hydrus.network
- on update, if you sync with the ptr, you will get a yes/no asking if you want to continue using it at the new location. on yes, it'll update your server's address automatically. on no, it'll leave it as-is and pause it. if you still have a connection to my old read-only file repo, that will be paused
- changed the auto repo setup command to be _help->add the public tag repository_. it points to the new location
- as repo processing and related maintenance is now nicer, and secondarily since bandwidth limits are less a problem for the ptr specifically, the default clientside hydrus bandwidth limit of 64MB/day is lifted to 512MB/day. any users who are still on the old default will be updated
- updated the help regarding the public tag repository, both in general description and the specific setup details
- a copy of the same sanitized and frozen PTR db used to start the new PTR, and convenient tag archives of its content, are now available at https://mega.nz/#F!w7REiS7a!bTKhQvZP48Fpo-zj5MAlhQ
- .
- the rest:
- fixed a small bug related to the new 'caught up' repository mechanic for clients that only just added (or desynced) a repository
- rewrote the tag migration startup job to handle specific 'x files' jobs better--they should now start relatively instantly, no matter the size of the tag service
- on 'all known files' tag migrations, a startup optimisation will now be applied if the tag service is huge
- fixed the tag filter's advanced panel's 'add' buttons, which were not hooked up correctly
- the internal backup job now leaves a non-auto-removing 'backup complete!' message when finished
- on update, server hydrus repositories will collapse all their existing content timestamps to a single value per update. also, all future content uploads will collapse similarly, meaning all update content has the same timestamp. this adds a further layer of anonymity and is a mid-step towards future serverside db compaction (I think I can ultimately reduce server.mappings.db filesize by ~33%). if you have a tag repo with 10M+ mappings, this will take some time
- hydrus servers now generate new cert/key files on boot if they are missing. whenever they generate a new cert/key, they now print a notification to the log
- misc help fixes and updates, and removed some ancient help that referred to old systems
- corrected journalling->journaling typo for the new experimental launch parameter

next week

The PTR work took much longer than I expected, and I was unable to get to modified date or file maintenance improvements, so I will have a re-do for those next week. For the ongoing tag work, I will start work on updating the tag 'censorship' system, which is still running on very old code, to more of a 'don't show these tags in places x, y, z', and see about a related database cache to speed up various sibling/display tag choices.

This week was a little bonkers and I fell behind on messages. I am sorry for the delay and will put some time aside to catch up when I can. I think my immediate busy period is done for a bit (although we'll be back to it for the wx->Qt conversion in mid-October), so I'll be grateful to take it a bit easier for a while.

Anonymous 09/22/2019 (Sun) 23:07:53 Id: ae028b [Preview] No.104 del
Trying to copy external share urls, and it throws an error. I can mangle it together myself by changing the ip from the internal to my public ip in the url but that's work

must be str, not int
Traceback (most recent call last):
File "include\ClientGUIDialogs.py", line 463, in EventCopyExternalShareURL
url = self._service.GetExternalShareURL( self._share_key )
File "include\ClientServices.py", line 464, in GetExternalShareURL
port = ':' + self._upnp_port
TypeError: must be str, not int

Anonymous 09/23/2019 (Mon) 01:29:22 Id: 58c0a9 [Preview] No.105 del
For the art perspective, the artist is putting out their raw finished work. Think of it this way, with videogames we have anti aliasing, and because its so processor intensive, we find ways to fuck around with it, some ways like fxaa where it blurs the whole screen, and some like msaa where it takes edges and renders them at a far higher resolution and then downscales them. the holy grail of aa is super sample, take a 4k image and down sample to 1080p, or an 8k and down sample to 4k, these methods objectively look the best.

art programs, largely, if you have the ram to work with it, it runs fairly smooth, so you typically work at a FAR larger resolution then you are going to output, this hides many flaws, or brush strokes that you don't want to be seen but were necessary. some patrons have this kind of image as their reward... I get why you may want to see it as an artist, but its never really worth having as someone who just wants to appreciate the art, unless you wanted to get it on a poster or something, at that point I can easily see the appeal of it... but I also use a 4k55inch as a monitor, some of the posters I have are smaller then my monitors size, and really, if they were 7680x4320 I honestly doubt I would be able to see where they were wrong.

I do already know that, the main thing is i'm using a macro pad, specifically the g13 and if I had a way to fast forward through videos by hotkeys I wouldn't have to use the mouse or open out of program, each take a bit of time to do which even laggy, if it could be done though hotkeys would ultimately be faster video to video.

mk2 09/23/2019 (Mon) 12:41:48 Id: 08ac19 [Preview] No.106 del
Duplicate processing now has a bug when the second image is displayed as if it had dimensions of the first one, which breaks scaling. Like if the second image is smaller, it is shown not appropriately scaled but at upper side, and if the second image has greater resolution, only upper part of it is shown.

Anonymous Board owner 09/25/2019 (Wed) 04:11:21 Id: 42a464 [Preview] No.107 del
Thank you for this report. This should be fixed in tomorrow's release. Please let me know if you have further trouble.

Anonymous Board owner 09/25/2019 (Wed) 04:44:01 Id: 42a464 [Preview] No.108 del
(836.34 KB 1446x1170 one.png)
(990.75 KB 1446x1170 two.png)
hot take alert
I get the overall nice feeling of having a 'perfect' image, and if that provides some people a sense of satisfaction, I am fine with it, but I personally can't see those sorts of giant pngs as defensible for any practical purpose vs a great jpeg at <50% the filesize. My general rule for media quality is that the value of an image is primarily in the human experience of observing it. My eyes aren't 40,000x16,000, and even if I zoom in to one area at 100%, I cannot tell the difference in a shaded image between a 98% jpeg and a png. I doubt, given a 'blind' test, that any human can with sight alone. If there aren't obvious jaggies (and 98% covers even these cases well), you have to load up photoshop and compare exact pixel values.

For a typical user simply consuming media, pngs are only appropriate for screenshots and vectors with pure flat colours and hard edges. Even for an enthusiast, the variation on a typical shaded piece of art between a very high quality jpeg and a png is far exceeded by the variation in a monitor's physical pixel quality, the grain on the printing paper, or the dust on the light bulbs in the room. And even then, that variability is further compounded by scratches and bumps on the lenses in my glasses, eyeballs, and my retinas. Not to mention that a stray molecule in one of my neurons is going to add tiny fuzziness to my perception.

To illustrate more what I mean, I'm attaching two pngs--one is a 98% jpeg of the other saved back to png. It is a clip from a 11kx7k png at 100% zoom. If you switch between them, can you spot which is the jpeg? You can cheat if you look at the filesizes.

Perhaps there are arguments like 'if it is a png, cloudflare won't scale its quality down automatically'. Dunno.
hydev spergout complete

Just for fun, I keep track of silly extremes I have encountered while working on hydrus. The current scoreboard is:

222MB gif, 4000x6000, 16 frames
131MB png, (7,819x5,222)
138MB webm, 1080p, 2.2s, 55 frames

Thank you for the note about scanning with shortcuts. I will make a job to add this. Something like jump x seconds forward/backward.

Anonymous Board owner 09/25/2019 (Wed) 04:47:28 Id: 42a464 [Preview] No.109 del
>the thumbnails are noticeably different, even if fullsize aren't
hydev btfo himself

interesting that two.png, the jpeg, has more blushline detail in the thumb

Anonymous Board owner 09/25/2019 (Wed) 04:54:30 Id: 42a464 [Preview] No.110 del
Thank you for this report. I changed this a little while ago, and the intention was to try to make them easier to compare even when the resolution ratio differed a little, like 1024x768 vs 1024x764. It is not supposed to stretch, but maintain correct ratio and adjust position as best it can. I think the two images are supposed to line up either with their top-left corner or the center-point--are you not seeing this? Is the bad line-up after you drag to move them, or from the start?

Can you post example screenshots of what you see? Or, if you do not want to share, can you post an example pair of resolutions that are showing badly for you, and their filetypes (jpg or png)?

Anonymous 09/25/2019 (Wed) 05:18:52 Id: 67fade [Preview] No.111 del
>138MB webm, 1080p, 2.2s, 55 frames
jesus christ

Anonymous 09/25/2019 (Wed) 10:51:23 Id: 58c0a9 [Preview] No.113 del
Oh, I fully agree, I also have an idea for implementable webm/gif conversion to mp4 but my knowledge on this side is a bit lacking, I have had gifs that went from 80mb down to 250kb due to an mp4 conversion and I cant tell the difference without an autistic level of detail.

My point was more explaining why they exist, artists work at a stupidly high resolution because when you save it down it AA's the thing via supersampling, along with makes working on the files easier because mistakes will be easier to hide/fix.

and also on the share side... the best examples I can tell you about were a few artists who did real art, not anime style or any of that shit but legitimate digital paintings, zooming in on them to study strokes or figure out why they did certain things is very helpful for learning technique, now one of the things you may overlook is how photoshop handles brushes which is it puts down a dot every x distance otherwise it lags to hell and back, so even seeing that would be helpful in learning how to set up a program.

I agree on the side of 'why the fuck would a normal person want this' to an extent, if I wanted to print a large poster of said art, the higher res the better, but that's the only reason I can see wanting it on a massive file but at that point I would rather have the psd then just a png.


as for the scanning, I do suggest 2/5/10 at least if we take these webm's are suppose to be gif replacements, having them longer than that would defeat the purpose and make them unusable, if you can user specify the time jump or even frame jump, that would be ideal so people who need short jumps and people who need longer 30 second jumps can both be covered, but no idea if that's possible.

some people want to see the world burn.

Anonymous Board owner 09/25/2019 (Wed) 17:45:26 Id: 42a464 [Preview] No.114 del
Looking at them again, two.png is 24 bit colour while the original png kept 32 bit. I wonder if that triggered a different scaling algorithm on endchan's side for the slightly different thumbs.

I think a lot of the silly files you see, like 1080p gifs, are due to some artists, while being great at drawing or modeling 3d, just happen to be ignorant about what file types and sizes actually are. They choose the wrong thing, have to rely on 'default' encoding settings, use some borked conversion software, or click the 'make this phone compatible' checkbox without realising. And there's a bit of auto-upload conversion scripting gone wrong. What used to be a pretty nice 15 second 22MB 1080p 60fps webm ends up a crazy 12fps super gif with a fucked palette. Thankfully, I would say the situation has improved in the past year or so. There are some fantastically well-optimised 4k webms floating around, even.

An idle dream of mine is to write a 'look at this media, see if it is massively unoptimised, and if possible make a "good" version' module that could take things like big pngs and make a human-imperceptible optimisation. It'd use pixel statistics to figure out the 'imperceptible' part, so you ideally wouldn't be able to tell which was the optimisation in a blind test. And ideally be completely deterministic, too, so the same operation on the same file on two computers produces the same byte-for-byte output file. Just an easy thing for anyone who was ignorant to say 'hey, I made/have this thing but don't understand the tech side, please make it nicer for sharing with an emphasis on human consumption'.

A bit like https://github.com/WebMBro/WebMConverter, but for more media types and an emphasis on no downgrades.

I'd plug it into hydrus, but it would be completely optional, of course. I have played around with ffmpeg a decent bit and have some ideas on this for video. It wouldn't be massively difficult to do, just would take some work, and the end product would be pretty CPU heavy. I am somewhat surprised the codec creators haven't written their own 'just make this shit good' parameter, but I generally figure they work for the industry more than anything, who are happy with obscure params for their different specific streaming tech needs and so on.

However, on the hydrus side, improvements to the duplicate system may supersede most of this. Lots of people are already doing this reencoding work manually or with their own scripts, the results of which we all end up downloading as dupes. If hydrus gets auto-duplicate-resolution tech that can mine your existing data for more optimal dupes, we won't have so much reason to generate our own.

And who knows, maybe waifu2x tech will get crazy in the next five years and the whole argument will be moot.

I still have to update the shortcut command panel ui and some of the backend, but I'll see if I can make it x seconds that you choose, including fractions, so you can set up 'forward 0.5s' or 'back 5.0 seconds', whatever you want.

Top | Return | Catalog | Post a reply