/hydrus/ - Hydrus Network

Hydrus Bunker

Boards | Catalog | Bottom

Check to confirm you're not a robot
Name
Email
Subject
Comment
Password
Drawing x size canvas
File(s)

Remember to follow the rules

Max file size: 350.00 MB

Max files: 5

Max message length: 4096


Extension Thread Anonymous 08/07/2019 (Wed) 19:04:24 Id: 32952a [Preview] No. 4 [Reply] [Last 50 Posts]
ITT; extensions and thing you can bolt on to Hydrus to make it better

http://archive.fo/rmx9d
http://archive.fo/C8ylc


Anonymous 09/28/2019 (Sat) 16:53:38 Id: bdc9e9 [Preview] No.130 del
partial bump



Co-development generals thread Anonymous 08/07/2019 (Wed) 18:56:36 Id: 5c0f11 [Preview] No. 2 [Reply] [Last 50 Posts]
Due to the recent take down of ExHentai, is pretty alarming that the same might happen to similar sites. So I propose to anons using Hydrus to save the doujins they like within the platform and tag them along so whenever Hydrus reaches an stage where sharing is easy, maybe you'll have not only a decent alternative to Sad Panda, but a lasting one.

Numerous ideas have been thrown around about it, but here is what the BO have confirmed:
1) Hydrus is not currently great at handling paged media. It can do it, but the workflow is a little awkward. I plan to improve this in future.
2) IPFS support is prototype, and mostly for advanced users who are already comfortable with the program. I'm working on it now.
3) Hydrus is not for everyone. If you have 10,000+ files and want to manage them better than files and folders, you may like it. If you want a clean and professional experience with beautiful UI, you may not.

So it's nothing but a project now. The main reason I do the thread is to keep anons up to support it and use Hydrus for managing whatever manga they downloaded to be used later when the project get to the point it can be used and to post resources and the such.

An index with tags and names:
https://mega.nz/#!dBxmUADB!4piJItp7ja7_9WbTmoLGo3nMLOP2Fr0AH_ju9W4-PLY

Hydrus+IPFS that supports concurrency
https://github.com/sakharovaan/hydrus-unleashed
https://github.com/rr-/szurubooru

IPFS+TOR/I2P is in the works

Message too long. Click here to view full text.



Anonymous 09/28/2019 (Sat) 16:53:00 Id: 4dfb77 [Preview] No.129 del
partial bump



Version 368 Anonymous Board owner 09/18/2019 (Wed) 23:54:49 Id: 42a464 [Preview] No. 96 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=KhWArBhPWu0 [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v368/Hydrus.Network.368.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v368/Hydrus.Network.368.-.Windows.-.Installer.exe
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v368/Hydrus.Network.368.-.OS.X.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v368/Hydrus.Network.368.-.Linux.-.Executable.tar.gz
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v368.tar.gz

I had a great week. The PTR has moved successfully, and I got multiple local tag services working.

PTR has moved

The Public Tag Repository has changed management. I am no longer running it or involved in working as a janitor. There is more information here:

Message too long. Click here to view full text.

8 posts and 2 images omitted.


Anonymous Board owner 09/25/2019 (Wed) 04:47:28 Id: 42a464 [Preview] No.109 del
>>108
>the thumbnails are noticeably different, even if fullsize aren't
hydev btfo himself

interesting that two.png, the jpeg, has more blushline detail in the thumb


Anonymous Board owner 09/25/2019 (Wed) 04:54:30 Id: 42a464 [Preview] No.110 del
>>106
Thank you for this report. I changed this a little while ago, and the intention was to try to make them easier to compare even when the resolution ratio differed a little, like 1024x768 vs 1024x764. It is not supposed to stretch, but maintain correct ratio and adjust position as best it can. I think the two images are supposed to line up either with their top-left corner or the center-point--are you not seeing this? Is the bad line-up after you drag to move them, or from the start?

Can you post example screenshots of what you see? Or, if you do not want to share, can you post an example pair of resolutions that are showing badly for you, and their filetypes (jpg or png)?


Anonymous 09/25/2019 (Wed) 05:18:52 Id: 67fade [Preview] No.111 del
>>108
>138MB webm, 1080p, 2.2s, 55 frames
jesus christ


Anonymous 09/25/2019 (Wed) 10:51:23 Id: 58c0a9 [Preview] No.113 del
>>108
Oh, I fully agree, I also have an idea for implementable webm/gif conversion to mp4 but my knowledge on this side is a bit lacking, I have had gifs that went from 80mb down to 250kb due to an mp4 conversion and I cant tell the difference without an autistic level of detail.

My point was more explaining why they exist, artists work at a stupidly high resolution because when you save it down it AA's the thing via supersampling, along with makes working on the files easier because mistakes will be easier to hide/fix.

and also on the share side... the best examples I can tell you about were a few artists who did real art, not anime style or any of that shit but legitimate digital paintings, zooming in on them to study strokes or figure out why they did certain things is very helpful for learning technique, now one of the things you may overlook is how photoshop handles brushes which is it puts down a dot every x distance otherwise it lags to hell and back, so even seeing that would be helpful in learning how to set up a program.

I agree on the side of 'why the fuck would a normal person want this' to an extent, if I wanted to print a large poster of said art, the higher res the better, but that's the only reason I can see wanting it on a massive file but at that point I would rather have the psd then just a png.

---------

as for the scanning, I do suggest 2/5/10 at least if we take these webm's are suppose to be gif replacements, having them longer than that would defeat the purpose and make them unusable, if you can user specify the time jump or even frame jump, that would be ideal so people who need short jumps and people who need longer 30 second jumps can both be covered, but no idea if that's possible.

>>111
some people want to see the world burn.


Anonymous Board owner 09/25/2019 (Wed) 17:45:26 Id: 42a464 [Preview] No.114 del
>>109
Looking at them again, two.png is 24 bit colour while the original png kept 32 bit. I wonder if that triggered a different scaling algorithm on endchan's side for the slightly different thumbs.

>>111
>>113
I think a lot of the silly files you see, like 1080p gifs, are due to some artists, while being great at drawing or modeling 3d, just happen to be ignorant about what file types and sizes actually are. They choose the wrong thing, have to rely on 'default' encoding settings, use some borked conversion software, or click the 'make this phone compatible' checkbox without realising. And there's a bit of auto-upload conversion scripting gone wrong. What used to be a pretty nice 15 second 22MB 1080p 60fps webm ends up a crazy 12fps super gif with a fucked palette. Thankfully, I would say the situation has improved in the past year or so. There are some fantastically well-optimised 4k webms floating around, even.

An idle dream of mine is to write a 'look at this media, see if it is massively unoptimised, and if possible make a "good" version' module that could take things like big pngs and make a human-imperceptible optimisation. It'd use pixel statistics to figure out the 'imperceptible' part, so you ideally wouldn't be able to tell which was the optimisation in a blind test. And ideally be completely deterministic, too, so the same operation on the same file on two computers produces the same byte-for-byte output file. Just an easy thing for anyone who was ignorant to say 'hey, I made/have this thing but don't understand the tech side, please make it nicer for sharing with an emphasis on human consumption'.

A bit like https://github.com/WebMBro/WebMConverter, but for more media types and an emphasis on no downgrades.

I'd plug it into hydrus, but it would be completely optional, of course. I have played around with ffmpeg a decent bit and have some ideas on this for video. It wouldn't be massively difficult to do, just would take some work, and the end product would be pretty CPU heavy. I am somewhat surprised the codec creators haven't written their own 'just make this shit good' parameter, but I generally figure they work for the industry more than anything, who are happy with obscure params for their different specific streaming tech needs and so on.

However, on the hydrus side, improvements to the duplicate system may supersede most of this. Lots of people are already doing this reencoding work manually or with their own scripts, the results of which we all end up downloading as dupes. If hydrus gets auto-duplicate-resolution tech that can mine your existing data for more optimal dupes, we won't have so much reason to generate our own.

And who knows, maybe waifu2x tech will get crazy in the next five years and the whole argument will be moot.

Message too long. Click here to view full text.




Release Tomorrow! Anonymous Board owner 09/25/2019 (Wed) 06:11:50 Id: 36e060 [Preview] No. 112 [Reply] [Last 50 Posts]
I had a good week. As well as some smaller fixes and improvements, I added file 'modified date' metadata to the client, letting you see and sort by it, and improved the file maintenance system (which will retroactively generate modified date info and catch up on the existing 'has audio' detection queue) to work quietly all the time in the background, so its gets much more work done with much less hassle.

The release should be as normal tomorrow.



Version 367 Anonymous Board owner 09/11/2019 (Wed) 23:00:04 Id: 4e5b9d [Preview] No. 77 [Reply] [Last 50 Posts]
https://youtube.com/watch?v=Z-eZl_CVQpA [Embed]
windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v367/Hydrus.Network.367.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v367/Hydrus.Network.367.-.Windows.-.Installer.exe
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v367/Hydrus.Network.367.-.OS.X.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v367/Hydrus.Network.367.-.Linux.-.Executable.tar.gz
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v367.tar.gz

I had a good week getting more of the tag migration working and adding some audio and ratings ui features.

sibling and parent migration

Last week's tag migration dialog now supports sibling and parent migration! Just like with tag mappings, you can copy between any combination of tag services and an external file type called a Hydrus Tag Pair Archive. There are tag filters for the left and right side of the pairs being migrated, so you can easily capture all the character->series parents or unnamespaced->creator siblings, for instance.

Message too long. Click here to view full text.

8 posts omitted.


Anonymous 09/15/2019 (Sun) 10:11:15 Id: a5806d [Preview] No.93 del
>>89
what I mean is if in qt there is something that makes it not a 1:1 change and there will be a difference due to qt, would it be possible to screenshot where it is in wx and then where it is in qt, the way you word it makes it sound like there may be a bit of this.

honestly i'm hoping most of the culling is done by the time the qt switch would be finalized, at least this round of it. still have a seperate more laggy one do deal with but this one is the most image open one to go though.


Anonymous 09/18/2019 (Wed) 03:38:24 Id: a5806d [Preview] No.94 del
ok, having some 'fun' going through some webm's,
separated them by duration
1-10 seconds ok, I can watch this
11-20 seconds ok, get on with it
21-30 seconds for fuck sake this is 2mb just get on with it
31+ seconds ... fuck my life

ok, more or less I have learned that going through some of these webm's that this is kind of fuckin painful to parse them without some kind of way to scrub through them.

with the way hydrus currently is, is it possible to have some controls over the video playback? Im not sure what would be good, seconds skipping or frame skipping, either could work if one was easier to implement, a 2/5/10 second skip, or 50/125/250 frame would probably cover almost anything, at least anything that's normally posted as a gif replacement, if someone added episodes of anime or something... lets say gelbooru had some mp4's that downloaded 15+ minute things... just for instance... those numbers would be a little weak... thinking of it, these don't need to be hard coded values, and you wouldn't necessarily be limited to 3 each way (every video program I use is like that so I did not consider you could have more and i'm not rewriting this) and you could have these all be user settable, either time wise or frame wise...

is this possible or not something that can currently work?

welp... back to watching webm's to see if they are worth keeping and then opening them up to see if the audio saves them.


Anonymous Board owner 09/21/2019 (Sat) 20:52:55 Id: 7191c6 [Preview] No.101 del
>>91
Thanks. I have had some other reports on this, so I will try to brush up the error reporting here on bad images and maybe add an option to turn 'load it even if it seems broken' mode back on.

I am not sure what the current state of this is, but I understand some users were working on waifu2x stuff with the Client API. Some sort of script that pulls images that you designate out of the client, does waifu2x (I have mostly seen blowing small images up, rather than your example here), and then reimports the result.

I definitely think there is a future for this stuff, and other file conversions, and am willing to expand the client api and other features to facilitate this stuff.

Thank you for the mega example. I haven't much personal experience with waifu2x specifically, but this is all neat tech. It will only get better, and I am definitely interested in hydrus talking to it more in future.

I don't really know what it looks like, but I'd be interested in adding 'this file A looks X SimilarUnits compared to file B' tech to the client. Probably by doing some stats on image diffs or similar. This will help us optimise png->jpg conversions and auto-duplicate resolution. If we know according to math that an image is human-imperceptably similar to another, that is very useful. I hate those 40,000x16,000 pngs put out by artists. There is no practical reason why an image that big needs to be pixel perfect.


Anonymous Board owner 09/21/2019 (Sat) 20:54:33 Id: 7191c6 [Preview] No.102 del
>>93
It should be basically the same layout, afaik. We are planning a 1:1 conversion as much as possible, so the differences will be in little things like button border widths and scrollbar style. I am not sure I'll be able to make nice screenshots, as I think I'll be scrambling to get it working. Let me know, once it comes out, if anything is confusing.


Anonymous Board owner 09/21/2019 (Sat) 20:58:26 Id: 7191c6 [Preview] No.103 del
>>94
Unfortunately, my video rendering pipeline is complete duct-tape hackery, so repositioning the scanbar caret has a short delay, and on some videos will not work at all.

Forgive me if you know this already, but just checking: the little scanbar beneath a video in hydrus lets you click and drag it. You can scroll back and forth and click to reposition it for a longer video. It is laggy to work with, but you don't have to watch things in real time.

You can also double-click on a video to open it in your default program.

After the Qt conversion, I am going to look at embedding mpv into the program. I do not know if this will work, but if it does it will be a very nice option that will play audio too.



The PTR will undergo a change of management in two weeks Anonymous Board owner 09/07/2019 (Sat) 20:56:55 Id: 0ac2e0 [Preview] No. 62 [Reply] [Last 50 Posts]
I have run the PTR since the start of hydrus, seven or so years now. I remember hitting the first 1,000 tag mappings, then being stunned at 1,000,000, and now we are at 650,000,000. It was always meant to be a test prototype, but I kept it going because it was so neat seeing the tech work, getting positive feedback, and dealing with all the interesting problems millions and then hundreds of millions of tags presented.

Unfortunately, as it has grown, it has become increasingly unwieldy to manage by myself. I have fallen behind on dealing with petitions, and I have increasing bandwidth problems you have no doubt noticed. I have never been an excellent server administrator, and I don't like being a single source of failure, either. A user who has more bandwidth has kindly offered to run it in my stead, so therefore I will be shutting it down on my end, likely on September 16th, in time for version 368.

I will sanitize the database so it has no more information than a syncing client currently knows (there have never been IP logs or anything, but I will do some additional work such as deleting the petition 'reasons' users have submitted over the years, some of which have personal joking messages to me) and put the frozen database up on a Mega account with scripts to generate new admin access keys and so on. Any user who wants to download it and continue their own tag repository with its data, whether that is private or public, will be able to. I will also create some Hydrus Tag Archives for the tags and parents and siblings for users who are separately interested in that data.
1 post omitted.


knioy Anonymous 09/07/2019 (Sat) 21:06:25 Id: f8538a [Preview] No.66 del
yoink


Anonymous 09/08/2019 (Sun) 16:45:31 Id: d33975 [Preview] No.67 del
Question: Why not have the new mod/admin of the PTR to start a process of "data cleaning" and "standardization"?
Surely the system ExHentai, *booru/e621/Sankaku/r34 and others uses (with voting + central moderation) works.

What would we need to be in-par with them? Firstly a tag dictionary (or better yet a tag wiki) would be needed.
That way people can easily define tags, such that misspelled duplicates can be ejected and siblings can be paired.
The extra benefit of using a wiki rather than a pure dictionary is that cross-linking and categorization is possible.
Trouble is, that none of the major Python/JS based wikis on the internet have enough API power to do this.

Secondly we need a basic "formal" process of handling and documenting definition and relations of the tags.
In this case we can learn from Python's PEP and Bitcoin's BIP, which are Enhancement/Improvement Proposals.
The issue is that a standard forum will be needed, as lack of archives, post and thread limits will make management hard.
A workaround would be to host a BBS that removes such limitations (like the one in yiff.party which is "plain")
Issue is, that nobody currently wants to host a BBS, and 8chan (an imageboard with archives) is down.

Thirdly we need an actual organization structure that will assess each proposal and give a verdict.
Of course since HyDev is giving up power to [insert name here] we can assume he is the benevolent dictator.
In that case a caucuses-like system to allow people to talk away the issues before the vote and possible vetoing.

Message too long. Click here to view full text.



Anonymous Board owner 09/11/2019 (Wed) 01:47:31 Id: c21c56 [Preview] No.72 del
>>67
The guy who I know will be running a successor is a regular feature on the hydrus discord at: https://discord.gg/3H8UTpb

It looks like he will be recruiting some janitors from there to help him out. He is mostly concerned about my bandwidth trouble and, afaik, does not seem to have big plans to implement strong new janitorial policies. If you are ok with discord, please feel free to join the hydrus discord server and start talking.

Otherwise, please feel free to set up one run by Anons, particularly if 8chan soon comes back and we return to regular order. I have never been very good or enthusiastic about setting up a proper/formal tag schema, so I will be glad to be out of the question entirely. I expect and hope in future to put some time into improving the user-admin interaction workflow (like Anonymously proposing changes to a tag repo's filter, once I have added that) so people who do like to work in groups and set up these sorts of things will be better able to figure out their own good solutions, including completely Anonymously, as well as the clientside sibling and tag filtering preferences I already have planned. All feedback on these issues is welcome.


Anonymous 09/13/2019 (Fri) 20:29:37 Id: d4179f [Preview] No.84 del
>>67
>>72
I will be taking it over as is, I do not plan for big changes. It shall stay the ways it was. Only want to host the PTR due to bandwidth limitations and the eternal new "features", that limit the data flow, that stem from the bandwidth limitations.

In the end, the data will go through a VPN and will end up on a server, somewhere no one needs to know. Basically, as good as any other Anon can do.

If someone wants to talk about security, I'm open to discussions.


Anonymous Board owner 09/19/2019 (Thu) 00:04:37 Id: 7ddf8f [Preview] No.100 del
It is done!

>>96



Release Tomorrow! Anonymous Board owner 09/18/2019 (Wed) 05:29:42 Id: a96983 [Preview] No. 95 [Reply] [Last 50 Posts]
I had a great week. The PTR has moved to its new higher bandwidth location and management successfully. If you sync with the PTR, tomorrow's client will offer to automatically move your settings to the new location and you will be able to continue syncing and uploading tags without skipping a beat. Also, I was able to get multiple local tag services working completely!

The release should be as normal tomorrow.



Anonymous 09/15/2019 (Sun) 10:01:46 Id: c9975b [Preview] No. 90 [Reply] [Last 50 Posts]
>>86
nope, running windows 7 and from installer, no idea why it wont import for you, but it was already in my db and not really sure where I would find it again to dick around.

also

I got somewhat done with the image thing I was doing

https://mega.nz/#!Dl4ABYwL!t53V4rjD7nqPYrPq-PYDT4ug-Uo67B0OfvnCOIpXdqc

its I think 9 images, with source files where I ran them through waifux2 at 1.0 magnification trying to see where they started to fall apart in quality, some of them go fairly low before really shitting the bed, some of them aren't able to handle the conversion to jpeg at all.

one of the images I went x2 on source because I have seen a lot of jpegs where they are twice the resolution, about the same file size but far higher quality then the one that should have been higher quality.

the main reason I did this was a thought. now, i'm not sure about you but I have a fuckload of images that I could run though this and get a reasonable file out of, but the only way to do this in bulk is to copy every image from the archive out, convert them 1 by 1 and zoom in on them which is honestly a shit work flow.

if this was capable in hydrus, you could have the source image convert the image, then save both images to a duplicate detector, hell if you wanted to get anal about it, you could have them go
20 - 40 - 60 - 80 - 100 and you pick where the image falls apart, then from there it goes into more fine grain. like lets say 40 was where it fell apart, then it would explode 40-60 with 40-45-50-55-60 and you figure out where the image goes to shit. most images seem to be good with about 1/5 to 1/10 their file size.

Message too long. Click here to view full text.



Anonymous 09/15/2019 (Sun) 10:04:13 Id: c9975b [Preview] No.92 del
meant this to go to the new thread not be a thread, wont let me remove.



Version 366 Anonymous Board owner 09/05/2019 (Thu) 01:16:27 Id: 4fac49 [Preview] No. 52 [Reply] [Last 50 Posts]
13 posts and 2 images omitted.


Anonymous 09/11/2019 (Wed) 10:14:36 Id: 807b48 [Preview] No.76 del
>>75
I can't even import this image, here's the error I'm getting:

Could not load the image--it was likely malformed!… (Copy note to see full error)
Traceback (most recent call last):
File "include\HydrusImageHandling.py", line 241, in GeneratePILImage
pil_image = PILImage.open( path )
File "site-packages\PIL\Image.py", line 2822, in open
OSError: cannot identify image file 'C:\\Users\\****\\AppData\\Local\\Temp\\hydrusvtrj3tg1'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "include\ClientImportFileSeeds.py", line 940, in ImportPath
self.Import( temp_path, file_import_options )
File "include\ClientImportFileSeeds.py", line 897, in Import
( status, hash, note ) = file_import_job.DoWork()

Message too long. Click here to view full text.



Anonymous 09/14/2019 (Sat) 02:28:48 Id: e974ef [Preview] No.85 del
>>71
On the PTR. Restarting the client doesn't hide the title from the infobar at the top of a file in the media viewer even if censored.


Anonymous Board owner 09/14/2019 (Sat) 19:15:27 Id: 7fe016 [Preview] No.86 del
>>75
Thank you for this report. I am afraid I get the same as >>76 . Are you perhaps on non-Windows or otherwise running from source, so you have a different version of an image library? If I go help->about, I get:

OpenCV: 4.1.0
Pillow: 6.1.0

A user helpfully pointed me to working versions of an alt of that file, both larger at ~300KB:

https://gelbooru.com/index.php?page=post&s=view&id=4486462
https://e621.net/post/show/1734511

It is alt because no saliva at those links. My guess is your image is a half-completed download, or an edit someone made and saved badly? Do you have a source URL for it?


Anonymous Board owner 09/14/2019 (Sat) 19:20:08 Id: 7fe016 [Preview] No.87 del
>>85
Thank you. I will check the code that fetches the tags for the thumb banner.


Anonymous Board owner 09/14/2019 (Sat) 19:29:04 Id: 7fe016 [Preview] No.88 del
>>74
No worries. This is definitely possible. You will need to have a little experience in running a server (things like opening a port on your router if you want people outside the home network to access it). I can help you do this if you are new to it.

Bear in mind however that hydrus servers are pretty debug-tier to run. The UI and process has mostly been used by me for my Public Tag Repository, so it just isn't a pretty process. I would welcome feedback if you give this a go so that I can improve the whole thing for regular users.

If you are new to hydrus, please use it a bit more so you are comfortable with the UI and workflows (and know that you like them!) before trying to run a server. Then, when you feel good, check out the help here:

https://hydrusnetwork.github.io/hydrus/help/server.html

The tl;dr: is you run server.exe and add it in the client's services->manage services. There's a bit of admin to do to set up access keys, and then your friends can talk to it as well. Then you just leave it running and all the clients will upload and sync to it without you having to do anything. Again, I am happy to help if you run into trouble.



Release Tomorrow! Anonymous Board owner 09/11/2019 (Wed) 02:38:31 Id: a47657 [Preview] No. 73 [Reply] [Last 50 Posts]
I had a good week. The new tag migration system now supports siblings and parents, including import/export to external files, and I added shortcuts to increment/decrement numerical ratings, and added sort and thumbnail display for the new 'has audio' property.

The release should be as normal tomorrow.