/operate/ - Endchan Operations

Let us know what's up

Boards | Catalog | Bottom

Check to confirm you're not a robot
Name
Email
Subject
Comment
Password
Drawing x size canvas
File(s)

Remember to follow the rules

Max file size: 350.00 MB

Max files: 5

Max message length: 4096


uptime drama continues odilitime Board volunteer 07/02/2018 (Mon) 07:44:32 [Preview] No. 9434 [Reply] [Last 50 Posts]
On June 28th, our hosting company started to migrate us to their Switzerland datacenter without notice, they estimated 5 days of downtime.
On June 29th, we were given some options. We could either stay down until monday and then moving everything ourselves over or wait a few couples and they would just copy what we have but we'd be given a new IP.
I opted for the few hours of down time, since we can change IPs quickly.
Late on the 29th, I asked how it was going, didn't get a response until the 30th that things were slow.
Late on the 30th, I asked about status, got an update on the 1st that the copy was done.
I was told we were back online but we weren't. I had to wait until late July 1st to get this resolved with their support.
Also the new server is a bit slow because they're still moving sites over to it.

At least we won't have the billing transfer issues on the 5th like we used to. Hopefully things will stay stable after all this nonsense.
4 posts and 1 image omitted.


Anonymous 07/07/2018 (Sat) 16:01:30 [Preview] No.9455 del
>>9444
MoonManDoom server when?


Anonymous 07/08/2018 (Sun) 01:41:27 [Preview] No.9456 del
>>9454
good point, I'll clean up that wording. Endchan is not DMCA free, just the hosting company does provide that.


Anonymous 07/18/2018 (Wed) 02:40:40 [Preview] No.9462 del
>Hopefully things will stay stable after all this nonsense.

The fuck just happened for the last 12 or so?


Anonymous 07/18/2018 (Wed) 02:47:33 [Preview] No.9463 del
(96.23 KB 500x380 tiresome.png)


Balrog Board owner 07/19/2018 (Thu) 01:32:27 [Preview] No.9464 del
>>9463
MongoDB shat its pants in a way that Lynxchan couldn't recover from, probably because it stepped in a document that didn't exist because it wasn't salvaged from the encryption clusterfuck. It didn't get caught immediately because both of us have day jobs. The 500 errors are probably related, since I've checked the server while the site is in 500 and it usually happens when mongod is eating obscene amounts of CPU time. (Of course it could also be 500ing because mongod is a hog and we're running on a hybrid VPS, so large operations take longer than lynxchan's database timeout and trick lynxchan into retrying with a new DB connection.)

To keep this shit from continuing we need to copy the valid documents (your boards, posts, accounts, and media) from our old broken database to a fresh database that doesn't have huge fucking holes in it. This would require experimentation to find how to do it without copying the same broken references over and would almost certainly require more downtime while the copy happens, so we haven't been able to do it yet. We will do it, though, because we hate this shit too.



odilitime Board volunteer 06/09/2018 (Sat) 13:53:26 [Preview] No. 9194 [Reply] [Last 50 Posts]
Things have been shit on the temporary servers we had to move to due to the lack of donations.
Sybil LTD ( http://sybil.li?ref=endchan ) has come to our rescue and offered us bigger VPS for free.

We've decided to simplify things and consolidate. We just finished this migration today, we had been working on it for a couple weeks now. We are now hosted in the EU on 4 core, 2.1ghz, 4gb of ram, 500gb SSD VPS

We now have a simpler config:
- no encrypted drives which means no need to wait for an admin to type in passwords for things to start automatically
- should have less power problems than our previous temporary servers
- no more frontend/backend servers, we now have one server and one server alone. This means:
- no more 5th of the month, bill isn't paid west frontend goes down
- no more Communication with Backend has gone down

We have removed i2p and eu access. We traded the EU VPS for this bigger one to run the entire site. Additionally, we're going to start forcing HTTPS. Also we're moving to block open proxies and botnets from posting.

We'll be upgrading LynxChan next.
7 posts and 2 images omitted.


odilitime Board volunteer 06/19/2018 (Tue) 21:59:44 [Preview] No.9340 del
>>9339
because the eu frontend got upgraded and turned into the main endchan server as well as we got rid of the frontend/backend mess.

i2p was removed because I don't think it ever worked right nor did we have the support or tools to test it correctly.


Attn: odilitime Anonymous 06/21/2018 (Thu) 06:29:54 [Preview] No.9344 del
Whiskey tango foxtrot, over.


Addendum Anonymous 06/21/2018 (Thu) 06:34:15 [Preview] No.9346 del
>>9344

'Nam get!


Anonymous 07/05/2018 (Thu) 07:22:13 [Preview] No.9445 del
How much does hosting cost now?


odilitime Board volunteer 07/05/2018 (Thu) 23:27:52 [Preview] No.9446 del
>>9445
It's close to free. We are going to basically give all the donations (after registrar, ssl (yes we pay for non-letsencrypt .xyz cert) and other costs) to the hosting company and they give us what we need.



(33.28 KB 1102x742 Screenshot.png)
(177.44 KB 1102x742 Screenshot 2.png)
(152.24 KB 1102x742 Screenshot 3.png)
Anonymous 06/13/2018 (Wed) 14:02:11 [Preview] No. 9285 [Reply] [Last 50 Posts]
Why am I having severe problems posting on certain boards like /news/ now? See pics. My threads no longer go through.
21 posts and 5 images omitted.


Anonymous 06/24/2018 (Sun) 09:14:46 [Preview] No.9364 del
>>9363
>source for the CAPTCHA module on gitgud if you're so inclined.
I did before I posted >>9338
I'm confident on both you&odili, not so much Lynx.
Hope it's been working out great where you are at. long long time no read.


Anonymous 06/24/2018 (Sun) 14:34:30 [Preview] No.9366 del
I have still been having the 500 error for a while, its really been rough this morning too. I have to copy/paste my intended comments in a utf-8 txt file before posting them just in case the server goes down as I post.


Anonymous 07/02/2018 (Mon) 15:36:53 [Preview] No.9438 del
>>9285
>>9366
I started my browsing today with that error.
Also I post something and it doesn't show up in the thread. On the first page it updates as it should be but not in the thread itself not after several auto and manual refresh. Not even after I close the thread and open it again.
Wtf.


Anonymous 07/02/2018 (Mon) 15:37:47 [Preview] No.9439 del
>>9438
I see it works fine here, not on /kc/.


Anonymous 07/02/2018 (Mon) 15:41:06 [Preview] No.9440 del
>>9439
I checked again and not all thread is affected even there.



(142.07 KB 1102x742 Screenshot - 06102018.png)
bot problem Anonymous 06/11/2018 (Mon) 01:31:26 [Preview] No. 9207 [Reply] [Last 50 Posts]
There are several bots spamming the shit out of random boards (like /test/) again with dick pictures, among other pics. Should the board owner or main site administration enforce global CAPTCHA for a while? I would not mind as long as it keeps the bots away.
90 posts and 25 images omitted.


Anonymous 06/17/2018 (Sun) 13:48:38 [Preview] No.9320 del
>>9319
>why would someone create a problem which justifies the solution they are demanding


Anonymous 06/17/2018 (Sun) 15:11:32 [Preview] No.9323 del
>>9316
He uses exclamations because he's an excitable spaz.


Anonymous 06/17/2018 (Sun) 16:05:14 [Preview] No.9325 del
>>9323
posting on endchan is literally all he has in life left.........


Anonymous 06/18/2018 (Mon) 01:22:29 [Preview] No.9331 del
>>9320
The spammer is a dick and you are blaming someone who is against the bot attacks. But go ahead, keep blaming the victim. Friendly fire is pretty easy as we are all anonymous on here.

>>9323
Nope! Because thats how I express my anger and grief in real life! I shout at people with a rasp voice these days. Don't like it? Too bad!!!

>>9325
I'm old and I get bored at times. I still have a lot to live for, and am very happy with my current life. I wish I could say the same for everyone else here but unfortunately for some that may not be true.


Anonymous 06/18/2018 (Mon) 16:53:22 [Preview] No.9336 del
(17.69 KB 458x305 6.jpg)
>>9331
>Don't like it? Too bad!!!
I'm on your side though. I was saying you're clearly not "the spammer because of exclamations". That's fucking retarded. You're just an excitable spaz. Be glad you're old and excitable. You could be a vegetable.



Anonymous 06/05/2018 (Tue) 02:21:05 [Preview] No. 9105 [Reply] [Last 50 Posts]
Dear Odilitime,

Some jerk just set up a bot on /meme/ and the bot keeps uploading disgusting pictures. This effects the whole image section on the front page. We would like endchan to block these kind of malicious bots, could you please enforce the CAPTCHA for all posts? This would stop this bot in its tracks.

Thank you.
55 posts and 18 images omitted.


Anonymous 06/07/2018 (Thu) 02:07:01 [Preview] No.9174 del
(26.51 KB 367x451 Ah_000621_565910.jpg)
>>9173
LoL. That's not even the log which shows how much of your faggotry I dealt with. Try this one: http://endchan.xyz/.global/logs/2018-06-05.html


Anonymous 06/07/2018 (Thu) 02:22:41 [Preview] No.9175 del
>>9174
thank you for proving my point


Anonymous 06/07/2018 (Thu) 03:28:40 [Preview] No.9181 del
(365.44 KB 2732x2048 3 (2).jpg)


Anonymous 06/09/2018 (Sat) 19:12:23 [Preview] No.9202 del
this site will never gain a community until cancer like OP are run off



Please Help! Anonymous 06/06/2018 (Wed) 17:13:40 [Preview] No. 9159 [Reply] [Last 50 Posts]
Odilitime, I have been having problems with this site since the last 12 hours.

First off, when I tried accessing this site in the morning I got the "502 Bad Gateway nginx" error.

Later on, I was able to access this site. Now my replies are not loading correctly and it seems I can no longer load images or media files either.

This leaves me in a massive rut! When I reply I get the "connection failed" popup, but eventually the reply will show up, if I am lucky. If I try to send another reply it freezes but will post double or triple replies, depending on the number of attempts.

WHAT IN THE NAME OF GOD HAPPENED!?

Proof, see image here: https://www90.zippyshare.com/v/BCizamDz/file.html
9 posts and 1 image omitted.


Anonymous 06/07/2018 (Thu) 01:09:33 [Preview] No.9172 del
>>9171
>not bullying the news aspie to suicide using the official admin tripcode
cmon now


Anonymous 06/07/2018 (Thu) 02:37:13 [Preview] No.9180 del
(66.98 KB 306x457 orang-han.jpg)
>>9170
thanks budy


Anonymous 06/07/2018 (Thu) 11:47:29 [Preview] No.9184 del
>>9170
OK, thanks. Keep up the good work. I'll check and see if I can post normally again every once in a while and wait it out.


Anonymous 06/09/2018 (Sat) 09:19:00 [Preview] No.9190 del
>>9170
Ah, thanks for the info. Without any the users cannot know what's going on, if the board is done for good, how much they should wait or if they should just give up and move to other chans. Or if you even know about the problems we're experiencing (you could be sleeping or working or whatever). Communication is essential. No one wants lengthy novels just a few words so we can keep calm. Uncertainty is shit.


odilitime Board volunteer 06/09/2018 (Sat) 13:56:04 [Preview] No.9195 del
>>9190
Yea, I've been trying to avoid posting when I'm stressed and pissed but you're right you deserve some sort of communication. I'll do my best.



/tech/ thread restoration/archive? Anonymous 06/29/2017 (Thu) 15:24:34 [Preview] No. 6619 [Reply] [Last 50 Posts]
There's a thread that had some useful information for something I'm working on but it has apparently vanished. Strangely, the thread is not in the logs as having been deleted, so did it just fall off due to being really old? It is my fault for not archiving, but I only just came across it about a week ago. Going through the logs it looks like /tech/ cleaned a bunch of shit on June 26, but this particular thread isn't listed anywhere as having been purged.

https://endchan.xyz/tech/res/597.html

Requesting backup or archive if possible. Thanks very much.


Anonymous 06/29/2017 (Thu) 15:29:28 [Preview] No. 6620 del
This thread?

https://web.archive.org/web/20170603035754/http://endchan.xyz/tech/res/597.html

The board was slid by a spammer. Endchan doesn't keep backups.


sage 06/29/2017 (Thu) 16:32:10 [Preview] No. 6621 del
Damn, you saved my ass, didn't even check if it was already archived. I'm an idiot. Thanks dude.


Anonymous 06/07/2018 (Thu) 02:26:21 [Preview] No.9177 del
woah


Anonymous 06/07/2018 (Thu) 02:29:36 [Preview] No.9178 del
valid?


Anonymous 06/07/2018 (Thu) 02:34:44 [Preview] No.9179 del
>>9177
dude woah, wut?



(187.20 KB 614x845 aj.jpeg)
Tor is FUCKED Anonymous 05/10/2018 (Thu) 20:43:30 [Preview] No. 9069 [Reply] [Last 50 Posts]
The globalists are shutting down Tor access to endchan on PURPOSE! The globalists are scum!
2 posts and 1 image omitted.


Anonymous 05/11/2018 (Fri) 17:24:01 [Preview] No.9072 del
I use a private VPN service so I can still access this fine without a need for Tor. Many private VPN services are pretty cheap if you purchase the yearly deal. Cost averages around $50 per year, and you can routinely change IPs anytime you please.


Anonymous 05/11/2018 (Fri) 19:04:03 [Preview] No.9073 del
>>9071
The old Tor address is broken for some reason.


Anonymous 05/15/2018 (Tue) 20:20:25 [Preview] No.9086 del
(42.48 KB 550x550 sad frog.jpeg)
Both onions are down now guise.

The globalists are cracking down.


odilitime Board volunteer 05/16/2018 (Wed) 10:19:03 [Preview] No.9087 del
these don't work:
/etc/init.d/tor start
service tor start

but this works:
after modifying debian-tor shell to bash
su debian-tor
nohup tor &

fuck. anyways it's up now.


Anonymous 06/07/2018 (Thu) 02:24:09 [Preview] No.9176 del
Both onions are down



odilitime Board owner 09/08/2016 (Thu) 05:12:04 [Preview] No. 4983 [Reply] [Last 50 Posts]
I had a development server breached that I had an old development copy of the Endchan database (without media).

All users are advised to change their passwords ASAP.

Development server was breached used an redis/ssh exploit. Redis was installed and usually ran as a user but recently doing some development work, I accidentally started it up as root to look something up and left it running. Redis then can write to your ssh keys and insert unwanted keys and allow root access. All files in /root and /home were removed and a note was left:

>Hi, please view here: http://pastebin.com/raw/vadfLyDS for information on how to obtain your files!

Luckily I have bandwidth logs on that box and I can see there was nothing transferred out of the box. So my guess is they just deleted the files. The nature in which they left the machine leads me to believe this was an automated attack (plenty of other meaningful data directories were left alone).

The copy of Endchan's data is left untouched on this development server. However the dump that was used to transfer the copy was still likely in the /root directory that was deleted. I will get the date of the data copy as soon as I can do some data recovery on that machine, I estimated the copy to be an early 2016 Q2 dump. This server is now offline.

At Endchan, we want to be as transparent as we possibly can and even though we do not believe anything was leaked, we cannot rule out nothing happen with 100%. And even if we could be certain that nothing was at risk, we still want to report anything of this nature to our users.

I fucked up, I'm sorry for any troubles this may and has caused any of you.

Please let us know any questions you may have.
24 posts and 4 images omitted.


Anonymous 05/15/2018 (Tue) 20:11:42 [Preview] No.9085 del
>>9074
Thank you. Let us know when this gets finished and when we can fix these issues. I would gladly re-upload all my video content (all very informative videos) as I have offline backups. Looking forward to this problem being fixed.


Anonymous 05/17/2018 (Thu) 00:01:39 [Preview] No.9089 del
Can the same fix be done for all media files in the near future?


odilitime Board volunteer 05/17/2018 (Thu) 00:04:06 [Preview] No.9090 del
>>9085
fix is in place.

>>9089
No sure what you mean but we don't have all media files. Users need to reupload what they want online.


Anonymous 05/22/2018 (Tue) 21:58:48 [Preview] No.9094 del
>>9090
Does this mean we're capable of now uploading what has been uploaded in the past without image and file corruption?


Anonymous 05/23/2018 (Wed) 08:41:32 [Preview] No.9095 del



Migration odilitime 04/26/2018 (Thu) 07:10:34 [Preview] No. 8874 [Reply] [Last 50 Posts]
I messed up during the migration. While waiting for the final database copy, I thought I'd upgrade/patch some servers, unfortunately I upgraded the box I was copying from. And I rebooted it without thinking about it. Now I manage around 20 servers that run hundreds of containers, so keeping track of everything isn't as easy as if I just had a couple. Also in my day job I get paid to professional move websites all the time. I make good money doing it, so I'm not a complete newb but I have no accounting for how I made such egregious error. I had a plan and just made a mistake.

Last year when we got the dedicated server, we built state of the art encryption into it. To the normal internet it looks like a website (unlike anything like Endchan) but with a special certificate our frontend server have, it runs the Endchan site. We split the drive into 3 sections, the camouflage (fake site / web server and relay), the app (lynxchan, megud, other site utilities), and the database. Then we encrypted the app and data drives. I held the keys to app, and Balrog the keys to data. We did have the foresight to give a back up of these password to other people just incase one of us got hit by a bus. Well Balrog and his backup StephenLynx both failed to find the correct password from a year ago.

So the two events combined means we can't access the latest copy of the Endchan data. So where does that leave us, well we had a full copy from 4/19 that we were syncing with the live data during the migration time. When it stop copying, it left the database in an inconsistent state. The mongo repair tool has done a great job recovering what you see;

we have most of posts but are missing a lot of database files (a great example of why you never store images in a database; we should only be missing the new files from last week on).

I fucked up, Balrog fucked up, and StephenLynx doesn't even remember us giving him the password. If just one of 4 events had gone another direction, we would not have been here.

I proposed throwing in the towel. My recent battles with pedospammers has not been fun but Balrog put it well: "we're still one of the only imageboards worth a damn and people need us goddamn it"

I considering just running the board with no content, even thought about switch imageboard software. Balrog said we should take it to a vote and let you guys decide. But to be honest we recovered more than I was expecting and the lynxchan repair process (rebuilding HTML files) is still running (going to take a long time on these older servers).

To be clear why we migrated, our donations levels were so low, we could no longer afford that server. Our previous server was awesome, a full year of uptime (Another reason why we never needed these passwords because the machine never went down).

In retrospect our set up was probably more complex than needed but it was just an attempt to help secure you're guys data the best we could. Our new drives are no longer encrypted.

Message too long. Click here to view full text.

Edited last time by odilitime on 04/26/2018 (Thu) 17:57:36.
139 posts and 39 images omitted.


Not that I'm complaining about your efforts but, you did ask. And, thank you. Ye olde /r9k/er 05/15/2018 (Tue) 02:56:32 [Preview] No.9080 del
>>9078
>Let me know if you have any problems

TFWNGF


Anonymous 05/15/2018 (Tue) 11:53:30 [Preview] No.9081 del
(275.40 KB 497x373 1488010265931.png)
>>9025
GIVE 76CHAN BACK
RRRRRRRRRRREEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE


Anonymous 05/15/2018 (Tue) 12:01:07 [Preview] No.9082 del
>>9081
https://youtube.com/watch?v=lqcvZ9xypts [Embed]
yu lift budy


Anonymous 05/15/2018 (Tue) 13:19:14 [Preview] No.9083 del
the onions are down


Anonymous 05/15/2018 (Tue) 14:37:25 [Preview] No.9084 del
>>9083
them onions be ally