Thread

And we've hit a bottleneck, we're out of space :joy:

Started by 1 - 28 Dec 2019, 02:29

22Replies
560Views
7Participants

Top posters

FamilyGuyFamilyGuy5 posts
We all knew it was coming, with the never ending amounts of data we have coming from here/the former AG/HiddenPalace/TCRF and all the other groups we're finally at a stage we cannot keep up with the data.

I've been looking into the next best option and feel we should maybe get our own server/NAS for data. Clearly we cannot have this online to the public as we have to respect developers/copyright still. But I'm thinking we maybe do something like betaarchive and have our own servers for the data.

How would users feel about this? I don't mind hosting it home for awhile but my uploads not th fastest so maybe colocated servers would be easier. It would remove the need for multiple file hosting accounts and relying on trust of third party services. We'd control it fully from hardware down to software.

I know other groups are looking into similar options and feel we're at that stage too.

A NAS would cost us around £800 along with about £400/14TB drive (it's like £100 less for 12TB oddly so cost wise I think 12TB is best.) The NAS I was looking at can take 120TB total which would give us years of growth too. if we did a raid setup we'd lose some space but I think around 40TB would cover most of what we have so far along with other bits.

Would anybody be interested in something like this? We'd have to setup some security etc, but it's at the stage there's too much data to manage on a single laptop. I'm also leaning at cold archives like tapes but this is pretty expensive too, more so than a NAS setup.

Please let me/us know your views below if this would be viable.

EDIT: So looking into this more we could get an 8 bay NAS for £600, this one also allows us to expand to 18 drives for an extra £800 while keeping it one system. HDD wise Toshiba enterprise HDD's at 14TB cost £20 more than the 12TB one's and these are around £380/14TB now. We would need least two to start so just to get started this project would need around £1360 with poor download speeds. (due to rubbish UK net ?)

EDIT2: For cold storage LTO-8 is £2500 for the drive alone and around £120/30TB tape. We'd need least 4 here and a cleaning tape too.
8 Edited 28 Dec 2019, 04:12
I like the idea of having a central location to store data. Hosting on sites will only get harder over time as the volume of data grows, and having it all in one unified location seems like a good idea to me. Would definitely be nice to have a RAID setup for redundancy.
2
kabojnk said:
. Would definitely be nice to have a RAID setup for redundancy.


Exactly, this was my thinking too with having it under our own control. We can use RAID etc which while we'll lose some space it'll keep data safer. If we went along cold storage route we can just do this to tape every few months or once a month etc. We'll then have that tape kept offsite / possibly sent encrypted to another staff member for safety.
1
I have never been a fan of tape storage myself.
I'm all for the off-site storage. As far as colocation goes, hit me up once you think you want to do that. My boss may have some room in the colo where I work, plus I can always ship something in a mid-tower form to Joe's Datacenter and have it hosted there for about $50-60/month, assuming we can keep it under 33TB/month and under 120W. Unmetered 1GBps connection is around $350 a month, which is a bit steep. Locally however I have been quoted $240 for 2U/1kW/unlimited bandwidth if I read the contract right.

I definitely also wouldn't mind being boots-on-ground for the tape swapping and back-up, there are a few other OG/old AG members around me as well that I bet would help. I've done lots of research on colo here in the US, so let me know if I can do anything for you!
1 Edited 28 Dec 2019, 06:16
The nas devices you are looking at sound expensive. Why not buy a microserver, they have 4 bays and are cheap and more flexible.
2
kabojnk said:
I have never been a fan of tape storage myself.


Tape would only be used for cold storage/long term without being online all the time. Ideally once a month tapes would be rotated with one being kept away in a safe or something.

Bad_Ad84 said:
The nas devices you are looking at sound expensive. Why not buy a microserver, they have 4 bays and are cheap and more flexible.


Unfortunately 4 BAY is too small for the data we have, even 2x 14TB isn't going to last a month once we sort just the data I have scattered here, The HW in question would be a Synology 8 BAY NAS with further support for another 10 bays upto 120TB (I think even that can be expanded if needed). This would future proof our storage needs without the need to rely on third partys for hardware/hosting, even the need to change HW for a few years atleast. It gives us full control with a system already setup out the box with encryption, file sharing features and even an option to link with another offsite NAS should we need. Meaning we won't need to dick around with custom linux configs that are prone to crashing if you breath.

If we were to just backup all existing betas I reckon it'll be around 20TB so really we'd need 4x 14TB and then backup storage for that. This is where the tape storage would come into play, We'd do 2 tapes/month with two already stored offsite. Each month the old two would be brought back, updated and then the newer tapes would be taken offsite and so on. I should mention this 20TB wouldn't cover archives I've been sent in private etc, alot of this can't be shared due to developers/companies asking for data to be kept private for the time being. With that, we'd be looking maybe 80TB.

There's so much already lost and the idea here would be to avoid any third party issues, along with maintaning full control over our hardware/data with also giving users access to it as needed.

Even reading this back sounds insane, 80TB before backups / redundancy, and I fear even this won't last a year ? I think I'm gonna further research and see if there's any old enterprise multibay servers that are cheap second hand. Drives clearly would be new but maybe older enterprise hw is best.

I do have an older Synology NAS in the mail along with a 3TB HDD I snagged for £9 as the USB port was snapped, I'll be doing some tests over the coming weeks to see what sort of speeds people would get if the NAS was hosted from home, if not I'll be calling the data center that's in norwich.

Simply putting it, commercial stuff isn't going to cut what we have already and alot of that's still not backed up properly. Enterprise level HDD's etc is where we're at as a whole with preservation. And even the current enterprise HDD's aren't big enough for us, We will need cold archives in which case tape is the only option and even this is around £120/30TB.
Better bang/buck would be to buy a used server or build a custom one, running Linux/unraid.

See http://servethehome.com

Those rack are also sweet and allow tons of drives, but are expensive if you have only a few drives: https://www.45drives.com/
1 Edited 28 Dec 2019, 17:13
If you're willing to spend more, or maybe get enough donations, may I suggest building a Backblaze Storage Pod?
You get 60 drive "bays", so can easily expand later without having to buy another device, and you can run whatever OS you want on it (you're not forced to whatever the NAS vendor uses):
https://www.backblaze.com/blog/open-source-data-storage-server/
https://f001.backblazeb2.com/file/Backblaze_Blog/Storage-Pod-6/Storage+Pod+6.0+Parts+List.pdf?_ga=2.53972655.680075291.1577570215-1897185910.1577570215
2
I have 8 drives in my microserver, as you can just buy external sas enclosures to sit next to it...

The whole Internet runs on Linux. I think your "custom configs that crash if you breath on them" is just scare mongering. The device you are talking about using is just Linux and open source software with a custom gui after all.

I have 4x 12tb drives and 4x 3tb, all of which are enterprise sas drives and this is just for a home setup. But what do I know about it.

Commercial stuff certainly will cut it. This amount of data is peanuts in a lot of enterprise situations.

I think you are certainly better off looking for older commercial hardware or building something custom with sata cards. Retail drives are priced significantly lower than commercial drives and with the right amount of drives and redundancy, you will get better storage and redundancy from the sata drives (it's exactly what back blaze do for the same reasons).
2 Edited 28 Dec 2019, 22:32
Bad_Ad84 said:
I have 8 drives in my microserver, as you can just buy external sas enclosures to sit next to it...

The whole Internet runs on Linux. I think your "custom configs that crash if you breath on them" is just scare mongering. The device you are talking about using is just Linux and open source software with a custom gui after all.

I have 4x 12tb drives and 4x 3tb, all of which are enterprise sas drives and this is just for a home setup. But what do I know about it.

Commercial stuff certainly will cut it. This amount of data is peanuts in a lot of enterprise situations.

I think you are certainly better off looking for older commercial hardware or building something custom with sata cards. Retail drives are priced significantly lower than commercial drives and with the right amount of drives and redundancy, you will get better storage and redundancy from the sata drives (it's exactly what back blaze do for the same reasons).

Yeah Linux is the way to go. It might have a steepier leaning curve, but it's the most robust and flexible.

In you're afraid to break stuff, unraid or FreeNAS are already well optimized for data sharing/VMs.

I personally have a Nas for my modest home setup, as it was a great deal for the drives included. But I do all of the sharing and management through a Linux box for ease of mind and flexibility.
Edited 28 Dec 2019, 23:06
If you are going with tape drives, I would recommend using older LTO since they are cheap used.
FamilyGuy said:
Those rack are also sweet and allow tons of drives, but are expensive if you have only a few drives: https://www.45drives.com/


I did look at 45 drives but sadly this is way too much, even my option was only $100 more and even manages to get 2x 14TB plus it gives us 5 more bays to use meaning no need to upgrade for many years.

45drives 15 bay NAS (includes 120GB BOOT SSD) > $2,735 before taxes/import fee's

Synology DS1815+ 8BAY NAS £578.84 + DX1215 12 Bay Expansion Unit £858.99 = $1,880.78 including taxes/shipping. Toshiba 14TB Enterprise MG07ACA14TE 2x £383.99 = $1,004.57 total for NAS's/2x 14TB: $2885.35

We could also use the Synology DS2419+ which is £1,239.99 and with the DX1215 at £858.99 giving us a total of 22 bays for $2,745.60. This one can also be expanded with 10gb network cards etc.

The Synology OS while based on linux is presetup, no need to configure, it allows user access control, encryption and now need to mess with configs/commandline. It allows file access by links/domain and can even have users upload to it, It's also used by other groups meaning we can sync our data for offsite backups. Itself even offering encryption.

pool7 said:
If you're willing to spend more, or maybe get enough donations, may I suggest building a Backblaze Storage Pod?
You get 60 drive "bays", so can easily expand later without having to buy another device, and you can run whatever OS you want on it (you're not forced to whatever the NAS vendor uses):
https://www.backblaze.com/blog/open-source-data-storage-server/
https://f001.backblazeb2.com/file/Backblaze_Blog/Storage-Pod-6/Storage+Pod+6.0+Parts+List.pdf?_ga=2.53972655.680075291.1577570215-1897185910.1577570215


Backblaze is something I've looked into before but the costs of it outweight the pros, the basic cost is $12,849.40 before adding drives/paying taxes.

Bad_Ad84 said:
I have 8 drives in my microserver, as you can just buy external sas enclosures to sit next to it...

The whole Internet runs on Linux. I think your "custom configs that crash if you breath on them" is just scare mongering. The device you are talking about using is just Linux and open source software with a custom gui after all.

I have 4x 12tb drives and 4x 3tb, all of which are enterprise sas drives and this is just for a home setup. But what do I know about it.

Commercial stuff certainly will cut it. This amount of data is peanuts in a lot of enterprise situations.

I think you are certainly better off looking for older commercial hardware or building something custom with sata cards. Retail drives are priced significantly lower than commercial drives and with the right amount of drives and redundancy, you will get better storage and redundancy from the sata drives (it's exactly what back blaze do for the same reasons).


The idea here is to buy once and be future proof for a few years while keeping same hardware. It's also easier with the Synology setup as our goal is to have users access older data from it. Out the box it offers everything we need and is cheaper than most options. We're already passed the 20TB mark and the Synology units can be expanded by 12 drives a time, this doesn' require formatting or anything, we literally plug the unit/drives in and it's ready to use and already part of the existing storage. We can add one/two/4 drives a time and it won't effect current data with no downtime/messing with rebuilding raid configs etc. Literally it's plug and play deal. There's nothing to install at all minus the first time install of the OS th rest is there once installed.

The comment about linux was due to OG's server bricking a few months back when MySQL changed. I spent hours rebuilding the server manaully as it corrupted everything. With a system like Synology there's no need to worry as much as it's tested before being pushed out.

FamilyGuy said:
Yeah Linux is the way to go. It might have a steepier leaning curve, but it's the most robust and flexible.

In you're afraid to break stuff, unraid or FreeNAS are already well optimized for data sharing/VMs.

I personally have a Nas for my modest home setup, as it was a great deal for the drives included. But I do all of the sharing and management through a Linux box for ease of mind and flexibility.


It'll still be linux based just less issues than a normal linux install.


The main goal of this project was to avoid all third party hosts which have proven unreliable/untrust worthy over the years. None have proven trust worthy, MS used to offer unlimited then that was gone, Mega used to offer 50GB and now that's 15GB with data removal something they promised wouldn't be done. Mediafire is a shitfest and has a buggy UI. We've lost lots from rapidshare closing, from the us goverment being retarded and stealing Megaupload and Mega.

Least if we control our own HW/setups we can link to other groups using same hardware, we can host the data for users without the worry some file hosting company is going to delete it as they're not making enough money.

Google drive business was proven not to be unlimited by a youtuber a few months back and I think it was limited at 10TB/day or something small like that.

Clealry my net couldn't manage uploading all this. We're already in the 80TB ranges once everything is transferred. I have another collector wishing to send a private archive of all their stuff too which is another 12TB alone there. This is why a 4 bay or smaller system isn't going to help us long term. If we're going to do this it has to be done right the first time, The bonus about using Synology too is we don't even have to fork out the £860 for the 12 bay expansion straight away as it'll take awhile to transfer existing data to the 8/12 drives. Literally everything we need is out the box, user access, upload options, encryption, backup options with remote nas's etc and I know a certain person won't mind syncing up with us.

While not as cheap as a home server long term is easier to manage this and also gives us peace of mind that we're not going to wakeup having to download 80TB of data. With proper cold storage options in place we can manage this. LTO-8 while expensive will come down in price once LTO-9/10 etc is out. And with this setup to allow users to dl/access we'll have offsite backups here too.
I'd really check the home server with FreeNAS/unraid option then.

It'll be cheaper and the NAS-specific focus of those OS are basically what you'd get with a Synology anyways.

With ZFS under FreeNAS or using unraid, you can also expand with new disks very really just as you described with the Synology.

45 drives are really only worth it when you actually plan to fill them eventually. IIRC each drive gets full sata and the included server is good.

You can do an off-site backup easily using syncthing, highly recommended.

If you want to pay for the simplicity/peace of mind of Synology, that's alright. But before buying anything, you should gather an old PC with old hard drives and create a FreeNAS/unraid testbench/sandbox to get a firsthand idea of the actual user experience with those.
Edited 29 Dec 2019, 18:35
FamilyGuy said:
I'd really check the home server with FreeNAS/unraid option then.

It'll be cheaper and the NAS-specific focus of those OS are basically what you'd get with a Synology anyways.

With ZFS under FreeNAS or using unraid, you can also expand with new disks very really just as you described with the Synology.

45 drives are really only worth it when you actually plan to fill them eventually. IIRC each drive gets full sata and the included server is good.

You can do an off-site backup easily using syncthing, highly recommended.

If you want to pay for the simplicity/peace of mind of Synology, that's alright. But before buying anything, you should gather an old PC with old hard drives and create a FreeNAS/unraid testbench/sandbox to get a firsthand idea of the actual user experience with those.


I've already tried these options in the past, and keeping things as simple and inline with what the rest are doing just doesn't cope as good. Most are using Synology because it's simple and almost bullet proof. Compared to other NAS options I've had 90% less hassle with bad configs, and the out of box user access/file sharing makes it more easy. Out the box we have encryption, user accounts which can be tied into OG's API I believe, we've got options to keep data seperate along with allowing certain users to directly upload to the system. We can expand as needed. And the main part, it's secure and doesn't require multiple external drives being hooked up, we only need 5 cables, 2 PSU, 2 ethernet, 1 link connector that doesn't use the ethernets connection to speed up HDD speed.

I've also yet to find a desktop style system I can tuck away and forget about that supports more than 8 drives. Cost wise, while Synology isn't the cheapest, but it's cheaper long term for us and offers us the storage we need faster. It also allow me to copy and paste all the data here faster too than having to connect to commandline crap which I'm trying to avoid. I cannot deal with command line at all it screws my head. With the expansion kit we can go from the base 112TB to 280TB total giving us least 5 years before we're in the need to upgrade again. We'll then be able to archive almost everything public along with alot of stuff we can't post public due to respecting developers etc.

EDIT: Fogot to mention but it does also cover almost all file services users use here meaning we can directly add content too without having to worry about leaving backdoors too.
As I said, if your reason for opting for Synology is peace of mind, fair enough, but it isn't the best bang for bucks if you're willing to learn. Compromises.


Damien said:
having to connect to commandline crap which I'm trying to avoid

I find your lack of faith disturbing.
As someone who runs a website, you should really get at least comfortable with the command line. It is daunting, true, but it is the best tool you've ever used. A Swiss Army knife on which each tool is a full-size version.

You know when you can't find an option in am abysmal GUI? Or when you know something should appear in a drop down list but doesn't? Or when there's just too many variables and your config page looks like a registry editor? The terminal is your best friend in those situations.

You can hit me up if you every need help in that department.
1
FamilyGuy said:
As I said, if your reason for opting for Synology is peace of mind, fair enough, but it isn't the best bang for bucks if you're willing to learn. Compromises.


This is the main reason yes, keeping it simple and using the same setup as the rest making it easier to share/merge archives etc. Running different systems can cause issues with filenames etc and one system may not understand a format etc.

FamilyGuy said:
I find your lack of faith disturbing.
As someone who runs a website, you should really get at least comfortable with the command line. It is daunting, true, but it is the best tool you've ever used. A Swiss Army knife on which each tool is a full-size version.

You know when you can't find an option in am abysmal GUI? Or when you know something should appear in a drop down list but doesn't? Or when there's just too many variables and your config page looks like a registry editor? The terminal is your best friend in those situations.

You can hit me up if you every need help in that department.


It's due to having Aspergers, Not sure why but for me "simple" is hard and the lack of color/too basic view screws with my head which is why I don't code etc. My brain can't think "simple" enough ? I just stick to why I know and can do and try to avoid anything that I could break, and I don't fancy risking hundreds of gigs of history to this.

For our needs server wise I'm fully able to manage this, but for large amounts of data like this I want it as simple as possible while maintaining ease of use to end users. I don't want to do something like betaarchive's ftp I want users to be able to see what they need on a web page etc. While also allowing users to upload etc and have it checked before going live, Synology covers this and with a few tweaks we can connect them to our API on OG.

I do understand their is better systems, I mean my NAS in France is using Debian, My dedi's use CentOS etc but the amount of issues I've had with linux breaking due to updates is a headache I want to avoid. Example with the MySQL update a few months back that bricked all but one of our servers and hundreds more as they changed 1 line of friggin code. It cost me hours in time manually rebuilding apache etc as I had to purge the lot and restart using bits from the old setup.

My paranoia of being updated doesn't help with that too, I cannot stand leaving a system not updated. I guess I got that from the 6+ years of repairing sites that had been hacked as they're not updated. The headaches of updating to newest versions normally resolve that need.
1
I see all of your points and they're fair. Although I don't agree with them all, it's a matter of personal preferences so who cares in the end. At long as the solution works and is robust.

Damien said:
Example with the MySQL update a few months back that bricked all but one of our servers and hundreds more as they changed 1 line of friggin code

That has nothing to do with Linux though.
FamilyGuy said:
That has nothing to do with Linux though.


Correct, just guess that all puts me off running straight up linux setup. I know it's more usable that way though. I just want to keep this all basic as possible while offering much as possible out the box including extra expansion with minimal costs.
Damien said:
Backblaze is something I've looked into before but the costs of it outweight the pros, the basic cost is $12,849.40 before adding drives/paying taxes.

The Parts List PDF I linked to says otherwise: $3494 (and you can go with a cheaper processor/motherboard and less RAM if you want it even cheaper).
In any case, it's not ideal in terms of you having to build it yourself.
Anyway, I'm not pushing it, just wanted to clarify regarding cost.
2 Edited 29 Dec 2019, 21:57