Virtual JBOD, is it possible?

M76

[H]ardForum Junkie
Joined
Jun 12, 2012
Messages
10,231
I'm not really familiar with how JBOD arrays works, my question is basically this:

Can I do a virtual JBOD where the drives all have physically separate file systems, so they can be accessed regularly as a single drive when connected one by one, and all the files on that particular drive are accessible, but when I connect them all they appear as one large combined storage?

Is there any software that does this for NTFS?

What I want to do is to use all my old and leftover drives as backup storage, but I Don't want the hassle of having to search which drive holds what backup. And since they are drives of various sizes and makes I can't put them in regular raid array, plus even if they were of the same size, a RAID has too many single points of failure for being a viable backup solution.
 

mikey976

[H]ard|Gawd
Joined
Dec 24, 2006
Messages
1,629
Take a look at Stablebit Drivepool, Used it for nearly 10yrs before recently changing to Unraid. will do exactly as you are imagining and then some.
 

acascianelli

[H]ardness Supreme
Joined
Feb 25, 2004
Messages
6,863
Maybe look into LVM?

Wouldn't work like you're describing, but it would be much more maintainable. Add as many disks as you want/can into a volume group, then create logical volumes to span across those disks.

I would recommend against using a RAID array as a backup, but if you absolutely must there are options for making an array more resilient. RAID 5 or even RAID 6 if you have enough drives. Adding a hot spare(s) to either of those will narrow the time frame down where you could potentially have a catastrophic drive failure during rebuild. If you have enough drives/money and controllers to support it, you could use nested RAID like RAID 50/60.

Or just use cloud storage and pay someone else to worry about all of it. :)
 

kdh

Gawd
Joined
Mar 16, 2005
Messages
799
I'm not really familiar with how JBOD arrays works, my question is basically this:

Can I do a virtual JBOD where the drives all have physically separate file systems, so they can be accessed regularly as a single drive when connected one by one, and all the files on that particular drive are accessible, but when I connect them all they appear as one large combined storage?

Is there any software that does this for NTFS?

What I want to do is to use all my old and leftover drives as backup storage, but I Don't want the hassle of having to search which drive holds what backup. And since they are drives of various sizes and makes I can't put them in regular raid array, plus even if they were of the same size, a RAID has too many single points of failure for being a viable backup solution.
Most storage subsystems, and no filesystem actually does this natively. Maybe some epic hacky unreliable setups. NTFS definitely does not do this, and no way LVM supports what the OP is asking at all. Maybe some ZFS, hadoop something or other solution but it would just be stupidly complicated...

@OP... Do not waste your time trying to get something like this to work, you will never have reliable access to your data. Don't hoop jump. Buy 1 drive that's larger then all your smaller drives combined. Put that 1 drive into your machine. Take your smaller drives and attached them to sata -> usb 3 converters. Then copy the data to your one large single drive. Toss the old smaller drives.
 

drescherjm

[H]ardForum Junkie
Joined
Nov 19, 2008
Messages
14,639
Snapraid may help with this depending on your use case.

What you describe is the pooling feature.

https://www.snapraid.it/

Note: you will need additional parity drives to protect from individual drive failures.

With this said you may just want to avoid the extra work and instead buy a few large 8+ TB external drives when they are on sale.
 
Last edited:
  • Like
Reactions: x509
like this

M76

[H]ardForum Junkie
Joined
Jun 12, 2012
Messages
10,231
@OP... Do not waste your time trying to get something like this to work, you will never have reliable access to your data. Don't hoop jump. Buy 1 drive that's larger then all your smaller drives combined. Put that 1 drive into your machine. Take your smaller drives and attached them to sata -> usb 3 converters. Then copy the data to your one large single drive. Toss the old smaller drives.
If I was made of money that is exactly what I'd do. But I'm in Eastern EU, there is no best buy or newegg here who sells external 10-14TB drives for scraps. Best Deal currently for a 10TB drive is $300. And minimum wage is about $600, so that is not happening.
 

Dajinn

[H]Lite
Joined
Aug 18, 2010
Messages
111
I would have to second stablebit drivepool, it is not that expensive at all, and does exactly what you want. It is a nice way to pool the combined total of storage, and using something like snapRaid (which I think is free) can give your 'array' some resiliency. The nice thing about these things are, though, when/if a drive fails, the entire array is not just completely toast. Of course, the main draw back is that you lose the combined throughput, since you are technically just writing to one drive at a time, unless you have multiple transfers in parallel that are writing to each disk. This is how unraid works, and for bulk storage of something like media, it is nice, because if something fails and you lose it, you could theoretically setup a script that keeps a 'log' of what is stored where, so if you have to restore any data, you know exactly what needs to be restored.
 

M76

[H]ardForum Junkie
Joined
Jun 12, 2012
Messages
10,231
It seems that stablebit is exactly what I was looking for, haven't tried it yet, but certainly will. Thanks for the suggestions.
 

M76

[H]ardForum Junkie
Joined
Jun 12, 2012
Messages
10,231
It turns out stablebit drivepool is not exactly as I thought. Files already on the drives are not added to the pool, and there is no option to do so. So it just creates a pool of the empty space on the drives, and the files in the pool are not visible on the individual drives and vica versa.
 

Deadjasper

[H]ard|Gawd
Joined
Oct 28, 2001
Messages
1,759
It turns out stablebit drivepool is not exactly as I thought. Files already on the drives are not added to the pool, and there is no option to do so. So it just creates a pool of the empty space on the drives, and the files in the pool are not visible on the individual drives and vica versa.
So what's the problem with that? Just copy the data to another drive, add it to the pool then copy it back. You can do this bucket brigade style till your pool is complete.
 

M76

[H]ardForum Junkie
Joined
Jun 12, 2012
Messages
10,231
So what's the problem with that? Just copy the data to another drive, add it to the pool then copy it back. You can do this bucket brigade style till your pool is complete.
The problem is that it takes tons of time to move the data, but that's the smaller issue. The bigger one is that I still want the option to access the files on the drives individually when they are not pooled. This doesn't seem possible with it. At least not officially. It stores files in some hidden folder as I've read, but since file placement is controlled by the software it is possible that files that go together are put on 8 different drives in a worst case scenario. ( I prefer sequential filling of the drives, it does some random load balancing)

This way when one of the pool drives fail it is literally impossible to assess what files are affected, I can't just copy one directory that was affected again, I have to refresh the entire pool taking ages because the files are completely randomly placed in the pool, and one drive failure means I loose a bit of everything almost.
 
Last edited:

Deadjasper

[H]ard|Gawd
Joined
Oct 28, 2001
Messages
1,759
The problem is that it takes tons of time to move the data, but that's the smaller issue. The bigger one is that I still want the option to access the files on the drives individually when they are not pooled. This doesn't seem possible with it. At least not officially. It stores files in some hidden folder as I've read, but since file placement is controlled by the software and is not disclosed it is possible that files that go together are put on 8 different drives in a worst case scenario. ( I want sequential filling of the drives, it does some random load balancing it seems)

It doesn't seem clear what happens when one of the pooled drives fails, how that affects the data in the pool.
It's true the files are stored in a hidden folder but you can unhide it and all your files are in that folder untouched. If your pool isn't duplicated and a drive fails, you lose only the files on that drive. The rest of the files are still accessible via the method above. This is why I use DrivePool, all is not lost if a drive fails. Also, files on a pooled drive can be read on any computer, not just the one with DrivePool installed.
 

M76

[H]ardForum Junkie
Joined
Jun 12, 2012
Messages
10,231
It's true the files are stored in a hidden folder but you can unhide it and all your files are in that folder untouched. If your pool isn't duplicated and a drive fails, you lose only the files on that drive. The rest of the files are still accessible via the method above. This is why I use DrivePool, all is not lost if a drive fails. Also, files on a pooled drive can be read on any computer, not just the one with DrivePool installed.
I edited the post, my main problem is that if a drive fails I want the damage to be limited to just a few folders, and not having to deal with randomly missing files everywhere. This makes it not muhc better than simple raid 0 where one drive failure means everything is lost.
 

drescherjm

[H]ardForum Junkie
Joined
Nov 19, 2008
Messages
14,639
I edited the post, my main problem is that if a drive fails I want the damage to be limited to just a few folders, and not having to deal with randomly missing files everywhere
SnapRaid in combination with stablebit should be able to do what you want. I recommend at least 2 parity drives. Note that the parity drives must be at least as large as the largest data drive in your array (probably slightly larger).
 

M76

[H]ardForum Junkie
Joined
Jun 12, 2012
Messages
10,231
There is a plugin that changes standard behavior of the software to fill the drives sequentially. So files that are copied at the same time will likely end up on the same drive, which is much better.
 

kdh

Gawd
Joined
Mar 16, 2005
Messages
799
If I was made of money that is exactly what I'd do. But I'm in Eastern EU, there is no best buy or newegg here who sells external 10-14TB drives for scraps. Best Deal currently for a 10TB drive is $300. And minimum wage is about $600, so that is not happening.
Damn, I'm sorry and I apologize for my tone. I spent a little bit of time searching around looking for an alternative solution to your issue but just couldnt find anything. Unfortunately NTFS wasn't built to smash together multiple different volumes into one common volume while retaining all the data. LVM isn't going to work because it wipes the headers on the drive. Most pooling software is based on free unused chunks of space on the disk so you'd have to shell game the hell out of your data. Unfortunately you'd be locked no into that pooling method with no ability to individually access the data on the drives if you chose to take them out of it one fails. Basically what you are looking for doesn't exist exactly how you want it to be. Fundamentally NTFS just doesn't work the way you want it to work. You'll have to make compromises. SnapRaid/Stablebit but might get you about 80% of the way there. I know its not a 10TB drive, but if your data is that important, you can send it to backblaze for a few dollars a month until you acquire a much larger drive.
 

M76

[H]ardForum Junkie
Joined
Jun 12, 2012
Messages
10,231
Damn, I'm sorry and I apologize for my tone. I spent a little bit of time searching around looking for an alternative solution to your issue but just couldnt find anything. Unfortunately NTFS wasn't built to smash together multiple different volumes into one common volume while retaining all the data. LVM isn't going to work because it wipes the headers on the drive. Most pooling software is based on free unused chunks of space on the disk so you'd have to shell game the hell out of your data. Unfortunately you'd be locked no into that pooling method with no ability to individually access the data on the drives if you chose to take them out of it one fails. Basically what you are looking for doesn't exist exactly how you want it to be. Fundamentally NTFS just doesn't work the way you want it to work. You'll have to make compromises. SnapRaid/Stablebit but might get you about 80% of the way there. I know its not a 10TB drive, but if your data is that important, you can send it to backblaze for a few dollars a month until you acquire a much larger drive.
No need to apologize, I'm used to people assuming everyone is from the continental US here. If anything I was too harsh at the comeback.

Stablebit seems OK, if I can make it behave the way I want it within the 30day trial period I'll probably buy it.

Thanks for the suggestions.
 

M76

[H]ardForum Junkie
Joined
Jun 12, 2012
Messages
10,231
Unfortunately stablebit failed the test. I'm better off using the drives as simple individual drive letters if I'm to have proper control over the file distribution. Honestly putting all the drives into one system already helped a lot in making backups more convenient. And all with parts I had laying around so literally zero cost solution.
 

x509

[H]ard|Gawd
Joined
Sep 20, 2009
Messages
1,930
Unfortunately stablebit failed the test. I'm better off using the drives as simple individual drive letters if I'm to have proper control over the file distribution. Honestly putting all the drives into one system already helped a lot in making backups more convenient. And all with parts I had laying around so literally zero cost solution.
How about using mount points? Disclaimer: I have no experience with this.
 

kdh

Gawd
Joined
Mar 16, 2005
Messages
799
How about using mount points? Disclaimer: I have no experience with this.

Actually.. not a bad idea. Pretty damn simple, and you have the ability remove any mount point you want at any point in time. There is no redundancy or safety net, but you do get the FS layout you want. Not hack at all. The best part, if he does acquire a larger drive to consolidate, this would be one of the most seamless ways to move the data over to a single volume.

https://www.windowscentral.com/how-mount-hard-drive-folder-windows-10
 

TheSmJ

2[H]4U
Joined
Dec 20, 2006
Messages
2,740
Is nobody else using the Drive Pool utility built in to Windows 10? I've been using it since 10's release on my server and never had any issues.
 

drescherjm

[H]ardForum Junkie
Joined
Nov 19, 2008
Messages
14,639
You mean Storage Spaces?

He wanted to keep the disks with separate file systems on them and independent of each other so if a drive dies you only loose what is on the drive the rest of the drives retain their full contents and are able to be used separately if needed.
 

UhClem

Limp Gawd
Joined
Jun 8, 2011
Messages
186
In addition to Mount Points (because Drive_Letters are so ... douchey), (re-)familiarize yourself with Symbolic Links. Now that you are completely controlling the physical locations, and hierarchical layout, of your files, SymLinks allow you to create alternate "virtual" layouts/hierarchies. "Have your cake--and eat it."
 

drescherjm

[H]ardForum Junkie
Joined
Nov 19, 2008
Messages
14,639
You can also use directory junction points

You may be interested in mklink

Code:
x:\x64.18\VC.142>mklink
Creates a symbolic link.

MKLINK [[/D] | [/H] | [/J]] Link Target

        /D      Creates a directory symbolic link.  Default is a file
                symbolic link.
        /H      Creates a hard link instead of a symbolic link.
        /J      Creates a Directory Junction.
        Link    Specifies the new symbolic link name.
        Target  Specifies the path (relative or absolute) that the new link
                refers to.
 

lilfiend

[H]ardness Supreme
Joined
Nov 28, 2008
Messages
6,464
I'm not really familiar with how JBOD arrays works, my question is basically this:

Can I do a virtual JBOD where the drives all have physically separate file systems, so they can be accessed regularly as a single drive when connected one by one, and all the files on that particular drive are accessible, but when I connect them all they appear as one large combined storage?

Is there any software that does this for NTFS?

What I want to do is to use all my old and leftover drives as backup storage, but I Don't want the hassle of having to search which drive holds what backup. And since they are drives of various sizes and makes I can't put them in regular raid array, plus even if they were of the same size, a RAID has too many single points of failure for being a viable backup solution.
You've basically described drive bender. I should have a few really old posts from ~2012 about it. I still use it daily.

upload_2020-1-8_15-30-20.png


I keep the HDD's mounted separate and run drive bender in snapraid mode so I can do parity with snapraid easily. The ssd's have folder level duplication enabled through drive bender for important things like family photos.

If you plan on running multiple arrays on the same PC, I highly recommend using the drive bender console and not the manager. It's still a GUI but its their older one that actually works.
 

M76

[H]ardForum Junkie
Joined
Jun 12, 2012
Messages
10,231
How about using mount points? Disclaimer: I have no experience with this.
I've used mount points before. I just hoped there is a more robust seamless option than that. But I've fallen back on that for now. It works and I know who is who.
 
Top