Raid 10 performance?

[Deleted member 330132]

Warthunder file: 3,384 files and 33.7gb(33.5 on disk)

That makes the average file size 10mb. So, 2mb chunk size?

Couldn't they logically make write on raid 10 write across all 6 disks and then do the redundancy like a file sync? It could split it across all disks and both stripes leaving a logical area to sync for the initial write. Then it could run sync the rest of the data after the fact in the form of a sync like activity. It could leave data vulnerable and hold up data, but it could get initial performance for home computers or certain applications. The data placed correctly across both stripes in the correct pattern could act like a mirror in various ways depending on the setup. This would make it write in the pattern of n1 instead of n2 with time afterwords to sync as long as the data is healthy. It would just have to read across both stripes intelligently. It could be useful for things with an internet or exterior check ability on the files afterwords like with steam or other offsite file backups or downloads. Maybe it could use an onsite backup like cache to hold commonly used data chunks for checking if desired.

That or make all file transfers a single large file and copy/check the old file(s) before syncing or deleting the origin of the copy job. You could move any checking of file bits till after the transfer and maintian the files physically in a large file. Maybe with data to find individual components by other software. The software could contain stop start data or have it added to the file by the file system.

I forgot. I read something about turning of or on direct something with file transfers to speed up writes. I wasn't sure what this was. If this is the write cache is it useful to turn off write cache for certain things?

 

[lopoetve]

It’s not raid or the filesystem; at least not precisely. Every 10MB it has to close a file, update metadata and access times and the like, update the Btree (or whatever BTRFS uses for a tracking system), find the next file on the source, open the destination handle, and start the transfer. You’re spending more time on all of those tasks than you are copying, as a copy is single threaded, which is reducing your throughput.

If you tried this with 100 500 MB files you’d see a speed somewhere between your folder test and the 50G test- but closer to the 50G one. Might have to look close but it would be there. You’re limited by IOPS during the metadata updates, and that those just take time.

The more files, the more updates there are and the more things it does other than actually moving data. Can you tune that? Possibly. Write caching on SSD or having SSD as a target would help, if the filesystem can do that, and turning off things like atime/ etc might too. But then again, how often are you copying 3400 10MB files?

You see different speeds because it isn’t actually 3400 10MB files. Some are bigger, and those pick up overall in speed, while others are batches of even smaller and waste more time.

It really is what it is. Not sure if this is really worth fighting with anymore on your side. The large block copy showed that the disks are working fine.

 

[lopoetve]

Oh, and there are systems that do what you describe. Most are non-free that I’m aware of, and generally enterprise grade.

Consumer wise, people buy an SSD and just tend to ignore it. Raid of spinners for anything other than reliability is a dead market.

 

[Deleted member 330132]

How much would one cost and do you know the name of any. I'm assuming it's not cheap. I'm technically using enterprise grade drives if it matters.

And who would not want those commonly used?! If it's money there have to be other ways to make money and still have them available for home users or other categories. I might pay 25 dollars for long use software. Especially if it has only upgrades for security and whatnot(IE minimal). Features would depend. My problem with most major software is expense. I might consider a version of windows without all the crap if it was near 25 bucks. Although I would want it more like linux out of the box.

Price addons for features are easy to do. Especially if it's relatively cheap and is based on things like what is needed for future hardware features as opposed to old. And they could always package in different ways. Different markets/eras could simply have different packages(maybe like cheap dlc's with cheap base software 10 buck or less.). Cheap minimal support or no support could work. Especially if the support is just going to a forum or having to use outside forums. They would just need to remove the costs on their ends.

Base software could be 10 bucks with special packages adding to the cost. Then you could buy more stuff like a dlc to add to your options. This could be split up as much as possible to make it as versatile to the user as possible. More expensive packages or monthly/yearly things could be added for phone or internet paid support etc with different prices for different arenas like business vs enterprise vs home etc. they could things like remote setup for various setups if a business doesn't want the hassle or for enterprise if they need support too. Although that might be worrisome knowing it can do it. I'd have it be a tag you can uninstall and confirm isn't usable after. Maybe if more linus repository like things are used it could be optional everywhere for security reasons. (some of the dlcs could be types of raid with specilized or funky aspects like maximising ecc or something. Or Sram for those who want to have fun with read enhanced ram. Or different more specific logic.)

Free is better though obviously. Or as cheap as possible. Especially if the free is the old hardly used rust disk version!! 8D

I still want to find a way to do asynchronous multi write across different drives for specific folders or applications. Then you could live update or treat it like a backup when done slower. Or use a backup to do live updates when needed. I have one program I would love to do that with.

 

[cjcox]

Consumer wise, people buy an SSD and just tend to ignore it. Raid of spinners for anything other than reliability is a dead market.

other than.... reliability, cost and density...

All of those things might be interesting even in the consumer market.

 

[Deleted member 330132]

Does anyone have any idea why my empty btrfs raid 10 shows 250gb used and only 500gb free. It's stopping me from writing to the disk over the amount linux thinks it's using. Assuming it's not.

$ sudo btrfs fi show
Label: 'Storage'  uuid: 87f48086-2bab-4e89-b04c-ff338fe263eb
        Total devices 6 FS bytes used 960.00KiB
        devid    1 size 233.82GiB used 1.17GiB path /dev/sdc
        devid    2 size 233.82GiB used 1.17GiB path /dev/sdd
        devid    3 size 233.82GiB used 1.17GiB path /dev/sde
        devid    4 size 233.82GiB used 1.17GiB path /dev/sdf
        devid    5 size 233.82GiB used 1.17GiB path /dev/sdg
        devid    6 size 233.82GiB used 1.17GiB path /dev/sdh

$ btrfs fi df /mnt/Storage
Data, RAID10: total=3.00GiB, used=832.00KiB
System, RAID10: total=7.88MiB, used=16.00KiB
Metadata, RAID10: total=511.88MiB, used=112.00KiB
GlobalReserve, single: total=3.25MiB, used=0.00B

$ sudo btrfs fi usage /mnt/Storage
Overall:
    Device size:                   1.37TiB
    Device allocated:              7.01GiB
    Device unallocated:            1.36TiB
    Device missing:                  0.00B
    Used:                          1.88MiB
    Free (estimated):            700.94GiB      (min: 700.94GiB)
    Data ratio:                       2.00
    Metadata ratio:                   2.00
    Global reserve:                3.25MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,RAID10: Size:3.00GiB, Used:832.00KiB (0.03%)
   /dev/sdc        1.00GiB
   /dev/sdd        1.00GiB
   /dev/sde        1.00GiB
   /dev/sdf        1.00GiB
   /dev/sdg        1.00GiB
   /dev/sdh        1.00GiB

Metadata,RAID10: Size:511.88MiB, Used:112.00KiB (0.02%)
   /dev/sdc      170.62MiB
   /dev/sdd      170.62MiB
   /dev/sde      170.62MiB
   /dev/sdf      170.62MiB
   /dev/sdg      170.62MiB
   /dev/sdh      170.62MiB

System,RAID10: Size:7.88MiB, Used:16.00KiB (0.20%)
   /dev/sdc        2.62MiB
   /dev/sdd        2.62MiB
   /dev/sde        2.62MiB
   /dev/sdf        2.62MiB
   /dev/sdg        2.62MiB
   /dev/sdh        2.62MiB

Unallocated:
   /dev/sdc      232.65GiB
   /dev/sdd      232.65GiB
   /dev/sde      232.65GiB
   /dev/sdf      232.65GiB
   /dev/sdg      232.65GiB
   /dev/sdh      232.65GiB

I rebalanced it and it took a few miliseconds:

$ sudo btrfs filesystem balance /mnt/Storage
Done, had to relocate 3 out of 3 chunks

$ sudo btrfs filesystem usage /mnt/Storage
Overall:
    Device size:                   1.37TiB
    Device allocated:              8.25GiB
    Device unallocated:            1.36TiB
    Device missing:                  0.00B
    Used:                          3.75MiB
    Free (estimated):            700.33GiB      (min: 700.33GiB)
    Data ratio:                       2.00
    Metadata ratio:                   2.00
    Global reserve:                3.25MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,RAID10: Size:3.00GiB, Used:1.75MiB (0.06%)
   /dev/sdc        1.00GiB
   /dev/sdd        1.00GiB
   /dev/sde        1.00GiB
   /dev/sdf        1.00GiB
   /dev/sdg        1.00GiB
   /dev/sdh        1.00GiB

Metadata,RAID10: Size:1.03GiB, Used:112.00KiB (0.01%)
   /dev/sdc      352.00MiB
   /dev/sdd      352.00MiB
   /dev/sde      352.00MiB
   /dev/sdf      352.00MiB
   /dev/sdg      352.00MiB
   /dev/sdh      352.00MiB

System,RAID10: Size:96.00MiB, Used:16.00KiB (0.02%)
   /dev/sdc       32.00MiB
   /dev/sdd       32.00MiB
   /dev/sde       32.00MiB
   /dev/sdf       32.00MiB
   /dev/sdg       32.00MiB
   /dev/sdh       32.00MiB

Unallocated:
   /dev/sdc      232.44GiB
   /dev/sdd      232.44GiB
   /dev/sde      232.44GiB
   /dev/sdf      232.44GiB
   /dev/sdg      232.44GiB
   /dev/sdh      232.44GiB

Do, I just need to zero the disks so it doesn't think they are used if it's an allocation issue. Or can i defrag them to get the data in a good spot.

With the ideal raid 10 idea. I was thinking you could probably have flags added to data or similar that increases file size a bit, but heavy use of ram could allow pre structuring of data for writes. You just need to get the flub away from the actual write function and then have it run ideally with various abilities to allocate the other functions to various times/resources around it. Then you could hypothetically get max read/write. But at the cost of space. and I would imagine flags or other markers could be small. And the idea of all data being structured into large files could use the flag with start stop flags for the entire thing or similar. That could also lead to ideal write location for read heads. Maybe load meta data into ram.

Speaking of which could I load my meta data into ram and make it not access it during the write functions but let it load it later into the drive after the main write function. Or added onto the write function in a chunk that can be written efficiently with the rest of the file basically or literally as a part of it's write job?

Then you would just need a read job of the entire metadata and organization in ram with the cpu or something to organize a write function and try to get rid of all other sloppy tasks so the write job is pure organized write(with an extra update to the metadata at the end potentially.). I think that would be near my dream raid10. The use of copy to not delete and double check the copy could be used for redundancy or cut checking before delete of old data.

Do I want a ramdisk installed while using btrfs in general? I had a tmpfs but remove it.

Edit: you could also make a raid 10 do all sync data via other sources and in essence do enhanced raid 0. It could be raid 0 with data patterns that can be simulated with faster resources on demand. Those data patterns, if uncorrupted, could be used to give raid 0 redundancy. Then you could take the actual writes from my raid 10 idea and run loose with them to double capacity also. While still allowing some minimal disk failure/rebuilding. This could be good for ssd's also as you could simulate raid space in ram and cpu on demand and lessen writes to the ssd. Assuming the accompanying hardware is strong enough. This might be more useful if we ever get more powerful computers like optical processing and whatnot where you may have near unlimited throughput or redundant task speed to do odd task like this. Optical may allow the use of small visual objects to be unpackaged on demand into larger files for use with smaller network space because of raw processing. This could also change the normal ratio of storage to computing power changing what is possible. So older hdd's might still be useful. Depends on how they are designed. You could also possibly run things like n3 or more stripes without loosing disk space if you have the accompanying power/drives to aid in using partial wider stripes across more drives. This could lead to further potentially lossless functionality. If you ran a 3/6 disk raid with 3 stripes could you use this for double rebuilding with massive compute power for even more redundancy on a single layer? Can't you hypothetically just spread the stripe data out more and then calculate like if you had stored it with sufficient calculation power? I wonder if that could lead to disk with internal raid like structures. Assuming that helps. I think at it's peak as long as you can live simulate the other parts of the disk you can avoid keeping them in writes on the drive. If the rest of the system is faster you have ideal raid10/0 as long as you can keep it's data correct via other means than writes or data/space usage. Then you could have drives ideally made for raid and throughput. Change or amend magnetic with optical and maybe you have something better than ssd in all categories. Assuming other things aren't superior. And that it can't have material or other changes to outdo those things. Since it could be the basis for optimized throughput devices it could be the basis for other different hdd like devices later. Especially it later things like optical allow for modifications of devices and not just single pre made objects. Or at least relatively more modifiable. Different and or more complex geometry could be used in endless combinations potentially. Optical geometric processors could be interesting. You could even use accurate or inaccurate shapes with pre calibrations and be used for both processing and other current functions in essence. It depends on the relative power to the function running through it. Getting a bit off track here. Either way, more processing power could hypothetically lead to simulated layers leading to greater and greater ideal lossless raid performance. with more power and more drives or innter tech in the drives you could increase the amount of redundancy per package in various ways to keep ideal functioning. You could even make a single package multiple drives with different power/connectors and possibly extra internal processing to alleviate cpu/ram or other functions at some point if needed. Just need a way to repair/replace. This could help as processing/physical needs fluctuation relative to each other and needed packaging. Lots of new fun things could be done though. If I'm not mistaken. this is where increased mod ability becomes more useful over time. Especially as optical can remove the double reality of connection in most or all of the system from also being data points. This increase potential versatility of a systems parts. I'm pretty sure optical will potentially lead to pure agnostic hardware/software realities to every single tiny point of each if desired. Or is that the opposite of agnostic. If both can be agnostic then I think it doesn't matter logically as they can both be developed to each other perfectly as desired. Hardware can be changed to software and vice versa. Or any other outcome as it is all changeable at that point. That and the potential endless ability to parallelize things. On some of these notes I wonder if you could go from 2d raids to 3d raids and increase the redundancy more. 9 disk could be more interesting. And what about 4d or higher? This could increase striping and functionality to even more interesting things. You could even gain space and gain functions you don't have now. Or at least the room for them.

 

[lopoetve]

other than.... reliability, cost and density...

All of those things might be interesting even in the consumer market.

Not enough for someone to write the software; ceph or gluster might do this, but I’ve avoided those rat holes for a reason. Async writes are extremely dangerous, and even then, metadata still has to be written.

Thinking in my head, ZFS with a very fast ZIL might pull it off- but we’re still talking metadata updates, and some of that is going to be processing time regardless of media.

Either way- reliability and density don’t imply performance, especially of metadata.

Fast, affordable, reliable. Pick two, not 3.

 

[lopoetve]

How much would one cost and do you know the name of any. I'm assuming it's not cheap. I'm technically using enterprise grade drives if it matters.

And who would not want those commonly used?! If it's money there have to be other ways to make money and still have them available for home users or other categories. I might pay 25 dollars for long use software. Especially if it has only upgrades for security and whatnot(IE minimal). Features would depend. My problem with most major software is expense. I might consider a version of windows without all the crap if it was near 25 bucks. Although I would want it more like linux out of the box.

Price addons for features are easy to do. Especially if it's relatively cheap and is based on things like what is needed for future hardware features as opposed to old. And they could always package in different ways. Different markets/eras could simply have different packages(maybe like cheap dlc's with cheap base software 10 buck or less.). Cheap minimal support or no support could work. Especially if the support is just going to a forum or having to use outside forums. They would just need to remove the costs on their ends.

Base software could be 10 bucks with special packages adding to the cost. Then you could buy more stuff like a dlc to add to your options. This could be split up as much as possible to make it as versatile to the user as possible. More expensive packages or monthly/yearly things could be added for phone or internet paid support etc with different prices for different arenas like business vs enterprise vs home etc. they could things like remote setup for various setups if a business doesn't want the hassle or for enterprise if they need support too. Although that might be worrisome knowing it can do it. I'd have it be a tag you can uninstall and confirm isn't usable after. Maybe if more linus repository like things are used it could be optional everywhere for security reasons. (some of the dlcs could be types of raid with specilized or funky aspects like maximising ecc or something. Or Sram for those who want to have fun with read enhanced ram. Or different more specific logic.)

Free is better though obviously. Or as cheap as possible. Especially if the free is the old hardly used rust disk version!! 8D

I still want to find a way to do asynchronous multi write across different drives for specific folders or applications. Then you could live update or treat it like a backup when done slower. Or use a backup to do live updates when needed. I have one program I would love to do that with.

For the ones I’m thinking, a couple of hundred thousand to start. Plus support. There aren’t many systems out there that try to optimize file ops like this- too much is dependent on metadata updates and that’s not something you can control.

Many small files sucks for almost any filesystem. Faster media is what you need- a spinner is good for 100 iops. Nvme is good for several hundred thousand.

 

[lopoetve]

Do, I just need to zero the disks so it doesn't think they are used if it's an allocation issue. Or can i defrag them to get the data in a good spot.

No idea, don't use BTRFS enough - might have to see if someone else knows.

With the ideal raid 10 idea. I was thinking you could probably have flags added to data or similar that increases file size a bit, but heavy use of ram could allow pre structuring of data for writes. You just need to get the flub away from the actual write function and then have it run ideally with various abilities to allocate the other functions to various times/resources around it. Then you could hypothetically get max read/write. But at the cost of space. and I would imagine flags or other markers could be small. And the idea of all data being structured into large files could use the flag with start stop flags for the entire thing or similar. That could also lead to ideal write location for read heads. Maybe load meta data into ram.

Extremely dangerous. Any power failure or system failure would result in corruption of the metadata and filesystem and a full restore from backup. The only systems that do this use NVDIMMs or NVRAM of some kind to guarantee data consistency, and often a second system as a checkpoint for journaling. Given the cost of developing/writing around that, it tends to be limited to enterprise systems - which are abandoning spinning disks almost entirely.

Speaking of which could I load my meta data into ram and make it not access it during the write functions but let it load it later into the drive after the main write function. Or added onto the write function in a chunk that can be written efficiently with the rest of the file basically or literally as a part of it's write job?

Extremely dangerous - this is asynchronous writes, which as I pointed out above, results in massive corruption if anything goes wrong. The latter part is what ZFS does (sorta) with the ZIL - which is why I suggested that with putting the ZIL on SSD (and it requires that be mirrored unless you're insane).

Then you would just need a read job of the entire metadata and organization in ram with the cpu or something to organize a write function and try to get rid of all other sloppy tasks so the write job is pure organized write(with an extra update to the metadata at the end potentially.). I think that would be near my dream raid10. The use of copy to not delete and double check the copy could be used for redundancy or cut checking before delete of old data.

This is the industry I work in - folks have done this, in various forms. Got 100k to buy one of the systems? The problem here is the folks that need this have money to spend, and the folks that develop it want to get paid a lot for writing it, which means very little trickles down to the consumer side. ZFS is the closest, but it also has limitations (unless you literally use an NVDIMM or the like for ZIL, and even then sync writes from things cause odd activity).

Do I want a ramdisk installed while using btrfs in general? I had a tmpfs but remove it.

Edit: you could also make a raid 10 do all sync data via other sources and in essence do enhanced raid 0. It could be raid 0 with data patterns that can be simulated with faster resources on demand. Those data patterns, if uncorrupted, could be used to give raid 0 redundancy. Then you could take the actual writes from my raid 10 idea and run loose with them to double capacity also. While still allowing some minimal disk failure/rebuilding. This could be good for ssd's also as you could simulate raid space in ram and cpu on demand and lessen writes to the ssd. Assuming the accompanying hardware is strong enough. This might be more useful if we ever get more powerful computers like optical processing and whatnot where you may have near unlimited throughput or redundant task speed to do odd task like this. Optical may allow the use of small visual objects to be unpackaged on demand into larger files for use with smaller network space because of raw processing. This could also change the normal ratio of storage to computing power changing what is possible. So older hdd's might still be useful. Depends on how they are designed. You could also possibly run things like n3 or more stripes without loosing disk space if you have the accompanying power/drives to aid in using partial wider stripes across more drives. This could lead to further potentially lossless functionality. If you ran a 3/6 disk raid with 3 stripes could you use this for double rebuilding with massive compute power for even more redundancy on a single layer? Can't you hypothetically just spread the stripe data out more and then calculate like if you had stored it with sufficient calculation power? I wonder if that could lead to disk with internal raid like structures. Assuming that helps. I think at it's peak as long as you can live simulate the other parts of the disk you can avoid keeping them in writes on the drive. If the rest of the system is faster you have ideal raid10/0 as long as you can keep it's data correct via other means than writes or data/space usage. Then you could have drives ideally made for raid and throughput. Change or amend magnetic with optical and maybe you have something better than ssd in all categories. Assuming other things aren't superior. And that it can't have material or other changes to outdo those things. Since it could be the basis for optimized throughput devices it could be the basis for other different hdd like devices later. Especially it later things like optical allow for modifications of devices and not just single pre made objects. Or at least relatively more modifiable. Different and or more complex geometry could be used in endless combinations potentially. Optical geometric processors could be interesting. You could even use accurate or inaccurate shapes with pre calibrations and be used for both processing and other current functions in essence. It depends on the relative power to the function running through it. Getting a bit off track here. Either way, more processing power could hypothetically lead to simulated layers leading to greater and greater ideal lossless raid performance. with more power and more drives or innter tech in the drives you could increase the amount of redundancy per package in various ways to keep ideal functioning. You could even make a single package multiple drives with different power/connectors and possibly extra internal processing to alleviate cpu/ram or other functions at some point if needed. Just need a way to repair/replace. This could help as processing/physical needs fluctuation relative to each other and needed packaging. Lots of new fun things could be done though. If I'm not mistaken. this is where increased mod ability becomes more useful over time. Especially as optical can remove the double reality of connection in most or all of the system from also being data points. This increase potential versatility of a systems parts. I'm pretty sure optical will potentially lead to pure agnostic hardware/software realities to every single tiny point of each if desired. Or is that the opposite of agnostic. If both can be agnostic then I think it doesn't matter logically as they can both be developed to each other perfectly as desired. Hardware can be changed to software and vice versa. Or any other outcome as it is all changeable at that point. That and the potential endless ability to parallelize things. On some of these notes I wonder if you could go from 2d raids to 3d raids and increase the redundancy more. 9 disk could be more interesting. And what about 4d or higher? This could increase striping and functionality to even more interesting things. You could even gain space and gain functions you don't have now. Or at least the room for them.

Huh? We generally don't care about lessening writes anymore - QLC/3d Nand/etc have enough endurance to handle, well, just about anything. The rest, I'm not following at all.

No one in that space is using spinning rust except as long-term archive/storage/backups - where no one worries about file op performance.

I ~think~ the fundamental part you're missing here is that unless you have NVRAM of some kind, the only "trusted" media is the media itself. Anything stored in cpu/memory isn't written yet, because it's ~unsafe~. That's why everything is moving to flash - it's faster trusted media. And doesn't require built in batteries or supercaps to try and survive a system failure (or at least give the chance of rebuilding the journal). Everything you're talking about here (the parts I can follow) are considered extremely dangerous as there's no way to guarantee the reliability of that data - and that'll trash the filesystem.

 

[Deleted member 330132]

I know it's dangerous. it's for a home computer that is running steam and video games. The point is that if you are using something that can purely be rescanned from a remote location it's not important as you can simply scan and regain the file anyway. It's just to gain pure performance. Plus I thought the idea might be useful in the future as stuff gets more powerful. Unless m.2 gets that cheap. But who knows. Maybe hdd's will find a way to keep up. It would just be fun to have a really good hdd raid solutions regardless. People might make use of it. I'm surprised it takes a lot of work to write. I'm using it as a medium storage area for moderate read write with a backup location that I run the files to when I want to use them. Or to write to the SSd when I really want a game to run faster. I also keep a backup copy on a different backup drive potentially. So, it's basically long term cache for game folders. I also use extremely cheap drives so it's cost effective. Plus if you take it to it's extreme it could be used for nvme raids also. And if the sell it for 100,000 they can easily make that much with smaller software to a much larger client base. Plus in a more advanced form you don't have to write it. You could simulate it. You could add safety with things like NVRam though in a more professional environment.

Does anyone have any idea what might be causing the used system thing. The btrfs tools are not reporting the used allocation. I'm wondering if it's something else in linux. Not sure where to start to figure it out.

$ df
df: /run/user/1000/doc: Operation not permitted
Filesystem                               1K-blocks       Used Available Use% Mounted on
devtmpfs                                   8173820          0   8173820   0% /dev
tmpfs                                      8226600     575164   7651436   7% /dev/shm
tmpfs                                      8226600       1712   8224888   1% /run
/dev/mapper/fedora_localhost--live-root   25671908   15529844   8814960  64% /
tmpfs                                      8226600        252   8226348   1% /tmp
/dev/sda1                                   999320     266152    664356  29% /boot
/dev/sdc                                 735526008       4480 490609664   1% /mnt/Storage
/dev/mapper/fedora_localhost--live-home  218244176  177558816  29529496  86% /home
/dev/dm-3                               1921800384 1081338056 742770376  60% /mnt/Backup
tmpfs                                      1645320         64   1645256   1% /run/user/1000

$ btrfs fi df /mnt/Storage
Data, RAID10: total=3.00GiB, used=1.00MiB
System, RAID10: total=96.00MiB, used=16.00KiB
Metadata, RAID10: total=1.03GiB, used=112.00KiB
GlobalReserve, single: total=3.25MiB, used=0.00B

Not sure what those tmpfs are either.(might be stuff on the boot disk.) I thought I got rid of the one I was using. It's drive SDC.

Is the metadata sloppy and filling up space?

 

[IdiotInCharge]

it's for a home computer that is running steam and video games.

M.2 is cheap enough for this.

Trying to make spinners faster is a fruitless endeavor.

 

[Deleted member 330132]

Stuff for home users are still very different than for servers. I can get very cheap hardware. In the range of 10-20 dollars a drive. It is much more cost effective than anything like an NVMe. Not to mention I'm using a 10 year old computer with a phenom 1100t. It's useful for people in different situations. Many people are on very old computers still because of the constant economic and other problems. There will always be people who can use this. Just need an HDD and SSD version if there is a difference.

And if you sell the software cheap or via certain clients, like business/home ones, you can get lots of money to equal the higher end market. Or add to it. I would love a more ideal raid solution. It gives some safety while still giving volume for me.

Not to mention they could come back in style if the market changes enough. You never now what could happen.

Edit: I reverted to mdadm and just put btrfs on top of it as a filesystem. I'm going to try this at 256k chunks then try a weird setup with 1n2 1f2 and 1o2 mirror in a raid 10.

https://serverfault.com/questions/241393/mdadm-change-raid-10-chunk-size-and-switch-to-far-layout

This was saying that the f2 to n2 offset it by 512. Maybe this will let it use more drives by automatically offsetting it. Should be interesting to see what it does. That or do a combination of f2 and n2 mirrors in a stripe. I could play with a 4 disk raid to see how it acts first. I wonder if I need to assume clean to not wipe the drives. Actually wouldn't I have to make them 2 disk raid 10's... Maybe the base raid 10's will be in 512k and the bigger raid in 4mb or the opposite or similarly 256k with 2mb. It's the closest I can find to a 6x ratio. That or make an f2 and an n2 3 disk stripe put into a mirror. I wonder if different chunk sized for the base raids would do anything useful. Edit: Tried n2 raid10 with f2 raid10 in a stripe. So far no difference. I'll try a mirror next.

The combo of n2, f2, o2 in a stripe seems to have increased file performance to a starting point of 320mb/s on the first copy job... Unless I missed something. Does that kill of the redundancy of the file system?

$ sudo mdadm -D /dev/md1*
/dev/md10:
           Version : 1.2
     Creation Time : Sun Aug 23 21:31:14 2020
        Raid Level : raid0
        Array Size : 734733312 (700.70 GiB 752.37 GB)
      Raid Devices : 3
     Total Devices : 3
       Persistence : Superblock is persistent

       Update Time : Sun Aug 23 21:31:14 2020
             State : clean
    Active Devices : 3
   Working Devices : 3
    Failed Devices : 0
     Spare Devices : 0

        Chunk Size : 512K

Consistency Policy : none

              Name : localhost-live.attlocal.net:10  (local to host localhost-live.attlocal.net)
              UUID : 3fbd282f:6176951f:84df7d10:f8984700
            Events : 0

    Number   Major   Minor   RaidDevice State
       0       9       11        0      active sync   /dev/md11
       1       9       12        1      active sync   /dev/md12
       2       9       13        2      active sync   /dev/md13
/dev/md11:
           Version : 1.2
     Creation Time : Sun Aug 23 19:49:20 2020
        Raid Level : raid10
        Array Size : 245043200 (233.69 GiB 250.92 GB)
     Used Dev Size : 245043200 (233.69 GiB 250.92 GB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

       Update Time : Sun Aug 23 21:34:32 2020
             State : clean
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0

            Layout : near=2
        Chunk Size : 512K

Consistency Policy : resync

              Name : localhost-live.attlocal.net:11  (local to host localhost-live.attlocal.net)
              UUID : 14e8926e:e0616843:4aea4031:d026f672
            Events : 18

    Number   Major   Minor   RaidDevice State
       0       8       32        0      active sync set-A   /dev/sdc
       1       8       48        1      active sync set-B   /dev/sdd
/dev/md12:
           Version : 1.2
     Creation Time : Sun Aug 23 19:49:36 2020
        Raid Level : raid10
        Array Size : 245043200 (233.69 GiB 250.92 GB)
     Used Dev Size : 245043200 (233.69 GiB 250.92 GB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

       Update Time : Sun Aug 23 21:34:33 2020
             State : clean
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0

            Layout : far=2
        Chunk Size : 512K

Consistency Policy : resync

              Name : localhost-live.attlocal.net:12  (local to host localhost-live.attlocal.net)
              UUID : 59c45319:4275aae7:161b9450:770ff7be
            Events : 17

    Number   Major   Minor   RaidDevice State
       0       8       64        0      active sync   /dev/sde
       1       8       80        1      active sync   /dev/sdf
/dev/md13:
           Version : 1.2
     Creation Time : Sun Aug 23 19:50:12 2020
        Raid Level : raid10
        Array Size : 245043200 (233.69 GiB 250.92 GB)
     Used Dev Size : 245043200 (233.69 GiB 250.92 GB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

       Update Time : Sun Aug 23 21:34:32 2020
             State : clean
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0

            Layout : offset=2
        Chunk Size : 1024K

Consistency Policy : resync

              Name : localhost-live.attlocal.net:13  (local to host localhost-live.attlocal.net)
              UUID : e66d1beb:83104a89:0cd3beb3:d1f3784e
            Events : 17

    Number   Major   Minor   RaidDevice State
       0       8       96        0      active sync   /dev/sdg
       1       8      112        1      active sync   /dev/sdh

I wonder if increasing the size of the o2 to higher than 1meg would help. maybe 6x the other 512k sizes or maxing it to 1g. Edit: Nope, seems to be back to normal speeds now. It runs about half the speed of the others variations. But it is staying 200+ for quite a long time. Edit: I stuck it in raid 0 and I'm still not getting faster speeds on actual file transfers.

Looking into xfs more again. It seems to be doing the best so far.

https://www.spinics.net/lists/raid/msg35524.html

Does anyone know what MD R1/10's is? Is this just mirrors stripped? Mdadm's default is two stripes mirrored I think. Is one better than the other?

Raid 0 is currently giving me a small boost in performance. But it's like 138mb/s compared to 133 or 125. Although one copy it boosted up to near 300 again.

Could I use xfs to use my gpu to deal with threads? 8p

 

[lopoetve]

I know it's dangerous. it's for a home computer that is running steam and video games.

Stick it all in a ramdisk - that's effectively what you're doing. By dangerous, I mean ANY crash (kernel, system, you held the power button down, power went out) corrupts the ENTIRE filesystem. Not one file, not one part - all of it. Garbage. That's why no one does this. The enterprise systems that do? Have NVRAM backing up multiple controllers to write all that data to NAND of some kind on failure - or an overpowered UPS to save it somewhere live until you boot up again.

The point is that if you are using something that can purely be rescanned from a remote location it's not important as you can simply scan and regain the file anyway.

There are two systems on the market I'm aware of that can do this; both cost 500k average to start.

It's just to gain pure performance.

Buy an NVMe drive. One NVMe drive is the equivalent of 1,000 spinners in RAID0 for this kind of operation. Why would anyone bother spending time optimizing that?

Plus I thought the idea might be useful in the future as stuff gets more powerful. Unless m.2 gets that cheap.

4TB drive = $400, give or take, for PCIE4. That's cheap.

But who knows. Maybe hdd's will find a way to keep up.

Not likely. DMA access of NAND flash natively will beat a physical head that has to seek... always. Spinners are archive - they're optimized now for sequential throughput on large files.

It would just be fun to have a really good hdd raid solutions regardless.

Why? I can beat it with a $50 SSD. NVMe will run rings around it.

People might make use of it. I'm surprised it takes a lot of work to write.

Filesystems are the hardest thing to possibly write - ever.

I'm using it as a medium storage area for moderate read write with a backup location that I run the files to when I want to use them. Or to write to the SSd when I really want a game to run faster. I also keep a backup copy on a different backup drive potentially. So, it's basically long term cache for game folders. I also use extremely cheap drives so it's cost effective. Plus if you take it to it's extreme it could be used for nvme raids also. And if the sell it for 100,000 they can easily make that much with smaller software to a much larger client base. Plus in a more advanced form you don't have to write it. You could simulate it. You could add safety with things like NVRam though in a more professional environment.

A hybrid Unity system with 16G FC will cost you around $30k. IF this is what you want, buy that. Nimble, etc exist in the same market. The fact is - no one in the consumer market pays for it; far easier to buy an NVMe SSD for cheaper and move on, so that's what they do. Why try to optimize inferior hardware?

Does anyone have any idea what might be causing the used system thing. The btrfs tools are not reporting the used allocation. I'm wondering if it's something else in linux. Not sure where to start to figure it out.


Not sure what those tmpfs are either.(might be stuff on the boot disk.) I thought I got rid of the one I was using. It's drive SDC.

Is the metadata sloppy and filling up space?

Skipping those for now.

 

[lopoetve]

Stuff for home users are still very different than for servers. I can get very cheap hardware. In the range of 10-20 dollars a drive. It is much more cost effective than anything like an NVMe.

Not for this it isn't. $20 / drive * 1000 drives = 20k. Buy the NVMe.

Not to mention I'm using a 10 year old computer with a phenom 1100t. It's useful for people in different situations. Many people are on very old computers still because of the constant economic and other problems. There will always be people who can use this. Just need an HDD and SSD version if there is a difference.

To be blunt - the vendors don't care about your use case. You're note spending money with them, they don't care. You're buying used/older hardware, you're not a revenue stream. Aside from that - this market died for the ENTERPRISE in the mid-2010s - it's all flash now or nothing.

And if you sell the software cheap or via certain clients, like business/home ones, you can get lots of money to equal the higher end market. Or add to it. I would love a more ideal raid solution. It gives some safety while still giving volume for me.

Who would use it? Flash is cheaper, especially since every system out there that they'd buy includes deduplication and compression on flash - that brings prices inline.

Not to mention they could come back in style if the market changes enough. You never now what could happen.

And pigs could fly. It ain't happening. The market moves forward - concepts and ideas are cyclical, but the technology itself is not, in general. HDD are for archive, backups, or low-IOPS workloads - there's no reason to try when a single flash drive will generate the performance of 1000+ spinners, natively, without trying

Edit: I reverted to mdadm and just put btrfs on top of it as a filesystem. I'm going to try this at 256k chunks then try a weird setup with 1n2 1f2 and 1o2 mirror in a raid 10.

https://serverfault.com/questions/241393/mdadm-change-raid-10-chunk-size-and-switch-to-far-layout

This was saying that the f2 to n2 offset it by 512. Maybe this will let it use more drives by automatically offsetting it. Should be interesting to see what it does. That or do a combination of f2 and n2 mirrors in a stripe. I could play with a 4 disk raid to see how it acts first. I wonder if I need to assume clean to not wipe the drives. Actually wouldn't I have to make them 2 disk raid 10's... Maybe the base raid 10's will be in 512k and the bigger raid in 4mb or the opposite or similarly 256k with 2mb. It's the closest I can find to a 6x ratio. That or make an f2 and an n2 3 disk stripe put into a mirror. I wonder if different chunk sized for the base raids would do anything useful. Edit: Tried n2 raid10 with f2 raid10 in a stripe. So far no difference. I'll try a mirror next.

The combo of n2, f2, o2 in a stripe seems to have increased file performance to a starting point of 320mb/s on the first copy job... Unless I missed something. Does that kill of the redundancy of the file system?


I wonder if increasing the size of the o2 to higher than 1meg would help. maybe 6x the other 512k sizes or maxing it to 1g. Edit: Nope, seems to be back to normal speeds now. It runs about half the speed of the others variations. But it is staying 200+ for quite a long time. Edit: I stuck it in raid 0 and I'm still not getting faster speeds on actual file transfers.

Looking into xfs more again. It seems to be doing the best so far.

https://www.spinics.net/lists/raid/msg35524.html

Does anyone know what MD R1/10's is? Is this just mirrors stripped? Mdadm's default is two stripes mirrored I think. Is one better than the other?

Raid 0 is currently giving me a small boost in performance. But it's like 138mb/s compared to 133 or 125. Although one copy it boosted up to near 300 again.

Could I use xfs to use my gpu to deal with threads? 8p

I'm going to be blunt. You are wasting your time. You're not going to make any real world difference on file-ops with a 7200RPM SATA drive (or RAID of sata drives) - that's IOPS and processor bound, and you don't have the IO to spare. You want to copy a bunch of small files fast? Put them on an SSD, turn off some metadata features, and let it go - that's not a real world use case for anything real.

 

[daglesj]

I'm going to be blunt. You are wasting your time. You're not going to make any real world difference on file-ops with a 7200RPM SATA drive (or RAID of sata drives) - that's IOPS and processor bound, and you don't have the IO to spare. You want to copy a bunch of small files fast? Put them on an SSD, turn off some metadata features, and let it go - that's not a real world use case for anything real.

Charming! I got in trouble for telling the guy he was wasting his time and got abuse and written warning! I guess he took the hint in the end?!?

Sounds to me he was getting paid a lot of money to install a crap setup and sponging for solutions.

 

[lopoetve]

Charming! I got in trouble for telling the guy he was wasting his time and got abuse and written warning! I guess he took the hint in the end?!?

Sounds to me he was getting paid a lot of money to install a crap setup and sponging for solutions.

It's a ~file~ copy. Big block shows the disks are working fine, so this is file ops - you can't really improve those except with better hardware.

Edit: And he nuked his account (or had it nuked). /shrug. Hope it helps if anyone else comes along; there's nothing you can do to make this stuff better. It literally is what it is.

 

[daglesj]

It's a ~file~ copy. Big block shows the disks are working fine, so this is file ops - you can't really improve those except with better hardware.

Edit: And he nuked his account (or had it nuked). /shrug. Hope it helps if anyone else comes along; there's nothing you can do to make this stuff better. It literally is what it is.

You do see a lot of dreamers on this forum.

 

[cjcox]

You do see a lot of dreamers on this forum.

My favorite dreamers are on r/techsupport. "My laptop is having issues. I accidentally poured my Cherry Coke on it. And it was working but I tried to remove the syrup from the screen using gasoline. The residual smell was awful, so I hosed it off outside, but accidentally dropped it, but only a few inches. However, before I could get to it my mom ran over it with our old Gremlin (not sure how much that car weighs). When I brought it back inside the screen was blank on power up. I figured maybe it just need a few more volts. So I used the power adapter from my old Dell Precision laptop. The connector didn't fit, so I mashed it to fit. So, right now, things are pretty dark. But my original problem is I'm getting 5 fps less than everyone else is on Fortnite. Any ideas?"

 

[daglesj]

My favorite dreamers are on r/techsupport. "My laptop is having issues. I accidentally poured my Cherry Coke on it. And it was working but I tried to remove the syrup from the screen using gasoline. The residual smell was awful, so I hosed it off outside, but accidentally dropped it, but only a few inches. However, before I could get to it my mom ran over it with our old Gremlin (not sure how much that car weighs). When I brought it back inside the screen was blank on power up. I figured maybe it just need a few more volts. So I used the power adapter from my old Dell Precision laptop. The connector didn't fit, so I mashed it to fit. So, right now, things are pretty dark. But my original problem is I'm getting 5 fps less than everyone else is on Fortnite. Any ideas?"

Ha yes, that's what I get told just as they are physically handing it over to me in pieces in a bag, after they called me an hour before to say "my laptop is going a bit slow, can you take a look?!"

 

[likeman]

just sounds like he is wanting more performance by using raid witch does not always give you more performance especially software raid (R for Redundant does not = speed)

be better if he just got a NVME SSD and backup it up via software (but he did not even seem to care about that as well so not sure why he is trying to use raid)

 

[_Gea]

just sounds like he is wanting more performance by using raid witch does not always give you more performance especially software raid (R for Redundant does not = speed)

Raid is not only for redundancy but also for performance.

A stripe raid (5/6/Z1/Z2/Z3) gives n * datadisks sequential performance unless CPU is not limiting. Only iops remains like a single disk. Iops on a mirror or multiple Raid 10 is 2n * mirrors on read and n * mirrors on write - does not matter if you use disks, SSDs or NVMes.

btw
With a modern CPU and filesystems, software raid is superiour.

 

[daglesj]

Yeah well guy was a whiny tattle-tale anyway!