Local storage vs NAS for photography (how to speed up workflow)

iroc409

[H]ard|Gawd
Joined
Jun 17, 2006
Messages
1,385
I have a Dell T110ii file server running FreeBSD with a ZFS RAID 10 array with all my documents, media, photos, etc. I work directly off the server instead of local storage.

A lot of the transfers I'm doing are from SD cards, which are fast-ish cards but often only seeing 50-60MB/s transfers. When I've done file transfers from my desktop in Windows to the server, I think it will reach 80-90MB speeds. If I scp a file from my Proxmox server to the SSD on the file server, I max out network speeds at about 114MB/s.

I now use Geeqie to do my initial cull and preview of photo sessions, but sometimes it can get hit with the slow downs that is kind of frustrating. When I'm doing more "art" I don't take a lot of photos, but when doing any action or wildlife some days I'll have 1500 photos to go through (and delete half of them). I would like to consider what I can do to improve the network performance of the server, or if I need to consider another storage strategy. I've even kind of looked at picking up something like a Dell T140 because I've had issues with the USB 3.0 add-on cards I use for backup drives on the server, but that's a lot of money to spend to replace a server that has (aside from the USB issues) been very reliable.

Some options I've considered:
-Moving all the files back to the desktop to WD Black or something, but then I lose the ECC/ZFS protection my server provides. My desktop runs OpenSUSE right now, so I could use btrfs and arguably have similar protections and then just use the server for backups.
-Adding a 10 gigabit network direct-connect between the desktop and server. I'm assuming I'd have to consider upgrading the server with cache drives or something to improve performance.
-Using local storage or adding a local storage drive specifically to import and do the initial cull, then move the files off to the server. It's the most cost-effective method, but adds extra steps in the workflow that I'm not overly excited about.

I am open to ideas, does anyone have suggestions?
 
I do not know the distance between your computer and the server, if you are very close (or sitting on the desk) you can direct connect 10Gb and call it a day. Might take a little to get working and since I see FreeBSD I would make sure the card you pickup has drivers/support for them.

Honestly the easier option is scripting out a pull/push Rsync setup that you can use before/after you are done. Might take a few to get use to and change your workload, but would be the cheapest/easiest to execute right now.
 
Hakaba, I think you're right that's going to be the easiest and most expedient way. All of my network gear is in a walk-in closet across the hall from my office and my desktop is near the office door so it's really only about ten feet. It wouldn't be much of an issue to run a 10G cable, but it's going to be an expense that may be difficult with the drivers and cards.

With a couple rsync scripts and some cron jobs I should be able to set it up to minimize the difficulty in working with it.
 
Different problems

1. sequential performance vs random performance
If you work from server and write/read a large file, your use case is a sequential load. For a sequential load a good harddisk can offer between 100 and 250 MB/s depending if you use the inner or outer tracks. In any case this is faster than a 1G network. In a Raid-10 your sequential and iops performance is 2x a single disk on writes and 4x on reads.

But as you use ZFS and CopyOnWrite, the ZFS datablocks are spread over the pool so even a pure sequential load result in a mix of seqzential and random access patterns. This is why you want to add a lot of RAM to ZFS that is used as read and write cache to avoid randon access pattern to disk. So best way to improve performance is either adding more RAM or use fast NVMe/Optane disks where performance is superiour even with less RAM than say 8-16GB.

So if you only have 4-8 GB RAM, think of 16-64 GB Ram. In such a case most (say >80%) random reads and reads of metadata come from RAM and all writes even small random ones are going to the rambased write cache of ZFS (default 10% Ram, max 4GB) to transform small and slow random writes to large and fast sequential ones.


2. SMB server
You see a difference between scp and SMB. This is mainly because SMB via SAMBA especially with several concurrent transfers is not as fast as other protocols like ftp or NFS. The reason is that SAMBA is not multithreaded and not really optimized to a special guest OS and it must run on everything that can add 1 and 1 in a binary way. A mostly faster option is the inkernel/in ZFS SMB server of Solaris or the free Solaris forks where SMB is mostly faster. The fastest and most feature rich ZFS server at all is Oracle Solaris 11.4 (where ZFS comes from) with native and genuine ZFS but this is a commercial datacenter OS and not free for commercial use. The fastest free option with Open-ZFS is maybe OmniOS, a minimalized ZFS distribution for storage production use with long term stable and security fixes mostly every two weeks. On Solarish you have not only a very fast SMB but the best integration of ZFS into the OS, the best compatibility to Windows ntfs ACL and the lowest ram needs and working ZFS snaps as Windows previous versions out of the box. The Solarish SMB server is mostly zero config, enable and it works.

So overall your server options are with 1G
- add more ram
- use faster disks (SSD or NVMe)
- use a faster SMB server or server OS

This will give you a quite stable server performance of 1G on any workload

If you want more than 1G, you can use 10G directly connected. With a Raid-10 from average disks you can expect around 200-300 MB/s write. Read depend the question if data is in RAM or must be readed from disk where you can expect up to 400 MB/s.


If you use a local disk of same type like in the Raid-10, you can get on a pure sequential load up to disk performance (say 200 MB/s) but on random reads you lack the superiour ZFS readcache what can mean only 40MB/s read. On small random writes it is the same (ZFS always faster).
 
Thanks for the reply, _Gea. You've given me a lot to think about. I was starting down the NFS path but the last thing I read said it was so close now with SMB performance it wasn't worth the effort. Browsing files on the machine is very quick, so I didn't think a lot of it. That might be the first step, however. The machine is a Xeon E3-1225v2 maxed out with 32Gb RAM, so there's not much more headroom I can get there.

Changing operating systems is a worthwhile consideration, I've just been doing this with FreeBSD for so long I trust it (and myself) with the files. I have plenty of extra hardware laying around, I might try playing with some alternatives and see where I get.

If I do the local drive, particularly for a "landing pad" for photos, they will be on SSD so I'd think I would see pretty high speeds there--but yes one of my options was running local spinning disks--which sounds like it won't buy much.

It's a lot to think about, and at some point in the not too distant future I'd like to make some upgrades--if for no other reason than my storage pool is at 72% and climbing. So if I put in some work now, maybe I'll have the right answer by the time I need the upgrade.
 
Back
Top