ZFS suitable for media streamer?

[ScottJ]

4 years ago I built a NAS using an Atom D510 CPU and 4-drive ZFS raidz1. It was unable to stream Blu-ray rips without occasional stutters in playback. I switched to UFS2 JBOD and have been using that ever since.

Should I try ZFS again? What do I need to improve this time? Faster CPU? 5 drive pool instead of 4? More RAM (had 4GB for 8TB last time)?

I don't really need high availability or reliability so I'm not sure I need raid at all. I can always go back and re-rip the original discs. But I like how ZFS can detect corruption instead of allowing silent bitrot. Perhaps I could do a sort of ZFS JBOD so I still get checksumming, at least. Would that help with performance?

I'm looking at running the upcoming Ubuntu 14.04 LTS plus ZFS. Hardware is TBD but probably an SAS Expander chassis plus a mini-ITX PC.

 

[Schro]

I'm using a bit more powerful of a setup, but it is working just fine for me to host multiple bluray streams from ZFS while running a bunch of other stuff. See folding rig 1 in the signature....

In general, ZFS will use as much RAM as you give it and most guides say you need at least 1GB per raw TB to be available.

 

[ScottJ]

I'm using a bit more powerful of a setup, but it is working just fine for me to host multiple bluray streams from ZFS while running a bunch of other stuff. See folding rig 1 in the signature....

In general, ZFS will use as much RAM as you give it and most guides say you need at least 1GB per raw TB to be available.

Just "a bit" more powerful? haha

What I'm afraid of is buying a more powerful CPU and more RAM and still not being able to reliably stream. I'm not going with a 4x CPU server board since that is extreme overkill for a simple NAS. Basically, Atom/i3/i5/i7 is the decision I'm trying to make.

Also the 1GB/TB sounds more like urban legend, based on this recent post.

 

[zrav]

I'd stay away from Atom. But with even a Sandy-Bridge Celeron you can saturate 1GE. Also media streaming should be one of the cases where ZFS's prefetching works beautifully.

The amount of RAM for ARC required for good performance is entirely dependent on the working set size.

Hopefully when Ubuntu 14.04 hits ZoL 0.6.3 will have been released (required for kernel 3.13 compat and a GCC related workaround).

 

[patrickdk]

My dual-core pentium D 930 worked well to saturate 2gbit, but that was about it's max.

ZFS doesn't really need much more cpu to make this work, over any other filesystem, and the amount of additional ram is going be realitively small.

The main issues with streaming something like that, is if a single disk starts to go slow, then it's going to cause the stream to get jittery.

 

[Rectal Prolapse]

zrav, I actually encountered issues with ZFS and prefetching for media streaming - by default, if more than 1 megabyte of data is being read, prefetching is no longer used - so basically when streaming videos prefetching is not being used (or only intermittently, depending). I suppose I could run DTrace to prove my case but for now I can say that if you use NFS UDP you may get a stall lasting a hundred milliseconds or so when ZFS decides not to prefetch and goes straight to disk.

On my system with RAIDZ2 and two vdevs of 10 drives each, this happens quite frequently (using DTrace on NFS reads while streaming a Blu-ray).

With that said, this shouldn't be an issue with TCP - at least on my Dune when it asks for 512KBytes

 

[brutalizer]

ZFS is 128 bit, so it does not like weak 32bit cpus. You need 64 bit cpu for full speed. I used a 32bit Pentium4 with 1GB RAM for a year, and got 20-30MB/sec. With 64 bit cpu, I got full speed.

With a decent 64 bit cpu and 7 disks you get 400MB/sec or so.

 

[ScottJ]

zrav, I actually encountered issues with ZFS and prefetching for media streaming - by default, if more than 1 megabyte of data is being read, prefetching is no longer used - so basically when streaming videos prefetching is not being used (or only intermittently, depending). I suppose I could run DTrace to prove my case but for now I can say that if you use NFS UDP you may get a stall lasting a hundred milliseconds or so when ZFS decides not to prefetch and goes straight to disk.

On my system with RAIDZ2 and two vdevs of 10 drives each, this happens quite frequently (using DTrace on NFS reads while streaming a Blu-ray).

With that said, this shouldn't be an issue with TCP - at least on my Dune when it asks for 512KBytes

I'm also using a Dune (Base 3.0). 100ms sounds like an extremely long delay, when HDDs have typical access times of 12-20ms. Do these delays cause stutters?

I've always used NFS UDP but I will keep in mind that TCP might work better.

 

[ScottJ]

ZFS is 128 bit, so it does not like weak 32bit cpus. You need 64 bit cpu for full speed. I used a 32bit Pentium4 with 1GB RAM for a year, and got 20-30MB/sec. With 64 bit cpu, I got full speed.

With a decent 64 bit cpu and 7 disks you get 400MB/sec or so.

Many Atoms (including my D510) are 64-bit. I don't think throughput is the problem, but worst-case latency. I don't need any more than about 10 MB/sec, but this Atom box couldn't sustain that without long gaps that caused stutters in playback.

 

[kidchunks]

I would recommend a faster cpu. I had similar issues when I was using an amd c-60 processor. Switched to an intel i3-3220 and immediately noticed a difference.

 

[westrock2000]

ZFS on OSX used for streaming. 12TB useable on Movies and I don't think I have ever seen the total system go above 4GB.

You can easily use ZFS in JBOD mode. You just add single disk pools. I have my main movie pool as RADZ1, but my other media are on single drives that are still using ZFS.

 

[FLECOM]

ZFS on OSX used for streaming.

ZEVO?

 

[spazoid]

ZFS is 128 bit, so it does not like weak 32bit cpus. You need 64 bit cpu for full speed. I used a 32bit Pentium4 with 1GB RAM for a year, and got 20-30MB/sec. With 64 bit cpu, I got full speed.

With a decent 64 bit cpu and 7 disks you get 400MB/sec or so.

This makes no sense, I'm afraid. ZFS is a 128 bit filesystem, yes, but it has nothing to do with your CPU being 32 or 64 bit compatible. The 128 bit refers to the amount of vdevs, filesizes, filesystems, files etc that ZFS can handle.

 

[brutalizer]

This makes no sense, I'm afraid. ZFS is a 128 bit filesystem, yes, but it has nothing to do with your CPU being 32 or 64 bit compatible. The 128 bit refers to the amount of vdevs, filesizes, filesystems, files etc that ZFS can handle.

This makes sense, as the zfs developers on the mail list confirmed this when I asked why I got so slow speeds with my 32bit cpu. They said I needed to go to 64bit cpus.

If you have a 32 bit cpu, and try to use 64 bit code, it will be slower because the 32 bit cpu will have to emulate 64 bits, worst case it will be half the speed in some unlucky circumstances. And if you have a 32 bit cpu and try to emulate 128 bits, it will be even slower, worst case, four times slower in some unlucky circumstances.

Another factor is that the 32 bit ZFS code had some problems, back then, according to the ZFS developers. I think later the 32 bit code was cut out, maybe by Oracle? I dont think Oracle ZFS will even work with 32 bit cpus anymore. But will OpenZFS work today with 32 bits, and has that code been improved?

ZFS creator Jeff Bonwick, Matt Ahrens explained back in the days, 1999-2000, they designed ZFS to be used on at least 1GB RAM and 64 bit cpus which was very controversial back then. But they reasoned as such that some years would pass before they released ZFS, and hardware development is very fast, so in a couple of years, every PC would have at least 1GB RAM and 64 bits. And they were right again. This design decision to target huge PCs (back then) allowed them to use better algorithms so they did not have to make special cases if you had less RAM, weak CPUs or being restrained in other ways. So they could go for the full monty, using powerful and RAM hungry algorithms which simplified the code and solution and made it more stable, which means they targeted 1GB RAM and 64 bit cpus. That is the reason they neglected 32 bit cpus.

But back then, people thought they were crazy and ZFS got a rumour of being very resource intensive, so you needed huge amounts of RAM to even use ZFS for basic usage - you needed 1GB RAM!!! Which was crazy! Yes, ZFS is very RAM hungry, so hungry that no one could use it, except big expensive Unix servers. I mean, who else had 1GB RAM just for basic ZFS usage?? And, then to get good perfomance, you need at least 2-4GB ARC, but preferably >8GB RAM. That is so crazy, everyone wondered what the ZFS devs was thinking, head up in the sky. And that is how ZFS got that rumour spread it needs lot of RAM, which is true - but it is not true anymore today, as everyone has 4GB RAM or more.

Another prediction they made, was that 64 bit filesystems will not be enough soon. 64bit is something like 1500 Petabytes. Today they there are 100PB storage systems. Every year you double according to Moores Law, so next year you will see 200PB storage systems. And 400PB. So in just some doublings (years) you will reach 1600PB. Back in 1999, when they layed out all design decisions, it would only take 20 years or so, before someone would hit the roof with 1500PB. And then ZFS would need to be modified to 65 bits, which would last another doubling (year). Every additional bit would last another year. So they made ZFS to 128 bits, which means ZFS would never need to be modified. 128 bits, is so much that if you created a zpool with 3TB hard disks and filled it up with 2^128 bits, the hard disks alone would weigh something like two moons. There is not even that much matter on earth. When you store data on hard disks, you move electrons. You would need so many electrons to move 2^128 bits, that the energy needed would literally boil all water on earth. I doubt mankind has ever created that much energy. The limitations of 128 bits, are physical. It is the laws of physics that sets the boundaries to us. We can not extract that much energy to fill a 128 bit zpool, nor create the number of hard disks, the earth does not hold that much iron needed.

But 64 bit filesystems (BTRFS) will need to be extended with another bit, soon. To 64 bits. And changing bit size, is quite nontrivial to do. Very messy can it be. You need to change the foundation, all data structs, which means huge changes everywhere. It is much easier to do 128 bits, 1GB RAM and 64 bit cpus, than to do extend number of bits later.

In short, 128 bit filesystems should be enough for anybody - because the next limitiation, are the laws of physics. There is no point of creating 129bit filesystems nor 256 bits, because the laws of physics will stop us before that. 128 bits are enough.

"64KB RAM should be enough for anybody" - Bill Gates

 

[TCM2]

If you have a 32 bit cpu, and try to use 64 bit code, it will be slower because the 32 bit cpu will have to emulate 64 bits, worst case it will be half the speed in some unlucky circumstances. And if you have a 32 bit cpu and try to emulate 128 bits, it will be even slower, worst case, four times slower in some unlucky circumstances.

That's a very naive and not really correct view. A 32bit CPU can't execute 64bit code. There is no emulation going on. You need to differentiate between code and data. A 32bit CPU can handle 64bit oder 128bit _data_ using more opcodes than a 64bit CPU would. But that makes no statement at all about the speed you get.

Saying a 32bit CPU is half the speed of a 64bit CPU is just wrong. It heavily depends on the data it handles. Try running a 32bit Linux and a 64bit Linux with some benchmarks on the same hardware.

And finally, ZFS data structures being 128bit doesn't mean the code is 128bit. The main difference in performance you see with 32bit vs. 64bit is not due to some magical speed halving, it's because if a CPU is so old that it doesn't even do 64bit, it's also much slower than a modern 64bit CPU in 32bit mode to begin with. Add to that the RAM limit of 4GB and that's where your limitations come from.

 

[SirMaster]

This is true. A 64-bit CPU isn't inherently faster than a 32bit CPU. At least not in any meaningful way. When I compile my code to a 32bit executable it doesn't run any slower than when it's compiled for 64bit. The RAM limit has always certainly always been the biggest difference in performance between a 32bit vs 64bit platform.

 

[drescherjm]

This is true. A 64-bit CPU isn't inherently faster than a 32bit CPU. At least not in any meaningful way. When I compile my code to a 32bit executable it doesn't run any slower than when it's compiled for 64bit. The RAM limit has always certainly always been the biggest difference in performance between a 32bit vs 64bit platform.

There are a few more SIMD type instructions, wider and more registers ... whether or not your code or compiler can or will utilize is a different matter.

 

[brutalizer]

That's a very naive and not really correct view. A 32bit CPU can't execute 64bit code. There is no emulation going on.

True. I didnt mean what I wrote.

I meant that if you need to handle 128 bits data structures on a 32 bit cpu, it will be problematic and you will loose performance.

 

[SirMaster]

There are a few more SIMD type instructions, wider and more registers ... whether or not your code or compiler can or will utilize is a different matter.

Yes, I should have mentioned the few things like that, but in my experience unless your code is very specialized these additions don't translate to much performance gain normally. Certainly not a 2x performance gain. And this is why the vast majority of software doesn't run much faster when it's simply recompiled for 64bit.

 

[drescherjm]

Yes, I should have mentioned the few things like that, but in my experience unless your code is very specialized these additions don't translate to much performance gain normally. Certainly not a 2x performance gain. And this is why the vast majority of software doesn't run much faster when it's simply recompiled for 64bit.

Fully agreed.

 

[SirMaster]

I can see how handling 128bit data structures on a 32bit cpu could potentially be 2x slower than a 64bit for those parts of the code. How much % slower that translates into the whole program... Well that depends on how much of the code is dealing with the manipulation of said data structures heh.

I still think the 4GB RAM limit which is usually limited to 2-3GB per executable for 32bit executables is a bigger problem for the ZFS performance, but certainly both the RAM and data structure issues add together to make everything slower.

 

[brutalizer]

Agreed that 4GB RAM limit is a much greater limit.

 

[Rectal Prolapse]

I'm also using a Dune (Base 3.0). 100ms sounds like an extremely long delay, when HDDs have typical access times of 12-20ms. Do these delays cause stutters?

I've always used NFS UDP but I will keep in mind that TCP might work better.

Yes there will be a very slight pause that will look like stutter when using NFS UDP.

With that said, it is possible that interrupt moderation enabled on the NIC may be at play here and may not just be slow ZFS disc access. It will be a while before I can prove this. I just discovered how to disable interrupt moderation in VMWare ESXI, so I will try to test this on the weekend.

I noticed that even when I modded UNFS3 to precache file data, there were at the time still some unexplained NFS read processing delays, which I am 90% sure are from e1000 interrupt moderation (on the order of 1 to 10 milliseconds). I've since disabled them from within VSphere Client. We'll see how my testing goes!

 

[spazoid]

True. I didnt mean what I wrote.

I meant that if you need to handle 128 bits data structures on a 32 bit cpu, it will be problematic and you will loose performance.

It is still completely incorrect. The two numbers has nothing to do with eachother.
The 128 bits in ZFS only has to do with file sizes, number of datasets, number of files and similar.

The 64 bits in a CPU has to do with memory address widths and integer sizes and stuff like that.

The two are COMPLETELY unrelated. If you insist it is otherwise, you need to back it up with data.

 

[brutalizer]

It is easy to back it up with data, just check the ZFS source code, it is open source.

Maybe I am not phrasing myself correclty. I am trying to say that if a 32 bit cpu tries to dabble with 128 bits, the cpu need additional administration. If you compare that to a 128 bit cpu.

Are you saying that 128 bits cpu and a 32bit cpu will do equal amount of work on 128 bit ZFS? I am claiming that 32 bit cpu needs to do additional work, you claim the opposite?

 

[spazoid]

Maybe I'm not making this clear: You're incorrect. Referring to the source code is not a valid argument.

There are no 128 bit CPU's that'll run an OS that in any way supports ZFS, so your point with that comment eludes me.

I'm saying that you're comparing apples and oranges. I'm claiming that the 128 bits referred to in the context of ZFS has nothing at all to do with the 32/64 bits referred to in the context of CPUs.
128 bits is just a number. In ZFS it's used to refer to the amount of filesystems, files etc. that the software (ZFS) can address, in CPU's it's used to refer to the memory address width, integer sizes etc.

 

[TCM2]

It is easy to back it up with data, just check the ZFS source code, it is open source.

Maybe I am not phrasing myself correclty. I am trying to say that if a 32 bit cpu tries to dabble with 128 bits, the cpu need additional administration. If you compare that to a 128 bit cpu.

Are you saying that 128 bits cpu and a 32bit cpu will do equal amount of work on 128 bit ZFS? I am claiming that 32 bit cpu needs to do additional work, you claim the opposite?

If ZFS was 100% crunching 128bit numbers, you'd have a point. It's not. The most CPU-intensive work is doing compression, which has nothing to do with 128bit. Before handling the 128bit numbers involved in ZFS, the CPU has done a thousand times the work managing the disk controller through the kernel driver, serving the data via NFS or running your userland application. All that has nothing to do with 128bit.

You are drawing conclusions from completely misunderstood facts.

 

[raiderj]

It is easy to back it up with data, just check the ZFS source code, it is open source.

Maybe I am not phrasing myself correclty. I am trying to say that if a 32 bit cpu tries to dabble with 128 bits, the cpu need additional administration. If you compare that to a 128 bit cpu.

Are you saying that 128 bits cpu and a 32bit cpu will do equal amount of work on 128 bit ZFS? I am claiming that 32 bit cpu needs to do additional work, you claim the opposite?

You're making some fundamental mistakes regarding processors, instruction lengths, and data handling.

 

[brutalizer]

Maybe I'm not making this clear: You're incorrect. Referring to the source code is not a valid argument.

I want to know in what way I am incorrect, of course.

Refering to the source code is a very valid argument, it is facit. But, no one of us has the time to study it, no. But everything you want to know, is there.

There are no 128 bit CPU's that'll run an OS that in any way supports ZFS, so your point with that comment eludes me.

Yes, but hyphothetically. See below.

I'm saying that you're comparing apples and oranges. I'm claiming that the 128 bits referred to in the context of ZFS has nothing at all to do with the 32/64 bits referred to in the context of CPUs. 128 bits is just a number.

Because ZFS is 128 bits, it must have variables in the data structs that are 128 bits. An hypothetical 128 bit cpu, would hold the entire variable in one register and do one calculation. An 32 bit cpu that are doing calculations with these 128 bit variables need to do four calculations because one register can not hold 128 bits, so it need to swap in different parts of the 128 bits and process them one part at a time. Do you agree on this?

 

[TCM2]

Because ZFS is 128 bits, it must have variables in the data structs that are 128 bits. An hypothetical 128 bit cpu, would hold the entire variable in one register and do one calculation. An 32 bit cpu that are doing calculations with these 128 bit variables need to do four calculations because one register can not hold 128 bits, so it need to swap in different parts of the 128 bits and process them one part at a time. Do you agree on this?

Whether or not that's true has no relevance to the real world since ZFS code doesn't consist of non-stop 128bit arithmetics. Look at it this way: If 128bit arithmetics make up maybe 1% of the whole code being executed on a file server, what does it matter if those 1% are four times slower? The CPU will be busy waiting for drives long before it's pegged doing 128bit arithmetics.

You are clinging to miniscule facts that play a very small role in the larger picture.

 

[raiderj]

Because ZFS is 128 bits, it must have variables in the data structs that are 128 bits. An hypothetical 128 bit cpu, would hold the entire variable in one register and do one calculation. An 32 bit cpu that are doing calculations with these 128 bit variables need to do four calculations because one register can not hold 128 bits, so it need to swap in different parts of the 128 bits and process them one part at a time. Do you agree on this?

This is precisely your error. Instruction length that a processor supports has nothing to do with the data structure size it can manipulate.

For a processor's speed in performing actual work, there is SO much more involved than instruction size.

 

[devman]

brutalizer you may just want to quit on this one. ZFS stability issues with 32 bit OS's are related to memory management.

The issue you bring up regarding portability of ZFS internal structs is not a real thing.

 

[DisposableHero]

how many bluray transcodings can you guys with an i3 do at once?

 

[brutalizer]

brutalizer you may just want to quit on this one. ZFS stability issues with 32 bit OS's are related to memory management.

I am not talking about ZFS stability anymore. I have left that discussion. I am interested to know more about bit size on cpus, so we are off topic, yes. But I want to learn.

Regarding the stability if we go back, the ZFS devs said there were some problems in the 32 bit code. I dont know really what they meant, if they refered to RAM limit, or if the code had some problems because they needed to cut some corners. But I got the impression the problems were not about the RAM limit, but in the code. See below (if you need to do four times more work, you need to try some optimizations)

Whether or not that's true has no relevance to the real world since ZFS code doesn't consist of non-stop 128bit arithmetics. Look at it this way: If 128bit arithmetics make up maybe 1% of the whole code being executed on a file server, what does it matter if those 1% are four times slower? The CPU will be busy waiting for drives long before it's pegged doing 128bit arithmetics.

Yes, that is true that there are other factors more important. But you are saying that a 128 bit cpu and a 32 bit cpu are doing exactly the same amount of work on 128 bit software - to which I object.

Assume ZFS stores all data in 128 bit blocks (because it is 128 bit), then a 128 bit cpu can calculate a checksum on all 128 bits at once. A 32 bit cpu needs to do 4 calculations on 32 bits a time. Hence four times more work, than a 128 bit cpu. And twice as much work as a 64bit cpu. Are you saying that I am wrong on this?

 

[TCM2]

Assume ZFS stores all data in 128 bit blocks (because it is 128 bit), then a 128 bit cpu can calculate a checksum on all 128 bits at once. A 32 bit cpu needs to do 4 calculations on 32 bits a time. Hence four times more work, than a 128 bit cpu. And twice as much work as a 64bit cpu. Are you saying that I am wrong on this?

Yes, that sounds completely ignorant of how ZFS and CPUs work.

 

[brutalizer]

Yes, that sounds completely ignorant of how ZFS and CPUs work.

Ok? In what way, can you please educate me? I would like to know.

 

[DPI]

Maybe I am not phrasing myself correclty. I am trying to say that if a 32 bit cpu tries to dabble with 128 bits, the cpu need additional administration. If you compare that to a 128 bit cpu.

Maybe you just have no clue what you're talking about, and don't really understand the concepts beyond buzzwords. Instead of derailing every thread with pointless arguments and wearing everyone's patience real thin, maybe chill out, take a step back and read for a while. You learn more by listening.

 

[brutalizer]

Thanks for your advice. If TCM2 explains a bit more on this, I will gladly read and learn. Or if you explain a bit more. Or if you post a link on this.

The thing is, I have a Bachelors and a Masters degree in computer science from a top notch university. In one of the classes we developed emulators for different cpus, and I learned a lot of cpu architectures. But my specialization is in algorithm theory. But as I work with computers, I would like to learn more on this, it seems that I have missed something big in my education as both you and TCM2 says so. BTW, I also have a Masters degree in math. So dont be afraid of complicated explanations, I think I am able to understand. So go ahead please and explain to us, or post some links. As I do when someone asks a question. I only want to help. Yes, this is actually how I see it: I tell people when they are wrong, and try to give good advice, and good links. Really. And I want learn too, if someone has something to teach me.

 

[raiderj]

Thanks for your advice. If TCM2 explains a bit more on this, I will gladly read and learn. Or if you explain a bit more. Or if you post a link on this.

The thing is, I have a Bachelors and a Masters degree in computer science from a top notch university. In one of the classes we developed emulators for different cpus, and I learned a lot of cpu architectures. But my specialization is in algorithm theory. But as I work with computers, I would like to learn more on this, it seems that I have missed something big in my education as both you and TCM2 says so. BTW, I also have a Masters degree in math. So dont be afraid of complicated explanations, I think I am able to understand. So go ahead please and explain to us, or post some links. As I do when someone asks a question. I only want to help. Yes, this is actually how I see it: I tell people when they are wrong, and try to give good advice, and good links. Really. And I want learn too, if someone has something to teach me.

Basically, you're confusing the instruction length that a processor can handle (32-bit/64-bit) with the size of the data structure that is being operated on. Think about it this way, if you manipulate a picture, say a JPG that is 2.0MB, your 32-bit processor can run an operation on that entire file. The data structure, a 2MB JPG, doesn't really matter. Same way with a 128-bit ZFS data structure. Loading the ENTIRE data structure into a CPU register isn't what happens anyway.

These topics are more in the realm of Computer or Electrical Engineering. There is some truth to the fact that a 64 or 128 processor could do things faster or more efficiently than a 32-bit processor, but that's an entirely different topic. CPU performance is enormously complex, and in no way can be simplified into the instruction length it supports.

If you really want to educate yourself, look into what makes a processor Turing Complete.

 

[brutalizer]

I dont think that you are loading the entire data struct into a CPU register. But the data struct has 128 bit elements, which separately gets loaded into several registers. Something like this:

struct {
128bitDataType Data1;
128bitDataType Data2;
...
}

so now, the entire data struct will not be loaded into register. But when Data1 gets processed it gets loaded into a cpu register. And if the cpu is 128 bits, the entire Data1 field gets loaded at once and the checksum can be calculated in one step.

If the cpu is 32 bit, the Data1 field can not be loaded into a cpu register, instead you need to load the first 32 bits, and do a checksum calculation, and then load bit 33-64 into the register, do a checksum, etc. You do four times the amount of work.

If you have a 128bit cpu and 2MB JPG file and load it into registers to work with, you need to load 2,000,000 / 128 = 15,625 times into a register before you have processed the entire file.

If you have a 32 bit cpu, and 2MB jpg file, you need to load 2,000,000 / 32 = 62,500 times into a register before you have processed the entire file. Four times as much work.

Frankly, I dont understand what you mean. In a 32 bit cpu, the registers are 32 bit, and you can only process data in chunks of 32 bits. If you have 128 bit cpu, you can process data in chunks of 128 bits because the registers are wider.

What has the instruction length to do with the data, a 2MB jpg file? A 32 bit cpu needs to do four times more load and stores, than a 128 bit cpu, and twice as many load/store than a 64 bit cpu. Do you not agree on this? Have I misunderstood this? I mean, a Nvidia card with 128 bit bus, compared to a 256 bit wide bus, uses twice as many operations when moving data - have I misunderstood this???

Turing complete - I now quite a lot about that stuff. I studied theoretical computer science. But this is more about low level programming, which I am going to do in project very soon, it needs to be fast. So I want to learn as much as I can.