Your Crashplan Upload Speeds?

Zarathustra[H]

Zarathustra[H]

Extremely [H]
Joined
Oct 29, 2000
Messages
38,877
Hey all,

For those of you who use Crashplan for your backup needs, I wanted to check what kind of speeds you are seeing, to see if my own install is performing as expected.

I have a 150/150 FiOS connection. I set my in app upload limit to 125Mbit (to make sure I had a little upstream bandwidth left for other things). My uploads started at ~85Mbps, but quickly dropped off, and now are averaging between 20 and 30 Mbps depending on when I check.

This is obviously a little disappointing. Is this just the way things are with Crashplan, or is it possibly Verizon is throttling me?

Could it be a local machine performance issue?

Backups are occurring from within a Ubuntu Server guest on my ESXi server. I ahve assigned it two cores of my 12 core (24 logical) low power Xeon system. (Base clock 2.26Ghz, turbo up to 2.8)

I told the client it can use 100% of the CPU, wheter idle or not.

Once core seems maxed out at 100%, the other core is hovering at about 20%.

I have disabled deduplication in the frontend.

Any thoughts, is this expected? Would throwing more cores at it have any effect?

Thanks,
Matt
 
Guess this doesn't help you. I only have 5mbit upload speed and CrashPlan at least maxes that out.
 
SirMaster said:
Guess this doesn't help you. I only have 5mbit upload speed and CrashPlan at least maxes that out.
Click to expand...

People and their problems huh?
I wish I could "only" get 20-30Mbps. :(

@OP
Yes, you will be throttled. That's just the nature of the business.
They let you burst, but ultimately throttle you.
 
spectrumbx said:
People and their problems huh?
I wish I could "only" get 20-30Mbps. :(
Click to expand...

Well, I pay a lot extra for the privilege, so it would be nice to take advantage of it.

spectrumbx said:
@OP
Yes, you will be throttled. That's just the nature of the business.
They let you burst, but ultimately throttle you.
Click to expand...

I was under the impression FiOS didn't throttle much. I have gotten sustained upstream use near my max for extended periods of time on big torrents like Ubuntu ISO's and in file transfers between friends, so I was assuming that if the congestion wasn't on Crash plans side, I'd get near full upstream utilization, but maybe they have singled out cloud backup services specifically.

Its kind of lame that they are happy to charge you extra for more bandwidth, but then negate that by not letting you actually use it... Makes you wonder what the point is :p

I'm still wondering if a contributing factor is that 448bit blowfish encryption they use is very CPU intensive, and only coded to use one CPU thread, based on the fact that one core is pinned and the other is only at ~20%
 
For what it's worth, I edited the settings XML file manually, as explained here and now I am going at full speed as fast as my connection allows.

Looks like setting dedup to "minimal" in the GUI wasn't sufficient to avoid CPU bottlenecks.
 
I have a 2gbit connection, and when I stopped using crashplan 2years ago, I could only substain 11mbit upload speeds during crashplans idle times, during their peeks it would drop to around 3mbit. That lasted for around 6months or so, when it went to 3mbit all day long. At this point, it couldn't keep up with my backups any longer, and when 12months came around, I didn't bother renewing.

The software though, is able to go much faster. I still use the crashplan software to backup my house to the datacenter, and datecenter to my house, and it has no problems doing >100mbit.
 
patrickdk said:
I have a 2gbit connection, and when I stopped using crashplan 2years ago, I could only substain 11mbit upload speeds during crashplans idle times, during their peeks it would drop to around 3mbit. That lasted for around 6months or so, when it went to 3mbit all day long. At this point, it couldn't keep up with my backups any longer, and when 12months came around, I didn't bother renewing.

The software though, is able to go much faster. I still use the crashplan software to backup my house to the datacenter, and datecenter to my house, and it has no problems doing >100mbit.
Click to expand...

Did you ever manually edit the config file to effectively disable deduplication as posted above?

The crashplan software assumes that your upstream connection is very slow, and your CPU is very fast, and in that scenario aggressive deduplication makes a lot of sense, as with a very slow connection, every file you don't need to upload saves A LOT of time.

Problem is deduplication algorithms are VERY CPU intense, and if you are one of the few home users in the U.S. that actually have decent upstream bandwidth, the dedup algorithm your CPU will be the bottleneck long before your upstream bandwidth is (it's only single threaded, so one core will be maxed). This is especially the case for me, running crashplan on a low power server XEON clocked only at 2.26Ghz.

Setting dedup to "minimal" in the front end appears to have no effect, but if you go in and edit the xml file and tell it to skip the dedup check on all files larger than one byte (essentially disabling it), the uploads should speed up significantly.

I uploaded at 90Mbps constantly all night, last night.

I could probably have uploaded faster if I changed server (and I still might do this, even if it requires you to start over). It appears as if Crashplan relies on the geolocation associated with ranges of IP addresses in picking which server to use.

I'm in Boston, but my IP - for some reason - is indicated as being in Kansas, so I probably wound up on the wrong server. (this causes problems on Speedtest.net as well, where I have to manually select a Boston based server to get accurate speed readings, otherwise I get poor readings to Kansas)

Crashplan probably would have been better off selecting the server based on actual metrics (like number of hops to server, or latency, or a brief bandwidth test, or a weighted combination of the three), but it is what it is.

THe link above speaks to switching servers, as well, and I may give it a try and report back.
 
Zarathustra[H];1041222435 said:
I could probably have uploaded faster if I changed server (and I still might do this, even if it requires you to start over). It appears as if Crashplan relies on the geolocation associated with ranges of IP addresses in picking which server to use.

I'm in Boston, but my IP - for some reason - is indicated as being in Kansas, so I probably wound up on the wrong server. (this causes problems on Speedtest.net as well, where I have to manually select a Boston based server to get accurate speed readings, otherwise I get poor readings to Kansas)

Crashplan probably would have been better off selecting the server based on actual metrics (like number of hops to server, or latency, or a brief bandwidth test, or a weighted combination of the three), but it is what it is.

THe link above speaks to switching servers, as well, and I may give it a try and report back.
Click to expand...

Actually scratch that.

Actually read the link on that site, and it turns out changing server just randomly selects another one, and doesn't help you choose one geographically close to you. My assumptions where wrong.

Well, at least I have the dedup thing sorted out.
 
Not sure what any of that has to do with my tests, as I stated, I can upload at >100mbit, but ONLY to my own servers.

Using crashplans servers (from 2 years ago) limited me to 11mbit.

They where likely overloaded.

But due to the upload speed issue, and the fact that twice they lost my data, and it had to resync. I have no faith in their service, but the software seems to work ok (if you don't use the dedup option).
 
patrickdk said:
Not sure what any of that has to do with my tests, as I stated, I can upload at >100mbit, but ONLY to my own servers.

Using crashplans servers (from 2 years ago) limited me to 11mbit.

They where likely overloaded.

But due to the upload speed issue, and the fact that twice they lost my data, and it had to resync. I have no faith in their service, but the software seems to work ok (if you don't use the dedup option).
Click to expand...

If you look at the xml config files you'll notice that by default they use much less dedup when uploading to non-crashplan servers (presumably to work around the CPU limitation when there is more assumed bandwidth) which would explain why you might be able to backup more quickly to your own servers.

Also, I wasn't using Crashplan 2 years ago, they may have more robust bandwidth now.

All I know is, my uploads dropped as low as 10Mbit. I stopped the service, edited the xml file and restarted it, and I've been averaging 90mbit since then.

Projected upload times went from a month to 4.5 days...
 
Last edited:
I've been using CrashPlan for 3 years and have never seen below my 5Mbit upload. So not sure why you could only get 3Mbit for a period.

Also, switching servers will make you start your backup over.
 
None of the budget/unlimited backup providers allow for very good speed. I have a 300 megabit upload and the best I was able to get was around 50-60 megabits via multiple connections (which was possible as they supported rsync for uploads) with OLS. Of course they went out of business... If yo udo actually find one with goood speed lets me know :)
 
Well, for what it's worth, after making the dedup config mod, I uploaded 5.4TB to Crashplan in 5 days.

Not too shabby, IMHO.

Did have to edit the config file to manually override the max ram on the Crashplan service though, as it would repeatedly crash during rescan with the default 1 gig. Now it has 4 gigs and appears to be running fine.
 
Last edited:
I know this isn't totally on topic, but for those of you who have attempted using crashplan with decent connections, have you had any issues with them booting you off for using excessive storage due to your IP not coming from a residential ISP? I once backed up about 8TB (gigabit upload speed) to a similar service offering unlimited backups, however due to me using a VPN my IP didn't come from a residential ISP and they decided I was a business customer and revoked my account with the option to switch to their business plans that were extremely costly.
 
bleomycin said:
I know this isn't totally on topic, but for those of you who have attempted using crashplan with decent connections, have you had any issues with them booting you off for using excessive storage due to your IP not coming from a residential ISP? I once backed up about 8TB (gigabit upload speed) to a similar service offering unlimited backups, however due to me using a VPN my IP didn't come from a residential ISP and they decided I was a business customer and revoked my account with the option to switch to their business plans that were extremely costly.
Click to expand...

Well, I just uploaded over 5TB with no problems, but I wasn't using a VPN.

I suspect the VPN may have been more to blame than the upload size, but I could be wrong
 
I started my backup to crashplan last night and have seen a fairly steady 65-70Mbit/s to them, reaching as high as ~100 and as low as 11 just depending on when I look over and time of day. This is almost all incompressible data. I disabled dedup as suggested by the blog post linked by Zarathustra[H]. My upload is gigabit but i'm happy with the speeds i'm getting now, especially for the price. I too tried backing up to them about a year (maybe 2?) ago and speeds were so abysmal back then i gave up, if I ever had to restore any data it would have taken far too long. Hopefully the current performance is here to stay for a while.
 
I have 17TB in my CrashPlan backup set personally and have not heard anything from them about this being a problem.
 
So,

Does anyone have any experience in doing full restores? What speeds did you get?

I did a web restore of a small folder just to test it, and averaged a rather miserable 14Mbit, which concerns me as if the worst happens, and I ever need to restore, that restore is going to take a month or more!

Hopefully restoring in app is faster.
 
I've done restores from the app on many GB before and it maxes out my 35Mbit downstream at least.
 
I stopped using CrashPlan awhile back, but I remember the initial seed being extremely slow. I switched to Acronis cloud backups and was able to do an initial whole hard drive backup in one day. It was about 150 GB. I didn't pay attention to the upload speed, but it didn't seem like Acronis was throttling the upload at all like CrashPlan (used to do, not sure if they still do.)
 
It's looking like they've capped me to 10Mbit uploads after uploading about 400GB. I haven't seen faster than 11Mbit in over a day now unfortunately.

*edit*
spoke too soon, back up to 28Mbit.
 
Last edited:
I'm not sure it makes sense to use such a service if you don't know what speed you'll get down the road, both upload and download. If you add to that the fact they don't backup...
 
bleomycin said:
It's looking like they've capped me to 10Mbit uploads after uploading about 400GB. I haven't seen faster than 11Mbit in over a day now unfortunately.

*edit*
spoke too soon, back up to 28Mbit.
Click to expand...

I'm not convinced it is a matter of "capping" per se.

I think several things affect speed:

1.) How heavily service is taxed.

I have noticed slowdowns during prime time. This could be due to ISP congestion, or due to limited bandwidth on the Crashplan side. After all, it is a relatively inexpensive all-you-can-eat service, so it wouldn't surprise me if they cut corners on bandwidth a little.

2.) Types of files.

Some files are compressible, some are not. Some files are caught by the dedup algorithm (if you haven't manually disabled it in conf file) some are not. (and either way, the speed formula in the client seems to count the full size as having been uploaded, as my upload speed indicator has on occasion jumped up to 800mbps. I have nowhere near that kind of bandwidth)

Also, just like hard drives with seek times, there must be some per-file overhead in the protocol, as I have noticed small amounts of large files upload MUCH faster than large amounts of small files.


Anyway, now that I have completed my initial upload, I have throttled my upstream bandwidth in the crashplan client to 15mbps. This seems to be sufficient to upload my daily additions without creating a backlog.

I just hope that if the time ever comes when I need to restore, it goes much faster than the 1.8MB/s I topped out at during my small web restore test.
 
Very valid points Zarathustra[H], in my case it would definitely seem it is just network or i/o saturation on their end that is causing the slowdowns, which is to be expected for the price point. I'm transferring large incompressible files with dedup disabled.

I'm using the service as a backup of a backup so the transfer and restore times aren't very important to me, its mostly just piece of mind. I just figured it would be helpful to others to know the ballpark speeds the service is capable of so they can plan accordingly.
 
Yeah I really doubt throttling. You have to realize they have tens of thousands of users and if there are people using a couple dozen Mbit/s of bandwidth all by themselves it quickly adds up and probably saturates their few hundred Gbit/s data center connection.
 
You must log in or register to reply here.
Back
Top