Network Caching

T

ToddW2

2[H]4U
Joined
Nov 8, 2004
Messages
4,017
Goals:

We only have a 1.5mbit connection, but it's guaranteed so I'd like to do some caching during the night when we're asleep to the common sites we visit/etc, and I'd also like the cache to work with windows updates, steam and origin games/patches if possible.

Semi-Plan

Since we only have 1.5mbit I'm hoping to run this on an ATOM or Intel low-low power CPU with 1 SSD to start, and then monitor writes/reads and possibly upgrade to RAID10 of S3500 drives.(Currently have ATOM with Linux already in ITX setup I could use)

I don't need this to do anything but act as a caching appliance.

We have cisco/zyxel router/firewall setup, so don't need any more duties like that.

Linux+Squid and then write a script to hit a url list I specify in the night to prime the cache?

Todd
 
I think you're going to find it hard to predicatively cache, but you'll likely be able to cache windows updates (which hopefully download at night anyway). A lot of sites will be serving cache-control headers to block caching, but hopefully you'll be able to cache css and javascript at least. (You can tell squid to ignore the rules, and cache anyway, but it's got a good chance of breaking login state or sharing information that shouldn't be shared, etc)

I wouldn't do raid 10 for cache storage either; you don't need redundancy, because the information is replaceable, and you don't need block level striping, you can tell squid to use multiple directories for storage, and it'll stripe more or less among them.
 
For Windows Update install a Windows Update server...it's by far the best method, though it does take a little admin to approve updates (or just auto-approve everything).

Your idea to script URL hits sounds feasible.

There are articles online about how to make Squid cache dynamic content but it's a pain to setup and can cause more headaches. Of course a lot of browsing is done over HTTPS nowadays (as it should be) and that stuff doesn't get cached by proxies anyway unless you setup an SSL intermediate proxy. Then you get into more grey areas. I liked Microsoft TMG for doing that but they discontinued it.

With Cisco stuff you can use WCCP to do a transparent Squid proxy.
 
toast0 said:
I think you're going to find it hard to predicatively cache, but you'll likely be able to cache windows updates (which hopefully download at night anyway). A lot of sites will be serving cache-control headers to block caching, but hopefully you'll be able to cache css and javascript at least. (You can tell squid to ignore the rules, and cache anyway, but it's got a good chance of breaking login state or sharing information that shouldn't be shared, etc)

I wouldn't do raid 10 for cache storage either; you don't need redundancy, because the information is replaceable, and you don't need block level striping, you can tell squid to use multiple directories for storage, and it'll stripe more or less among them.
Click to expand...

I'm not predicting anything, I'm providing a URL list. If I wanted to predict I could always check the proxy logs and write an algorithm though, that wouldn't be uncommon.

I'm more hoping to cache images than CSS/JS, as well as Win Updates and other Updates (games) just to save any re-download headache and add that slight edge to browsing performance on common sites :)

Replaceable... sure but at 1.5mbit replacing 4gigs for 1 game update takes a while let alone a half dozen, but you do have a point that I will have to think on about my biggest "cache hit" if I can get game updates to work or not.


MysticRyuujin said:
For Windows Update install a Windows Update server...it's by far the best method, though it does take a little admin to approve updates (or just auto-approve everything).

Your idea to script URL hits sounds feasible.

There are articles online about how to make Squid cache dynamic content but it's a pain to setup and can cause more headaches. Of course a lot of browsing is done over HTTPS nowadays (as it should be) and that stuff doesn't get cached by proxies anyway unless you setup an SSL intermediate proxy. Then you get into more grey areas. I liked Microsoft TMG for doing that but they discontinued it.

With Cisco stuff you can use WCCP to do a transparent Squid proxy.
Click to expand...

Thanks I'll have to check into that.
 
For windows update a windows update server is the best solution as mentioned above.
 
You must log in or register to reply here.
Back
Top