Exchange 2007 Offline Maintenance and more.

[bigstusexy]

Hello, lots of questions.


I don't know much about Exchange and didn't really want to manage it but I have to. We have Exchange 2007, don't know if its SP1 or RTM. The problem is that our storage database drive has ran out of space and currently we are running on THE last bit of space we can get.

My boss wants to run the offline defrag, he said compression but it seems like offline defrag is what we want to do by cleaning up old items that are no longer needed. I have a few questions about this that I haven't found in a search here yet and I'm still looking online of course.

1. According to the way I've read on one site defrag works it creates a new database and basically reads all needed information to it. then places this new database inplace of the old. Does this mean that I should technially have double the space of the Storage group database available before doing this?

2. Anything else you think we should try?

3. I'm thinking we should delete attachments that are 1yr or older?

4. We have 728 boxes currently and our new superintendent is gaga over communication via email, so far not with attachments but its likely that there will be 6 or more All user emails sent per month, no telling how many departmental emails are sent weekly. What size of storage would you recommend in that case?

5. Because it will take time to get more storage and to move to it we are looking at maybe setting up NAS to move the larger boxes onto to test and if okay moving a sum of boxes over there to give us space. We have a few server laying around and I'm going to look at freenas and openfilder to possibly get that done, any other ways (NAS or not) to have remote storeage of Exchange datastores without having a second Exchange server?


Like I said I'm very new to exchange and to Powershell at the same time. I'm going to read up on ESUTIL now and I'm a bit worried because we have zero backups of exchange and no real space to do it right now and I have no time I could try to sneak and make one less I can do it while its online.


EDIT:
According to http://technet.microsoft.com/en-us/library/aa998863.aspx They recomend that you have 110% of the space of the resultant database.
According to http://technet.microsoft.com/en-us/library/aa997972.aspx, you can determine how much space you have after an online defrag that happens automatically and that would mean that it would be 175GB... we only have 7.11GB free and dropping


EDIT2:
I'm taking a look at http://support.microsoft.com/kb/240145 after replying to k1pp3r's request of file sizes and noticing large amounts of 1MB log files in there. If they turn out to be transaction logs and the DB is only 35GB or so then I can save a HUGE ammount of space by moving older logs to another drive, and they will be deleted when we do a proper backup.

 

[corge]

Why not just expand the storage? If you're referring to physical disks, hard drives are cheap. If you're referring to the storage database size limit, are you running Exchange Standard or Enterprise? The default database size limit for Exchange 2007 Standard Edition is 50 gigabytes and there is no default size limit in Enterprise version.

It may be that you just need to either add drives or upgrade to the Enterprise version. I would also recommend upgrading to the latest service pack and the hotfix rollup packs.

Go look for Exchange 2007 SP2 and get it installed. The following link goes to Microsoft's page on getting the latest updates for Exchange 2007.

http://technet.microsoft.com/en-us/library/ee221180.aspx

Are you running on multiple exchange servers or are all roles on one server? Might provide some more details if you're able.

 

[gimp0]

Why not just expand the storage? If you're referring to physical disks, hard drives are cheap. If you're referring to the storage database size limit, are you running Exchange Standard or Enterprise? The default database size limit for Exchange 2007 Standard Edition is 50 gigabytes and there is no default size limit in Enterprise version.

It may be that you just need to either add drives or upgrade to the Enterprise version. I would also recommend upgrading to the latest service pack and the hotfix rollup packs.

Go look for Exchange 2007 SP2 and get it installed. The following link goes to Microsoft's page on getting the latest updates for Exchange 2007.

http://technet.microsoft.com/en-us/library/ee221180.aspx

Are you running on multiple exchange servers or are all roles on one server? Might provide some more details if you're able.

The database storage limitation for Exchange 2007 is 16 TB not 50 GB and you can have 5 storage groups with 5 databases each so upgrading to Enterprise is not needed.

http://www.microsoft.com/exchange/2007/evaluation/editions.mspx


Bigstusexy,

You need to expand physical storage using either new drives in a seperate RAID array or with some form of direct attached storage or network storage (iSCSI or Fiber SAN).

Once you have the additional storage you can create new storage groups (up to 5) and move mailboxes to the new storage group/database that is located on the new storage drive.

If you already have multiple storage groups or databases you can migrate them to the new storage drives very easily.

Once that is all done you will have enough space to do online and off line defrags.

P.S. get those backups working and tested before proceeding with anything.

 

[gimp0]

4. We have 728 boxes currently and our new superintendent is gaga over communication via email, so far not with attachments but its likely that there will be 6 or more All user emails sent per month, no telling how many departmental emails are sent weekly. What size of storage would you recommend in that case?

5. Because it will take time to get more storage and to move to it we are looking at maybe setting up NAS to move the larger boxes onto to test and if okay moving a sum of boxes over there to give us space. We have a few server laying around and I'm going to look at freenas and openfilder to possibly get that done, any other ways (NAS or not) to have remote storeage of Exchange datastores without having a second Exchange server?

The size of storage you need is going to depend on your mailbox policy such as if you limit each mailbox to 2 GB etc.

You can control those mailbox sizes by removing e-mails older than a certain time or auto archiving to .PST located on a file server or SAN.

I would recommend either an iSCSI (cheaper but still fast) or fiber (faster but more expensive) SAN.

Load it up with 300-450 GB drives and you attach them to the exchange server using the built in iSCSI initiator and it will look like a normal storage drive.

This does take a little bit of engineering but there is a lot of info and recommendations.

 

[k1pp3r]

How many mail databases do you have and what are the sizes of the .edb and .stm files?

BTW, with 700+ mailbox's i'm going to assume your .edb files are beyond huge. Offline defrag is going to take you forever.

What is the average size of a user mailbox?

What is your labor budget? How many IT employees can you throw on this?

And most important, why wasn't this caught before it reached a critical level?

 

[bigstusexy]

To corge: I know I need more space, thats no doubt. We are talking physical space. We have already expanded the Array to another drive in a slot that is free but no more slots are avaliable. Question #4 was to get an idea of what we should be looking at. I can't tell you what level it is because... I can't find it, in the console it just says Exchange 2007, we are already well past 50GB but I would think the limits are in TB since its 64bit. This is one server with all rolls.

To gimp0: Oh Gosh I don't loose sleep because I'm not the director but I do sweat a little that we have no backups, this is a known issue to my director I have done my CYA and can do no more. Currently we have 3 storage groups only 1 for actual mail. If we do come to need a external array we will most likely contract that out, we just need an ASAP solution because this thing is going to eat its way to nonworking status very soon. Will archived mail in PST files still be accessible to users? Do you have any guess on what the size would be for that many users, like if someone told you they want an Exchange box with that many users what range would you start looking in, I know things like size limits come in to play but what about a guess? I would like to throw 750 at it really, at least.


Thanks for the input guys, oh during all these changes that have to come I will try to get in the service pack if its not in there.

 

[bigstusexy]

How many mail databases do you have and what are the sizes of the .edb and .stm files?

BTW, with 700+ mailbox's i'm going to assume your .edb files are beyond huge. Offline defrag is going to take you forever.

What is the average size of a user mailbox?

What is your labor budget? How many IT employees can you throw on this?

And most important, why wasn't this caught before it reached a critical level?

We have 3, one for actual mail, one where we send accounts to die and a second storage group for... Public storage group.

As for why it wasn't caught... Department politics That about all that I should say publicly Ithink, we added another drive to the array months ago and it has been known since the system was put in that there was no backup system. File storage server thanks, I'm taking my 20days of sick leave and month a vacation

First Storage (main mail) .edb 35.9GB* .stm Can't find it
Second .edb 36MB
Disabled .edb 1.25GB and 2MB

* in here we have a TON of Log files, 1MB each, they make up most of the space taken, some date back to a year ago or more, there are so many of them it slows down explorer they have names like E00000000A0.log, E0000000A00.log, etc

P.S. See the second edit that I'm making.

 

[gimp0]

We have 3, one for actual mail, one where we send accounts to die and a second storage group for... Public storage group.

As for why it wasn't caught... Department politics That about all that I should say publicly Ithink, we added another drive to the array months ago and it has been known since the system was put in that there was no backup system. File storage server thanks, I'm taking my 20days of sick leave and month a vacation

First Storage (main mail) .edb 35.9GB* .stm Can't find it
Second .edb 36MB
Disabled .edb 1.25GB and 2MB

* in here we have a TON of Log files, 1MB each, they make up most of the space taken, some date back to a year ago or more, there are so many of them it slows down explorer they have names like E00000000A0.log, E0000000A00.log, etc

P.S. See the second edit that I'm making.

Those are transaction logs that are created by exchange and are NOT cleared until there is a backup. Once you backup exchange all of those logs will be removed.

They do this so that in the event of a failure the logs can be replayed to restore the database.

How much space are the logs taking up? would clearing them solve all the issues?

2 Things to consider

1. Get your backups set up and the problem is gone as the logs are cleared everytime a backup is done.

2. Enable circular logging (not recommended) as you lose the ability to replay logs and restore your database without a full backup

Circular logging limits the amount of log files that can be created by overwriting the oldest log so it does not use the entire disk like it is doing to your server. This is not meant to be used on your exchange database server.

DO NOT DELETE THOSE LOGS MANUALLY

 

[Captain Colonoscopy]

you need to run a backup to clear out those old log files. that should free up a good chunk of space. DO NOT manually delete any log files, you could seriously break crap and have to cal MS to get it working again.

If you're running server2003 then you can just run ntbackup to a network drive or a USB drive as a quick fix.
Posted via [H] Mobile Device

 

[k1pp3r]

We have 3, one for actual mail, one where we send accounts to die and a second storage group for... Public storage group.

As for why it wasn't caught... Department politics That about all that I should say publicly Ithink, we added another drive to the array months ago and it has been known since the system was put in that there was no backup system. File storage server thanks, I'm taking my 20days of sick leave and month a vacation

First Storage (main mail) .edb 35.9GB* .stm Can't find it
Second .edb 36MB
Disabled .edb 1.25GB and 2MB

* in here we have a TON of Log files, 1MB each, they make up most of the space taken, some date back to a year ago or more, there are so many of them it slows down explorer they have names like E00000000A0.log, E0000000A00.log, etc

P.S. See the second edit that I'm making.

I feel like i'm missing something here.

You have under 40 GB of exchange DB's, minus whatever the streaming DB is. How large are your drives, and how do you manage to have such a small edb with over 700 mailbox's? That would make each mailbox 55 MB's or so, thats one hell of a restriction.

How much drives space is on that logical drive?

Like said before me, run an exchange backup to clean the log files

 

[bigstusexy]

Thanks gimp0 and Captain Colonoscopy, I completely understand your warning and I did read that KB from the start to the finish. I would never delete them but I would consider moving them off the drive but it still would mean that I'd need enough space for all the logs plus the DB if I need to replay them.

How much space are they taking up? 138GB of the 175GB of the entire folder (besides the index which is like 1.2GB or something)

I'm going to test one of these network storage guys and see if it will work well enough, if not I have more boxes I can use and lots of storage and I'll do a backup starting after hours.


I'm going to go on a little rant here. Since the new Director came in we let him know "our dirty little secrete" that we have NO backups and no retention system. Had one meeting with a vendor about a solution and he though it was too pricey but it would give us 100% coverage and retention along with a good amount if not 100% bare metal restore. That was almost a year ago and we have nothing... I asked for power strips so I could deploy a few computers back before the summer... still waiting.

 

[gimp0]

Those numbers look way off to me as well. When you originally asked me what size storage to look at for 750 users I would say in the 4-7 TB range but you are only using ~80 GB when you factor in the .stm file.

Are these student mailboxes that hardly get used and have 20 e-mails in them?

How many staff members do you have and how many students? That will give us a better look at the size requirements for your exchange environment. You can limit students to a 500 MB - 1 GB mailbox and staff to a 5 GB and depending on the ratio of teachers to sutdents your storage requirements are reduced drastically.

Look at current mailbox sizes as well and see what they looks like.

 

[gimp0]

Thanks gimp0 and Captain Colonoscopy, I completely understand your warning and I did read that KB from the start to the finish. I would never delete them but I would consider moving them off the drive but it still would mean that I'd need enough space for all the logs plus the DB if I need to replay them.

How much space are they taking up? 138GB of the 175GB of the entire folder (besides the index which is like 1.2GB or something)

I'm going to test one of these network storage guys and see if it will work well enough, if not I have more boxes I can use and lots of storage and I'll do a backup starting after hours.


I'm going to go on a little rant here. Since the new Director came in we let him know "our dirty little secrete" that we have NO backups and no retention system. Had one meeting with a vendor about a solution and he though it was too pricey but it would give us 100% coverage and retention along with a good amount if not 100% bare metal restore. That was almost a year ago and we have nothing... I asked for power strips so I could deploy a few computers back before the summer... still waiting.

I wouldn't bother with an extra box at this point, you just need to get some form of backup running on the Excahnge database and you will reclaim all that space.

Grab a Western Digital 2 TB External USB Drive and it will be better than what you are using / not using now. Windows / NTBackup (built in to the server) and you have some form of temporary protection until you can get your real back up solution in place.

http://www.newegg.com/Product/Product.aspx?Item=N82E16822136281

 

[bigstusexy]

I'm just going to back up to the other server gmp0, I'd love to order that guy but 1. Can't order from newegg because they don't take P.O.s and we need this done like now. I've just seen an all user email and we are down to 6.91GB, not saying it did that alone.

I'm very familiar with most MS products and windows in general (not server 2008 boy oh boy did they change things in AD!) I didn't know NTbackup would suffice for doing this sort of backup and removing the logs for me until you all told me.

NTbackup will be work with exchange to do this while its online? Will it be backing up just a clean database or is it going to get the database and all log files like a regular file copy? If its just going to use VSS to copy the files then I will want to do the manual remove later which means I'll have to take it down so I need to announce that soon.

This is just temporary but you know how those things can go sometimes.


Thank you everyone for helping me, I think this is the biggest server related problem that has worried me yet.

 

[Captain Colonoscopy]

just wait until after hours and do your ntbackup then. Keep in mind that server 2008 ntbackup can't backup exchange unless you're running SBS2008. So, are you running server 2003 or server 2008 as the OS on the Exchange server?

 

[bigstusexy]

Server 2003 x64 is the host OS.

 

[gimp0]

I'm just going to back up to the other server gmp0, I'd love to order that guy but 1. Can't order from newegg because they don't take P.O.s and we need this done like now. I've just seen an all user email and we are down to 6.91GB, not saying it did that alone.

I'm very familiar with most MS products and windows in general (not server 2008 boy oh boy did they change things in AD!) I didn't know NTbackup would suffice for doing this sort of backup and removing the logs for me until you all told me.

NTbackup will be work with exchange to do this while its online? Will it be backing up just a clean database or is it going to get the database and all log files like a regular file copy? If its just going to use VSS to copy the files then I will want to do the manual remove later which means I'll have to take it down so I need to announce that soon.

This is just temporary but you know how those things can go sometimes.


Thank you everyone for helping me, I think this is the biggest server related problem that has worried me yet.

You can grab that External drive at any store locally and possibly from your supplier. I bought one for some temporary use at Best Buy or Compusa can't remember.

You will want to schedule the backups to run overnight and it will only backup the database and not the logs, unless you manually select the logs on accident.

This will back up the database while it is online so no need to prepare downtime for it, but I wouldn't do it during heavy usage during the day.

edit: also, you can download an evaluation copy of a backup suite like Symantec Backup Exec 12.5 and backup exchange using it, think they have a 60 day trial. I have used it for a while and the granular restore of individual e-mails is pretty nifty.

 

[Captain Colonoscopy]

Okay, you should be good to go then. Just grab a USB drive or find some space on another server to backup to using NT backup and flush the logs. Should be golden, for a while, after that.

 

[bigstusexy]

Thanks all, gimp0 I can't do that. I don't think I explained the politics and my position I can't purchases anything on my own and we are a school district, to buy even a pen requires a ton of paperwork and approval unless you are the right person or know the right person. Luckly we have some servers with space that aren't in use... yet So I have repurposed it with OpenFiler and now I just have to figure out iscsi or I'm going to call it quits and just use the SMB share.

I was thinking about setting it go off around 6 maybe, I know no one wants to stick around after work today like I am.

 

[gimp0]

Thanks all, gimp0 I can't do that. I don't think I explained the politics and my position I can't purchases anything on my own and we are a school district, to buy even a pen requires a ton of paperwork and approval unless you are the right person or know the right person. Luckly we have some servers with space that aren't in use... yet So I have repurposed it with OpenFiler and now I just have to figure out iscsi or I'm going to call it quits and just use the SMB share.

I was thinking about setting it go off around 6 maybe, I know no one wants to stick around after work today like I am.

Gotcha, sounds like a bitch

I would just toss windows server 2003/2008 on that spare box and back up to a share if it is just a temporary thing. No need to mess with iSCSI unless you want to do it to learn.

 

[bigstusexy]

It would be nice to learn but i want to go home, I think I'm halfway home though I found a nice article on msexchange.com about iscsi.

Yeah it does suck hard, I remember years ago when I hadn't been here for that long I had a server die a week before I was going to replace it. Turns out one of the (at that time) stupid contractors somehow killed one the IDE channels. I thought it was a software raid drive done by windows,which it was, but it was raid 5 not raid 1 or 0. All I needed was a cheap IDE card and I could have the server limp along until it was replaced but I couldn't get the okay. Luckly I found out it was raid 5 and I put the two data drives in and I could force it to come up and I copied all the files and settings to the new guy and retired it.

 

[bigstusexy]

This is just a follow up. THANK YOU EVERYONE!

I got openfiler up and running after a few tires. I worked through a tutorial on msexchange.com (some place I will be reading lots from) and fumbled through to connect back to Openfiler via ISCSI except that there was one problems that the volumes weren't showing up. Turns out like most interactions with linux services you had to restart it and I made it. I took instructions fromThis windowsnetowrking.com article to figure that out.

I selected the entire exchange drive (the mail database has its own drive) from the top and system state just for GP and backed it up, maybe I shouldn't as it took all the log files as well but I don't care. I later removed the logs upto about two months ago and that left me with 131GB to use. If we drop below 100GB or at Christmas break I plan to take the database off line check its state and back it up again ditching the current Emergency back for that clean one, then I'll be small enough that I will set it to update once a week until something proper is purchased and maybe I can sneak out important FS server on there too.


Thanks again.

 

[k1pp3r]

Well there ya go, plenty of drive space now!

maybe I can sneak out important FS server on there too.

Given the department political crap you deal with, i don't suggest "sneaking" anything, that may end with you looking for future employment. You will have to play the politics game now.

Good luck!