Store.exe hangs up when performing an online backup with NTbackup.

[Starriol]

Good morning guys.

This is the second day that we encounter this beauty; the backup server performs an online backup of Exchange (remotely, they are 2 different PCs) and between 2 and 5 minutes later, Microsoft Exchange Information Store (store.exe) crashes.

The events went like this:

First day:


Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 220
Date: 14/01/2008
Time: 20:01:51
User: N/A
Computer: TS001
Description:
Information Store (4032) First Storage Group: Beginning the backup of the file D:\Exchsrvr\mdbdata\priv1.edb (size 31642 Mb).

For more information, click http://www.microsoft.com/contentredirect.asp.


************************
Event Type: Error
Event Source: Microsoft Exchange Server
Event Category: None
Event ID: 1000
Date: 14/01/2008
Time: 20:02:37
User: N/A
Computer: TS001
Description:
The description for Event ID ( 1000 ) in Source ( Microsoft Exchange Server ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: store.exe. 6.5.7650.28. 43dfe228. ese.dll. 6.5.7638.2. 43433bbb. 0. 00046f0a.
Data:
0000: 41 00 70 00 70 00 6c 00 A.p.p.l.
0008: 69 00 63 00 61 00 74 00 i.c.a.t.
0010: 69 00 6f 00 6e 00 20 00 i.o.n. .
0018: 46 00 61 00 69 00 6c 00 F.a.i.l.
0020: 75 00 72 00 65 00 20 00 u.r.e. .
0028: 20 00 73 00 74 00 6f 00 .s.t.o.
0030: 72 00 65 00 2e 00 65 00 r.e...e.
0038: 78 00 65 00 20 00 36 00 x.e. .6.
0040: 2e 00 35 00 2e 00 37 00 ..5...7.
0048: 36 00 35 00 30 00 2e 00 6.5.0...
0050: 32 00 38 00 20 00 34 00 2.8. .4.
0058: 33 00 64 00 66 00 65 00 3.d.f.e.
0060: 32 00 32 00 38 00 20 00 2.2.8. .
0068: 69 00 6e 00 20 00 65 00 i.n. .e.
0070: 73 00 65 00 2e 00 64 00 s.e...d.
0078: 6c 00 6c 00 20 00 36 00 l.l. .6.
0080: 2e 00 35 00 2e 00 37 00 ..5...7.
0088: 36 00 33 00 38 00 2e 00 6.3.8...
0090: 32 00 20 00 34 00 33 00 2. .4.3.
0098: 34 00 33 00 33 00 62 00 4.3.3.b.
00a0: 62 00 62 00 20 00 66 00 b.b. .f.
00a8: 44 00 65 00 62 00 75 00 D.e.b.u.
00b0: 67 00 20 00 30 00 20 00 g. .0. .
00b8: 61 00 74 00 20 00 6f 00 a.t. .o.
00c0: 66 00 66 00 73 00 65 00 f.f.s.e.
00c8: 74 00 20 00 30 00 30 00 t. .0.0.
00d0: 30 00 34 00 36 00 66 00 0.4.6.f.
00d8: 30 00 61 00 0d 00 0a 00 0.a.....
*******************************************
Event Type: Error
Event Source: Application Error
Event Category: (100)
Event ID: 1000
Date: 14/01/2008
Time: 20:13:36
User: N/A
Computer: TS001
Description:
Faulting application store.exe, version 6.5.7650.28, faulting module ese.dll, version 6.5.7638.2, fault address 0x00046f0a.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 41 70 70 6c 69 63 61 74 Applicat
0008: 69 6f 6e 20 46 61 69 6c ion Fail
0010: 75 72 65 20 20 73 74 6f ure sto
0018: 72 65 2e 65 78 65 20 36 re.exe 6
0020: 2e 35 2e 37 36 35 30 2e .5.7650.
0028: 32 38 20 69 6e 20 65 73 28 in es
0030: 65 2e 64 6c 6c 20 36 2e e.dll 6.
0038: 35 2e 37 36 33 38 2e 32 5.7638.2
0040: 20 61 74 20 6f 66 66 73 at offs
0048: 65 74 20 30 30 30 34 36 et 00046
0050: 66 30 61 f0a
*******************************************


Second day:

Event Type: Information
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 220
Date: 15/01/2008
Time: 20:00:10
User: N/A
Computer: TS001
Description:
Information Store (3840) First Storage Group: Beginning the backup of the file D:\Exchsrvr\mdbdata\priv1.edb (size 31642 Mb).

For more information, click http://www.microsoft.com/contentredirect.asp.

***********************************************
Event Type: Error
Event Source: Microsoft Exchange Server
Event Category: None
Event ID: 1000
Date: 15/01/2008
Time: 20:01:43
User: N/A
Computer: TS001
Description:
The description for Event ID ( 1000 ) in Source ( Microsoft Exchange Server ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: store.exe. 6.5.7650.28. 43dfe228. ese.dll. 6.5.7638.2. 43433bbb. 0. 000047e0.
Data:
0000: 41 00 70 00 70 00 6c 00 A.p.p.l.
0008: 69 00 63 00 61 00 74 00 i.c.a.t.
0010: 69 00 6f 00 6e 00 20 00 i.o.n. .
0018: 46 00 61 00 69 00 6c 00 F.a.i.l.
0020: 75 00 72 00 65 00 20 00 u.r.e. .
0028: 20 00 73 00 74 00 6f 00 .s.t.o.
0030: 72 00 65 00 2e 00 65 00 r.e...e.
0038: 78 00 65 00 20 00 36 00 x.e. .6.
0040: 2e 00 35 00 2e 00 37 00 ..5...7.
0048: 36 00 35 00 30 00 2e 00 6.5.0...
0050: 32 00 38 00 20 00 34 00 2.8. .4.
0058: 33 00 64 00 66 00 65 00 3.d.f.e.
0060: 32 00 32 00 38 00 20 00 2.2.8. .
0068: 69 00 6e 00 20 00 65 00 i.n. .e.
0070: 73 00 65 00 2e 00 64 00 s.e...d.
0078: 6c 00 6c 00 20 00 36 00 l.l. .6.
0080: 2e 00 35 00 2e 00 37 00 ..5...7.
0088: 36 00 33 00 38 00 2e 00 6.3.8...
0090: 32 00 20 00 34 00 33 00 2. .4.3.
0098: 34 00 33 00 33 00 62 00 4.3.3.b.
00a0: 62 00 62 00 20 00 66 00 b.b. .f.
00a8: 44 00 65 00 62 00 75 00 D.e.b.u.
00b0: 67 00 20 00 30 00 20 00 g. .0. .
00b8: 61 00 74 00 20 00 6f 00 a.t. .o.
00c0: 66 00 66 00 73 00 65 00 f.f.s.e.
00c8: 74 00 20 00 30 00 30 00 t. .0.0.
00d0: 30 00 30 00 34 00 37 00 0.0.4.7.
00d8: 65 00 30 00 0d 00 0a 00 e.0.....

**********************************************
Event Type: Error
Event Source: Application Error
Event Category: (100)
Event ID: 1000
Date: 15/01/2008
Time: 20:04:55
User: N/A
Computer: TS001
Description:
Faulting application store.exe, version 6.5.7650.28, faulting module ese.dll, version 6.5.7638.2, fault address 0x000047e0.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 41 70 70 6c 69 63 61 74 Applicat
0008: 69 6f 6e 20 46 61 69 6c ion Fail
0010: 75 72 65 20 20 73 74 6f ure sto
0018: 72 65 2e 65 78 65 20 36 re.exe 6
0020: 2e 35 2e 37 36 35 30 2e .5.7650.
0028: 32 38 20 69 6e 20 65 73 28 in es
0030: 65 2e 64 6c 6c 20 36 2e e.dll 6.
0038: 35 2e 37 36 33 38 2e 32 5.7638.2
0040: 20 61 74 20 6f 66 66 73 at offs
0048: 65 74 20 30 30 30 30 34 et 00004
0050: 37 65 30 7e0
**************************************************

Didn't find much information on this, except doing an offline defrag of the database, using eseutil. This haven't been done for about 2 years, so it looks like a good idea, if only to recover wasted disk space.

Any other ideas?

 

[Starriol]

Found this KB article, http://support.microsoft.com/kb/944254

Looks right, I have SP2, run public folders, but I get only 2 of the 3 events listed there. Also, it sounds too much of a coincidence that this happens 2-5 minutes after ntbackup starts.

And I'm not fan of installing hot fixes on a production server like nothing.

 

[4saken]

We use backupexec but experienced a somewhat similar problem on one of our exchange servers.
Im not sure how you do your backups, full/inremental/etc.

But when we missed a Full backup and a incremental would kick in the next day, our Information store would hang.
It would continue this behavior until we reran a full exchange backup, flushing logs etc.

So i am curious if you guys missed a Full backup, and if this is happening during your incrementals.

Does this happen if you manually initiate a Full exchange backup job? Also we are running Exchange 2003 at the moment.

 

[Starriol]

Nope, we always use full backups, so must be something else.
Thanks for the idea anyway

 

[Starriol]

Bump, third day that does this and can't do online backups... this is starting to worry me.
Also, about the same time this problem started, we see with my coworkers some "ghost" mails in a shared mailbox. I see "2 new mails", for example. With outlook (2000 or 2007, we use both), you can't see the mails. But when we go to OWA, they are there.

Any clues?