[SteveE]

Since patching my E2010 boxes with Windows Updates this week, the last 4 evenings I've seen the BlackBerryAgent.exe miss heartbeats then fail to start with error code 5305. The problem starts after midnight, and once it starts failing, I can't get it to start properly / behave unless I restart all BES services.

This link discusses that error, but all of those solutions imply that it never s tarts properly. Mine just starts failing during the evenings. I've just been getting up early to restart BES servicse, at which point it's fine the rest of the day.

Has anyone encountered this?

MAGT log snippet:

[30480] (02/01 03:45:09.354):{0x1280} [CFG] BES failover mode is ACTIVE
[41120] (02/01 03:45:09.354):{0x1280} [CFG] Address Lookup is enabled
[41122] (02/01 03:45:09.354):{0x1280} [CFG] LDAP search is enabled, LDAP PIM search is disabled, LDAP ALP search is disabled
[35035] (02/01 03:45:09.573):{0x1280} LDAP: Create 0x00a40a44
[30460] (02/01 03:45:09.573):{0x1280} [CFG] LDAP Info: Host Name = DCServerName.domain.local
[41123] (02/01 03:45:09.573):{0x1280} [CFG] Using up to 10 external CDO Helpers in keep-alive mode
[41124] (02/01 03:45:09.573):{0x1280} [CFG] AddOwnerApptID is disabled
[41125] (02/01 03:45:09.573):{0x1280} [CFG] SetLocaleIDs is disabled
[30302] (02/01 03:45:09.573):{0x1280} [CFG] OTAFM hard-delete server support is disabled, rescans are 0x00
[40532] (02/01 03:45:09.573):{0x1280} [CFG] vCard conversion support has been enabled
[40892] (02/01 03:45:09.573):{0x1280} [CFG] GlobalRescan config: messaging:O, calendar:O, pim:O
[41126] (02/01 03:45:09.573):{0x1280} [CFG] EnableExceptionStackTrace is disabled
[40000] (02/01 03:45:09.573):{0x1280} [CFG] MAPI Encoding is 1
[10277] (02/01 03:45:09.573):{0x1280} BlackBerry Messaging Agent BESServerName Agent 1 failed to start. Error code 5305
[50106] (02/01 03:45:09.573):{0x1280} Stopping BlackBerry Mailbox Agent 1 for Server BESServerName
[40507] (02/01 03:45:09.589):{0x1A88} ExtUDPLogThread: Code on closing = 0
[30000] (02/01 03:45:09.589):{0x1A88} [DIAG] EVENT=Unregister_thread, THREADID=0x1A88, THREADNAME="ExtUDPLogThread"
[35035] (02/01 03:45:09.620):{0x1280} LDAP: Delete 0x00a40a44
[30065] (02/01 03:45:09.651):{0x1280} BlackBerry Mailbox Agent 1 for Server BESServerName shutdown complete
====== END OF LOG ======

[knottyrope]

are you stopping BES during the updates?

Is BES installed on mail server or on dedicated box

proper boot sequence
DC, GC, SQL and Exchange, BES

[SteveE]

I am not. To be honest I used to, but the last couple of years BES recovered without issue so I stopped since it was a massive headache. Still, with multiple days of this, and the intermittent (yet scheduled) nature, I'm naturally suspect that a simple DC reboot would fix this. I'll give it a go either way, but I suspect something else is at play.

[SteveE]

I'm also finding that handheldcleanup.exe is not running. Despite typing in the exact BESName, I get:

Open() failed for BESName
OpenDB failed to open Database

I have run this before without issue, and wonder if it is related.

[knottyrope]

could be related, check with SSMS to see if you can access it

[SteveE]

The credentials check out, and I can telnet to the SQL server from the BES on the SQL port. Plus profiler shows the BES doing its usual work.

I'm going to investigate NSPI tonight, since we are mixed between 2003 and 2008R2 DCs, though why that issue would crop up now is beyond me given our user count has been pretty static / low at ~100 users. Our throttling policies are all kosher.

[knottyrope]

NSPI limit fix here

http://www.blackberry.com/btsc/KB17325

[SteveE]

No luck with the NSPI limit fix. Also re-set the throttling policies and re-created the MAPI profile.

Going to re-set permissions and possibly do a repair of BES files. Curioser and curioser what's causing this morning MAPI logon failures. The times are inconsistent outside of the 12AM-7AM consistency.

I wonder if upgrading from MAPI/CDO 1.2.1 (6.5.8211.0) to (6.5.8244.0) would be worthwhile. They're both 2011 releases.

Thanks for all your help.

[knottyrope]

I would use latest version or CDO if possible

[SteveE]

I've made a little progress - I have not yet updated MAPI/CDO, but did do a network capture. At the time we got this error in the ALRT log:

[30000] (02/10 02:14:13.728):{0x20A4} [Alarm::ActivateAlarm] Queuing alarm: <N/A> | BlackBerry Messaging Agent BESServer Agent 1 (Application Event Log on BESServer.domain.local) | 02/10/2013 02:14:13 (EFFF2825) -> BlackBerry Messaging Agent BESServer Agent 1 failed to start. Error code 5305

The network capture recorded:

1064653 2:14:13 AM 2/10/2013 8963.3886465 BlackBerryAgent.exe CAS.Domain.Com BESServer MSRPC MSRPC:c/o Fault: Call=0xE Context=0x0 Status=0x1C010014 Cancels=0x0 {MSRPC:11522, TCP:11520, IPv4:7}

That status (0x1C010014) indicates the server (one of the CASes) is too busy to fulfill the request.

Combine this with the IOS 6.1 issues with Exchange we're seeing, and I wonder if they're not related, since that update was released a day before we started seeing issues.

Anyway, we'll see what I find out next. Just need to collect some perf logs from the CASes.

[knottyrope]

IOS issues have killed my IIS a few times in past weeks from the nefarious syncs.

BYOD Nightmare: iOS Devices Hobble Exchange Servers When They Synch - CIO.com

iOS 6.1 banned from corporate servers due to Exchange snafu | ZDNet

[SteveE]

Originally Posted by knottyrope

IOS issues have killed my IIS a few times in past weeks from the nefarious syncs.

BYOD Nightmare: iOS Devices Hobble Exchange Servers When They Synch - CIO.com

iOS 6.1 banned from corporate servers due to Exchange snafu | ZDNet

Regrettably, throttling our IOS 6.1 users did not resolve the nightly issues.

MAPI / CDO upgrade is on the table for the weekend, but I wanted to get your thoughts on CAS Arrays. Both our MAPI profile and EWSCASURL/EWSSCPURL point to an NLB'ed address. Would you recommend pointing at a single CAS for those, or is the method we're using supported? We'd had no issues up until this month with pointing directly to the load-balanced address, but finding solid documentation is somewhat tricky.

Oooh, another possibility: http://support.microsoft.com/kb/2810616.

Endless!

[SteveE]

After using our hosts file to point the BES at a single CAS instead of the array, we no longer see the nightly 5305s. It appears one of the CASes (SCOM hints at which) is the culprit! Why it gets busier at night is beyond me, but that's an issue for a different forum.