OpsMgr by Example: Configuring Baselines

We are starting a series of “OpsMgr by Example” blog posts. The intent is to provide both the 5,000 foot/meter perspective on what to do and show the details for a particular type of tuning performed in an example deployment.This first one is related to an Exchange MP deployment (not covering all of Exchange). We hope to follow-up with OpsMgr by Example posts for the AD MP, SQL MP, and potentially the Exchange Management Pack.Let’s begin by talking about configuring baselines. While tuning the OpsMgr 2007 Exchange 2003 Management Pack, the majority of the alerts generated were a result of the calculated baseline rules. The following is a step-by-step for how to configure the sensitivity of these rules to decrease the alert volume. But first a huge thanks to the explanation on this which started us down the path on how to do this tuning:http://www.eggheadcafe.com/software/aspnet/29844092/tuning-baselining-monitor.aspxThe following were the primary alerts that caused large amounts of volume in the OpsMgr environment:

  • Information Store Transport Temp Table is outside the calculated baseline
  • Mailbox Store Send Queue is outside the calculated baseline
  • SMTP Local queue is outside calculated baseline
  • SMTP Messages in the Queue Directory is outside calculated baseline
  • SMTP Remote Queue is outside the calculated baseline
  • SMTP Remote Retry Queue is outside the calculated baseline
  • IS Virtual Bytes is outside the calculated baseline
  • Number of RPC requests is outside the calculated baseline

Perform the following steps for all Alerts that are causing significant volume in your environment. Implement these one at a time. We recommend following the order listed, as it groups together the types of rules to make them easier to find. The steps that refer to the Exchange Queue will vary depending upon the rule and monitor changed. The first six alerts above are all part of the Exchange Queue, the last two are part of Exchange IS Service. Change each on both the monitor and rule level.Also, we strongly recommend you save your changes to an unsealed MP other than the Default management pack.Steps to resolve: (perform all of these steps for each Alert in your environment which needs to be tuned)

  1. Find the rule that applies to the alert. (To find the rules, it’s easiest to change the scope to filter by the two areas that we need – which are the Exchange Queue and Exchange IS Service. Both of these are available when you click on scope and choose the option to view all targets. Then find rules with “Baseline Collection” as the start. This scopes it down to about 17 rules versus over 6000.) Details on the names of each of the above rules are listed below. Disable the rule by creating an override to set Enabled to False. 
  2. Change the rule sensitivity to 2.81 (Right-click on the rule, Overrides, Override the rule, For all Objects of type: Exchange Queue, check the Sensitivity parameter and set it to 2.81 if it’s not already set to that value, click OK).
  3. Find the monitor that applies to the alert. This can be found by searching or scoping to the type of object identified for the rule. Disable the monitor (Right-click on the monitor, Overrides, Disable the monitor for all objects of type: Exchange Queue, click yes to accept).
  4. Change the monitor inner sensitivity to 2.81 (Right-click on the monitor, Overrides, Overrides the monitor, For all Objects of type: Exchange Queue, check the Inner Sensitivity parameter and set it to 2.81 if it’s not already set to that value, click Ok).
  5. Change the monitor outer sensitivity to 3.31 (Right-click on the monitor, overrides, Overrides the monitor, For all Objects of type: Exchange Queue, check the Outer Sensitivity parameter and set it to 3.31 if it’s not already set to that value, click Ok).
  6. Re-enable the monitor. (Right-click on the monitor, click on Overrides Summary, delete the override that says Type, Exchange Queue, Enabled, False).
  7. Go back to the rule identified in step #1 and re-enable the rule. (Right-click on the rule, click on Overrides Summary, delete the override that says Type, Exchange Queue, Enabled, False).

Mapping for Alerts, Rules and Monitors:

ALERT=Information Store Transport Temp Table is outside the calculated baseline
RULE=Baseline Collection Rule for Information Store temp table number of entries (Rules, of type Exchange Queue)
MONITOR=IS Transport Temp Table Monitor (Exchange Queue, Entity Health, Performance)
 
ALERT= Mailbox Store Send Queue is outside the calculated baseline
RULE=Baseline Collection Rule for Mailbox Store Send Queue Length (Rules, of type Exchange Queue)
MONITOR=MB Store Send Queue Monitor (Exchange Queue, Entity Health, Performance)
 
ALERT=SMTP Local queue is outside calculated baseline
RULE=Baseline Collection Rule for SMTP Server Local Queue (Rules, of type Exchange Queue)
MONITOR=SMTP Local Queue Monitor (Exchange Queue, Entity Health, Performance)
 
ALERT=SMTP Messages in the Queue Directory is outside calculated baseline
RULE=Baseline Collection for SMTP Message Queue Directory (Rules, of type Exchange Queue)
MONITOR=SMTP Message Queue Directory Monitor (Exchange Queue, Entity Health, Performance)
 
ALERT=SMTP Remote Queue is outside the calculated baseline
RULE=Baseline Collection Rule for SMTP Server Remote Queue Length (Rules, of type Exchange Queue)
MONITOR= SMTP Remote Queue Monitor (Exchange Queue, Entity Health, Performance)
 
ALERT=SMTP Remote Retry Queue is outside the calculated baseline
RULE=Baseline Collection Rule for SMTP Server Remote Retry Queue Length (Rules, of type Exchange Queue)
MONITOR=SMTP Remote Retry Queue Monitor (Exchange Queue, Entity Health, Performance)
 
ALERT=IS Virtual Bytes is outside the calculated baseline
RULE=Baseline Collection Rule for IS Virtual Bytes (Rules, of type Exchange IS Service)
MONITOR=IS Virtual Bytes Monitor (Exchange IS Service, Entity Health, Performance)
 
ALERT= Number of RPC requests is outside the calculated baseline
RULE=Baseline Collection Rule for IS RPC Requests (Rule, of type Exchange IS Service)
MONITOR=IS RPC Requests Monitor (Exchange IS Service, Entity Health, Performance)

UPDATE for the IS Virtual Bytes and RPC requests:After configuring the inner and outer sensitivities (Steps 4 and 5 in that article), several of the rules were still  generating large volumes of alerts. Those rules include:

  • IS Virtual Bytes is outside the calculated baseline
  • Number of RPC requests is outside the calculated baseline

The alerts were identified as “Above Inner Envelope.” To minimize their frequency, we previously changed both the rule and the monitor’s sensitivity from 2.81 to 3.31 on the overrides.From reading up on this sensitivity concept, it appears that increases to this value decreases the frequency of the alerts, as it decreases the sensitivity to the difference from the calculated baseline.In theory, if the 3.31 override was not sufficient, then one should next try 3.81; this is because the increase from 2.81 to 3.31 is an increase of .5; therefore another .5 increase seems logical if another value change is required. This is an extrapolation based on what we have seen so far, as we do not know the internal workings of the algorithm! The feedback we have seen indicates that 3.81 works quite well.

This entry was posted in Tuning and Configuration. Bookmark the permalink.

8 Responses to OpsMgr by Example: Configuring Baselines

  1. Ben S says:

    Thanks for this. These automated baselines have been causing a real headache for me so it\’s great to now have a way to control them.

  2. Unknown says:

    It\’s worth noting that as of MP 6.0.6278.0, these sensitivity settings are the defaults for the Exchange management pack.

  3. sebastian.haraldsson says:

    Yes, thanks alot for this, it very much explains the reason for all these alerts I\’m getting.However, I do have these values by default as Scottv said and I still get tons of alerts. When I try to make it even less sensitive (e.g. changing 3.31 to 3.81) I can\’t put in the correct value. When I enter 3.81 and click apply it changes to 381 and that is by far too insensitive to be of any use. I get so many alerts that I have now had to resort to disabling the rule and the monitor for almost all of these until it can be configured properly.If you have any tips regarding this please don\’t be afraid to post.Thanks again for a great article!/Sebastian

  4. Operations says:

    Hi Sebastian,
    The set-tuning threshold counters were a very creative idea by Microsoft, but as you (and others) have pointed out, there is much confusion regarding them.
    We actually are in the process of talking to Microsoft about these, and as we get guidance, we will be sure to post it to the blog.
    3.81 does appear to actually set the envelope so large that it will neve alert again! 
    The other point is that a set of values that works for one organization may not work for someone else, so there will be some trial and error associated for each of us as we try to tune the counters. It may be easiest to start with all them disabled and tune them one by one.

  5. thomas.chittenden says:

    First off, thanks for these articles, they are very clear and helpful in working with SCOM.My question is about how the latest Exchange 2003 for SCOM MP (6.0.6387.0) should either replace or work with these Baseline rules. After importing the new MP, I see both the baseline rules and the Performance Collection rules. If we\’ve lost the faith of the STT and just want to apply monitors to static thresholds for say SMTP queue monitoring, should we just disable all monitors on the Baseline rules and configure/enable/tune the monitors on the Performance collection rules? I guess I\’m just unclear as to if the performance collection rules need or depend on the baseline rules. Any instruction on best steps to tune the new MP would be awesome too. Thanks again for these posts!

  6. Operations says:

    Another great reference on STTs is Kevin Holman\’s blog posting at http://blogs.technet.com/kevinholman/archive/2008/03/19/self-tuning-thresholds-love-and-hate.aspx. Note as he says at the top of his article that the newer version of the Exchange Management Pack alleviates many of these issues.

  7. Kyle says:

    Hey guys,

    really good article. I would like to point out something though…

    You said:
    “we strongly recommend you save your changes to an unsealed MP other than the Default management pack.”

    but then later you go through the steps and say:
    “disable the rule for all objects of type: Exchange Queue, click yes to accept”

    I’ve been told that disabling a rule saves the changes to the default management pack, but overriding a rule, or a monitor to be disabled will save the changes to the specific management pack that was created for that monitor or rule.

    Please correct me if im wrong.

    thank you,

Leave a comment