Operations Manager: Tuning and Configuration

OpsMgr 2007 Unleashed Errata - the EnableAdIntegration Registry Key

Fri, 15 Aug 2008 17:35:00 GMT

Page 390 of System Center Operations Manager 2007 Unleashed discusses modifying the Registry settings for the EnableADIntegration key on the RMS and management servers. This information was based on our own experiences and testing during earlier versions of OpsMgr 2007, as well as recommendations from Microsoft. Microsoft has since changed the behavior such that this Registry key hack is no longer recommended, and can actually cause problems.

We will be changing the information in the next printing of the book and the errata to say:

To complete the process of activating OpsMgr integration with AD, validate that the registry key HKLM\SYSTEM\CurrentControlSet\Services\HealthService\Parameters\ConnectionManager\EnableADIntegration is set to 0.This is the default setting.

There has been some confusion regarding how to configure this setting on management servers. However, do not change it to 1; the value should actually be the default configuration of 0. Changing the value to 1 is not required and actually may cause issues on the management servers.

More on OpsMgr and I/O

Fri, 08 Feb 2008 18:46:56 GMT

We received the following email from "Jay", but his communication preference settings do not allow a reply to his email. Jay said:

I'm attempting to build a system that can host 10-20 concurrent Operation Consoles. I already plan to get dedicated fast drives (10K SAS), and have separate drives (or "spindles") for each of: OS/apps/swap; DB data; DB logs; DB backup.

My deliberation is whether to have the RMS, Reporting/SRS (not the data warehouse DB), and operational DB/DBS all on one server with 4 dual-core Opterons and 64 GB RAM (OS, DBS, and SCOM, all 64-bit) OR

Put the operational DB/DBS on it's own server -- still with 4 dual-core Opterons and 64 GB RAM, and have the RMS and reporting on it own server with 4 dual-core Opterons and 16 GB RAM OR

go with 3 servers and put the reporting/SRS on it's own server.

Microsoft says to "scale up" or "scale out", but my instinct says "scale up" may be better for reducing latency for an interactive app like the console, as long as you have "enough" CPUs and RAM.

With around 1000 managed servers, or operational DB is about 50 GB, so the large amount of memory is intended for caching the DB. With the DB "heavily" cached, and the DB/DBS on its own server, even with GB network, I would think the network latency, including the OS network stack latency would be significant.

With the one server solution, the "thread execution delay" may be significant, as even 8 processors may not be enough to keep all of the functions: RMS, reporting, SRS, DBS, running without "latency".

What I don't know, is what is more significant -- the network latency or the "thread execution delay" latency.

I could also build the single server with 4 quad-cores to cut down the "thread execution delay" latency, but the quad-cores are still a bit pricey.

Salient points:

The question is where to position the database servers. There are three scenarios (if we read this correctly):
- Have the RMS and SRS on one server with the operational database and data warehouse on a second server
- Use three servers - RMS and SRS on one server, Operations database on a second server, data warehouse on the third server, Operations database on the third server
- Use three servers - RMS on on server, Operations database on second server, data warehouse and SRS on on a third server
He has 1000 managed servers, and a 50GB Operational database.
The plan is to have 10-20 concurrent Operations Consoles open.

Thoughts:

We vote for option number 3. With 1000 managed servers, it is definitely best to have the RMS all by itself. This also separates the Operational database and the data warehouse, which are updated simultaneously by the management server(s).

Jay did not mention the number of planned management servers. With 1000 agents, you would want to install multiple management servers. A rule of thumb is once you have at least 100 managed nodes, you will want to install a second management server. (A second management server is always a good idea in case the RMS goes down and you need to promote another server to that role.) While Microsoft supports up to 2000 agents per management server, you may want to add additional management servers, depending on the type of data being collected. For example, a single management server is unlikely to be capable of supporting 2000 Exchange servers due to the particularly heavy load these agents place on it. After the first two management servers, you may want to add an additional management server for every increase of 250-500 nodes.

As the number of active consoles grows, the database load also grows. This is because consoles, either operator or web-based, increase the number of database queries on both the operations and data warehouse databases. Having that many consoles is another reason to separate the two databases to different servers. Console performance improves quite a bit with Service Pack 1, which will be released shortly.

In addition, 50GB is fairly big for the Operations database. You may want to tune the grooming settings to get the database down to 40GB or even perhaps 30GB. While the database size officially has no limits, a number of OpsMgr sites (including Microsoft), have suggested a database of 40GB or below.

A good article on network bandwidth is one by Satya Vel at http://blogs.technet.com/momteam/archive/2007/10/22/network-bandwidth-utilization-for-the-various-opsmgr-2007-roles.aspx

In addition, you may want to look at Chapter 4 of System Center Operations Manager 2007 Unleashed, which discusses planning your OpsMgr deployment.

Anyone else have comments or suggestions for Jay?

OpsMgr by Example: CPU Percentage Utilization Monitors in OpsMgr 2007

Tue, 05 Feb 2008 01:58:27 GMT

In many of our OpsMgr by Example articles, we recommend you install the Windows Server management pack to be able to get access to underlying operating system performance data for applications including SQL Server, Exchange, and IIS. However, per KB article 948097 (http://support.microsoft.com/kb/948097), the CPU percentage utilization monitors do not work on Windows 2003 Server!

These monitors are targeted to the Windows Server 2003 processor class - but by default, there is no available Windows Server 2003 processor class instance, since the corresponding discovery rule (Discover Windows CPUs) is disabled by default. To enable the rule, perform the following steps:

In the Operations console, navigate to the Authoring space. Open Authoring -> Management Pack Objects -> Objects Discoveries.
Narrow your scope, then search for the "Discover Windows CPUs" discovery rule.
In the Properties window of the rule, go to Overrides. Set overrides on all Windows Server 2003 Operations System role type objects. Check "override" on the "Enable" parameter.
Save the override setting to an unsealed management pack, preferably not the Default management pack.

Once the Processor class is functional, the CPU percentage utilization monitors will also start working.

OpsMgr and I/O

Wed, 09 Jan 2008 17:07:25 GMT

A friend of ours, Walter Chomak, was chatting the other day about OpsMgr I/O and that he had just finished working on his performance model (not sure you ever really "finish" these things, though). The cornerstone of the model is that regardless of the number of agents, the I/O is a 85/15 split. With 85% reads, if your average disk sec/read is greater than 50-75ms, you will observe latency in the console. This will hold true for 100 agents or 6000 agents (of course, as agents increase [load], keeping those times down requires better disk subsystems.)

Walter modeled this in many sizes of environments, ranging from 500 to over 7000 agents. He has published an article on his blog, OpsMgr 2007 I/O Considerations, that you will definitely want to read. Check out http://wchomak.spaces.live.com/blog/cns!F56EFE25599555EC!610.entry.

OpsMgr by Example: Synthetic OWA Testing

Sat, 13 Oct 2007 13:29:41 GMT

If you have already tried everything that you can do to get rid of the OWA logon failure (other than disabling it) in Exchange 2003, this may be of assistance.

Alert: OWA: Outlook Web Access logon failure: Authentication error

Issue: The OpsMgr script in the Exchange 2003 management pack will not work if you are using a custom URL and HTTPS with certificates. If you're not already familiar with this issue, Andy Dominey previously blogged this at http://myitforum.com/cs2/blogs/adominey/archive/2007/04/10/mom-2005-and-om-2007-exchange-2003-management-pack-issue.aspx). For example, if you have a server named SERVER1 at ABCCO and your webmail address is https://webmail.abcco.com on SERVER1 the current MP cannot perform the check on this web location correctly, as the server name (server1.abcco.com) does not match the certificate name of webmail.abcco.com.

Resolution: Create a custom simple monitor with two views to monitor the OWA front-end functionality.

Perform the following steps:

1. Open the Operations Console -> Authoring -> Web Applications. Right-click and choose Add monitoring wizard.

2. Choose the Web Application monitoring type.

3. Enter a name (OWA Web Test) and description and choose the management pack (preferably not the default management pack, we created our own called OWA Web Test).

4. Enter the URL to test. For our example company of ABCCO.com we will be using https://webmail.abcco.com to match the organization’s existing external name that is assigned to the SSL certificate.

5. Choose a watcher node (the MS or RMS does well on this if it isn’t too busy) and the frequency (defaults to 2 minutes).

6. Create the web application.

7. Highlight the new Web Application and choose Edit web application settings under actions.

8. Start a capture and go through the following process: (As an FYI we were not able to select and preview messages/it caused too many issues with the monitor)

· Log in to the OWA server using specified credentials

· Create and sends a new message to the email of the specified credentials

· Delete the message which was sent

· Logs out of the OWA browse session.

9. Remove any failed responses that are not required. As an example, we removed the the links section on ours. We removed this by going to the Properties -> Monitoring tab, and unchecking “Enable health evaluation and performance collection for Internal links”. We also needed to remove other conditions which failed regularly, do this by highlighting the URL that failed and deleting it under the Actions selection.

10. Now that the Web application is monitoring the OWA site, we can see the state of the monitor either under the management pack we specified (Administration -> OWA Web Test in our case) or under the Administration -> Web Application -> Web Applications State.

11. We can also right-click on the particular state view and choose to open the alert view or the performance view. The performance view is especially useful, and it is a good idea to go ahead and create a performance view so that you can easily access these counters (be sure to limit the performance counters shown to the name that you created such as our OWA Web Test example otherwise there are a lot of counters). An example of the performance view is shown here.

12. Now that we are effectively monitoring OWA functionality we can now disable the original “OWA: Outlook Web Access logon failure: Authentication error” alert.

We would like to give a huge thanks to Tony Greco who pointed out this issue, and found this creative approach to resolve it!

OpsMgr by Example: The Dell Management Pack

Wed, 26 Sep 2007 07:58:23 GMT

This blog entry is the next in a series of Operations Manager-related items which review the steps performed to install, configure and tune management packs in real-world environments.

The Dell management pack guide is part of the Dell Management Pack download available at http://support.dell.com/FileLib/Format.aspx?c=us&l=en&ReleaseID=R158716. The OpsMgr Dell management pack actually only takes what information is provided by the Dell OpenManage agent and integrates it with OpsMgr. As a result, the alerts raised are directly related to hardware issues that are shown in the logs available through the OpenManage interface.

Prerequisites:
Before you install the Dell MP, you must install the updated Microsoft Operations Manager 2005 Backward Compatibility Management Pack. You can download the Backward Compatibility MP at http://go.microsoft.com/fwlink/?LinkId=98874. Without the updated Backward Compatibility MP, you may experience CPU spikes!

Lessons Learned:

Make sure that all of the systems in your environment that have the Dell OpenManage software installed are at least version 5.2. A variety of errors occur if you try to monitor using the OpsMgr management pack and an older version of Dell OpenManage (including a lot of Script or Executable Failed to run alerts). You can check the version either by running the Dell Server Administrator and checking the version it lists or through checking the registry key available under HKLM\SOFTWARE\Dell Computer Corporation\OpenManage\Applications\SystemsManagement, in the Version field.

Tuning/Alerts to Look for: The following are alerts that were found and resolved during the tuning of Dell Management pack.

Alert: Dell.Connections.ServerAdministrator.Alert.1306.Critical

Issue: Redundancy lost Redundancy unit: System Power Unit Chassis location: Main System Chassis Previous redundancy state was Normal Number of devices required for full redundancy: 2. Checked with the Launch Server Administrator task, and did not find any current issues on the server. The actual issue was in the alert log not in the hardware log.

Resolution: Found the issue through the alert log on the Dell OpenManage. This appears to be an issue with a sensor or a power supply on the system. Entered company knowledge on the time and server that the alert occurred on to determine if this is a component that may be failing.

Alert: Dell.Connections.ServerAdministrator.Alert.1104

Issue: Fan sensor detected a failure Sensor location: ESM MB Fan7 RPM Chassis location: Main System Chassis Previous state was: OK (Normal) Fan sensor value (in RPM): 0. Checked with the Launch Server Administrator task, and did not find any current issues on the server. The actual issue was in the alert log not in the hardware log.

Resolution: Found the issue through the alert log on the Dell OpenManage. This appears to be an issue with a sensor or a power supply on the system.

Issue: DellStorageDiscovery.vbs failing on an Exchange 2003 server, process exited with 0. Searched the XML files to validate that this is part of the Dell management pack. Checked the Dell Server Administrator on the system and it was running version 1.9 (5.2 is required).

Resolution: Upgrade the version of the Dell Server Administrator software or disable the alert on this system.

Alert: Script or Executable Failed to run

Issue: DellServerFansUnitUnitMonitor.vbs failing on a Windows 2003 server. Checked the Dell Server Administrator on the system and it was running version 1.8 (5.2 is required).

Resolution: Upgrade the version of the Dell Server Administrator software or disable the alert on this system.

Alert: Dell.Connections.ServerAdministrator.Alert.1554

Issue: Log size is full Log type: ESM

Resolution: Validated that the log was full (used the Launch Server Administrator task) and then use the Clear ESM Logs task to clear out the logs as the items were not current but were historical. Closed the alert.

Alert: Dell.Connections.ServerAdministrator.Alert.1553

Issue: Log size is near or at capacity Log type: ESM

Resolution: Used the Clear ESM Logs task after reviewing them with the Launch Server Administrator task. Closed the alert.

OpsMgr by Example: Tuning Management Packs

Sun, 23 Sep 2007 01:42:33 GMT

Philosophy

Deploying management packs involves planning, evaluating in a test environment, and finally importing the management pack along with any changes from test to your production environment. You also will want to decide which management packs you want to deploy, and in what order. Some applications have dependencies on others, so you may want to implement the related management packs as you deploy those products.

For example, say you use Exchange Server. Exchange requires Active Directory, which in turn utilizes DNS. You would want to first test and implement in order the dependencies, e.g. the DNS MP, then AD, and finally Exchange. Another consideration is where you might get the most "bang for the buck" - which management packs give you the most benefit for the least amount of cost (tuning, resources, or effort). The order in which you implement management packs will depend on the priorities of your organization and your goals for monitoring.

TIPS

It is best to introduce a single management pack at a time, as this practice makes it easier to deal with management pack issues as they occur. When there is more than one management pack involved it may be difficult to determine what initially caused a problem.
We also suggest you only install those management packs you need, as extra rules and monitors impose a cost on your system resources; increased memory utilizations of the agents targeted by the management pack, and increased traffic between the managed computers and the management server.

Initial Tuning - By Function

Once you have determined your strategy and order for deploying management packs, it is time to import selected management packs one at a time into your test environment. As part of your approach, be sure to refer to that MP's management pack guide. The MP guides discuss particulars for installing, configuring, and tuning that particular management pack. The management pack guides are typically included in the download package with the management pack.

It is of course, best to tune in a test environment as the impact of a badly-performing rule or monitor is less than in production. Testing pre-production also helps minimize the information load and unnecessary work for your production computer operators.

As you evaluate a management pack's behavior you may decide to tune one or more parameters to meet your organization's needs. For example, you may have a performance monitor generating an alert or threshold value that is inappropriate for your particular environment. You can tune that setting by overriding the default settings for that monitor.

You will either work on a server-by-server or application-by-application basis, tuning from the highest severity alerts and dependencies to the lowest.

A server-by-server approach addresses issues identified while deploying servers into OpsMgr. Once that is complete, your process should be an application-by-application / service-by-service basis, focusing on the overall health of the application or service. Look at alerts first, then open he Health Explorer to drill down specifically into the problem.

Managing Alerts

When you implement a management pack, it will generate alerts that you will want to review and evaluate for tuning. Some rules and monitors may generate low severity alerts, depending on your specific environment these may not be worth investigating or resolving, and you may consider disabling that rule or monitor. Any changes made are saved to an unsealed management pack. You can document your actions using the Company Knowledge section of the object.

Review the Operations console regularly to see whether information is captured that is unnecessary for your environment, as this could take up an excessive amount of storage within Operations Manager.

Tuning Tips

Review any new alerts reported for servers monitored with the new management pack. You can use the Alerts and Most Common Alerts reports to help you discover your most common alerts.
Resolve the issue generating the alert. Use the product knowledge base information regarding the specific error. When you first install a management pack, it tends to discover a multitude of previously unknown issues. Monitor the alerts to determine potential areas of concern
Override the monitor or rule as applicable for a particular object type, a group, or a specific object.
Disable the monitor or rule if the issue is not severe enough to warrant an alert and you do not need to be made aware of the specific situation being monitored.
Change the threshold of the monitor that is generating the alert if you want the underlying condition to be monitored, but the alert is being generated before the condition is actually a problem for your particular environment. Remember OpsMgr incorporates self-tuning thresholds, so in many instances the system will do this for you.
If a new management pack generates a ton of alerts, you may want to start by disabling monitors or rules within the that management pack. You can turn them on gradually, making the new management pack easier to tune and troubleshoot.

Finished?

After you reach a comfort level with each management pack, it is time to put it into production. Because you've already "tuned" the MP and want to keep your changes, export your customizations from the test environment and then import that into production along with the vendor-provided management pack.

This is an abbreviated excerpt of tuning information from our forthcoming book, System Center Operations Manager 2007 Unleashed.

OpsMgr by Example: The Exchange 2007 Management Pack

Mon, 17 Sep 2007 22:24:44 GMT

This blog entry continues our series of Operations Manager-related items that review the steps performed to install, configure and tune management packs in real-world environments.

Since there is not currently an Exchange 2007 management pack for Operations Manager 2007 (but there is one for MOM 2005), the following results are from a tuning exercise on Exchange 2007 using a version of the management pack converted from MOM 2005 to OpsMgr 2007. The core functionality appears to have converted well (excellent call on this one/thanks to our co-author John Joyner for the idea and his help getting this put together!). Tuning on this version of the management pack should provide a jump-start for tuning on the Exchange 2007 management pack designed specifically for OpsMgr 2007. The management pack guide is available at: http://technet.microsoft.com/en-us/library/bb217782.aspx.

Issues:

· No reports (it is a converted management pack so that is why - there should be reports in the Exchange 2007 MP when it is released by Microsoft).

· Cannot edit company knowledge on the monitors within the Exchange 2007 management pack (there is no Company Knowledge tab within the Alert properties).

· No Exchange wizard for configuration: Whoho!

Tuning/Alerts to Look for: The following are alerts that we found and resolved while tuning the converted Exchange 2007 Management pack.

Alert: LDAP Search Time - sustained for 5 minutes - Red(>100msec).

Issue: This condition occurs sporadically on the servers in the environment.

Resolution: First tried to create an override for the rule [LDAP Search Time – sustained for 5 minutes – Red(>100msec)] to move this from sustained for 5 minutes to sustained for 10 minutes. (Also in another environment we tried setting the override to 15 minutes). Re-configured this threshold to a higher value (200msec) as no network issues were found on the systems (gigabit linked on each side, direct to the switch), no bottlenecks were found on the systems, and this is occurring in multiple environments. Changed the threshold [right-clicked on the LDAP Search Time – sustained for 5 minutes – Red(>100msec) and choose View or edit the settings of this Monitor, on the configuration tab, within the XML changed 100 to 200]. Also renamed to now say >200msec and renamed the alert as well.

Alert: LDAP Search Time - sustained for 5 minutes - Yellow(>50msec).

Issue: This condition occurs sporadically on the servers in the environment.

Resolution: First tried to create an override for the rule [LDAP Search Time – sustained for 5 minutes – Yellow(>50msec)] to move this from sustained for 5 minutes to sustained for 10 minutes. (Also in another environment we tried setting the override to 15 minutes). Re-configured this threshold to a higher value (150msec) as no network issues were found on the systems (gigabit linked on each side, direct to the switch), no bottlenecks were found on the systems, and this is occurring in multiple environments. Changed the threshold [right-clicked on the LDAP Search Time – sustained for 5 minutes – Yellow(>50msec) and choose View or edit the settings of this Monitor, on the configuration tab, within the XML changed 100 to 200]. Also renamed to now say >150msec and renamed the alert as well.

Alert: Application log size.

Issue: Exchange 2007 application log size was 16 MB, per the Exchange 2007 MP this should be at least 40 MB for Exchange servers.

Resolution: Increased the application log size on the servers indicated.

Alert: Crash upload logging disabled.

Issue: Exchange fatal information is not being sent to Microsoft.

Resolution: Per the knowledge link, this can be changed with the Exchange UI (http://technet.microsoft.com/en-us/library/2582b127-b826-4eac-88b6-47a79ed49c6d.aspx) to resolve the issue. For those environments where there is a requirement to not send this information, the rule can be disabled.

Alert: WebServices connectivity (Internal) transaction failure - The credentials cannot be used to test Web Services.

Additional Alerts:

Error occurred while executing the Test-ExchangeSearch diagnostic cmdlet.

Error occurred while executing the Test-Mailflow (Remote) diagnostic cmdlet.

Error occurred while executing the Test-Mailflow (Local) diagnostic cmdlet.

Error occurred while executing the Test-MAPIConnectivity diagnostic cmdlet.

Exchange ActiveSync connectivity (Internal) transaction failure - The test credentials cannot be used to test Exchange ActiveSync.

Outlook Web Access connectivity (External) transaction failure - The test credentials cannot be used to test Outlook Web Access.

Outlook Web Access connectivity (Internal) transaction failure - The test credentials cannot be used to test Outlook Web Access.

Issue: Exchange 2007 management pack configuration required.

Resolution: Ran the new-TestCasConnectivityUser.ps1 on the Exchange server from the Exchange Management Console within the Exchange shell on the Mailbox server. To run the utility, enter a temporary password for the system, press enter to continue, and specify an OrganizationUnit to put this in (the OU name needs to be unique or you need to point it to the full name of the OU). This creates the account in the OU that you specify. CAS_{sid}

Alert: The Microsoft Exchange Replication Service requires re-seeding a storage group on the passive node.

Issue: Passive node

Resolution: Microsoft provides product knowledge on how to fix this, available at: http://technet.microsoft.com/en-us/library/63367703-1226-44b2-a4b8-205ed7222da0.aspx

Alert: MSExchange Replication: ReplayQueueLength - sustained for 5 minutes - Red(>15).

Issue: Problems were occurring with Cluster Continuous Replication where the passive node required a re-seeding

Resolution: See the “The Microsoft Exchange Replication Service requires re-seeding a storage group on the passive node.” alert.

Alert: Edge Synchronization transaction failure - Recipients are out of sync.

Issue: Edge Synchronization issue

Resolution: http://technet.microsoft.com/en-us/library/7b5897c5-9c72-40ee-b977-4f4f6821d1ed.aspx

Alert: Percentage of Committed Memory in Use is too high

Issue: Several Microsoft products including Exchange, SQL Server, and the Operations Manager RMS will use all available memory. This is especially noticeable on 64-bit platforms where memory can scale-out more effectively for the applications.

Resolution: Configured servers with Exchange, SQL or Operations Manager RMS to have a 95% threshold instead of 80%.

Additional Notes:

It may be required to configure URL monitoring to work correctly on the managed Exchange 2007 box: (the following command sets the configuration is documented at http://technet.microsoft.com/en-us/library/bb691294.aspx)

set-owavirtualdirectory "Server01\owa (Default Web Site)" -externalurl:"https://Server01.Domain.contoso.com/owa"

Alerts Not Resolved:

Alert: Delay DSNs - increase over 60 minutes - Red(>20) - Hub Transport.

Issue: Issue did not recur.

Resolution: No resolution found.

Alert: Delay DSNs - increase over 60 minutes - Yellow(>10) - Hub Transport.

Issue: Issue did not recur.

Resolution: No resolution found.

Alert: Failure DSNs Total - increase over 60 minutes - Red(>40) - Hub Transport.

Issue: Issue did not recur.

Resolution: No resolution found.

Alert: Failure DSNs Total - increase over 60 minutes - Yellow(>30) - Hub Transport.

Issue: Issue did not recur.

Resolution: No resolution found.

Alert: Inbound direct trust certificate has expired. Run New-ExchangeCertificate to generate a new direct trust certificate.

Issue: Unknown.

Resolution: Unknown.

OpsMgr by Example: The Jalasoft Management Pack

Tue, 11 Sep 2007 03:03:19 GMT

This blog entry is the next in a series of Operations Manager-related items which review the steps performed to install, configure and tune management packs in real-world environments.

What is Jalasoft and why would people want to deploy it with Operations Manager 2007? Jalasoft provides extensions which allow OpsMgr to monitor routers & switches as well as Unix based operating systems. The full list includes:

APC UPS
Availability (ICMP only)
Cisco PIX/ASA
Cisco Routers
Cisco Switches
Cisco VPN Concentrators
Cisco Wireless
F5 Big Ip
Generic Network Device
HP Procurve Switches
Linux MySQL
Linux Servers
Solaris Servers
VMware ESX
VMware VirtualCenter

There are a variety of vendors which provide competitive products to the Jalasoft offerings. These vendors include:

eXc Software’s management packs and virtual agents
Quest’s Management Xtensions
nworks VMware Management

For the purposes of this example, we used the standard Jalasoft management packs and the following additional management packs:

Cisco Switches & Routers
Linux Servers
General Availability

Jalasoft Installation:

Install and configure Operations Manager 2007, including the reporting components.
Read the guides on the Jalasoft products, which are available at: http://www.jalasoft.com/jalasoftweb/jsp/products/xianio/ under the More Information section.
Downloaded the Jalasoft evaluation from the same site (http://www.jalasoft.com/jalasoftweb/jsp/products/xianio/).
Extracted the files from the XianIo.zip file.
Ran the Index.hta and then chose Install Xian Network Manager Io. Installed with a custom installation including all of the programs available.
Prerequisite included MSMQ as it is used for communication between the Xian services. This was added through Control Panel Add/Remove programs -> Add/Remove Windows Components -> Application Server -> Details -> Message Queueing.
During the installation chose a single server installation, connecting to the existing SQL server in the OpsMgr environment, default database name of XIAN on the default port of 8586, with defaults on the remainder of the configurations.
This installs the Xian console which is available on via Start -> Programs -> Xian Network Manager Io -> Xian Network Manager Io. After launching the console, we added the evaluation license provided as part of the download.
Installed the management packs for evaluation from the directory where Jalasoft was extracted to within the Management Packs folder. In our case, we added each of the MPs on the top level of the folder but none of the subfolders (APCUPS, Availability, Cisco, F5BigIP, HPProCurveSwitch, NetworkDevice, Solaris, Linux, MySQLServer, and VMWare).
Added the management packs (and reporting management packs) for the folders we were implementing: Cisco Routers, Cisco Switches, Linux.
Ended up also adding the availability MPs to test with the minimal configuration and impacts to the first sets of clients.

Jalasoft usage:

Availability Monitoring: Adding the availability monitoring (IP Test) worked great. We just added the IP address through the Xian console as part of the availability view. The device appeared in the Operations console under Monitoring -> Xian Network Manager -> All Xian Monitored Network Devices. We removed it from the Xian console and it disappeared out of OpsMgr as well. Testing the availability pieces of this included addition of systems which either could not have the agent deployed or were currently not deployed due to network restrictions on the devices. We added several devices, and then configured an ICMP availability rule to create a critical alert if the device was offline (and to create a warning if response time was too slow).
Network Monitoring: Deploying network monitoring for Cisco switches and Cisco routers was very straightforward. Adding the IP address and the read-only community string integrated our switches with the Cisco Switches section of the Jalasoft console, which in turn appear within the Operations console -> Monitoring -> Xian Network Manager -> All Xian Monitored Network Devices State. The default configuration provides up/down information only. Addition of rules to the device in the Jalasoft console provided testing for temperature status and device availability.
Unix Monitoring: The Unix monitoring component requires installing software on the Unix system (a daemon) (XianServer-3.1.727.5-727.i386.rpm). Once the daemon is deployed, Unix systems are easily added into Jalasoft from a process very similar to the one used for networking devices.

General thoughts on Jalasoft’s solution:

The Xian console provides an easy method to integrate with OpsMgr 2007. It is unfortunate however that a separate product needs to be installed on the environment rather than integrating with existing OpsMgr network monitoring functionality.
The evaluation version appears to be fully functional and supports up to 10 devices for up to 60 days.
Once devices are configured to be monitored by Jalasoft, the actual monitoring does not take place until you configure the specific rules that are active for the device (as an example, adding the ICMP availability active rule to the system being monitored within availability).
The integration with OpsMgr was easy to use and was able to provide not only up/down information, but also gather and track performance metrics for the various devices.

OpsMgr by Example: The Secure Vantage Management Pack

Fri, 31 Aug 2007 19:10:13 GMT

This blog entry is the next in a series of Operations Manager-related items that review the steps performed to install, configure and tune management packs in real-world environments. Historically we have only discussed management packs from Microsoft, but beginning with this blog posting we digress a bit and look at the Secure Vantage Management Pack.

What is SecureVantage and why would people want to deploy it with Operations Manager 2007? SecureVantage has a variety of products that enhance the capabilities of Operations Manager focused around the areas of Security and Audit Collection Services (ACS). These products include solutions for archiving information from the ACS database, and management packs focusing on security information that can provide reports for regulations such as HIPAA and SOX (among others).

General information about SecureVantage and its product line is available at http://www.securevantage.com/. SecureVantage also provides a free management pack for download that provides alerting for the top Windows security audit scenarios. You can download this management pack at http://www.securevantage.com/ProductsSTAMP.html.

For the purposes of this article, we are using the IT Auditors Express for reports and the following SecureVantage management packs:

Security Base Library
Security Top Alerts
Group Policy Auditor
Windows Security Auditor

Installation:

Identify the Auditing requirements for your organization (understood, this is a really big high-level bullet, but you want to have a good idea of the particular items you want to audit in your environment).
Install and configure Operations Manager 2007, including the reporting components.
Deploy the OpsMgr agent to the systems that plan to you monitor with ACS and SecureVantage.
Install and configure Audit Collection Services for Operations Manager 2007.
Enable Auditing on the servers that you will be auditing.
Validate the functionality of ACS by opening the Performance Monitor (perfmon) and monitoring the ACS Collector object/Connected Clients Counter. If ACS is installed correctly and clients are reporting in to the server, this counter should be greater than 0.
Install the SecureVantage management packs on the Root Management Server (RMS).
Install IT Auditors Express on the Operations Database Server (not on the RMS).
The SecureVantage Management pack information is available at http://www.securevantage.com/ComplianceSecuritySuite.html. High-level information on the management packs and the download links are available at http://www.microsoft.com/technet/prodtechnol/mom/catalog/catalog.aspx?kw=&vs=2007&ca=&co=Secure%20Vantage%20Technologies.
Read the guides on the SecureVantage products, available at http://www.securevantage.com/ProductsDocuments2007.html

We ran into a few interesting tidbits and caveats to be aware of with the SecureVantage functionality:

The SecureVantage product uses both the Operations Manager functionality (event log gathering, etc) and the ACS functionality to provide the alerts and reports that are provided with the product.
Currently the group membership rule actually provides all changes made to group memberships, not just the changed in high security groups (such as domain administrators). The Admin Group Membership view (Operations Console -> Monitoring -> Security Operations -> Windows Security Operations -> Server Security -> Account Management -> Admin Group Membership) also currently displays all group changes. This is scheduled to be resolved shortly.
The SecureVantage management pack creates its alerts in an informational state. The number of alerts will vary depending upon a variety of factors that include the number of servers you are auditing, what is being audited, and how active the servers are which you are monitoring. For our particular environment with approximately 30 domain controllers, approximately 1000 informational alerts were listed.
When using the SecureVantage reports, if you choose the Expose Details option prior to running the report the system will pause for a several seconds before you can run the report.

OpsMgr by Example: The IIS Management Pack

Wed, 15 Aug 2007 14:51:14 GMT

This blog entry is another in a series of Operations Manager related items that review the steps that we performed to install, configure and tune management packs in real-world environments. With this entry we focus on the IIS MP.

Installation:

Download the IIS management pack (http://www.microsoft.com/downloads/details.aspx?FamilyId=D351BCA8-182B-4223-8C9E-627E184BA02B&displaylang=en), and the IIS Management Pack Guide (http://download.microsoft.com/download/7/4/d/74deff5e-449f-4a6b-91dd-ffbc117869a2/OM2007_MP_IIS.doc).
Read the Management Pack guide from cover to cover. There are important pieces to know that the document spells out in detail.
Import the IIS management pack. This consists of the Windows Server Internet Information Services Library, and individual management packs for IIS 5 (Internet Information Services 2000 with Windows 2000) and IIS 6 (Internet Information Services 2003 with Windows Server 2003). Import the Library (which is a prerequisite), plus the appropriate management pack for the version of IIS that you will be monitoring.
We recommend you also import the appropriate version of the Windows Server management pack (Windows 2000 or 2003). Some of the views provided with the IIS MP require the MPs for the appropriate level of operating systems to have data to display.
Even if you do not have any custom web applications using IIS, remember that Exchange, SQL Server Reporting Services, and Operations Manager itself have components that use IIS; you will want to implement the IIS MP as part of rolling out and monitoring those applications.
The IIS management pack does not support agentless monitoring. Verify that the OpsMgr agent is installed on your IIS servers.
The IIS MP collects data from the IIS logs. If logging is not enabled, the MP will only collect and analyze service data. The IIS logs must be set to the W3C Extended Log File format. Enable logging for each type of site and virtual server that you want to collect monitoring data for. This can include FTP sites, Web sites, SMTP virtual servers, and NNTP virtual servers. (Enable logging for a virtual server in the IIS Services Manager by double-clicking the local computer, right-click the SMTP or NNTP Virtual Server folder you want to enable logging for, select Properties, then on the General tab, select Enable logging. Be sure to select W3C Extended Log File format on the Active log format drop-down list.)

Rolling up Health

If you have used the IIS MP in MOM 2005, the Health rollup is a new feature. Use the Health Explorer to examine Health:

The IIS MP for OpsMgr 2007 has the abiilty to tell you whether a specific Web site is health, in addition to being able to know if the Web server is healthy.
The health of the IIS Sever is dependent on the health of the objects at the next lower level - the IIS Web, FTP, NNTP, and SMTP Servers. If any of these servers is in a critical health state, the IIS Server will display in a critical health state.
By default, the health of the IIS FTP, NNTP, SMTP, and Web Servers are not dependent on the health of the objects at the next lower level. If one or more Web Sites is in a critical state, the Web Server object will not change state.

Tuning/Alerts to Look for: The following are alerts found and resolved while tuning the IIS management pack.

Alert: An unknown token name (s-event) was encountered.

Issue: IIS logging is configured by default on Windows Server 2000 to include Process Accounting extensions for Web sites.

Resolution: Disable logging Process Accounting Extensions. (In IIS Service Manager, select the Default Web Site, Properties, select Enable logging on the Web Site tab, from the Active log format drop-down list, select W3C Extended Log File Format, select properties, choose Extended Properties, and clear Process Accounting from the Extended Logging Options list box.)

Issue: IISReset causes a ton of alerts.

Resolution: Put the IIS object in maintenance mode before doing the IISReset. You could use PowerShell (use the command New-MaintenanceWindow to put the server into maintenance mode and then start the IISReset cmd operation) to automate this.

Issue: IIS MP does not work well with clusters. This can be a real issue when monitoring Exchange.

Resolution: None currently available. This may be addressed in Service Pack 1 / the next release of the IIS MP.

Issue: If you are using the Exchange management pack, you will encounter a number of IIS-related issues with Exchange.

Resolution: Check our "OpsMgr by Example: the Exchange Management Pack" entry (http://ops-mgr.spaces.live.com/blog/cns!3D3B8489FCAA9B51!220.entry) for a detailed list.

OpsMgr by Example: The SQL Management Pack

Wed, 08 Aug 2007 14:57:29 GMT

Installation:

Download the SQL Management Pack (http://www.microsoft.com/downloads/details.aspx?FamilyID=8c0f970e-c653-4c15-9e51-6a6cadfca363&DisplayLang=en), and the SQL Server Management Pack Guide (http://download.microsoft.com/download/7/4/d/74deff5e-449f-4a6b-91dd-ffbc117869a2/OM2007_MP_SQLSrvr.doc).
Read the Management Pack guide – cover to cover. There are important pieces to know that the document spells out in detail.
Import the SQL Server Management Pack. The management pack for each monitored version of SQL Server (2000 and 2005) consists of two .mp files. These files provide logic for discovery and monitoring, meaning you can use a smaller management pack to discover the existence of SQL Server; deploying the monitoring MP to the agent after OpsMgr has discovered SQL Server there. There is also a SQL Server Library MP, which is a prerequisite for the other management packs.
We recommend you also import the appropriate version of the Windows Server management pack (Windows 2000 or 2003). The Windows Server management packs monitor various aspects of the OS that can influence the performance of those computers running SQL Server! This includes disk capacity, disk performance, memory utilization, network adapter utilization, and processor performance.
Running the SQL Server Studio and SQL Profiler tasks from the OpsMgr console requires that you have installed that software on all OpsMgr computers where these tasks will execute, or you will receive an error message “the system cannot find the file specified.” Installing the Management Studio and Profiler are not required unless you want to run those tasks.
The SQL Server MP supports agentless monitoring with the exception of tasks that start and stop SQL Server services and SQL Server mail.
The management pack installs two Run As Profiles: the SQL Server Discovery account and the SQL Server Monitoring account. By default, the management pack uses the Default Action account.

Optional Configuration:

The SQL Server MP does not automatically discover all object types. Go to the Authoring Pane of the Operations console to enable discovering additional components. Components not discovered include:

SQL Server 2005 Publisher
SQL Server 2005 Subscriber
SQL Server 2005 Subscription
SQL Server 2005 Agent Job
SQL Server 2000 Agent Job
SQL Server 2005 DB File Group
SQL Server 2005 DB File

What this means - you will not receive alerts for these objects failing since they are not even discovered objects! For example, if you have scheduled SQL backups using the SQL Agent and the job fails, OpsMgr won't tell you about it. If an agent job failed in MOM 2005, the SQL MP generated an alert. So these behaviors are not necessarily the same between MOM 2005 and OpsMgr 2007.

You can use overrides to change the settings for automatic discovery to enable these object types. Be sure to change your settings in an unsealed MP other than the Default management pack.

Tuning/Alerts to Look for: The following are alerts found and resolved while tuning the SQL Server management pack.

Alert: The SQL Server Service Broker or Database Mirroring transport is disabled or not configured. (EventID 9666)

Issue: This alert may occur even if the broker IS enabled.

Resolution: Verify the broker is enabled by running the following query in Management Studio, connected to the Master database:

SELECT is_broker_enabled FROM sys.databases WHERE name = 'OperationsManager'

If the result=1, the broker is enabled. If result=0 enable the broker as follows:

Stop the SDK, Config, and Health Services on the RMS, and the Health Service on any secondary management servers
Execute the following statement from SQL Management Studio
ALTER DATABASE OperationsManager SET ENABLE_BROKER
Restart the services

If the alert continues to reoccur, disable the rule using an override.

Issue: Clustered virtual servers are discovered and display as agentless managed, but the SQL Server database engine on the cluster does not appear to be monitored.

Resolution: Only the virtual SQL Servers are discovered (the cluster and not the individual cluster nodes). In the Monitoring tab under Windows Server, check that each Virtual Server shows up as a Windows Server with the property "Is Virtual Server" set to True. Restart the Health Service on the RMS and any other management servers after adding the cluster. You may need to restart the Health Service on the cluster as well, which will rerun the discovery.

It is also possible that you are having RPC issues. See KB article 306985 (http://support.microsoft.com/kb/306985) for additional information.

Alert: 8957 Monitor Name: DBCC executed found and repaired errors – but found 0 errors and repaired 0.

Issue: When DBCC runs it generates this event log message with the same event ID if any problems were found or not.

Resolution: Disable the rule and create your own. For the new rule, copy all of the settings the same from the original but set the description to not contain "found 0 errors." For all other events with this ID, it will generate an alert to indicate a problem was found.

Alert: Health Monitor Description: Service Pack Compliance - MSSQLSERVER (SQL 2005 DB Engine) Warning (against ACS database)

Issue: SQL Server 2005 Service Pack 2 is installed, which is acceptable for the ACS database server. SP2 has been approved for all OpsMgr database components.

Resolution: Created an override (for specific object of type SQL Engine DB) to allow this configuration for this server/set the enabled parameter to False for this server. Reset the health for this health monitor on this server, and refreshed and the state updated to green from yellow.

Issue: The Management Server Action account is used as the Default Data Warehouse Action Account, rather than the DW Action account you specified during setup.

Resolution: This will be fixed in SP1. In the interim, create a RunAs account, type Simple, and set the username and password to a single space. In the same-name profile associate this account to all management servers, including the RMS. Also, be sure that the Data Warehouse Action account profile is correctly associated with an account for all management servers to be used as the Window authentication account. This information was obtained from the newsgroups (nntp://msnews.microsoft.com/microsoft.public.opsmgr.setup/7363632A-A650-4367-9DCE-27CC2887B786@microsoft.com).

Issue: SQL Server 2000 database engine health is not monitored. This is an aggregate monitor that includes the SQL Service State terminated unexpected monitor and the SQL Service terminated unexpectedly monitor.

Resolution: The SQL DB Engine Service Health Rollup monitor is not enabled by default. Use the Authoring pane of the OpsMgr console to enable the aggregate rollup monitor (Under Management Pack objects, select Monitors, change the scope to SQL 2000 DB Engine, search, then expand the SQL 2000 DB Engine, expand Entity Health, expand Availability, select SQL DB Engine Serve Health Rollup, and create an override to Override the monitor for all objects of type SQL 2000 DB Engine. See KB article 938991 (http://support.microsoft.com/kb/938991) for additional information.

The following issues are related to specific applications you may have installed:

Issue: Alert Rule or Alert Monitor: Auto Shrink Flag Alert Description: The auto shrink flag for database SUSDB in SQL instance MSSQL SERVER on computer 123.abc.com is not set according to best practice.

Resolution: This is a standard Microsoft application (WSUS) and a default configuration. Created an override to exclude this database.

Issue: Alert Rule or Alert Monitor: Auto Shrink Flag Alert Description: The auto shrink flag for database BEDB in SQL instance MSSQL SERVER on computer 123.abc.com is not set according to best practice.

Resolution: This is the standard configuration for Backup Exec's database.

Issue: Alert Rule or Alert Monitor: Auto Shrink Flag Alert Description: The auto shrink flag for database MSCUPTDB in SQL instance MSSQL SERVER on computer 123.abc.com is not set according to best practice.

Resolution: This is a standard Microsoft application (patch Management for SMS and Configuration Manager) and a default configuration. Created an override to exclude this database.

Issue: Alert Rule or Alert Monitor: Auto Close Flag Alert Description: The auto close flag for database MSCUPTDB in SQL instance MSSQL SERVER on computer 123.abc.com is not set according to best practice.

Resolution: This is a standard Microsoft application (patch Management for SMS and Configuration Manager) and a default configuration. Created an override to exclude this database.

OpsMgr by Example: The Exchange Management Pack

Tue, 07 Aug 2007 15:52:08 GMT

This blog entry is the next in a series of Operations Manager-related items that review the steps performed to install, configure and tune management packs in real-world environments.

Installation:

Download the Exchange 2003 Management Pack (http://www.microsoft.com/downloads/details.aspx?FamilyId=9FF454F4-6D34-4FB9-9E0B-F5B68C6EDC4F&displaylang=en), and the Exchange Management Pack Guide (http://download.microsoft.com/download/7/4/d/74deff5e-449f-4a6b-91dd-ffbc117869a2/om2007_mp_exsrvr2003.doc).
Read the Management Pack guide – cover to cover. There are important pieces you need to know that this document spells out in detail.
Import the Exchange Management Pack (either using the Operations console or PowerShell).
Deploy the OpsMgr agent to all Exchange Servers. The agent must be deployed to all Exchange Servers. Agentless configurations will NOT work for the Exchange Management Pack.
Get a list of all Exchange Servers from the Operations console. In the Authoring node, navigate to Authoring -> Groups -> Microsoft Exchange 2003 Server Group. Right-click on the group(s) and select View Group Members.
Enable Agent Proxy configuration on all Exchange Servers identified from the groups. This is in the Administration node under Administration -> Device Management -> Agent Managed. Right-click on each domain controller, select Properties, then the Security tab, and check the box to “Allow this agent to act as a proxy and discover managed objects on other computers.” This has to be done for EVERY EXCHANGE SERVER, even if the server is added after your initial configuration of OpsMgr.
Download and run the Exchange 2003 MP Wizard (http://go.microsoft.com/fwlink/?LinkId=82103) on one of the Exchange servers in the environment. Run the wizard using an Exchange Full Administrator and take the default configurations.
Enable the Exchange Topology View in the Operations console -> Authoring -> Management Pack Objects -> Object Discoveries. Find the Exchange 2003 Topology Discovery and override it for a specific object choosing the Exchange server that you want to perform this role (set it to True).
Enable the mailbox and mailflow rules. To enable these rules, go to Authoring / Rules and search on “message tracking”. Sort the results by the “Enabled by Default” field, and find the following two rules: (There are 8 reports based on these two rules. Because the rules are not enabled by default, the reports are not visible until you set up an override. Thanks to Bernie Chouinard for pointing this out! There is also an error in the collect message tracking statistic vbscript which generates an error in the OpsMgr event log.)
- Performance Collection Rule to Collect Message Tracking Log Statistics – Top Destinations by Message Count
- Performance Collection Rule to Collect Message Tracking Log Statistics – Top Destinations by Size
Configure overrides to Enable these rules for all objects of Type: Exchange Database Storage.
Check to make sure that Exchange shows up under Monitoring -> Distributed Applications as a distributed application which is in the Healthy, Warning or Critical state. If it is in the “Not Monitored” state, check for Exchange servers which are not installed or are in a “gray” state. This may take some time to populate after all of the above tasks have been completed.
Several of the "Top" 100" reports return blank data. This is because the Rule IDs associated with the reports are misconfigured and must be manually edited. Perform the following steps:
- On the Report Server, open a browser and navigate to http://localhost/reports. Select Microsoft.Exchange.Server.2003.Monitoring
- Find and select the "Report.Exchange.Top100MailboxesbySize" report (it does not have a rpdl extension)
- Select the Properties tab, then select the Parameters link on left-hand margin
- Scroll down and find the RuleID String parameter, and replace the value with 2EE6F2C1-4C8B-AFA9-D615-238F6AA73E8C
- Click Apply, then run the Top100 mailboxes report to verify that data is now being returned.
- Repeat these actions for the following Rule IDs:
  - Performance Collection Rule to Collect Mailbox Statistics - Top 100 Mailboxes by Message Count
    New RuleID = 55BBEDA5-C09C-7C06-602F-20C85723EACE
  - Performance Collection Rule to Collect Mailbox Statistics - Top 100 Mailboxes by Size
    New RuleID = 2EE6F2C1-4C8B-AFA9-D615-238F6AA73E8C
  - Performance Collection Rule to Collect Public Folder Statistics - Top 100 Public Folders by Size
    New RuleID = 5D3DAEDA-56E6-909A-FAB8-AF021AA1A61E
  - Performance Collection Rule to Collect Public Folder Statistics - Top 100 Public Folders by Message Count
    New RuleID = B2032940-E1E0-975F-42F0-302C7B5F21DB

This is also documented at http://support.microsoft.com/kb/948096.

Tuning/Alerts to Look for: The following are alerts we encountered and resolved while tuning the Exchange Management pack.

Alert: Multiple/any alert with “Baseline” in the title

Issue: Default sensitivity levels within the Exchange management pack.

Resolution: See blog articles: http://ops-mgr.spaces.live.com/blog/cns!3D3B8489FCAA9B51!183.entry and http://ops-mgr.spaces.live.com/blog/cns!3D3B8489FCAA9B51!189.entry for details on tuning baseline counters.

Alert: The Internet Information Service NNTP Virtual Server named NNTPSVC/1 is unavailable as the virtual server has been stopped.

Issue: On Exchange servers this service is required to install but it is not required after it is installed.

Resolution: If this service is disabled/not in use you can remove it. To remove the service, log into the server and use “sc delete NNTPSvc”. Or you can create an override to ignore this on Microsoft Exchange 2003 Server Group, as NNTP was required for the installation but can be disabled after the installation has been completed.

Alert: Verify Test Mailboxes: This Exchange Server does not have any MOM test mailboxes.

Issue: Test mailboxes are created by the Exchange Configuration Wizard.

Resolution: Run the Configuration Wizard to create the mailboxes.

Alert: No MOM test mailbox account for some mailbox databases

Issue: Test mailboxes are created by the Exchange Configuration Wizard.

Resolution: Run the configuration Wizard creating test mailboxes on each database or disable the rule.

Alert: Replication is not occurring – All replication partners have failed to synchronize

Issue: The Alert Description is the key on this alert.

Resolution: Alert description of “AD Replication Monitoring : All replication partners are now replicating successfully” is a success condition and does not require any intervention other than closing the alert.

Alert: Some replication partners have failed to synchronize

Issue: A domain controller was offline and unable to be synchronized with.

Resolution: Bring the domain controller back online.

Alert: Outlook Web Access logon failure: Unexpected error during synthetic Outlook Web Access logon

Issue: OWA Logon failure: OWA can only be configured to be monitored if the site runs on HTTPS.

Resolution: Disabling the rule (For all objects of type: Exchange OWA), as this environment only runs with HTTP on the OWA configuration.

Alert: Exchange ActiveSync logon failure: Unexpected Error

Issue: Exchange EAS not required in the environment.

Resolution: Disabled the rule for all types of type Exchange EAS, as this functionality is not used in the environment.

Alert: The 3GB virtual address space option is not enabled

Issue: The 3GB configuration should be used for Exchange servers except for those which are functioning as bridgeheads or front-end servers (per the Exchange Best Practices Analyzer [BPA]).

Resolution: Disabling this rule for the front-end servers or bridgehead servers in the environment.

Alert: Failed to probe the state of monitored services

Issue: This was occurring on the SMTP services on an Exchange server which the administrators has manually restarted.

Resolution: The alert was notifying on a true business-impacted situation. Requested the administrators to put the server into maintenance mode prior to making changes like this, unless it is an emergency situation.

Alert: Data Publisher object is not installed

Issue: This was a system which was misidentified as an Exchange sever that was using a third party product to provide Exchange restoration functionality.

Resolution: Disabled the rule for this system through an override.

Alert: Microsoft Windows Internet Information Server 2003 NNTP Virtual Server is Unavailable.

Issue: NNTP Service Down on non-active cluster node

Resolution: The NNTP service is supposed to be down since it is running on a cluster and the system showing this error is not the active node in the cluster. Created a group for these servers which are running Exchange and are part of the cluster and disabled the rules for the group. NNTP was not used on Exchange and could also have been removed as a service from the systems.

Alert: Microsoft Windows Internet Information Server 2003 SMTP Virtual Server is Unavailable.

Issue: SMTP Service Down on non-active cluster node

Resolution: The SMTP service is supposed to be down as it is running on a cluster and the system showing this error is not the active node in the cluster. Created a group for these servers which are running Exchange and are part of the cluster and disabled the rules for the group.

Alert: Microsoft Windows Internet Information Server 2003 Web Site is Unavailable

Issue: Web Service Down on non-active cluster node

Resolution: The Web service is supposed to be down as it is running on a cluster and the system showing this error is not the active node in the cluster. Created a group for these servers which are running Exchange and are part of the cluster and disabled the rules for the group.

Alert: Check Services FE Monitor reported a problem

Issue: Product knowledge on this: “Services State monitoring with this registry key is a legacy from the MOM 2005 Exchange 2003 MP. This monitor is included since configuration is possible from within the Exchange Configuration Wizard. OpsMgr 2007 provides a dedicated health model for monitoring Windows Service Health.”.

Resolution: Right-click and choose Overrides, Disable the Monitor for all objects of type: Exchange 2003 Role.

Alert: Exchange EAS monitor reported a problem

Issue: Synthetic Exchange ActiveSync requires SSL

Resolution: Closed the alert as it had not repeated for 2 days and had a 15 minute schedule to run. Issue repeated. EAS logon verification: Cannot measure EAS availability for the following URL: 0x80131537(-214233033) Invalid URI: The format of the URI could not be determined. Found the following information at MyItForum:

This script problem is caused by OMA and EAS virtual directories not being SSL-enabled. So in order to correct it, simply enable SSL:

Open Internet Information Services (IIS Manager).
Connect to the server name of your front-end Exchange server.
Drill down to Web Sites, then to the web site.
Locate the two virtual directories named OMA and Microsoft-Server-ActiveSync.
Open the properties of the virtual directories, choose the Directory Security tab.
Under Secure communications, click Edit.
Check the box labeled "Require security channel (SSL)".

Alert: No MOM test mailbox account for some mailbox databases

Issue: No MOM mailboxes were created on a per-storage group when running the configuration Wizard. The alert is being created expecting that per-store monitoring will be configured which is not the case in this environment.

Resolution: Disable this rule for all objects (of type Exchange 2003 role) because this rule is monitoring on a per-store basis but we are monitoring on a per-server basis. Closed the alerts.

Alert: SSL is not configured on this Exchange server

Issue: This occurs on servers which have SSL enabled if they do not require usage of SSL within IIS. Back-end servers communicate with front-end servers via HTTP not HTTPS so SSL should not be required on the back-end Exchange servers. We found the following information at Notes from the Underground…

"SSL in a Front-End/Back-End Scenario
Although it’s possible to implement SSL on a front-end (FE) server, resulting in all transmitted data between the FE and your client browsers being encrypted, you should be aware that you can’t use SSL between any FE and back-end (BE) servers—it simply doesn’t work. This means that if your FE server is placed in a perimeter network (also known as a demilitarized zone, or DMZ), all traffic between the FE and BE would be unencrypted. So if you’re planning such a scenario, consider using IPSec between the FEs and BEs. More and more organizations place their FEs directly on their private networks (and instead place an ISA server or similar in the DMZ), which eliminates this security risk.”

Resolution: Disabled the alert on Exchange back-end servers.

Alert: Calendaring agent failed with error while saving appointment

Issue: Calendaring agent failed with error code 0x8004010f while saving appointment.

Resolution: Good links on this: http://www.eventid.net/display.asp?eventid=8206&eventno=1103&source=EXCDO&phase=1. Lots of product knowledge on this related to virus scanners, registry settings, etc. This is a result of an event ID of 8206 on the Exchange server.

Alert: Disabled user does not have a master account SID.

Issue: The user does not have “Associated external account” permission and the Exchange server does not have the hotfix available to resolve this issue.

Resolution: To resolve this, open the user account in Active Directory Users and Computers, go to Properties, Exchange Advanced, Mailbox Rights. For the Self account we added the “Associated external account” permission which resolves the error. The error itself does re-appear, but it appears with the next user identified in the environment which had the issue. If there are a large number of these in your environment you can also locate them by going to each Exchange back-end server, and doing a Filter on event number 9548 within the application event log. A hotfix is available for this, available at: http://support.microsoft.com/kb/916783. (This information is a subset of what was originally posted at http://cameronfuller.spaces.live.com/blog/cns!A231E4EB0417CB76!835.entry.)

Alert: Low Free Disk Space

Issue: Part of the Exchange Management Pack checks for free space on all drives including those which do not have Exchange directories or files on them. This activates a warning at less than 5% free disk space and less than 1000 MB of free disk space on Exchange server drives that do NOT have the transaction logs or queue files on them.

Resolution: Free disk space on the drive. See the “Logical Disk Free Space is Low” entry for potential approaches to free disk space on the drive.

Alert: Very low free disk space

Issue: Part of the Exchange Management Pack checks for free space on all drives including those which do not have Exchange directories or files on them. This activates an error at less than 2% free disk space and less than 400 MB of free disk space on Exchange server drives which do NOT have the transaction logs or queue files on them.

Resolution: Free disk space on the drive. See the “Logical Disk Free Space is Low” entry for potential approaches to free disk space on the drive.

Alert: Logical Disk Free Space is Low

Issue: Low disk space on a drive within a server being monitored by OpsMgr.

Resolution: Can either free up disk space on the drive or configure an override for the drive to change the monitoring configurations for the drive (see http://cameronfuller.spaces.live.com/blog/cns!A231E4EB0417CB76!1001.entry for details on how to do this override). Other items to consider:

If the page file is currently on the drive which is critical on drive space, it can be moved to another drive.
The “disk cleanup” wizard can also be used to provide methods to free up disk space (right-click on the drive, go to properties, click the disk cleanup button).
If the drive is critical on available free disk space, automatic updates can be turned off in the control panel and the c:\windows\softwaredistribution\download folder can be removed (of course, automatic updates will not occur after this change is made).
The default IIS configuration puts the IIS log files under C:\WINDOWS\system32\LogFiles\W3SVC1. These can be moved within the Internet Information Services (IIS) Manager by clicking on the properties of the web sites, under the properties of the log files. The log files can either be moved or disabled if required.
Exchange log files can take up a large amount of disk space on a drive if the Exchange server is not being backed up regularly. When the Exchange server has a full backup completed the log files are removed. If an Exchange server is critical on space on the log drive, determine if backups are occurring and if they are not, perform an ntbackup of the Exchange files to truncate the logs. Circular logging (which removes this type of a situation) can also be enabled in some configurations but is not recommended if there is any mailbox data on the system.

Alert: MAPI Logon Failure.

Issue: This occurred almost immediately after running the Exchange 2003 Management Pack configuration wizard.

Resolution: The issue was resolved when the Wizard completed its configurations and had only repeated once. Ran the “MAPI Logon” task to validate that the issue had been resolved and confirmed no errors. Closed out the alert.

Alert: MAPI session closed due to excessive number of store objects in use.

Issue: Exceeded the maximum of 250 objects of type “objtMessage” (1 repeat). Or exceeded the maximum of 32 objects of type “session” (0 repeats). Or Exceeded the maximum of 500 objects of type “objtFolder”.

Resolution: Microsoft resolutions in the Product Knowledge. Eventid.net has http://www.eventid.net/display.asp?eventid=9646&eventno=3449&source=MSExchangeIS&phase=1 on this. Microsoft KB article on this: http://support.microsoft.com/default.aspx/kb/830836.

Alert: Outlook Web Access logon failure: Unexpected error during synthetic Outlook Web Access logon

Issue: OWA Logon failed. Cannot measure OWA availability. Unexpected error. No Exchange virtual servers and virtual directory (SSL enabled) can be found on this server to form a valid URL. Try providing the url in the custom urls registry key.

If the name in URL matches the name in the certificate, we learned that when SSL is enabled, the MP reports an error like this when 'Require SSL' checkbox is not checked on the Directory Security tab of the website. See Andy Dominey’s blog writeup on this: http://myitforum.com/cs2/blogs/adominey/archive/2007/04/10/mom-2005-and-om-2007-exchange-2003-management-pack-issue.aspx

This rule requires OWA to be installed with SSL and to have require SSL checked on the system. It will NOT work without both of these configured. This also requires that the name matches the name on the certificate.

Resolution: Enable SSL and require SSL on the OWA server. If the name of the URL doesn't match the certificate this rule will not work.
Update: Microsoft has a resolution to this which is available at http://support.microsoft.com/default.aspx/kb/919356 for the error 0x80131502(-2146233086) Index was out of range.

Alert: The MAD Monitoring thread was unable to read the CPU usage information.

Issue: This had repeated 8 times in 5 days/16 hours. The MAD Monitoring thread was unable to read the CPU usage information, error ‘0x800706be’. From the summary, if this happens occasionally it can be safely ignored. If it happens every five minutes then there is an issue.

Resolution: Closed the alert as it was not occurring “frequently”.

Alert: The Offline Address List (OAL) Generator could not generate full details for some entries in the OAL. To see which entries are affected, event logging for the OAL must be set to at least medium.

Issue: MSExchangeSA event id 9320.

Resolution: Eventid link on this: http://www.eventid.net/display.asp?eventid=9320&eventno=3692&source=MSExchangeSA&phase=1. The Microsoft article on this is available at http://support.microsoft.com/default.aspx/kb/908496.

Alert: The Offline Address List Generator could not generate full details because the total size of the details information is greater than 64 kilobytes.

Issue: See the Microsoft support article.

Resolution: The Microsoft article on this is available at http://support.microsoft.com/default.aspx/kb/908496.

OpsMgr by Example: Monitoring Web Applications

Sat, 14 Jul 2007 10:23:53 GMT

Operations Manager 2007 includes built-in website monitoring functionality (similar to that provided by MOM 2005’s Web Sites and Web Services MP), using the "Web Application" Management Pack Template. This functionality is quite useful for monitoring web sites. The template records where you go with your browser (to use this functionality you need to configure your browser, in Internet Explorer under Tools -> Internet Options -> Advanced -> Enable third party browser extensions (requires restart) in both IE 6 and IE 7). Web Applications are created in the Operations console under Authoring -> Management Pack Templates -> Web Application.

We decided to start simple, and then move into more complex monitoring configurations.

Starting simple was developing a web application that monitors a single webpage (such as www.google.com or www.microsoft.com) without requiring authentication. There is a great write-up available at http://www.technotesblog.com/?p=432 which provides an step-by-step process to create monitoring for a single web page. We used www.google.com in our particular example, and as in the TechNotesBlog example (which uses the Microsoft website), we disabled link tracking.

Getting more complex: once our application was in a working state we went to the next step that we wanted to test - monitoring the OpsMgr Operations Web console. Since the Operations Web console requires authentication, the monitoring setup is more difficult. We created a new Web Application called Operations Web console (and stored it in a new, non-default management pack), and had the application browse to http://(servername):51908/default.aspx for the Operations Web console. We created the Web Application using the default configurations and ran it on Windows 2003, Windows XP and Windows Vista workstation systems (one of each for testing purposes). Each of these systems went to a critical status due to an access is denied message.

You can check the status of the monitored web sites by navigating in the Operations console to Monitoring -> Web Application -> Web Application State. You can also right-click and open the performance view for any of these and receive a large number of performance information is collected, check the graphic for an example.

Resolving the security issue required creating a Run As Account of type Windows (under Administration -> Security -> Run As Accounts), using an account with permissions to access the Operations Web console. We then configured this account to be used by the Web Application in the Authoring section under Authoring -> Management Pack Templates -> Web Application:

Edit the web application settings for the Operations Web console just created
Select the General tab to configure its settings, select the authentication method of NLTM and specify the account created to monitor the web site

After going back to monitoring section (Operations Console -> Monitoring -> Web Application -> Web Application State) and waiting a little bit, the Operations Web console monitor went to green.

To get even more complex, we created a web test that used the recorder. The Reporting Server was a good test for this. The URL for the Reporting Server is located under Administration -> Settings -> Reporting. In our testing environment this has a value of http://QUICKSILVER:80/ReportServer. To record, we started with http://QUICKSILVER/Reports and worked from that point. We opened up a graphic, and a folder, and a report during the capture process. Running a report would also be an option, but as this would run on a regular basis (every few minutes) we did not want to create that level of overhead with our monitoring. We configured the authentication method (NTLM and the account we previously created) and the watcher node. We then checked its status in the Health Explorer (see graphic), all were green.

Lessons Learned:

The systems performing the watcher function did not have any customizations made to their browsers, such as adding the browser location to the trusted sites. Some servers would work well as watchers and other would not (in our case the Root Management Server). We were unable to identify a specific reason for this.

Don’t test authentication items within the Web Application creator. It brings up a pop-up that warns “Running a test of this web application may fail. While running the test, credentials that have been configured for this web application will not be used. If the site you are testing does not explicitly require authentication, the test may still succeed.” Test these by actually checking their alerts and status on the monitoring tab.

If the site requires authentication to get to it, you need to configure authentication for the web tests. Check IIS to see what type it allows and provide a match (NTLM = Integrated Authentication in this particular case).