Operations Manager

Version 6.0.62623.0 of the Exchange Server management pack requires Operations Manager 2007 R2 and is specifically for the Exchange 2007 product.

How to Install the Exchange 2007 MP

1. Download the Exchange Server 2007 Management Pack (URL). The Exchange Server 2007 Management Pack Guide is included in the download and labeled “OM2007_MP_EX2007_R2.doc.”
2. Read the Management Pack Guide[md]it’s just over 100 pages so all the configurations are not discussed here. This is a complicated management pack, and the document spells some important pieces of information you will need to know including how to enable discovery, as well as optional configurations available for the Exchange 2007 management pack. Marnix Wolf also has a good write-up on this, available at http://thoughtsonopsmgr.blogspot.com/2009/07/quick-guide-to-get-native-exchange-2007.html.
3. Import the Exchange Server 2007 Management Pack (using either the Operations console or PowerShell). If you have Exchange clusters, add the cluster management pack before adding the Exchange Server 2007 management pack.
4. Deploy the OpsMgr agent to all Exchange Servers. The agent must be deployed to all DCs. Agentless configurations will NOT work for this management pack.
5. Enable Agent Proxy configuration on all Exchange Servers identified from the groups. This is in the Administration node, under Administration -> Device Management -> Agent Managed. Right-click each Exchange 2007 server, select Properties, click the Security tab, then check the box labeled “Allow this agent to act as a proxy and discover managed objects on other computers.” Perform this action for every Exchange 2007 server, even if you add server after your initial configuration of OpsMgr.
6. Check to make sure Exchange 2007 Service shows up under Monitoring -> Distributed Applications as a distributed application that is in the Healthy, Warning, or Critical state. If it is in the “Not Monitored” state, check for Exchange servers that are not installed or in a “gray” state.
7. Windows PowerShell and the Operations console need to be installed on the RMS. If not already installed, do so at this time.
8. The Exchange 2007 management pack is designed to not automatically discover any Exchange 2007 Server roles so you can phase in monitoring. See the management pack guide for details on how to enable discoveries.
9. Create an Exchange2007_Overrides management pack to contain any overrides required for the MP.

Exchange MP Tuning / Alerts to look for

The following alerts were encountered and resolved the following alerts while tuning the Exchange 2007 management pack (listed alphabetically by Alert name):

Alert: Exchange 2007 ExBPA Generate Alert Rule

Issue: Application log size should be increased to 40MB.

Resolution: Logged into each system, opened the event viewer and changed the properties of the application log to 40960, and closed the alerts because they were generated by a rule not a monitor.

Determined the default domain policy was set to 12160. To change this, implemented a new group policy linked to the Exchange servers OU to set the application log file size to 40960. See http://technet.microsoft.com/en-us/library/cc778402(WS.10).aspx for details on how to configure this policy.

Alert: Exchange 2007 ExBPA Generate Alert Rule

Issue: There are multiple different alerts generated based upon the ExBPA. In this case, the issue was the Exchange fatal information on the server is not set to automatically send to Microsoft for analysis.

Resolution: As this particular environment does not allow these servers to have Internet connectivity, the rule had to be disabled. To do so, open the Operations console and navigate to Authoring -> Management Pack Objects -> Rules; look for Exchange 2007 Best Practice and sort by the rule name of Crash upload logging is disabled. Overwrote this rule to disable it for all Exchange 2007 servers then closed the alert.

Alert: RPC latency is above the threshold

Issue: One Exchange server in the environment was reporting this frequently and automatically closing itself as part of the IS RPC Latency Monitor alert monitor. Per product knowledge, these commonly occur due to issues communicating with the Active Directory service or due to disk bottlenecks. This specific server is experiencing high disk queue lengths on the C drive and both storage group drives.

Resolution: This server cannot be upgraded and is scheduled for replacement with new mailbox servers. In the interim, configured an override for this object of class: Exchange IS Service, and set the new threshold to 100 from the default value of 50 to minimize the number of alerts on this issue on this server. The override value will vary depending upon what the actual daily average is for the particular server, which can be determined by right-clicking on the alert, opening the performance view and seeing the RPC Averaged Latency counter over a period of time.

Alert: Exchange 2007 Test Local Mail Flow Alert

Issue: An Exchange 2007 mailbox server was failing when attempting to perform a local mail flow test. Logged into the server and reviewed the event logs but found no relevant information. Opened the Exchange Management Shell and ran the “test-mailflow” command. Received a message that “No mailbox databases were found on SERVERNAME to perform the operation. Check if the user has permission to read the Exchange configuration from Active Directory.” This server does not have any user mailboxes on it (it is scheduled for decommissioning).

Resolution: Created an override to disable this alert and reset the health for this through the Exchange 2007 distributed application.

The Office SharePoint Server 2007 management pack and SharePoint Portal Server 2003 management pack are separate management packs. This discussion focuses on the Office SharePoint Server 2007 management pack, as this is the more current version of the two.

How to Install the SharePoint MP

1. If your environment is running SharePoint 2007, download the Office SharePoint Server 2007 Management Pack from the Management Pack Catalog. The Office SharePoint Server 2007 Management Pack Guide is included in the download and labeled “Microsoft_Office_SharePoint_Server_Management_Pack_Guide.doc.”
2. Read the Management Pack guide, which includes tips such as resizing the Windows System and Application logs to at least 10240 in size.
3. Import the Office SharePoint Server 2007 management pack (using either the Operations console or PowerShell).
4. Create a SharePoint_Overrides management pack to contain any overrides required for the MP.

The Office SharePoint 2007 management pack does not support agentless monitoring.

SharePoint MP Tuning / Alerts to look for

The following alerts were encountered and resolved when tuning the various SharePoint management packs (listed in alphabetical order by Alert name). Alerts from the SharePoint Portal Server 2003 management pack are also provided as additional information in the event similar issues are found in the SharePoint 2007 version of the management pack.

Alert: Can not load virus scanner

Issue: SharePoint 2003 alert. This looks for event number 1000 from the source of Windows SharePoint Services 2.0 containing the description.

Resolution: There is no virus scanner on this system specifically for SharePoint, so this rule was disabled for this system.

Alert: Failed to load index

Issue: This appears to be a side effect of an underlying corruption that occurred on an index. See http://objectmix.com/sharepoint/298641-sharepoint-search-index-corrupt.html.

Resolution: This environment had three SharePoint servers; two of them were for SharePoint content and the third server to provide indexing. In SharePoint Central administration it was determined there was no indexing server. Configured the third server to perform indexing and these issues no longer occurred.

Alert: Failure in loading assembly

Issue: Per the management pack guide: Discovery for the SharePoint Portal Server Management Pack relies on the file wssDiscovery.exe. This file is run automatically when you import the Management Pack. You must configure the Agent Action Account so that is has administrative access to the SharePoint Portal Server API and administrator rights to the SQL Server databases.

Resolution: For this to work correctly, you must enable Proxy. Be sure to enable Proxy for each of the SharePoint (including SharePoint 2003) servers.

Alert: IIS Stop Command

Issue: IIS was stopped on the server while a portal protection program was installed on the system. This action was performed by a system administrator. This is an alert rule so it will not close automatically.

Resolution: Created a web monitor for the SharePoint website. Changed the severity of this item from 2 to 1 (critical to warning) as this is by itself not a critical situation. If the website was down as a result, that would be a critical situation but these are checked separately with web monitors.

Alert: Index is corrupt

Issue: Index corruption identified on multiple SharePoint servers.

Resolution: This environment had three SharePoint servers; two of them were for SharePoint content and the third server to provide indexing. In SharePoint Central administration, it was determined there was no indexing server. Configured the third server to perform indexing and these issues no longer occurred.

Alert: Query server removed from rotation

Issue: This appears to be a side effect of an underlying corruption that occurred on an index. See http://objectmix.com/sharepoint/298641-sharepoint-search-index-corrupt.html.

Resolution: This environment had three SharePoint servers; two of them were for SharePoint content and the third server to provide indexing. In SharePoint Central administration, it was determined there was no indexing server. Configured the third server to perform indexing and these issues no longer occurred.

Alert: Sweep Synch failed

Issue: Errors reported on various SharePoint servers in the server farm. This appears to be a side effect of an underlying corruption that occurred on an index. http://objectmix.com/sharepoint/298641-sharepoint-search-index-corrupt.html

Resolution: This environment had three SharePoint servers, two of them were for SharePoint content and the third server to provide indexing. In SharePoint Central administration, it was determined there was no indexing server. Configured the third server to perform indexing and these issues no longer occurred.

Alert: The Microsoft Single Sign-on Service State

Issue: This service is not running on the SharePoint 2007 servers in the environment. This service is used to store login credentials for 3rd party applications so you can create links from SharePoint to those apps and auto login users if they aren’t using Windows Authentication. That is the only reason you would need to have it running.

Resolution: This functionality was not in use in the environment, so created an override to disable the alert, and stored it in a Sharepoint_Overrides management pack.

Alert: The Office SharePoint Server Search Service State

Issue: Office SharePoint Server Search is not running. This appears to be a side effect of an underlying corruption that occurred on an index. See http://objectmix.com/sharepoint/298641-sharepoint-search-index-corrupt.html.

Resolution: This environment had three SharePoint servers; two of them were for SharePoint content and the third server to provide indexing. In SharePoint Central administration, it was determined there was no indexing server. Configured the third server to perform indexing and these issues no longer occurred.

Alert: Unable to discovery sharepoint components

Issue: Per the management pack guide: Discovery for the SharePoint Portal Server Management Pack relies on the file wss Discovery.exe. This file is run automatically when you import the management pack. You must configure the Agent Action Account so that is has administrative access to the SharePoint Portal Server API and administrator rights to the SQL Server databases.

Resolution: For discovery to work correctly, you must enable Proxy. Be sure to enable Proxy for each of the SharePoint (including SharePoint 2003) servers.

The Windows Print Server management pack is available as a single download that contains different libraries to monitor Windows Print services on Windows Server 2000, 2003 and 2008 operating systems.

How to Install the Windows Print Server MP

1. Download the Windows Print Server management pack from the Management Pack Catalog (http://technet.microsoft.com/en-us/opsmgr/cc539535.aspx). The Windows Print Server Management Pack Guide is included in the download and labeled “OM2007_MP_PrintSvr.doc.”
2. Read the Management Pack guide to gather additional information such as that the current version only supports monitoring clustered instances of the Print Server role on Windows Server 2008.
3. Import the Print Server Management Pack (using either the Operations console or PowerShell) after importing the Windows Server management pack.
4. Create a PrintServer_Overrides management pack to contain any overrides required for the MP.

The Windows Print Server MP supports agentless monitoring with the exception of tasks.

Windows Print Server MP Tuning / Alerts to look for

The following alerts were encountered and resolved while tuning the various print server management packs (these are listed in alphabetical order by Alert name):

Alert: Document failed to print

Issue: Document failed to print, categorized as a critical alert.

Resolution: This should not be a critical alert. Downgraded to a warning level alert through an override in the PrintServer_Overrides management pack.

Alert: Printer: Publish Error

Issue: The print server had a single DNS server defined for its location and the DNS server was down.

Resolution: The DNS server was brought back online and a second DNS server was configured for the system that had reported this error. Manually closed the alert.

Alert: Shared Printer Availability Alert

Issue: An error was generated for each of the shared printers on the server. These were generated by the Shared Printer: Restart the print spooler fix sharing problems and check Group Policy alert.

Resolution: This is an alert not a monitor so it does not automatically resolve itself. There does not appear to be an equivalent number for the same source in the event logs which indicates that the print spool is back online (which may be why this is an alert rather than a monitor).

Per the knowledgebase, restarted the spooler service on the server with the issues when this was caused by a situation that was not a reboot.

This also occurred when the print server was rebooted. Verified the ability to print to the printer specified after the reboot completed.

Print Server Management Pack Evolution

While the Windows Server 2008 version of this management pack functions well, the earlier versions of this management pack do not work well on a clustered server (either Windows 2000 or 2003). There are also issues with the approach to discovery on Windows 2000/2003 servers so that many servers are identified as being print servers that may not actually be print servers. The evolution of this set of management packs would be to bring the same functionality now available in the Windows Server 2008 management pack to the Windows 2000 and 2003 versions.

The Windows DHCP Server management pack is available as a single download that contains different libraries to monitor Windows DHCP on Windows Server 2000, 2003 and 2008 operating systems.

How to Install the DHCP MP

1. Download the Windows DHCP Server management pack from the Management Pack Catalog (http://technet.microsoft.com/en-us/opsmgr/cc539535.aspx). The Windows DHCP Server Management Pack Guide is included in the download and labeled “OM2007_MP_DHCP_2003_2008 QFE110408.doc.” Read the Management Pack guide, as this covers items to be aware of with the DHCP such as how DHCP Clustering and multicast scopes are not supported.
2. Import the Windows DHCP Server Management Pack (using either the Operations console or PowerShell).
3. Create a WindowsDHCP_Overrides management pack to contain any overrides required for the MP.

DHCP MP Tuning / Alerts to look for

The following alerts were encountered and resolved while tuning the various DHCP management packs (these are listed in alphabetical order by Alert name):

Alert: DHCP IPv4 Runtime Authorization Needed Alert

Issue: DHCP scopes (both IPv4 and IPv6) were showing up as turned off offline/not authorized. This server had been authorized and then the IP address of the server was changed. The authorization was listing the previous IP address.

Resolution: Unauthorized the current server name/wrong IP address and re-authorized it with the correct IP address. This was occurring on a domain controller in a child domain. To do this change, logged into the root domain and authorized/re-authorized the server, and then restarted services on the domain controller after a short period of time.

Alert: DHCP Scope Addresses Available Monitor

Issue: Alert description is that the available scope addresses have fallen below the specified threshold. This is raised by the DHCP Scope Addresses Available Monitor. This monitor goes to warning level when there are less than 10 available IP addresses in the pool, and to critical when there are no remaining IP addresses in the pool. This environment originally had one DHCP server that contained the entire scope of available addresses. To provide redundancy, the scope was split between two different DHCP servers. This unfortunately leads to a tendency for the original DHCP server scope to fill while the other scope remains with a large number of available addresses in the range.

Resolution: For this environment, it was necessary to match up the two different scopes to determine if the lack of addresses was really a lack of addresses or just half of the scope filled while the half remained open. Performed the following actions to make this more readily apparent:

• Configured the warning states on this monitor to go to yellow when there are less than 2 available IP addresses in the scope.
• Monitored within the Microsoft Windows DHCP Server -> Scope Health view and ordered by display name to validate that the address range was not critical on both halves of the scope.
• Used the Microsoft Windows DHCP Server -> DHCP Performance Views -> Scopes & Superscopes -> Scope Free Addresses view sorted by Instance and color coded to match the colors for each half of the scope (so that as an example both halves of the Data Network on floor three show up as blue). Added this to the My Workspace view with the Y access limited to a maximum of 10 (to more easily identify scopes with less than 10 available addresses). This is useable but pretty unwieldy with a large number of DHCP scopes.

Alert: DHCP Service Bound to Static IP Address

Issue: The alert description on the Alert Context tab shows that “The DHCP service is not servicing any clients because none of the active network interfaces have statically configured IP addresses, or there are no active interfaces.”

Resolution: The product knowledge provided an effective resolution for this issue. The DHCP service was not bound to any IP addresses on the system. In this case this DHCP scope was not required and as it was the only DHCP scope on the system, removing the DHCP service from the system was an acceptable solution after deactivating the DHCP scope for a period of time.

Alert: Performance Threshold: Process\Working Set threshold exceeded.

Issue: DHCP management pack error. Occurring sporadically on the DHCP server in the environment, but not seeing any errors or issues as a result of the condition. The rule (Performance Threshold: Process\Working Set threshold exceeded.) is configured to work over a 5 minute interval, and to measure over three samples by default (per the overrides). The management pack says that the utilization is measured over 5 samples. In a 24-hour period, there were approximately 22 of these alerts occurring and self-closing. Increased the number of samples to measure over from 3 to 5 and tracking the result (as these were usually closing with 5-10 minutes automatically). This did not help the issue.

Resolution: In this environment, this is NOT affecting the ability of the DHCP server to function, but could affect it at higher-level values. This value is in place because exceeding this threshold can be an issue, so do not disable or override this rule unless you are sure that it is NOT impacting your environment.

Determined the trend of this value based upon the alerts in the environment (tried tracking it monitoring the performance counter for this variable with no luck as there does not appear to be one). For each alert, went to the alert context tab and tracked the values which appeared (17806131, 17780028, 17809408, 17823061, 17793024, 17788928, 17791658, 17800330, 17786197, 17787562, 17828522, 17772544, 17783466, 17824426, 17829888, 17821696, 17788928, 17870028). Determined the average, maximum (17870028) and minimum (17772544) values to determine where this threshold should be for the environment/found that closed alerts were not relevant as they showed values less than the threshold.

Created an override to change this value for this server (on the monitor) from the default of 17830000 to 18070000.

Alert: Script or Executable Failed to run

Issue: Script failure for Nslookuptest.js. Reporting for tests to Microsoft.com, localhost ip address, and the fully qualified name of the server all three failed at the same date and time.

Resolution: Noted the alert and the date/time to see if a root cause could be tracked back. Reviewed the event logs on the system to track back potential issues/none found. Reviewed the performance counters gathered by OpsMgr, but no bottlenecks identified during that timeframe. Closed the alerts.

Alert: The DHCP service has determined that it is not authorized on this domain.

Issue: Description says the DHCP/Binl service on the local machine belonging to the windows administrative domain (domain name) has determined that it is authorized to start. It is servicing clients now. This appears as a critical alert, but is actually stating that the DHCP server is working.

Resolution: The only option for the override on this is to disable it so the criticality of the alert cannot be changed. This should actually be an informational level alert. The only option currently is either to close the alerts or to disable the alert.

Alert: The DHCP Service is not servicing any clients because none of the active network interfaces have statically configured IP addresses, or there are no active interfaces

Issue: DHCP server cannot be a DHCP client.

Resolution: Hard-coded an IP address for the DHCP server.

DHCP Management Pack Evolution

It would be extremely useful if in future revisions of this management pack it could effectively match scopes (based upon name, or matching subnet potentially) and gather the information to provide a critical alert when the each of the different scopes was nearing empty.

Another useful enhancement would be to provide the number of available addresses in the range within the alert description text.

The Windows Server Group Policy management pack is available as a single download that contains different libraries to monitor Windows Server Group Policy on Server 2003 and 2008 operating systems.

How to Install

1. Download the Windows Server Group Policy management pack from the Management Pack Catalog (http://technet.microsoft.com/en-us/opsmgr/cc539535.aspx). The Windows Server Group Policy Management Pack Guide is included in the download and labeled “OM2007_MP_GP2008.doc.”
2. Read the Management Pack guide, which points out solid tips like the installation order (windows server management packs, group policy 2008 management packs, and then group policy 2003 management packs).
3. Import the Group Policy 2008 Management Pack (using either the Operations console or PowerShell), and then the Group Policy 2003 Management Pack.
4. Create a GroupPolicy_Overrides management pack to contain any overrides required for the MP.

Agentless monitoring is not supported by the Windows Server Group Policy management pack.

Tuning / Alerts to look for

The following alerts were encountered and resolved while tuning the various Group Policy management packs (these are listed in alphabetical order by Alert name):

Alert: Application of Group Policy Alert

Issue: The alert monitor on the alert was the Time Skew Monitor. The computer in question was in the wrong time zone.

Resolution: Changed the time zone on the server reporting the alert.

Alert: Application of Group Policy Alert

Issue: Alert that a user in a different forest than the computer account is logging on and that Group Policy from the other forest is not currently allowed.

Resolution: This environment has two different forests. One of them is a new replacement forest and for it group policies are being built to replace the group policies used in the original forest. While users will log into the new forest with credentials for the old forest, the old forest group policies should not apply. This will eventually be resolved when the old forest is decommissioned. In the mean time, the monitor was overridden to not be enabled (override, parameter enabled = false) for All objects of type: Group Policy 2008 Runtime.

Alert: Folder Redirection CSE ProcessedWithErrors

Issue: Group policy client failed 1085 and event 107 (which showed the user that had the issue) before it. This was occurring on a terminal server (citrix).

Resolution: User did not have their home folder mapped correctly.

Alert: GPO Data Retrieval Error

Issue: Event log (application) userenv 1058 error on group policy.

Resolution: Found article #828760 that implies that ACSL sysvol issues with the domain controllers and service pack 1. Used gpupdate /force on the system to see if we could recreate the event. Found that it creates a 1704 message in the event log (information) that it succeeded. Tested accessing of this path from the domain name, and from each of the domain controllers that it should be using to authenticate. There were differences in the dates of the folders indicated within the error message itself. The actual content was consistent however. This was occurring on the all domain computers policy. No indication that this occurred because of a WAN outage.

Alert: GPO Data Retrieval Error

Issue: Every 5 minutes errors were occurring in the application log for Userenv for 1058 and then 1030.

Resolution: Determined that the domain controller had not been patched or rebooted in over six months (checked the system log for the event source of eventlog). Patched and rebooted the DC and the group policy errors stopped occurring.

Alert: Group Policy Preprocessing (Active Directory) Alert

Issue: DNS Issues occurred in the environment causing an inability to resolve names in the environment.

Resolution: Fixed the DNS resolution issue so the environment could resolve names.

Alert: Group Policy Preprocessing (Networking) Alert

Issue: This alert occurs when an event of 1058 is created in the system log for the source of GroupPolicy. This occurs when the system is unable to connect to \\abc.com\SysVol\Policies\abc.com\Policies\{guid}\gpt.ini (where abc.com is the domain name and guid is the guid provided in the alert). Issues like this are caused by network connectivity or network resolution, or FRS latency, or if the DFS client is not running (per the knowledge in the alert). Information on this event is available at http://technet.microsoft.com/en-us/library/cc727259(WS.10).aspx.

Resolution: In this case this was an errorcode number of 5, which is access is denied. From the details on the event copied the file path and verified that the system could open the file with notepad. Logged into a server that had the last event in the System log from the source of GroupPolicy and opened a command prompt (run-as administrator) and did a gpupdate /force. Verified successful creation of a 1502, and 1503. Verified that the majority of these alerts all occurred at the same time. Closed this alert.

Also verified that DNS was providing this information correctly. Opened nslookup and did a resolution for abc.com. Copied the name of the file shown (the gpt.ini file) and replaced the abc.com domain name with the actual IP address (\\1.1.1.1\SysVol\Policies\abc.com\Policies\{guid}\gpt.ini) and verified that each of the domain controllers not only had the gpt.ini file but that it was readable from the path specified.

Alert: Group Policy Preprocessing (Security) Alert

Issue: This alert appears to occur when there is an inability to resolve DNS from the system identified or group policy fails to apply. It is stating that the specified domain either does not exist or could not be contacted.

Resolution: Researching this alert from the system log event number 1054 found this article from Microsoft: http://technet.microsoft.com/en-us/library/cc727331(WS.10).aspx. After researching this, it appears that an event 1500 that has occurred since the 1054 occurred indicates that group policy is now functional.

Copied the name of the server from the alert detail pane, changed the view to Monitoring -> Computers and pasted the name of the server into the filter. Used the Computer Management Action to connect and remotely review the event logs for the server. Closed the alert after verifying that the 1500 has occurred since the 1054 occurred in the system log where the alert occurred.

Logged into a server which had the last event in the System log from the source of GroupPolicy and opened a command prompt (run-as administrator) and did a gpupdate /force. Verified successful creation of a 1502, and 1503. Closed this alert.

Group Policy Management Pack Evolution

The Group Policy File Access Monitor in the Group Policy 2008 management pack version 6.0.6648.0 should be a two-state monitor with a health condition of the 1500 event (or 1051 or 1052 or 1053) and a warning or critical for the 1058 event. This could be accomplished by creating a new custom monitor and disabling the original monitor included in the management pack.

The Machine Account Determination Monitor in the Group Policy 2008 management pack version 6.0.6648.0 should be a two-state monitor with a health condition of the 1500 event (or 1051 or 1052 or 1053) and a warning or critical for the 1054 event. This could be accomplished by creating a new custom monitor and disabling the original monitor included in the management pack.