Operations Manager

The following message was posted as an email to this blog last Thursday:

In Operations Manager 2007 Unleashed, you say that it is possible to create synthetic transactions using VBscript. You go on to work through a few examples of synthetic transactions based on pre-defined templates in Operations Manager. As part of this, you describe how to use watcher nodes to run these transactions.
Unfortunately I've not been able to find many examples of people who are actually doing this, so I'm turning to your site in desperation. I have an FTP server and need to prove that I can establish a connection and transfer a file to it. What I thought was the hard bit - I'm not a coder! - is done. A small VBscript establishes a connection, transfers a files and returns the status code to Operations Manager in a property bag.
What I'm struggling with is the simple bit: where to run it. I don't want to run it on the FTP server itself, as this wouldn't prove anything. Ideally I would like to use a watcher node, but I can't find a way to do this without using one of the pre-defined templates for synthetic transactions, none of which allow me to run my own VBscript. Am I missing something obvious?
By the way, and regardless of whether you can answer this post or not, System Center Operations Manager 2007 Unleashed is a truly excellent book. In my 12 year career I've read through many IT titles. This one really stands as an example of what a good IT book should be: it covers the theory, operation and real-life scenarios (for example database sizes) in just the right level of detail.
Andy

Unfortunately, Andy's communication preference settings don't allow us to respond to him, and we would like some additional details as we may need to have him try some things.

Andy, can you email ops-mgr@hotmail.com with an email address that we can reply to? Thanks!

At MMS 2008 last May, Microsoft announced their direction to use Operations Manager to manage non-Windows systems (for more information, see Kerrie’s articles “Of Flying Pigs” at http://www.networkworld.com/community/node/27600 and “The Dynamic Datacenter” at http://www.networkworld.com/community/node/27354). This article discusses our experiences testing a beta version of the (Cross Platform) X-Plat software.

The Conventional OpsMgr Gateway Role

Let’s say you have computers at a branch office, in the offices of a partner or customer, or in a datacenter that resides on an untrusted and/or unconnected network. You put an OpsMgr gateway server on that remote network and connect it to your main OpsMgr management group with certificate-based authentication. Cool technology, and you are now monitoring those remote systems from your main location without standing up any new connectivity and potentially increasing the attack surface.

New OpsMgr/X-Plat Gateway Scenario

Before Microsoft introduced the Cross-Platform beta 1 refresh, you could not leverage that secure yet lightweight OpsMgr gateway service for monitoring any Linux computers at your remote location with anything more than a basic SNMP heartbeat. This article reviews this new feature of the Microsoft System Center Operations Manager 2007 Cross Platform Extensions Public Beta 1 Refresh. The software allows OpsMgr gateway servers to discover and fully manage non-Windows computers at remote network locations. This capability opens a new market for Operations Manager with a novel solution to extend management to Linux and other X-Plat systems such as HP-UX or Solaris and even AIX, which were previously out of reach of native System Center tools.

Note: We review here the second released beta for X-Plat. Features and function will change in the released product. Microsoft plans to release X-Plat as part of an update to OpsMgr in 2009.

Demo environment

An OpsMgr management group with Internet-facing gateway servers includes a gateway server at a remote datacenter. All gateway servers trust the same Certificate Authority (CA) and use unique identity certificates issued by the mutually trusted NOC CA for encryption and authentication. There is a Red Hat Enterprise Linux server (RHEL) at the remote site. We want to use the gateway server to monitor the Linux server from the NOC.

Here are the steps we took to discover and manage the RHEL box at the remote datacenter:

1. Install the X-Plat extensions on a selected management server and consoles. The official name of the installable is “System Center Operations Manager Cross-Platform Extensions.” Prerequisites include OpsMgr 2007 SP1 and WS-Management (WS-Man) 1.1.
   
   Something we liked a lot is that you don’t need to touch the RMS or any high-value management servers to use X-Plat. You only need to install X-Plat extensions on the management server you will run the discovery wizard from.
   
   There are 32-bit and X64 versions of X-Plat, and also full server and console only versions (a total of four .MSI files to select from). Install the console-only executable on other OpsMgr consoles you will use to monitor the cross-platform systems from.
2. Import the desired X-Plat management packs. The server X-Plat extensions setup defaults to dumping about 14 management packs (for all the operating systems supported by X-Plat) to the %programfiles%\System Center Management Packs folder. You only need to import the libraries and management packs needed to manage your target systems. To manage the RHEL 5 box, we imported these management packs:
   • WS-Management Library
   • Linux Operating System Library
   • Unix View Library
   • Red Hat Operating System Library
   • Red Had Enterprise Linux Server 5 Operating System management packs
3. Run ImportXSLT.cmd on those computers where you installed the X-Plat extensions (management server and consoles). This small step changes how the task output and diagnostic and recovery messages generated by Health Explorer on Unix and Linux computers are displayed. This step has to take place after the X-Plat management packs are imported or you will receive an error.
4. Install the X-Plat extensions on the gateway server. Repeat the installation, similar to the management server. An additional step is that we create a UnixAgents folder in the AgentManagement folder of the gateway server. Extract the UnixAgents.zip that comes with X-Plat to that folder. When the gateway pushes the agent to the Linux server at the datacenter, the Linux bits will come from that folder.
5. Configure the management group Run As Accounts. There is some manual work for the OpsMgr administrator to let the X-Plat extensions on the gateway server know what the credentials are to access the Linux computer.
   1. In the Administration -> Security -> Run As Accounts node of the Operations console, create two new Run As Accounts of the Basic Authentication type. One is a normal user account on the Linux computer and one is a privileged account. For the demo, we used the same root account and password for both Run As accounts. Name the accounts something that identifies them with the gateway server.
   2. In the Security -> Run As Profile node, locate the Unix Privileged Account and associate it with the privileged Run As Account and the target of the gateway server with X-Plat Extensions. Similarly, associate the Unix Action Account Run As Profile with the normal user Run As Account and the target of the gateway server.
   3. This beta release of X-Plat extensions only provides for a single pair of Run As Accounts per management server or per gateway server that performs the discovery and monitoring. To monitor other Linux computers with different sets of credentials requires an additional management server or gateway server for each set of credentials. This is a product limitation we hope is overcome in future releases.

6. Discover and accept the Linux server from the management server. This is just like using the Discovery Wizard from the Administration space of the Operations console, except you launch the X-Plat discovery process from the Overview page of the Cross Platform management pack in the Monitoring space. (In later releases X-Plat discovery is expected to migrate to the Administration space and integrate with Windows computer and network device discovery.)
   • An issue with this beta release of X-Plat is that support for discovery of the most current versions of some Linux distributions isn't there. In our environment where the demo Linux computer is located, datacenter security polices require Linux distributions be kept current.

     While RHEL 5.2 is the current release, X-Plat only discovers up to RHEL 5.1. (Our hope and assumption is that the RHEL 5.1 agent will work on 5.2.) We expect that with future releases of X-Plat, there will be a community effort to keep X-Plat management packs updated with discovery support for more versions and releases.

     There is a manual install option for the X-Plat agent, which in this case would be as follows (the RPM file can be found in the UnixAgents folder on the gateway server):

     rpm -i scx.1.0.1-151.rhel.x86.rpm

     Another solution that enables use of the automatic discovery and integrated features of the X-Plat management packs is to 'trick' the discovery into thinking that the RHEL 5.1 version is installed on the target computer. We used this method, and pushed the version RPM file for 5.1 to the target computer running RHEL 5.2 with this command:

     rpm -i --force redhat-release-5Server-5.1.0.2.i386.rpm

     The --force switch is used since there is a file version downgrade. That RPM file is part of the RHEL 5.1 Server distribution. To later restore the RHEL 5.2 version file, it's enough to run the command "yum update redhat-release-5server" for the single package, or "yum update" to update any other pieces with patches since it was installed.

   • Perform the discovery from the console of a management server where X-Plat Extensions is installed. You need privileged access to the Linux server to push the agent. If you don’t have a superuser account, you need to provide the root user password. After you specify the IP address and privileged account information for the target, if the computer is discoverable, it will shortly appear as seen in this screenshot of the Select Computers to Manage step in the Unix and Linux Computer Management Wizard:

After approving the discovered Linux computer, the gateway server uses SSH to push the System Center Cross-Platform (SCX) agent to the /tmp folder of the Linux computer. After a few minutes you can query the state of the two services that are started by the SCX agent. See this screen shot of an SSH session from the gateway server to the managed Linux server, confirming that the WS-Man daemon and the CIM server are up:

Managing Red Hat Linux with Operations Manager

Soon after completing these actions, the RHEL computer appeared in the Linux Servers state view of the OpsMgr console. Next, data started appearing in the memory and processor-related views. Some hours later, the disk and network views were populated. We received some alerts regarding invalid SSH authentication attempts, and we immediately had a solid feeling about our ability to really manage Linux boxes from Windows with OpsMgr.

Here is a screenshot of an alert related to security of the SSH services on the RHEL box:

An Internet-facing web server is going to get a lot of intrusion attempts against any open service. We secured the SSH services on the RHEL box with these host rules (and the alerts stopped!):

1. Edit /etc/ssh/ssh_config
   1. “vi /etc/ssh/ssh_config”
   2. Press “i” to allow modification of file contents

2. Modify line to restrict SSH protocol to version 2
   1. Locate line “# Protocol 2,1”
   2. Remove “#” from beginning of line, and “,1” from end of line.

3. Save the file
   Press “:wq” and press enter

4. Modify hosts.deny file to deny all hosts access to SSH
   1. “Vi /etc/hosts.deny”
   2. Press “i” to allow modification of file contents
   3. Add this to the next available blank line: “sshd: ALL”
   4. Press “:wq” and press enter

5. Modify hosts.allow file to permit specific hosts to connect via SSH
   1. “vi /etc/hosts.allow”
   2. Press “i” to allow modification of file contents
   3. Add this to the next available blank line: “sshd: <ip address of permitted host> <ip address of permitted host> …..” (…. = etc, not literal)
   4. Press “:wq” and press enter

Monitoring Views

The next screenshot expands all the branches in the Cross Platform Servers view folder (left) created when you import the X-Plat management packs for Red Hat Linux. Focus (right) is on a 24-hour performance view of Physical Disk target “sda” in the RHEL server.

Reports

When you select a Linux server in the Linux Server State view folder, in the Actions pane you will see a dozen targeted Unix Computer Reports available for on-the-fly generation. Here is the 7-day Memory Performance History (Pages per Sec) report for the RHEL computer:

Distributed Application Possibilities

X-Plat Extensions creates OpsMgr objects for monitored components of discovered Linux computers. This expands the universe of objects available to create Distributed Applications (DAs) to include Linux disks, processors, network interfaces and the like.

• We created a DA that contains two components of classes Windows 2008 Logical Disks and Linux Logical Disks. This DA represents the health of the logical disks of all the web farm members, regardless of their OS.
  
• Relationships are defined as Web Server Farm Logical Disks Uses Linux Logical Disk and Web Server Farm Logical Disks Uses Windows 2008 Logical Disk. See the screenshot of the DA below, open in the Distributed Application Designer:

True Cross-Platform Performance Monitoring

By creating a Performance view that targets the DA we created, we can assess aggregated logical disk performance across Windows and Linux members of a web server farm in a remote data center. Now we have "apples to apples" metrics in the same pane of management glass! See this screenshot of X-Plat in full motion:

Remote Task Execution

A final systems management value-add we find in the current X-Plat release is a small collection of Unix Computer Tasks, which are available in both the Operations console and Web console. These tasks are:

• Run VMStat (a short report on virtual memory statistics, paging block I/O, traps, system and CPU usage),\
• Memory Information (paging and swap data)
• Top 10 CPU Processes

In this screenshot we demonstrate listing the top 10 CPU processes on the Linux server:

Contributors: Thanks to Jacob Linscott, Linux Guru at datacenter provider Softlayer for help on the RHEL versioning; and to Kevin Clark, NOC Manager at managed services provider ClearPointe for the command list that secured the SSH service.

Our friend Walter Chomak (http://wchomak.spaces.live.com/) previously posted September 16th, 2008 that his blog would become less active due to some internal projects he was taking on at Microsoft (see http://wchomak.spaces.live.com/blog/cns!F56EFE25599555EC!1657.entry). Well a week later, he's back! - but at a new location. See http://blogs.technet.com/wchomak/ for his latest postings.

On August 28, this blog passed the 200,000 mark for page views since its creation! Thank you to everyone who has contributed to the blog via articles, comments, or questions. We are glad this blog serves a useful purpose :). Thank you also to everyone who has sent words of thanks and encouragement regarding System Center Operations Manager 2007 Unleashed and Microsoft Operations Manager 2005 Unleashed. Authoring books is a lot of work, but hearing how it has helped so many makes that all worthwhile!

- Kerrie, Cameron, John, and Andy   8/28/08

This is the final of a five part series discussing lessons learned through installing System Center Operations Manager onto a fully Windows 2008 environment (DC, RMS, SQL, and Reporting servers). You can see previous posts in this series at:

• http://ops-mgr.spaces.live.com/blog/cns!3D3B8489FCAA9B51!695.entry  OpsMgr by Example - Server 2008 POC – Part 1 (Domain Controller)
• http://ops-mgr.spaces.live.com/blog/cns!3D3B8489FCAA9B51!710.entry  OpsMgr by Example - Server 2008 POC - Part 2 (Database Server)
• http://ops-mgr.spaces.live.com/blog/cns!3D3B8489FCAA9B51!719.entry  OpsMgr by Example - Server 2008 POC - Part 3 (Root Management Server)
• http://ops-mgr.spaces.live.com/blog/cns!3D3B8489FCAA9B51!728.entry  OpsMgr by Example - Server 2008 POC – Part 4 (Data Warehouse Server) 

At this time, we have successfully completed all of the required pieces of the environment other than the reporting components. This post discusses installing the Reporting Server on Windows Server 2008. The Reporting Server installation is definitely the most error-prone part of the entire OpsMgr installation, be it on Windows 2003 or Windows 2008.

Hotfixes for OpsMgr 20007 – Windows 2008 Servers with Agents

Each of the servers in this configuration needed to have three hotfixes applied to them:

• 951327
• 952664
• 953290

After applying these hotfixes, you will need to reboot the system.

OpsMgr Prerequisites

The .NET Framework 3.0 components are not installed by default. You can install these in the Server Manager by adding the Application Server role.

Prior to installing the reporting components for Operations Manager, follow the steps identified in KB article 938245 (http://support.microsoft.com/kb/938245/) to configure reporting services on Windows Server 2008.

Do not attempt to install the reporting components on a system until you can successfully browse to both http://localhost/reports and http://localhost/reportserver on the reporting server. Configuration is required within the Reporting Services Configuration. Browsing of either of the above URL’s will not work until the Reporting Services Configuration is working, we display a functional example below. During our configurations we needed to create a new Report Server Virtual Directory, configure the Database Setup section, and perform an IISReset of the website to get it to a green state for the first six items, as shown in this screenshot:

A successful browse of http://localhost/reports and http://localhost/reportserver will look like this:

Do not pass go, do not collect $200/We spent several hours trying to resolve issues that were actually related to configurations necessary for SQL 2005 Reporting Services to work on Windows Server 2008. Installing the reporting components on a system that does not already have functional reporting services will just make the situation worse.

The prerequisites for the Reporting Server component include SQL Server 2005 with Reporting Services (which in turn requires the Web role, etc) with SQL 2005 Service Pack 2 applied.

OpsMgr Reporting Server installation

The installation of Operations Manager’s Reporting Server worked just the same as on a Windows 2003 platform (once the prerequisites had been configured correctly).

What’s Next?

• Validate reporting functionality both from the existence of the reports and the ability to get valid data from the reports (the Management Pack ODR report is a simple one to validate this with).
• Deploy Operations Manager agents into the environment.
• Integrate additional management packs.
• Wait on official support for Windows Server 2008 as a platform for all of the Operations Manager components.

Lessons Learned

• It is possible you will get an error "Unable to connect to the remote server" when browsing to http://localhost/reports or http://<servername>/reports. For steps on how to address this, check out http://r0nwilliams.spaces.live.com/blog/cns!62A0019E9E556103!190.entry.
• Without question, getting SQL 2005 Reporting Services working on Windows 2008 was the most complex piece of the installation. Focus on getting a functional SQL Reporting server before continuing (see http://support.microsoft.com/kb/938245/ for details).