At MMS 2008 last May, Microsoft announced their direction to use Operations Manager to manage non-Windows systems (for more information, see Kerrie’s articles “Of Flying Pigs” at http://www.networkworld.com/community/node/27600 and “The Dynamic Datacenter” at http://www.networkworld.com/community/node/27354). This article discusses our experiences testing a beta version of the (Cross Platform) X-Plat software.

The Conventional OpsMgr Gateway Role

Let’s say you have computers at a branch office, in the offices of a partner or customer, or in a datacenter that resides on an untrusted and/or unconnected network. You put an OpsMgr gateway server on that remote network and connect it to your main OpsMgr management group with certificate-based authentication. Cool technology, and you are now monitoring those remote systems from your main location without standing up any new connectivity and potentially increasing the attack surface.

New OpsMgr/X-Plat Gateway Scenario

Before Microsoft introduced the Cross-Platform beta 1 refresh, you could not leverage that secure yet lightweight OpsMgr gateway service for monitoring any Linux computers at your remote location with anything more than a basic SNMP heartbeat. This article reviews this new feature of the Microsoft System Center Operations Manager 2007 Cross Platform Extensions Public Beta 1 Refresh. The software allows OpsMgr gateway servers to discover and fully manage non-Windows computers at remote network locations. This capability opens a new market for Operations Manager with a novel solution to extend management to Linux and other X-Plat systems such as HP-UX or Solaris and even AIX, which were previously out of reach of native System Center tools.

Note: We review here the second released beta for X-Plat. Features and function will change in the released product. Microsoft plans to release X-Plat as part of an update to OpsMgr in 2009.

Demo environment

An OpsMgr management group with Internet-facing gateway servers includes a gateway server at a remote datacenter. All gateway servers trust the same Certificate Authority (CA) and use unique identity certificates issued by the mutually trusted NOC CA for encryption and authentication. There is a Red Hat Enterprise Linux server (RHEL) at the remote site. We want to use the gateway server to monitor the Linux server from the NOC.

Here are the steps we took to discover and manage the RHEL box at the remote datacenter:

1. Install the X-Plat extensions on a selected management server and consoles. The official name of the installable is “System Center Operations Manager Cross-Platform Extensions.” Prerequisites include OpsMgr 2007 SP1 and WS-Management (WS-Man) 1.1.

   Something we liked a lot is that you don’t need to touch the RMS or any high-value management servers to use X-Plat. You only need to install X-Plat extensions on the management server you will run the discovery wizard from.

   There are 32-bit and X64 versions of X-Plat, and also full server and console only versions (a total of four .MSI files to select from). Install the console-only executable on other OpsMgr consoles you will use to monitor the cross-platform systems from.
2. Import the desired X-Plat management packs. The server X-Plat extensions setup defaults to dumping about 14 management packs (for all the operating systems supported by X-Plat) to the %programfiles%\System Center Management Packs folder. You only need to import the libraries and management packs needed to manage your target systems. To manage the RHEL 5 box, we imported these management packs:
   • WS-Management Library
   • Linux Operating System Library
   • Unix View Library
   • Red Hat Operating System Library
   • Red Had Enterprise Linux Server 5 Operating System management packs
3. Run ImportXSLT.cmd on those computers where you installed the X-Plat extensions (management server and consoles). This small step changes how the task output and diagnostic and recovery messages generated by Health Explorer on Unix and Linux computers are displayed. This step has to take place after the X-Plat management packs are imported or you will receive an error.
4. Install the X-Plat extensions on the gateway server. Repeat the installation, similar to the management server. An additional step is that we create a UnixAgents folder in the AgentManagement folder of the gateway server. Extract the UnixAgents.zip that comes with X-Plat to that folder. When the gateway pushes the agent to the Linux server at the datacenter, the Linux bits will come from that folder.
5. Configure the management group Run As Accounts. There is some manual work for the OpsMgr administrator to let the X-Plat extensions on the gateway server know what the credentials are to access the Linux computer.
   1. In the Administration -> Security -> Run As Accounts node of the Operations console, create two new Run As Accounts of the Basic Authentication type. One is a normal user account on the Linux computer and one is a privileged account. For the demo, we used the same root account and password for both Run As accounts. Name the accounts something that identifies them with the gateway server.
   2. In the Security -> Run As Profile node, locate the Unix Privileged Account and associate it with the privileged Run As Account and the target of the gateway server with X-Plat Extensions. Similarly, associate the Unix Action Account Run As Profile with the normal user Run As Account and the target of the gateway server.
   3. This beta release of X-Plat extensions only provides for a single pair of Run As Accounts per management server or per gateway server that performs the discovery and monitoring. To monitor other Linux computers with different sets of credentials requires an additional management server or gateway server for each set of credentials. This is a product limitation we hope is overcome in future releases.

6. Discover and accept the Linux server from the management server. This is just like using the Discovery Wizard from the Administration space of the Operations console, except you launch the X-Plat discovery process from the Overview page of the Cross Platform management pack in the Monitoring space. (In later releases X-Plat discovery is expected to migrate to the Administration space and integrate with Windows computer and network device discovery.)
   • An issue with this beta release of X-Plat is that support for discovery of the most current versions of some Linux distributions isn’t there. In our environment where the demo Linux computer is located, datacenter security polices require Linux distributions be kept current.

     While RHEL 5.2 is the current release, X-Plat only discovers up to RHEL 5.1. (Our hope and assumption is that the RHEL 5.1 agent will work on 5.2.) We expect that with future releases of X-Plat, there will be a community effort to keep X-Plat management packs updated with discovery support for more versions and releases.

     There is a manual install option for the X-Plat agent, which in this case would be as follows (the RPM file can be found in the UnixAgents folder on the gateway server):

     rpm -i scx.1.0.1-151.rhel.x86.rpm

     Another solution that enables use of the automatic discovery and integrated features of the X-Plat management packs is to ‘trick’ the discovery into thinking that the RHEL 5.1 version is installed on the target computer. We used this method, and pushed the version RPM file for 5.1 to the target computer running RHEL 5.2 with this command:

     rpm -i –force redhat-release-5Server-5.1.0.2.i386.rpm

     The –force switch is used since there is a file version downgrade. That RPM file is part of the RHEL 5.1 Server distribution. To later restore the RHEL 5.2 version file, it’s enough to run the command "yum update redhat-release-5server" for the single package, or "yum update" to update any other pieces with patches since it was installed.

   • Perform the discovery from the console of a management server where X-Plat Extensions is installed. You need privileged access to the Linux server to push the agent. If you don’t have a superuser account, you need to provide the root user password. After you specify the IP address and privileged account information for the target, if the computer is discoverable, it will shortly appear as seen in this screenshot of the Select Computers to Manage step in the Unix and Linux Computer Management Wizard:

After approving the discovered Linux computer, the gateway server uses SSH to push the System Center Cross-Platform (SCX) agent to the /tmp folder of the Linux computer. After a few minutes you can query the state of the two services that are started by the SCX agent. See this screen shot of an SSH session from the gateway server to the managed Linux server, confirming that the WS-Man daemon and the CIM server are up:

Managing Red Hat Linux with Operations Manager

Soon after completing these actions, the RHEL computer appeared in the Linux Servers state view of the OpsMgr console. Next, data started appearing in the memory and processor-related views. Some hours later, the disk and network views were populated. We received some alerts regarding invalid SSH authentication attempts, and we immediately had a solid feeling about our ability to really manage Linux boxes from Windows with OpsMgr.

Here is a screenshot of an alert related to security of the SSH services on the RHEL box:

An Internet-facing web server is going to get a lot of intrusion attempts against any open service. We secured the SSH services on the RHEL box with these host rules (and the alerts stopped!):

1. Edit /etc/ssh/ssh_config
   1. “vi /etc/ssh/ssh_config”
   2. Press “i” to allow modification of file contents

2. Modify line to restrict SSH protocol to version 2
   1. Locate line “# Protocol 2,1”
   2. Remove “#” from beginning of line, and “,1” from end of line.

3. Save the file
   Press “:wq” and press enter

4. Modify hosts.deny file to deny all hosts access to SSH
   1. “Vi /etc/hosts.deny”
   2. Press “i” to allow modification of file contents
   3. Add this to the next available blank line: “sshd: ALL”
   4. Press “:wq” and press enter

5. Modify hosts.allow file to permit specific hosts to connect via SSH
   1. “vi /etc/hosts.allow”
   2. Press “i” to allow modification of file contents
   3. Add this to the next available blank line: “sshd: <ip address of permitted host> <ip address of permitted host> …..” (…. = etc, not literal)
   4. Press “:wq” and press enter

Monitoring Views

The next screenshot expands all the branches in the Cross Platform Servers view folder (left) created when you import the X-Plat management packs for Red Hat Linux. Focus (right) is on a 24-hour performance view of Physical Disk target “sda” in the RHEL server.

Reports

When you select a Linux server in the Linux Server State view folder, in the Actions pane you will see a dozen targeted Unix Computer Reports available for on-the-fly generation. Here is the 7-day Memory Performance History (Pages per Sec) report for the RHEL computer:

Distributed Application Possibilities

X-Plat Extensions creates OpsMgr objects for monitored components of discovered Linux computers. This expands the universe of objects available to create Distributed Applications (DAs) to include Linux disks, processors, network interfaces and the like.

• We created a DA that contains two components of classes Windows 2008 Logical Disks and Linux Logical Disks. This DA represents the health of the logical disks of all the web farm members, regardless of their OS.
  
• Relationships are defined as Web Server Farm Logical Disks Uses Linux Logical Disk and Web Server Farm Logical Disks Uses Windows 2008 Logical Disk. See the screenshot of the DA below, open in the Distributed Application Designer:

True Cross-Platform Performance Monitoring

By creating a Performance view that targets the DA we created, we can assess aggregated logical disk performance across Windows and Linux members of a web server farm in a remote data center. Now we have "apples to apples" metrics in the same pane of management glass! See this screenshot of X-Plat in full motion:

Remote Task Execution

A final systems management value-add we find in the current X-Plat release is a small collection of Unix Computer Tasks, which are available in both the Operations console and Web console. These tasks are:

• Run VMStat (a short report on virtual memory statistics, paging block I/O, traps, system and CPU usage),\
• Memory Information (paging and swap data)
• Top 10 CPU Processes

In this screenshot we demonstrate listing the top 10 CPU processes on the Linux server:

Contributors: Thanks to Jacob Linscott, Linux Guru at datacenter provider Softlayer for help on the RHEL versioning; and to Kevin Clark, NOC Manager at managed services provider ClearPointe for the command list that secured the SSH service.