IBM System p5 595

(Frame-mount)
The IBM System p5 595 server uses fifth-generation 64-bit IBM POWER5 technology in up to 64-core symmetric multiprocessing (SMP) configurations with IBM Advanced POWER Virtualization and offers the performance, flexibility, scalability, manageability and security features needed for the consolidation of mission-critical AIX 5L and Linux applications on a single system to save on hardware, software, energy and space costs.

Processor cores		16 to 64 IBM POWER5+
Clock rates (Min/Max)		2.1 / 2.3 GHz
System memory (Std/Max)		8GB / 2TB
Internal storage (Std/Max)		146.8GB / 28.1TB (using optional I/O drawers)
Performance (rPerf range)*		80.96 to 393.55

IBM System p5 570

(Rack-mount)
Easily scale from 2- to 16-cores with the IBM System p5™ 570. Unique IBM modular SMP architecture lets you add more powerful IBM POWER5+™ processing capability exactly when needed.

Processor cores		2, 4, 8, 12, 16 POWER5+
Clock rates (Min/Max)		1.9 GHz / 2.2 GHz
System memory (Std/Max)		2GB / 512GB
Internal disk storage (Std/Max)		73.4GB / 79.2TB (with optional I/O drawers)
Performance (rPerf range)*		12.27 / 95.96

IBM System p 570

(Rack-mount)
Easily scale from 2- to 16-cores with the IBM System p™ 570. Unique IBM modular SMP architecture lets you add more powerful IBM POWER6™ 3.5, 4.2 or 4.7 GHz processing capability exactly when needed. Innovative RAS features and leadership virtualization capabilities make the p570 well suited as a mid-range application or database server, or for server consolidation. And the flexibility to use both the leading-edge AIX® and Linux® operating systems broadens the application offerings available and increases the ways clients can manage growth, complexity and risk.

Processor cores		2, 4, 8, 12, 16 POWER6
Clock rates (Min/Max)		3.5 GHz / 4.7 GHz
System memory (Std/Max)		2 GB / 768 GB
Internal storage (Std/Max)		73.4 GB / 79.2 TB (with optional I/O drawers)
Performance (rPerf range)*		15.85 / 134.35

HACMP Scripts

The following scripts are supplied with the HACMP software.

Startup and Shutdown Scripts

Each of the following scripts is involved in starting and stopping the HACMP software.

/usr/es/sbin/cluster/utilities/clstart

The /usr/es/sbin/cluster/utilities/clstart script, which is called by the /usr/es/sbin/cluster/utilities/rc.cluster script, invokes the AIX System Resource Controller (SRC) facility to start the cluster daemons. The clstart script starts HACMP with the options currently specified on the Start Cluster Services SMIT panel.

There is a corresponding C-SPOC version of this script that starts cluster services on each cluster node. The /usr/es/sbin/cluster/sbin/cl_clstart script calls the HACMP clstart script.

At cluster startup, clstart looks for the file /etc/rc.shutdown. The system file /etc/rc.shutdown can be configured to run user specified commands during processing of the AIX /usr/sbin/shutdown command.

Newer versions of the AIX /usr/sbin/shutdown command automatically call HACMP's /usr/es/sbin/cluster/etc/rc.shutdown, and subsequently call the existing /etc/rc.shutdown (if it exists).

Older versions of the AIX /usr/sbin/shutdown command do not have this capability. In this case, HACMP manipulates the /etc/rc.shutdown script, so that both /usr/es/sbin/cluster/etc/rc.shutdown and the existing /etc/rc.shutdown (if it exists) are run. Since HACMP needs to stop cluster services before the shutdown command is run, on cluster startup, rc.cluster replaces any user supplied /etc/rc.shutdown file with the HACMP version. The user version is saved and is called by the HACMP version prior to its own processing. When cluster services are stopped, the clstop command restores the user's version of rc.shutdown.

/usr/es/sbin/cluster/utilities/clstop

The /usr/es/sbin/cluster/utilities/clstop script, which is called by the SMIT Stop Cluster Services panel, invokes the SRC facility to stop the cluster daemons with the options specified on the Stop Cluster Services panel.

There is a corresponding C-SPOC version of this script that stops cluster services on each cluster node. The /usr/es/sbin/cluster/sbin/cl_clstop script calls the HACMP clstop script.

Also see the notes on /etc/rc.shutdown in the section on clstart above for more information.

/usr/es/sbin/cluster/utilities/clexit.rc

If the SRC detects that the clstrmgr daemon has exited abnormally, it executes the /usr/es/sbin/cluster/utilities/clexit.rc script to halt the system. If the SRC detects that any other HACMP daemon has exited abnormally, it executes the clexit.rc script to stop these processes, but does not halt the node.

You can change the default behavior of the clexit.rc script by configuring the /usr/es/sbin/cluster/etc/hacmp.term file to be called when the HACMP cluster services terminate abnormally. You can customize the hacmp.term file so that HACMP will take actions specific to your installation. See the file for full information.

/usr/es/sbin/cluster/etc/rc.cluster

If the Start at system restart option is chosen on the Start Cluster Services SMIT panel, the /usr/es/sbin/cluster/etc/rc.cluster script is called by the /etc/inittab file to start HACMP. The /usr/es/sbin/cluster/etc/rc.cluster script does some necessary initialization and then calls the usr/es/sbin/cluster/utilities/clstart script to start HACMP.

The /usr/es/sbin/cluster/etc/rc.cluster script is also used to start the clinfo daemon on a client.

There is a corresponding C-SPOC version of this script that starts cluster services on each cluster node. The /usr/es/sbin/cluster/sbin/cl_rc.cluster script calls the HACMP rc.cluster script.

See the man page for rc.cluster for more information.

/etc/rc.net

The /etc/rc.net script is called by the /usr/es/sbin/cluster/etc/rc.cluster script to configure and start the TCP/IP interfaces and to set the required network options. The /etc/rc.net script is used in the boot process to retrieve interface information from the ODM and to configure all defined interfaces. If IP address takeover is configured, the /etc/rc.net script is called from the /usr/es/sbin/cluster/etc/rc.cluster script at cluster startup instead of during the boot process.

AIX Files Modified by HACMP

The following AIX files are modified to support HACMP. They are not distributed with HACMP.

/etc/hosts

The cluster event scripts use the /etc/hosts file for name resolution. All cluster node IP interfaces must be added to this file on each node.

If you delete service IP labels from the cluster configuration using SMIT, we recommend that you remove them from /etc/hosts too. This reduces the possibility of having conflicting entries if the labels are reused with different addresses in a future configuration.

Note that DNS and NIS are disabled during HACMP-related name resolution. This is why HACMP IP addresses must be maintained locally.

HACMP may modify this file to ensure that all nodes have the necessary information in their /etc/hosts file, for proper HACMP operations.

/etc/inittab

During installation, the following entry is made to the /etc/inittab file to start the Cluster Communication Daemon at boot:

clcomdES:2:once:startsrc -s clcomdES >dev/console 2>&1  

The /etc/inittab file is modified in each of the following cases:

• HACMP is configured for IP address takeover

• The Start at System Restart option is chosen on the SMIT Start Cluster Services panel

• Concurrent Logical Volume Manager (CLVM) is installed with HACMP 5.2.

Modifications to the /etc/inittab File due to IP Address Takeover

The following entry is added to the /etc/inittab file for HACMP network startup with IP address takeover:

harc:2:wait:/usr/es/sbin/cluster/etc/harc.net # HACMP network startup  

When IP address takeover is enabled, the system edits /etc/inittab to change the rc.tcpip and inet-dependent entries from run level “2” (the default multi-user level) to run level “a”. Entries that have run level “a” are processed only when the telinit command is executed specifying that specific run level.

Modifications to the /etc/inittab File due to System Boot

The /etc/inittab file is used by the init process to control the startup of processes at boot time. The following line is added to /etc/inittab during HACMP install:

clcomdES:2:once:startsrc -s clcomdES >/dev/console 2>&1  

This entry starts the Cluster Communications Daemon (clcomd) at boot.

The following entry is added to the /etc/inittab file if the Start at system restart option is chosen on the SMIT Start Cluster Services panel:

hacmp:2:wait:/usr/sbin/etc/rc.cluster -boot> /dev/console 2>&1 # Bring
up Cluster  

When the system boots, the /etc/inittab file calls the /usr/es/sbin/cluster/etc/rc.cluster script to start HACMP.

Because the inet daemons must not be started until after HACMP-controlled interfaces have swapped to their service address, HACMP also adds the following entry to the end of the /etc/inittab file to indicate that /etc/inittab processing has completed:

clinit:a:wait:/bin/touch /usr/es/sbin/cluster/.telinit  
#HACMP for AIX These must be the last entry in run level “a” in inittab! 
pst_clinit:a:wait/bin/echo Created /usr/es/sbin/cluster/ .telinit >
/dev/console 
#HACMP for AIX These must be the last entry in run level “a” in inittab!  

See Chapter 8: Starting and Stopping Cluster Services, for more information about the files involved in starting and stopping HACMP.

/etc/rc.net

The /etc/rc.net file is called by cfgmgr to configure and start TCP/IP during the boot process. It sets hostname, default gateway and static routes. The following entry is added at the beginning of the file for a node on which IP address takeover is enabled:

# HACMP for AIX 
# HACMP for AIX These lines added by HACMP for AIX software 
[ "$1" = "-boot" ] && shift || { ifconfig 1o0 127.0.0.1 up; exit 0; }
#HACMP for AIX 
# HACMP for AIX  

The entry prevents cfgmgr from reconfiguring boot and service addresses while HACMP is running.

/etc/services

The /etc/services file defines the sockets and protocols used for network services on a system. The ports and protocols used by the HACMP components are defined here.

#clinfo_deadman             6176/tcp 
#clm_keepalive              6255/udp 
#clm_pts                    6200/tcp 
#clsmuxpd                   6270/tcp 
#clm_lkm                    6150/tcp 
#clm_smux                   6175/tcp 
#godm                       6177/tcp 
#topsvcs    6178/udp 
#grpsvcs    6179/udp 
#emsvcs    6180/udp 
#clver    6190/tcp 
#clcomd    6191/tcp  

/etc/snmpd.conf

Note: The version of snmpd.conf depends on whether you are using AIX 5L v.5.1 or v.5.2. The default version for v.5.2. is snmpdv3.conf.

The SNMP daemon reads the /etc/snmpd.conf configuration file when it starts up and when a refresh or kill -1 signal is issued. This file specifies the community names and associated access privileges and views, hosts for trap notification, logging attributes, snmpd-specific parameter configurations, and SMUX configurations for the snmpd. The HACMP installation process adds the clsmuxpd password to this file. The following entry is added to the end of the file, to include the HACMP MIB managed by the clsmuxpd:

smux  1.3.6.1.4.1.2.3.1.2.1.5  "clsmuxpd_password" # HACMP clsmuxpd  

HACMP supports SNMP Community Names other than “public.” If the default SNMP Community Name has been changed in /etc/snmpd.conf to something different from the default of “public” HACMP will function correctly. The SNMP Community Name used by HACMP is the first name found that is not “private” or “system” using the lssrc -ls snmpd command.

The Clinfo service also gets the SNMP Community Name in the same manner. The Clinfo service supports the -c option for specifying SNMP Community Name but its use is not required. The use of the -c option is considered a security risk because doing a ps command could find the SNMP Community Name. If it is important to keep the SNMP Community Name protected, change permissions on /tmp/hacmp.out, /etc/snmpd.conf, /smit.log and /usr/tmp/snmpd.log to not be world readable.

/etc/snmpd.peers

The /etc/snmpd.peers file configures snmpd SMUX peers. The HACMP install process adds the following entry to include the clsmuxpd:

clsmuxpd 1.3.6.1.4.1.2.3.1.2.1.5  "clsmuxpd_password" # HACMP clsmuxpd  

/etc/syslog.conf

The /etc/syslog.conf file is used to control output of the syslogd daemon, which logs system messages. During the install process HACMP adds entries to this file that direct the output from HACMP-related problems to certain files.

# example: 
# "mail messages, at debug or higher, go to Log file. File must exist." 
# "all facilities, at debug and higher, go to console" 
# "all facilities, at crit or higher, go to all users" 
#  mail.debug           /usr/spool/mqueue/syslog 
#  *.debug              /dev/console 
#  *.crit                       * 
# HACMP Critical Messages from HACMP 
local0.crit /dev/console 
# HACMP Informational Messages from HACMP 
local0.info /usr/es/adm/cluster.log 
# HACMP Messages from Cluster Scripts 
user.notice /usr/es/adm/cluster.log  

The /etc/syslog.conf file should be identical on all cluster nodes.

/etc/trcfmt

The /etc/trcfmt file is the template file for the system trace logging and report utility, trcrpt. The install process adds HACMP tracing to the trace format file. HACMP tracing applies to the following daemons: clstrmgr, clinfo, and clsmuxpd.

/var/spool/cron/crontab/root

The /var/spool/cron/crontab/root file contains commands needed for basic system control. The install process adds HACMP logfile rotation to the file.

Monitoring the Cluster

By design, failures of components in the cluster are handled automatically. But you need to be aware of all such events. Chapter 9: Monitoring an HACMP Cluster, describes various tools you can use to check the status of an HACMP cluster, the nodes, networks, and resource groups within that cluster, and the daemons that run on the nodes.

The HACMP software incudes the Cluster Information Program (Clinfo), an SNMP-based monitor. The HACMP for AIX software provides the HACMP for AIX MIB, associated with and maintained by the HACMP for AIX management agent, the Cluster SMUX peer daemon (clsmuxpd). Clinfo retrieves this information from the HACMP for AIX MIB through the clsmuxpd.

Clinfo can run on cluster nodes and on HACMP for AIX client machines. It makes information about the state of an HACMP cluster and its components available to clients and applications via an application programming interface (API). Clinfo and its associated APIs enable a developer to write applications that recognize and respond to changes within a cluster.

The Clinfo program, the HACMP MIB, and the API are documented in Programming Client Applications (part of the HACMP for AIX documentation set).

Although the combination of HACMP and the high availability features built into the AIX system keeps single points of failure to a minimum, there are still failures that, although detected, can cause other problems. See chapter 15 in the Planning and Installation Guide, for suggestions on customizing error notification for various problems not handled by the HACMP events.

Maintaining an HACMP Cluster

The following maintenance tasks for an HACMP system are described in detail in subsequent chapters:

• Starting and stopping cluster services

• Managing shared LVM and CLVM components

• Managing the cluster topology

• Managing cluster resources

• Managing cluster resource groups

• Managing users, groups and security in a cluster

• Saving and restoring an HACMP configuration

Configuration Tasks of HACMP

The easiest way to do the initial configuration of the cluster is to use the Online Planning Worksheet program (OLPW). See Chapter 16 in the Planning and Installation Guide for instructions.

You can also use the Two-Node Cluster Configuration Assistant to configure a basic two-node HACMP cluster. You supply the minimum information required to define a cluster, and HACMP discovers the remainder of the information for you. See Chapter 11 in the Planning and Installation Guide for instructions.

If you have a snapshot of the HACMP cluster configuration taken before an upgrade, you can use the Cluster Snapshot utility to do the initial configuration. See Chapter 17: Saving and Restoring Cluster Configurations.

The HACMP configuration tasks are described in detail in subsequent chapters. You can choose to use either the standard or the extended path for the initial configuration, though the standard configuration path is recommended. These are the major steps in the process:

• Configure the cluster topology, then HACMP resources and resource groups using the standard configuration path

or

Configure the cluster topology, then HACMP resources and resource groups using the extended configuration path

• (Optional) Configure pre- and post-events, pager notification, HACMP File collection, auto-corrective verification, other optional settings

• Verify and synchronize the HACMP configuration

• Test the cluster.

Planning for a CSM for AIX management server

The following requirements apply to the management server:

• The machine you use for your management server must have a CD-ROM drive or equivalent (for example, a DVD-ROM or DVD-RAM drive).
• With CSM for AIX the management server must be a workstation, capable of running AIX 5L Version 5.2 or higher.
• A sufficient number of network adapters are required to accommodate the required management, cluster and public VLANs. For information on the network options and requirements, see the Hardware and network requirements.

Memory and disk space

On the management server, a minimum of 512 MB of memory and 250 MB of disk space is required for installing CSM. When using an AIX CSM management server to manage and install Linux nodes, an additional 100MB of space is required in the /var directory . Additional disk space on the management server is required to support installation of the AIX operating system and CSM on the managed nodes. (Typically, for every version of AIX that you have installed, you need at least 2.0 GB of storage.)

Network requirements

IBM suggests creating one or more virtual local area network (VLAN) for the CSM management server and hardware control points, and one or more separate VLAN for the CSM management server and cluster nodes. Although cluster hardware control points and nodes can be on the same VLAN, limiting access to the management VLAN reduces security exposure for IP traffic on the management VLAN and access to hardware control points. For more information about network requirements, see Hardware and network requirements. For more information on VLANs, see Virtual LANs (VLANs).

Your pSeries nodes do not have to be in the same subnet as your install server. You can install them across one or more gateways. If your install server is located across one or more gateways from your pSeries Linux nodes, then you will need to set up your networking as follows:

• Add a stanza to the /etc/dhcpd.conf file on the install server that contains the node's subnet.
• On the gateway machine, add a permanent arp entry for each node.
• On the gateway machine, configure and enable the dhcp-relay daemon.

If you have more than one gateway between the install server and the node, the gateway to set up is the one that is "closest" to the node. For instance, if you have the install server connected to gateway A, Gateway A connected to gateway B, and gateway B connected to the nodes, you would add the arp entries and the dhcp-relay daemon to gateway B.

Planning for CSM for AIX Install Servers

Install servers are CSM nodes that provide basic operating system installation services and NFS file services during CSM node installs. An AIX install server is a NIM master that is used to install AIX on CSM nodes. Your AIX management server can also be your AIX install server. However, you may need to set up a separate AIX install server if you want to install AIX nodes at a higher operating system level than exists on the CSM for AIX management server. NIM requires that the NIM master be at the same or higher level of AIX as the level to be installed on the NIM clients. You will also need an AIX install server if you have a Linux management server and you want to install operating systems on your AIX nodes in an heterogeneous cluster.

The requirements for a CSM for AIX install server, which are similar to those for a CSM for AIX management server, are the following:

• The install server must be capable of running AIX 5L Version 5.2 or higher, and CSM 1.4.1 or higher
• A sufficient number of network adapters are required to accommodate the network installation of all nodes to be served by this install server. See Network requirements.
• The nodes to be served must be network connected to this install server either directly through a local subnet or indirectly through a gateway.
• A minimum of 512 MB of memory and 250 MB of disk space is required for CSM. Additional disk space is required to support the installation of the AIX operating system and CSM on the nodes to be served by this install server. (For every version of AIX that you want to install you will need at least 2.0 GB of storage.)

Limitations & Considerations of LPAR management server

If you use an LPAR as a CSM management server, consider the following limitations. Note that these limitations apply only if the CSM management server is an LPAR and not a separate physical machine:

1. The CSM management server can be brought down inadvertently by a user on the HMC who deactivates the LPAR. Even if a user does not have access to the CSM management server, a user with access to the HMC can power off the management server or move resources such as CPU or I/O from the LPAR.
2. If the firmware needs to be upgraded, the LPAR management server might also go down when the system is quiesced. However, bringing the CEC back up returns the system to normal.
3. There is no direct manual hardware control of the CSM management server. You must use the HMC for power control of the management server.
4. An LPAR management server cannot have an attached display. This limitation can affect the performance of your CSM GUIs.
5. In machines such as the p690, you can assign a CD-ROM drive to one LPAR on the CEC, (the management server LPAR).
6. Do not define an LPAR management server as a managed node.
7. A cluster that is installed and configured can still function even if the management server goes down. For example, cluster applications can continue to run, and nodes in the cluster can be rebooted. However, tasks including monitoring, automated responses for detecting problems in the cluster, and scheduled file and software updates cannot occur while the management server is down.
8. If the cluster contains a 9076 SP Node or 7026 server, you cannot define an LPAR management server for the cluster.

IBM System p5 570

* Up to 16-core scalability with modular architecture and leadership POWER5+ technology

* IBM Advanced POWER™ Virtualization features increase system utilization and reduce the number of overall systems required

* Capacity on Demand features enable quick response to spikes in processing requirements

The IBM System p5 570 mid-range server is a powerful 19-inch rack mount system that can be used for database and application serving, as well as server consolidation. IBM’s modular symmetric multiprocessor (SMP) architecture means you can start with a 2-core system and easily add additional building blocks when needed for more processing power (up to 16-cores) I/O and storage capacity. The p5-570 includes IBM mainframe-inspired reliability, availability and serviceability features.

The System p5 570 server is designed to be a cost-effective, flexible server for the on demand environment. Innovative virtualization technologies and Capacity on Demand (CoD) options help increase the responsiveness of the server to variable computing demands. These features also help increase the systems utilization of processors and system components allowing businesses to meet their computing requirements with a smaller system. By combining IBM’s most advanced leading-edge technology for enterprise-class performance and flexible adaptation to changing market conditions, the p5-570 can deliver the key capabilities medium-sized companies need to survive in today’s highly competitive world.

Specifically, the System p5 570 server provides:

Common features Hardware summary

* 19-inch rack-mount packaging
* 2- to 16-core SMP design with unique building block architecture
* 64-bit 1.9 or 2.2 GHz POWER5+ processor cores
* Mainframe-inspired RAS features
* Dynamic LPAR support
* Advanced POWER Virtualization1 (option)
o IBM Micro-Partitioning™ (up to 160 micro- partitions)
o Shared processor pool
o Virtual I/O Server
o Partition Load Manager (IBM AIX 5L™ only)
* Up to 20 optional I/O drawers
* IBM HACMP™ software support for near continuous operation*
* Supported by AIX 5L (V5.2 or later) and Linux® distributions from Red Hat (RHEL AS 4 or later) and SUSE Linux (SLES 9 or later) operating systems
* System Cluster 1600 support with Cluster Systems Management software*



* 4U 19-inch rack-mount packaging
* One to four building blocks
* Two, four, eight, 12, 16 1.9 or 2.2 GHz 64-bit POWER5+ processor cores
* L2 cache: 1.9MB to 15.2MB (2- to 16-core)
* L3 cache: 36MB to 288MB (2- to 16-core)
* 1.9 GHz systems: 2GB to 256GB of 533 MHz DDR2 memory; 2.2 GHz systems: 2GB to 256GB of 533 MHz or 32GB to 512GB of 400 MHz DDR2 memory
* Six hot-plug PCI-X adapter slots per building block
* Six hot-swappable disk bays per building block provide up to 7.2TB of internal disk storage
* Optional I/O drawers may add up to an additional 139 PCI-X slots (for a maximum of 163) and 240 disk bays (72TB additional)
* Dual channel Ultra320 SCSI controller per building block (internal; RAID optional)
* One integrated 2-port 10/100/1000 Ethernet per building block
* Optional 2 Gigabit Fibre Channel, 10 Gigabit Ethernet and 4x GX adapters
* One 2-port USB per building block
* Two HMC, two system ports
* Two hot-plug media bays per building block

IBM System p 570 with POWER 6

* Advanced IBM POWER6™ processor cores for enhanced performance and reliability

* Building block architecture delivers flexible scalability and modular growth

* Advanced virtualization features facilitate highly efficient systems utilization

* Enhanced RAS features enable improved application availability

The IBM POWER6 processor-based System p™ 570 mid-range server delivers outstanding price/performance, mainframe-inspired reliability and availability features, flexible capacity upgrades and innovative virtualization technologies. This powerful 19-inch rack-mount system, which can handle up to 16 POWER6 cores, can be used for database and application serving, as well as server consolidation. The modular p570 is designed to continue the tradition of its predecessor, the IBM POWER5+™ processor-based System p5™ 570 server, for resource optimization, secure and dependable performance and the flexibility to change with business needs. Clients have the ability to upgrade their current p5-570 servers and know that their investment in IBM Power Architecture™ technology has again been rewarded.

The p570 is the first server designed with POWER6 processors, resulting in performance and price/performance advantages while ushering in a new era in the virtualization and availability of UNIX® and Linux® data centers. POWER6 processors can run 64-bit applications, while concurrently supporting 32-bit applications to enhance flexibility. They feature simultaneous multithreading,1 allowing two application “threads” to be run at the same time, which can significantly reduce the time to complete tasks.

The p570 system is more than an evolution of technology wrapped into a familiar package; it is the result of “thinking outside the box.” IBM’s modular symmetric multiprocessor (SMP) architecture means that the system is constructed using 4-core building blocks. This design allows clients to start with what they need and grow by adding additional building blocks, all without disruption to the base system.2 Optional Capacity on Demand features allow the activation of dormant processor power for times as short as one minute. Clients may start small and grow with systems designed for continuous application availability.

Specifically, the System p 570 server provides:

Common features Hardware summary

* 19-inch rack-mount packaging
* 2- to 16-core SMP design with building block architecture
* 64-bit 3.5, 4.2 or 4.7 GHz POWER6 processor cores
* Mainframe-inspired RAS features
* Dynamic LPAR support
* Advanced POWER Virtualization1 (option)
o IBM Micro-Partitioning™ (up to 160 micro-partitions)
o Shared processor pool
o Virtual I/O Server
o Partition Mobility2
* Up to 32 optional I/O drawers
* IBM HACMP™ software support for near continuous operation*
* Supported by AIX 5L (V5.2 or later) and Linux® distributions from Red Hat (RHEL 4 Update 5 or later) and SUSE Linux (SLES 10 SP1 or later) operating systems



* 4U 19-inch rack-mount packaging
* One to four building blocks
* Two, four, eight, 12 or 16 3.5 GHz, 4.2 GHz or 4.7 GHz 64-bit POWER6 processor cores
* L2 cache: 8 MB to 64 MB (2- to 16-core)
* L3 cache: 32 MB to 256 MB (2- to 16-core)
* 2 GB to 192 GB of 667 MHz buffered DDR2 or 16 GB to 384 GB of 533 MHz buffered DDR2 or 32 GB to 768 GB of 400 MHz buffered DDR2 memory3
* Four hot-plug, blind-swap PCI Express 8x and two hot-plug, blind-swap PCI-X DDR adapter slots per building block
* Six hot-swappable SAS disk bays per building block provide up to 7.2 TB of internal disk storage
* Optional I/O drawers may add up to an additional 188 PCI-X slots and up to 240 disk bays (72 TB additional)4
* One SAS disk controller per building block (internal)
* One integrated dual-port Gigabit Ethernet per building block standard; One quad-port Gigabit Ethernet per building block available as optional upgrade; One dual-port 10 Gigabit Ethernet per building block available as optional upgrade
* Two GX I/O expansion adapter slots
* One dual-port USB per building block
* Two HMC ports (maximum of two), two SPCN ports per building block
* One optional hot-plug media bay per building block
* Redundant service processor for multiple building block systems2

IBM System Cluster 1350

Reduced time to deployment

IBM HPC clustering offers significant price/performance advantages for many high-performance workloads by harnessing the advantages of low cost servers plus innovative, easily available open source software.

Today, some businesses are building their own Linux and Microsoft clusters using commodity hardware, standard interconnects and networking technology, open source software, and in-house or third-party applications. Despite the apparent cost advantages offered by these systems, the expense and complexity of assembling, integrating, testing and managing these clusters from disparate, piece-part components often outweigh any benefits gained.

IBM has designed the IBM System Cluster 1350 to help address these challenges. Now clients can benefit from IBM’s extensive experience with HPC to help minimize this complexity and risk. Using advanced Intel® Xeon®, AMD Opteron™, and IBM PowerPC® processor-based server nodes, proven cluster management software and optional high-speed interconnects, the Cluster 1350 offers the best of IBM and third-party technology. As a result, clients can speed up installation of an HPC cluster, simplify its management, and reduce mean time to payback.

The Cluster 1350 is designed to be an ideal solution for a broad range of application environments, including industrial design and manufacturing, financial services, life sciences, government and education. These environments typically require excellent price/performance for handling high performance computing (HPC) and business performance computing (BPC) workloads. It is also an excellent choice for applications that require horizontal scaling capabilities, such as Web serving and collaboration.

Common features		Hardware summary
Rack-optimized Intel Xeon dual-core and quad-core and AMD Opteron processor-based servers Intel Xeon, AMD and PowerPC processor-based blades Optional high capacity IBM System Storage™ DS3200, DS3400, DS4700, DS4800 and EXP3000 Storage Servers and IBM System Storage EXP 810 Storage Expansion Industry-standard Gigabit Ethernet cluster interconnect Optional high-performance Myrinet-2000 and Myricom 10g cluster interconnect Optional Cisco, Voltaire, Force10 and PathScale InfiniBand cluster interconnects Clearspeed Floating Point Accelerator Terminal server and KVM switch Space-saving flat panel monitor and keyboard Runs with RHEL 4 or SLES 10 Linux operating systems or Windows Compute Cluster Server Robust cluster systems management and scalable parallel file system software Hardware installed and integrated in 25U or 42U Enterprise racks Scales up to 1,024 cluster nodes (larger systems and additional configurations available—contact your IBM representative or IBM Business Partner) Optional Linux cluster installation and support services from IBM Global Services or an authorized partner or distributor Clients must obtain the version of the Linux operating system specified by IBM from IBM, the Linux Distributor or an authorized reseller

IBM System Cluster 1600

IBM System Cluster 1600 systems are comprised of IBM POWER5™ and POWER5+™ symmetric multiprocessing (SMP) servers running AIX 5L™ or Linux®. Cluster 1600 is a highly scalable cluster solution for large-scale computational modeling and analysis, large databases and business intelligence applications and cost-effective datacenter, server and workload consolidation. Cluster 1600 systems can be deployed on Ethernet networks, InfiniBand networks, or with the IBM High Performance Switch and are typically managed with Cluster Systems Management (CSM) software, a comprehensive tool designed specifically to streamline initial deployment and ongoing management of cluster systems.

Common features

· Highly scalable AIX 5L or Linux cluster solutions for large-scale computational modeling, large databases and cost-effective data center, server and workload consolidation

· Cluster Systems Management (CSM) software for comprehensive, flexible deployment and ongoing management

· Cluster interconnect options: industry standard 1/10Gb Ethernet (AIX 5L or Linux), IBM High Performance Switch (AIX 5L and CSM) SP Switch2 (AIX 5L and PSSP); 4x/12x InfiniBand (AIX 5L or SLES 9); or Myrinet (Linux)

· Operating system options: AIX 5L Version 5.2 or 5.3, SUSE Linux Enterprise Server 8 or 9, Red Hat Enterprise Linux 4

· Complete software suite for creating, tuning and running parallel applications: Engineering & Scientific Subroutine Library (ESSL), Parallel ESSL, Parallel Environment, XL Fortran, VisualAge C++

· High-performance, high availability, highly scalable cluster file system General Parallel File System (GPFS)

· Job scheduling software to optimize resource utilization and throughput: LoadLeveler®

· High availability software for continuous access to data and applications: High Availability Cluster Multiprocessing (HACMP™)

Hardware summary

· Mix and match IBM POWER5 and POWER5+ servers:
· IBM System p5™ 595, 590, 575, 570, 560Q, 550Q, 550, 520Q, 520, 510Q, 510, 505Q and 505

· IBM eServer™ p5 595, 590, 575, 570, 550, 520, and 510



· Up to 128 servers or LPARs (AIX 5L or Linux operating system images) per cluster depending on hardware; higher scalability by special order

File system types

The following types of file systems are supported on an AIX 5L Version 5.3:
Journaled file system
This type of file system is named journaled because the
system uses journaling techniques to maintain the
integrity of control structures. Each journaled file system
must reside on a distinct jfs logical volume. Therefore, the
file system size will be a multiple of the size of a logical
partition.
Enhanced journaled file system
This is the enhanced version of the initial journalized file
system. It uses extent based allocation to allow higher
performance, larger file systems, and a larger file size.
Each enhanced journaled file system must reside on a
distinct jfs2 logical volume. When the operating system is
installed using the default options, it creates JFS2 file
systems.
Network file system The network file system (NFS) is a distributed file system
that allows users to access files and directories located on
remote computers and use those files and directories as
though they are local.
CD-ROM file system The CD-ROM file system (CDRFS) is a file system type
that allows you to access the contents of a CD-ROM
through the normal file system interfaces

Storage management concepts

The fundamental concepts used by LVM are physical volumes, volume groups,
physical partitions, logical volumes, logical partitions, file systems, and raw
devices. Some of their characteristics are presented as follows:
 Each individual disk drive is a named physical volume (PV) and has a name
such as hdisk0 or hdisk1.
 One or more PVs can make up a volume group (VG). A physical volume can
belong to a maximum of one VG.
 You cannot assign a fraction of a PV to one VG. A physical volume is
assigned entirely to a volume group.
 Physical volumes can be assigned to the same volume group even though
they are of different types, such as SCSI or SSA.
 Storage space from physical volumes is divided into physical partitions (PPs).
The size of the physical partitions is identical on all disks belonging to the
same VG.
 Within each volume group, one or more logical volumes (LVs) can be defined.
Data stored on logical volumes appears to be contiguous from the user point
of view, but can be spread on different physical volumes from the same
volume group.
 Logical volumes consist of one or more logical partitions (LPs). Each logical
partition has at least one corresponding physical partition. A logical partition
and a physical partition always have the same size. You can have up to three
copies of the data located on different physical partitions. Usually, physical
partitions storing identical data are located on different physical disks for
redundancy purposes.
 Data from a logical volume can be stored in an organized manner, having the
form of files located in directories. This structured and hierarchical form of
organization is named a file system.
 Data from a logical volume can also be seen as a sequential string of bytes.
This type of logical volumes are named raw logical volumes. It is the
responsibility of the application that uses this data to access and interpret it
correctly.
 The volume group descriptor area (VGDA) is an area on the disk that contains
information pertinent to the volume group that the physical volume belongs to.
It also includes information about the properties and status of all physical and
logical volumes that are part of the volume group. The information from VGDA
is used and updated by LVM commands. There is at least one VGDA per
physical volume. Information from VGDAs of all disks that are part of the
same volume group must be identical. The VGDA internal architecture and
Chapter 6. Disk storage management 213
location on the disk depends on the type of the volume group (original, big, or
scalable).
 The volume group status area (VGSA) is used to describe the state of all
physical partitions from all physical volumes within a volume group. The
VGSA indicates if a physical partition contains accurate or stale information.
VGSA is used for monitoring and maintained data copies synchronization.
The VGSA is essentially a bitmap and its architecture and location on the disk
depends on the type of the volume group.
 A logical volume control block (LVCB) contains important information about
the logical volume, such as the number of the logical partitions or disk
allocation policy. Its architecture and location on the disk depends on the type
of the volume group it belongs to. For standard volume groups, the LVCB
resides on the first block of user data within the LV. For big volume groups,
there is additional LVCB information in VGDA on the disk. For scalable volume
groups, all relevant logical volume control information is kept in the VGDA as
part of the LVCB information area and the LV entry area.

Boot process of IBM AIX

This chapter describes the boot process and the different stages the system
uses to prepare the AIX 5L environment.
Topics discussed in this chapter are:
 The boot process
 System initialization
 The /etc/inittab file
 How to recover from a non-responsive boot process
 Run levels
 An introduction to the rc.* files

WSM Objectives

The objectives of the Web-based System Manager are:
• Simplification of AIX administration by a single interface
• Enable AIX systems to be administered from almost any client platform with a browser
that supports Java 1.3 or use downloaded client code from an AIX V5.3 code
• Enable AIX systems to be administered remotely
• Provide a system administration environment that provides a similar look and feel to the
Windows NT/2000/XP, LINUX and AIX CDE environments
The Web-based System Manager provides a comprehensive system management
environment and covers most of the tasks in the SMIT user interface. The Web-based
System Manager can only be run from a graphics terminal so SMIT will need to be used in
the ASCII environment.
To download Web-based System Manager Client code from an AIX host use the address
http:///remote_client.html
Supported Microsoft Windows clients for AIX 5.3 are Windows 2000 Professional version,
Windows XP Professional version, or Windows Server 2003.
Supported Linux clients are PCs running: Red Hat Enterprise Version 3, SLES 8, SLES 9,
Suse 8.0, Suse 8.1, Suse 8.2, and Suse 9.0 using desktops KDE or GNOME only.
The PC Web-based System Manager Client installation needs a minimum of 300 MB free
disk space, 512 MB memory (1GB preferred) and a 1 GHZ cpu.

HACMP COurse Details

· Explain what High Availability is.

· Outline the capabilities of HACMP for AIX.

· Design and plan a highly available cluster.

· Install and configure HACMP for AIX or HACMP/ES in the

following modes of operaton:

• Cascading.
• Mutual Takeover.
• Cascading without Fallback.
• Rotating.
• Concurrent Access (optional).

• Perform basic system administration tasks for HACMP.

• Perform basic customisation for HACMP.
• Carry out problem determination and recovery.

System management

Cluster Systems Management (CSM) for AIX and Linux
CSM is designed to minimize the cost and complexity of administering clustered and partitioned systems by enabling comprehensive management and monitoring of the entire environment from a single point of control. CSM provides:

• Software distribution, installation and update (operating system and applications)
• Comprehensive system monitoring with customizable automated responses
• Distributed command execution
• Hardware control
• Diagnostic tools
• Management by group
• Both a graphical interface and a fully scriptable command line interface

In addition to providing all the key functions for administration and maintenance of distributed systems, CSM is designed to deliver the parallel execution required to manage clustered computing environments effectively. CSM supports homogeneous or mixed environments of IBM servers running AIX or Linux.

Parallel System Support Programs (PSSP) for AIX
PSSP is the systems management predecessor to Cluster Systems Management (CSM) and does not support IBM System p servers or AIX 5L™ V5.3 or above. New cluster deployments should use CSM and existing PSSP clients with software maintenance will be transitioned to CSM at no charge.

Very usefull Command

svmon
svmon -P 

Further:
use can user svmon command to monitor memory usage as follows;

(A) #svmon -P -v -t 10 | more         (will give top ten processes)
(B) #svmon -U -v -t 10 | more         ( will give top ten user)

smit install requires "inutoc ." first.  It'll autogenerate a .toc for you
I believe, but if you later add more .bff's to the same directory, then
the inutoc . becomes important.  It is of course, a table of contents.

dump -ov /dir/xcoff-file

topas, -P is useful # similar to top

When creating really big filesystems, this is very helpful:
chlv -x 6552 lv08
Word on the net is that this is required for filesystems over 512M.

esmf04m-root> crfs -v jfs -g'ptmpvg' -a size='884998144' -m'/ptmp2'
-A''`locale yesstr | awk -F: '{print $1}'`'' -p'rw' -t''`locale yesstr |
awk -F: '{print $1}'`'' -a frag='4096' -a nbpi='131072' -a ag='64'
Based on the parameters chosen, the new /ptmp2 JFS file system
is limited to a maximum size of 2147483648 (512 byte blocks)
New File System size is 884998144
esmf04m-root>

If you give a bad combination of parameters, the command will list
possibilities.  I got something like this from smit, then seasoned
to taste.

If you need files larger than 2 gigabytes in size, this is better.
It should allow files up to 64 gigabytes:
crfs -v jfs -a bf=true -g'ptmpvg' -a size='884998144' -m'/ptmp2' -A''` |
|   locale yesstr | awk -F: '{print $1}'`'' -p'rw' -t''`locale yesstr | aw |
|   k -F: '{print $1}'`'' -a nbpi='131072' -a ag='64'  

Show version of SSP (IBM SP switch) software:
lslpp -al ssp.basic

llctl -g reconfig - make loadleveler reread its config files

oslevel (sometimes lies)
oslevel -r (seems to do better)

lsdev -Cc adapter

pstat -a looks useful

vmo is for VM tuning

On 1000BaseT, you really want this:
chdev -P -l ent2 -a media_speed=Auto_Negotiation

Setting jumbo frames on en2 looks like:
ifconfig en2 down detach
chdev -l ent2 -a jumbo_frames=yes
chdev -l en2 -a mtu=9000
chdev -l en2 -a state=up

Search for the meaning of AIX errors:
http://publib16.boulder.ibm.com/pseries/en_US/infocenter/base/eisearch.htm 


nfso -a shows AIX NFS tuning parameters; good to check on if you're
getting badcalls in nfsstat.  Most people don't bother to tweaks these
though.

nfsstat -m shows great info about full set of NFS mount options

Turn on path mtu discovery
no -o tcp_pmtu_discover=1
no -o udp_pmtu_discover=1
TCP support is handled by the OS.  UDP support requires cooperation
between OS and application.

nfsstat -c shows rpc stats

To check for software problems:
lppchk -v
lppchk -c
lppchk -l

List subsystem (my word) status:
lssrc -a
mkssys
rmssys
chssys
auditpr
refresh
startsrc
stopsrc
traceson
tracesoff

This starts sendmail:
startsrc -s sendmail -a "-bd -q30m"

This makes inetd reread its config file.  Not sure if it kills and
restarts or just HUP's or what:
refresh -s inetd

lsps is used to list the characteristics of paging space.

Turning off ip forwarding:
/usr/sbin/no -o ipforwarding=0

Detailed info about a specific error:
errpt -a -jE85C5C4C
BTW, Rajiv Bendale tells me that errors are stored in NVRAM on AIX,
so you don't have to put time into replicating an error as often.

Some or all of these will list more than one number.  Trust the first,
not the second.

lslpp -l ppe.poe
...should list the version of poe installed on the system

Check on compiler versions:
lslpp -l vac.C
lslpp -l vacpp.cmp.core

Check on loadleveler version:
lslpp -l LoadL.full

If you want to check the bootlist do bootlist -o -m normal if you want to
update bootlist do bootlist -m normal hdisk* hdisk* cd* rmt*

prtconf

Run the ssadiag against the drive and the adapter and it will tell you if it
fails or not. Then if its a hot plugable it can be replace

AIX Control Book Creation

List the licensed program productslslpp -L
List the defined devices lsdev -C -H
List the disk drives on the system lsdev -Cc disk
List the memory on the system lsdev -Cc memory (MCA)
List the memory on the system lsattr -El sys0 -a realmem (PCI)
lsattr -El mem0
List system resources lsattr -EHl sys0
List the VPD (Vital Product Data) lscfg -v
Document the tty setup lscfg or smit screen capture F8
Document the print queues qchk -A
Document disk Physical Volumes (PVs) lspv
Document Logical Volumes (LVs) lslv
Document Volume Groups (long list) lsvg -l vgname
Document Physical Volumes (long list) lspv -l pvname
Document File Systems lsfs fsname
/etc/filesystems
Document disk allocation df
Document mounted file systems mount
Document paging space (70 - 30 rule) lsps -a
Document paging space activation /etc/swapspaces
Document users on the system /etc/passwd
lsuser -a id home ALL
Document users attributes /etc/security/user
Document users limits /etc/security/limits
Document users environments /etc/security/environ
Document login settings (login herald) /etc/security/login.cfg
Document valid group attributes /etc/group
lsgroup ALL
Document system wide profile /etc/profile
Document system wide environment /etc/environment
Document cron jobs /var/spool/cron/crontabs/*
Document skulker changes if used /usr/sbin/skulker
Document system startup file /etc/inittab
Document the hostnames /etc/hosts
Document network printing /etc/hosts.lpd
Document remote login host authority /etc/hosts

What is Hot Spare

What is an LVM hot spare?

A hot spare is a disk or group of disks used to replace a failing disk. LVM marks a physical
volume missing due to write failures. It then starts the migration of data to the hot spare
disk.
Minimum hot spare requirements
The following is a list of minimal hot sparing requirements enforced by the operating
system.
- Spares are allocated and used by volume group
- Logical volumes must be mirrored
- All logical partitions on hot spare disks must be unallocated
- Hot spare disks must have at least equal capacity to the smallest disk already
in the volume group. Good practice dictates having enough hot spares to
cover your largest mirrored disk.
Hot spare policy
The chpv and the chvg commands are enhanced with a new -h argument. This allows you
to designate disks as hot spares in a volume group and to specify a policy to be used in the
case of failing disks.
The following four values are valid for the hot spare policy argument (-h):
Synchronization policy
There is a new -s argument for the chvg command that is used to specify synchronization
characteristics.
The following two values are valid for the synchronization argument (-s):
Examples
The following command marks hdisk1 as a hot spare disk:
# chpv -hy hdisk1
The following command sets an automatic migration policy which uses the smallest hot
spare that is large enough to replace the failing disk, and automatically tries to synchronize
stale partitions:
# chvg -hy -sy testvg
Argument Description
y (lower case)
Automatically migrates partitions from one failing disk to one spare
disk. From the pool of hot spare disks, the smallest one which is big
enough to substitute for the failing disk will be used.
Y (upper case)
Automatically migrates partitions from a failing disk, but might use
the complete pool of hot spare disks.
n
No automatic migration will take place. This is the default value for a
volume group.
r
Removes all disks from the pool of hot spare disks for this volume