Oracle 10g RAC On Linux Using VMware Server
This article describes the installation of Oracle 10g release 2 (10.2.0.1) RAC on Linux (CentOS 4) using VMware Server with no additional shared disk devices.

Introduction
One of the biggest obstacles preventing people from setting up test RAC environments is the requirement for shared storage. In a production environment, shared storage is often provided by a SAN or high-end NAS device, but both of these options are very expensive when all you want to do is get some experience installing and using RAC. A cheaper alternative is to use a FireWire disk enclosure to allow two machines to access the same disk(s), but that still costs money and requires two servers. A third option is to use VMware Server to fake the shared storage.
Using VMware Server you can run multiple Virtual Machines (VMs) on a single server, allowing you to run both RAC nodes on a single machine. In additon, it allows you to set up shared virtual disks, overcoming the obstacle of expensive shared storage.
Before you launch into this installation, here are a few things to consider.

Download Software
Download the following software.

VMware Server Installation
For this article, I used CentOS 4.3 as both the host and guest operating systems. Regardless of the host OS, the setup of the virtual machines should be similar.
First, install the VMware Server software. On Linux you do this with the following command as the root user.

# rpm -Uvh VMware-server-*.rpm
Preparing... ########################################### [100%]
1:VMware-server ########################################### [100%]
#

Then finish the configuration by running the vmware-config.pl script as the root user. Most of the questions can be answered with the default response by pressing the return key. The output below shows my responses to the questions.

# vmware-config.pl
Making sure services for VMware Server are stopped.

Stopping VMware services:
Virtual machine monitor [ OK ]

You must read and accept the End User License Agreement to continue.
Press enter to display it.

VMWARE, INC.
SOFTWARE BETA TEST AGREEMENT

*** Editied out license agreement ***

Do you accept? (yes/no) yes

Thank you.

Configuring fallback GTK+ 2.4 libraries.

In which directory do you want to install the mime type icons?
[/usr/share/icons]

What directory contains your desktop menu entry files? These files have a
.desktop file extension. [/usr/share/applications]

In which directory do you want to install the application's icon?
[/usr/share/pixmaps]

Trying to find a suitable vmmon module for your running kernel.

The module bld-2.6.9-5.EL-i686smp-RHEL4 loads perfectly in the running kernel.

Do you want networking for your virtual machines? (yes/no/help) [yes]

Configuring a bridged network for vmnet0.

The following bridged networks have been defined:

. vmnet0 is bridged to eth0

All your ethernet interfaces are already bridged.

Do you want to be able to use NAT networking in your virtual machines? (yes/no)
[yes]

Configuring a NAT network for vmnet8.

Do you want this program to probe for an unused private subnet? (yes/no/help)
[yes]

Probing for an unused private subnet (this can take some time)...

The subnet 172.16.210.0/255.255.255.0 appears to be unused.

The following NAT networks have been defined:

. vmnet8 is a NAT network on private subnet 172.16.210.0.

Do you wish to configure another NAT network? (yes/no) [no]

Do you want to be able to use host-only networking in your virtual machines?
[yes] no

Trying to find a suitable vmnet module for your running kernel.

The module bld-2.6.9-5.EL-i686smp-RHEL4 loads perfectly in the running kernel.

Please specify a port for remote console connections to use [902]

Stopping xinetd: [ OK ]
Starting xinetd: [ OK ]
Configuring the VMware VmPerl Scripting API.

Building the VMware VmPerl Scripting API.

Using compiler "/usr/bin/gcc". Use environment variable CC to override.

The installation of the VMware VmPerl Scripting API succeeded.

Do you want this program to set up permissions for your registered virtual
machines? This will be done by setting new permissions on all files found in
the "/etc/vmware/vm-list" file. [no] yes

Generating SSL Server Certificate

In which directory do you want to keep your virtual machine files?
[/var/lib/vmware/Virtual Machines] /u01/VM

Do you want to enter a serial number now? (yes/no/help) [no] yes

Please enter your 20-character serial number.

Type XXXXX-XXXXX-XXXXX-XXXXX or 'Enter' to cancel: ENTER-YOUR-SERIAL-NUMBER

Starting VMware services:
Virtual machine monitor [ OK ]
Virtual ethernet [ OK ]
Bridged networking on /dev/vmnet0 [ OK ]
Host-only networking on /dev/vmnet8 (background) [ OK ]
NAT service on /dev/vmnet8 [ OK ]
Starting VMware virtual machines... [ OK ]

The configuration of VMware Server e.x.p build-22874 for Linux for this running
kernel completed successfully.

#

The VMware Server Console is started by issuing the command "vmware" at the command prompt, or by selecting it from the "System Tools" menu.

On the "Connect to Host" dialog, accept the "Local host" option by clicking the "Connect" button.

You are then presented with the main VMware Server Console screen.

The VMware Server is now installed and ready to use.
Virtual Machine Setup
Now we must define the two virtual RAC nodes. We can save time by defining one VM, then cloning it when it is installed.
Click the "Create a new virtual machine" button to start the "New Virtual Machine Wizard". Click the "Next" button onthe welcome page.

Select the "Custom" virtual machine configuration and click the "Next" button.

Select the "Linux" guest operating system option, and set the version to "Red Hat Enterprise Linux 4", then click the "Next" button.

Enter the name "RAC1" and the location should default to "/u01/VM/RAC1", then click the "Next" button.

Select the required number of processors and click the "Next" button.

Uncheck the "Make this virtual machine private" checkbox and click the "Next" button.

Select the amount of memory to associate with the virtual machine. Remember, you are going to need two instances, so don't associate too much, but you are going to need approximately 1 Gig (1024 Meg) to compete the installation successfully.

Accept the "Use bridged networking" option by clicking the "Next" button.

Accept the "LSI Logic" option by clicking the "Next" button.

Select the "Create a new virtual disk" option and click the "Next" button.

Accept the "SCSI" option by clicking the "Next" button. It's a virtual disk, so you can still use this option even if your physical disk is IDE or SATA.

Set the disk size to "10.0" GB and uncheck the "Allocate all disk space now" option. The latter will make disk access slower, but will save you wasting disk space.

Accept "RAC1.vmdk" as the disk file name and complete the VM creation by clicking the "Finish" button.

On the "VMware Server Console" screen, click the "Edit virtual machine settings" button.

On the "Virtual Machine Settings" screen, highlight the "Floppy 1" drive and click the "- Remove" button.

Click the "+ Add" button and select a hardware type of "Ethernet Adapter", then click the "Next" button.

Accept the "Bridged" option by clicking the "Finish" button.

Click on the "Options" tab, highlight the "Startup/Shutdown" setting and select the "Don't power on virtual machine" in the "On host startup" option. Finish by clicking the "OK" button.

The virtual machine is now configured so we can start the guest operating system installation.
Guest Operating System Installation
Place the first CentOS 4 disk in the CD drive and start the virtual machine by clicking the "Power on this virtual machine" button. The right pane of the VMware Server Console should display a boot loader, then the CentOS installation screen.

Continue through the CentOS 4 installation as you would for a normal server. A general pictorial guide to the installation can be found here. More specifically, it should be a server installation with a minimum of 2G swap, secure Linux disabled and the following package groups installed:

To be consistent with the rest of the article, the following information should be set during the installation:

You are free to change the IP addresses to suit your network, but remember to stay consistent with those adjustments throughout the rest of the article.
Once the basic installation is complete, install the following packages whilst logged in as the root user.

# From CentOS4 Disk 1
cd /media/cdrecorder/CentOS/RPMS
rpm -Uvh setarch-1*
rpm -Uvh compat-libstdc++-33-3*
rpm -Uvh make-3*
rpm -Uvh glibc-2*
cd /
eject

# From CentOS4 Disk 2
cd /media/cdrecorder/CentOS/RPMS
rpm -Uvh openmotif-2*
rpm -Uvh compat-db-4*
rpm -Uvh gcc-3*
cd /
eject

# From CentOS4 Disk 3
cd /media/cdrecorder/CentOS/RPMS
rpm -Uvh libaio-0*
rpm -Uvh rsh-*
rpm -Uvh compat-gcc-32-3*
rpm -Uvh compat-gcc-32-c++-3*
rpm -Uvh openmotif21*
cd /
eject

Oracle Installation Prerequisites
Perform the following steps whilst logged into the RAC1 virtual machine as the root user.
The /etc/hosts file must contain the following information.

127.0.0.1 localhost.localdomain localhost
# Public
192.168.2.101 rac1.localdomain rac1
192.168.2.102 rac2.localdomain rac2
#Private
192.168.0.101 rac1-priv.localdomain rac1-priv
192.168.0.102 rac2-priv.localdomain rac2-priv
#Virtual
192.168.2.111 rac1-vip.localdomain rac1-vip
192.168.2.112 rac2-vip.localdomain rac2-vip

Add the following lines to the /etc/sysctl.conf file.

kernel.shmall = 2097152
kernel.shmmax = 2147483648
kernel.shmmni = 4096
# semaphores: semmsl, semmns, semopm, semmni
kernel.sem = 250 32000 100 128
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000
net.core.rmem_default=262144
net.core.rmem_max=262144
net.core.wmem_default=262144
net.core.wmem_max=262144

Run the following command to change the current kernel parameters.

/sbin/sysctl -p

Add the following lines to the /etc/security/limits.conf file.

* soft nproc 2047
* hard nproc 16384
* soft nofile 1024
* hard nofile 65536

Add the following line to the /etc/pam.d/login file, if it does not already exist.

session required pam_limits.so

Disable secure linux by editing the /etc/selinux/config file, making sure the SELINUX flag is set as follows.

SELINUX=disabled

Alternatively, this alteration can be done using the GUI tool (Applications > System Settings > Security Level). Click on the SELinux tab and disable the feature.
Set the hangcheck kernel module parameters by adding the following line to the /etc/modprobe.conf file.

options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180

To load the module immediately, execute "modprobe -v hangcheck-timer".
Create the new groups and users.

groupadd oinstall
groupadd dba
groupadd oper

useradd -g oinstall -G dba oracle
passwd oracle

Create the directories in which the Oracle software will be installed.

mkdir -p /u01/crs/oracle/product/10.2.0/crs
mkdir -p /u01/app/oracle/product/10.2.0/db_1
mkdir -p /u01/oradata
chown -R oracle.oinstall /u01

During the installation, both RSH and RSH-Server were installed. Enable remote shell and rlogin by doing the following.

chkconfig rsh on
chkconfig rlogin on
service xinetd reload

Create the /etc/hosts.equiv file as the root user.

touch /etc/hosts.equiv
chmod 600 /etc/hosts.equiv
chown root:root /etc/hosts.equiv

Edit the /etc/hosts.equiv file to include all the RAC nodes:

+rac1 oracle
+rac2 oracle
+rac1-priv oracle
+rac2-priv oracle

Login as the oracle user and add the following lines at the end of the .bash_profile file.

# Oracle Settings
TMP=/tmp; export TMP
TMPDIR=$TMP; export TMPDIR

ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
ORACLE_HOME=$ORACLE_BASE/product/10.2.0/db_1; export ORACLE_HOME
ORACLE_SID=RAC1; export ORACLE_SID
ORACLE_TERM=xterm; export ORACLE_TERM
PATH=/usr/sbin:$PATH; export PATH
PATH=$ORACLE_HOME/bin:$PATH; export PATH

LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH

if [ $USER = "oracle" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
fi

Install VMware Client Tools
Login as the root user on the RAC1 virtual machine, then select the "VM > Install VMware Tools..." option from the main VMware Server Console menu.

This should mount a virtual CD containing the VMware Tools software. Double-click on the CD icon labelled "VMware Tools" to open the CD. Right-click on the ".rpm" package and select the "Open with 'Install Packages'" menu option.

Click the "Continue" button on the "Completed System Preparation" screen and wait for the installation to complete.

Once the package is loaded, the CD should unmount automatically. You must then run the "vmware-config-tools.pl" script as the root user. The following listing is an example of the output you should expect.

# vmware-config-tools.pl

Stopping VMware Tools services in the virtual machine:
Guest operating system daemon: [ OK ]
Trying to find a suitable vmhgfs module for your running kernel.

The module bld-2.6.9-11.EL-i686up-RHEL4 loads perfectly in the running kernel.

pcnet32 30409 0
Unloading pcnet32 module

Trying to find a suitable vmxnet module for your running kernel.

The module bld-2.6.9-11.EL-i686up-RHEL4 loads perfectly in the running kernel.

Detected X.org version 6.8.

Please choose one of the following display sizes (1 - 13):

[1] "640x480"
[2] "800x600"
[3] "1024x768"
[4] "1152x864"
[5] "1280x800"
[6] "1152x900"
[7] "1280x1024"
[8] "1376x1032"
[9] "1400x1050"
[10] "1680x1050"
[11] "1600x1200"
[12]< "1920x1200"
[13] "2364x1773"
Please enter a number between 1 and 13:

[12] 3

X Window System Version 6.8.2
Release Date: 9 February 2005
X Protocol Version 11, Revision 0, Release 6.8.2
Build Operating System: Linux 2.6.9-11.EL i686 [ELF]
Current Operating System: Linux rac1.localdomain 2.6.9-22.EL #1 Sat Oct 8 17:48:27 CDT 2005 i686
Build Date: 07 October 2005
Build Host: x8664-build.home.local

Before reporting problems, check http://wiki.X.Org
to make sure that you have the latest version.
Module Loader present
OS Kernel: Linux version 2.6.9-22.EL (buildcentos@louisa.home.local)
(gcc version 3.4.4 20050721 (Red Hat 3.4.4-2)) #1 Sat Oct 8 17:48:27 CDT 2005 P
Markers: (--) probed, (**) from config file, (==) default setting,
(++) from command line, (!!) notice, (II) informational,
(WW) warning, (EE) error, (NI) not implemented, (??) unknown.
(++) Log file: "/tmp/vmware-config0/XF86ConfigLog.3674", Time: Thu Apr 13 21:17:37 2006
(++) Using config file: "/tmp/vmware-config0/XF86Config.3674"

X is running fine with the new config file.

(WW) VMWARE(0): Failed to set up write-combining range (0xf0000000,0x1000000)
Starting VMware Tools services in the virtual machine:
Switching to guest configuration: [ OK ]
Guest filesystem driver: [ OK ]
Guest vmxnet fast network device: [ OK ]
DMA setup: [ OK ]
Guest operating system daemon: [ OK ]

The configuration of VMware Tools e.x.p build-22874 for Linux for this running
kernel completed successfully.

You must restart your X session before any mouse or graphics changes take
effect.

You can now run VMware Tools by invoking the following command:
"/usr/bin/vmware-toolbox" during an XFree86 session.

To use the vmxnet driver, restart networking using the following commands:
/etc/init.d/network stop
rmmod pcnet32
rmmod vmxnet
depmod -a
modprobe vmxnet
/etc/init.d/network start

Enjoy,

--the VMware team

#

The VMware client tools are now installed.