Cold Failover Single Instance DB to Different Node on ODA

server-disaster

Oracle Database Appliance (ODA) is the entry level of Oracle Engineering System. It is like Exadata, but it does not have cell nodes and InfiniBand. It does have two db nodes and a shared storage with 24 disks. Both ODA and Exadata are using the same Grid Infrastructure to manage ASM across multiple database nodes. As the ASM db files can be accessed by all db nodes, usually companies are implementing RAC or RAC ONE options to handle the failover in case of hardware failure. As both ODA and Exadata can support single instance database, it makes me wonder how the failover works and how I can bring up the database on a different node in case there is a hardware failure or a failure show above.

As many people know, the failover on RAC and RAC ONE is super easy and you don’t need do anything if the db node is down, the db can continue to run on the other db node. The difference for RAC and RAC ONE is that RAC perform hot failover while RAC ONE perform cold failover. I wouldn’t expect single instance or non-rac db can do the hot failover, but some of kind of cold failover is fine with me. It’s better than nothing.

At first, I thought just shutdown the db and created some necessary directories, add an entry to oratab file on another node should work. I remember reading one article about this procedure long time ago by changing certain db parameters from the pfile, and bring up the db by using the modified pfile. Search the internet and could not find it anymore. The most closed article I can find is, How To Setup ASM (11.2) On An Active/Passive Cluster (Non-RAC). (Doc ID 1296124.1). The document discusses a lot of steps to install and configure GI on the 2nd db node, so I basically can skip those steps. When I worked on the most important step, srvctl add database, I got the following errors:

oracle:testdb@odap2 ~ > /u01/app/oracle/product/11.2.0.3/dbhome_1/bin/srvctl add database -d testdb -n odap2 -o /u01/app/oracle/product/11.2.0.3/dbhome_1 -p +data/TESTDB/spfiletestdb.ora -s OPEN -y AUTOMATIC -a DATA,RECO,REDO -t IMMEDIATE
PRCS-1007 : Server pool testdb already exists
PRCR-1086 : server pool ora.testdb is already registered

It looks weird that this db exists on db node 2 even before I created it. Later I realized the srvctl command can see the resource across all db nodes. Obviously this document is not going to work for me. So let me try another way. What if I shutdown the db on node 1 and bring it up on node 2 directly. Wrong! After I did that, I got many errors during mount stage as follows:

SQL> startup nomount
ORACLE instance started.

Total System Global Area 4275781632 bytes
Fixed Size 2235208 bytes
Variable Size 838862008 bytes
Database Buffers 3388997632 bytes
Redo Buffers 45686784 bytes
SQL> alter database mount
2 ;
alter database mount
*
ERROR at line 1:
ORA-03113: end-of-file on communication channel
Process ID: 2838
Session ID: 584 Serial number: 1

The alert log also shows the following errors:
Thu Sep 05 19:56:38 2013
alter database mount
Thu Sep 05 19:56:38 2013
NOTE: Loaded library: System
ORA-15025: could not open disk “/dev/mapper/SSD_E0_S20_805849611p1”
ORA-27041: unable to open file
Linux-x86_64 Error: 13: Permission denied
Additional information: 9
ORA-15025: could not open disk “/dev/mapper/SSD_E0_S21_805849344p1”
ORA-27041: unable to open file
Linux-x86_64 Error: 13: Permission denied
Additional information: 9
ORA-15025: could not open disk “/dev/mapper/SSD_E0_S22_805849349p1”
ORA-27041: unable to open file
Linux-x86_64 Error: 13: Permission denied
Additional information: 9
ORA-15025: could not open disk “/dev/mapper/SSD_E0_S23_805849238p1”
ORA-27041: unable to open file
Linux-x86_64 Error: 13: Permission denied
Additional information: 9
Thu Sep 05 19:56:38 2013
SUCCESS: diskgroup REDO was mounted
Thu Sep 05 19:56:38 2013
NOTE: dependency between database testdb and diskgroup resource ora.REDO.dg is established
Errors in file /u01/app/oracle/diag/rdbms/testdb/testdb/trace/testdb_ckpt_2813.trc (incident=24145):
ORA-00600: internal error code, arguments: [kfioTranslateIO03], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/testdb/testdb/incident/incdir_24145/testdb_ckpt_2813_i24145.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/testdb/testdb/trace/testdb_ckpt_2813.trc (incident=24146):
ORA-00600: internal error code, arguments: [17090], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/testdb/testdb/incident/incdir_24146/testdb_ckpt_2813_i24146.trc

Out of ideas, I posted my question to Enkitec‘s internal technical distribution email list. My colleague, Andy Colvin, came back with an interesting idea. He recommended to use srvctl modify database command to tell the OCR to use the other node. After that, just run srvctl start database to start the database. It is an interesting idea and seem make sense to me. However, when I checked out the Oracle document about srvctl modify database, there is no parameter that can specify the hostname. The srvctl is used to use -n to specify hostname in many occasion. For this command, -n means db name. Here is the syntax about this command from the document.

srvctl modify database -d db_unique_name [-o oracle_home] [-u oracle_user]
[-m domain_name] [-n db_name] [-p spfile] [-s start_options]
[-t stop_options] [-r {PRIMARY | PHYSICAL_STANDBY | LOGICAL_STANDBY |
SNAPSHOT_STANDBY}] [-y {automatic | manual}] [-a disk_group_list] [-z]

It looks weird to me and I pretty sure it must have some ways to specify hostname somewhere when creating the db. Luckily, whenever I created a db using dbca, I always save a copy of the creation script. So I run the following to search the creation scripts.

oracle:testdb@odap1 scripts> cd /u01/app/oracle/admin/testdb/scripts
oracle:testdb@odap1 scripts> grep odap1 *
testdb.sql:host /u01/app/oracle/product/11.2.0.3/dbhome_1/bin/srvctl add database -d testdb -o /u01/app/oracle/product/11.2.0.3/dbhome_1 -p +DATA/testdb/spfiletestdb.ora -n testdb -x odap1 -a “DATA,RECO”

Ah, here is the answer, it does have undocument parameter -x that can specify the hostname. So I did the following:
/u01/app/oracle/product/11.2.0.3/dbhome_1/bin/srvctl modify database -d testdb -o /u01/app/oracle/product/11.2.0.3/dbhome_1 -p +DATA/testdb/spfiletestdb.ora -n testdb -x odap2 -a “DATA,RECO”

Then the database can be brought up by using sqlplus on the 2nd db node.

Sounds like the end of story. Later on, another colleague, Frits Hoogland, brought up a even nicer and cleaner solution than what I did. He mentioned the approach can be used on Exadata. Basically on 2nd db node, I don’t need to do make change in oratab file, and create directories, but must use srvctl command instead of sqlplus to do the work. Another interesting idea, so I reproduced his recommendation and it works easily without any issue. To make the example to use for different scenarios, I created a db on node 1 with the following information:

db name : mytestdb
db unquie name : mytestprd
service : mytest

Here are the steps:
Step 1. Verificate some information about the db and directories
on node 1

oracle:mytestdb@odap1 ~> srvctl config database -d mytestprd
Database unique name: mytestprd
Database name: mytestdb
Oracle home: /u01/app/oracle/product/11.2.0.3/dbhome_1
Oracle user: oracle
Spfile: +DATA/mytestprd/spfilemytestdb.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: mytestprd
Database instance: mytestdb
Disk Groups: DATA,RECO,REDO
Mount point paths:
Services: mytest
Type: SINGLE
Database is administrator managed

SYS@mytestdb> show parameter name

NAME TYPE                   VALUE
--------------------------- ----------- ------------------------------
db_file_name_convert        string
db_name                     string	 mytestdb
db_unique_name              string	 mytestprd
global_names                boolean      TRUE
instance_name               string	 mytestdb
lock_name_space             string
log_file_name_convert       string
processor_group_name        string
service_names               string	 mytest

On db node 2, check the entry and db directories exist or not. Of course, they should not exit.

oracle:testdb@odap2 ~> grep mytest /etc/oratab

oracle:testdb@odap2 ~> ls -l /u01/app/oracle/diag/rdbms/mytestprd/mytestdb/trace
ls: /u01/app/oracle/diag/rdbms/mytestprd/mytestdb/trace: No such file or directory

oracle:testdb@odap2 ~> ls -l /u01/app/oracle/admin/mytestprd/adump
ls: /u01/app/oracle/admin/mytestprd/adump: No such file or directory

Step 2. Configure the db to move from db node 1 to node 2

oracle:testdb@odap2 ~> srvctl status database -d mytestprd
Instance mytestdb is running on node odap1

oracle:testdb@odap2 ~> srvctl status service -d mytestprd -s mytest
Service mytest is running on instance(s) mytestdb

oracle:testdb@odap2 ~> srvctl stop database -d mytestprd
oracle:testdb@odap2 ~> srvctl modify database -d mytestprd -x odap2
oracle:testdb@odap2 ~> srvctl start database -d mytestprd

oracle:testdb@odap2 ~> srvctl status database -d mytestprd
Instance mytestdb is running on node odap2
oracle:testdb@odap2 ~> srvctl status service -d mytestprd -s mytest
Service mytest is running on instance(s) mytestdb

oracle:testdb@odap2 ~> srvctl config database -d mytestprd
Database unique name: mytestprd
Database name: mytestdb
Oracle home: /u01/app/oracle/product/11.2.0.3/dbhome_1
Oracle user: oracle
Spfile: +DATA/mytestprd/spfilemytestdb.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: mytestprd
Database instance: mytestdb
Disk Groups: DATA,RECO,REDO
Mount point paths:
Services: mytest
Type: SINGLE
Database is administrator managed
[/code]

Right now, we can verify whether the oratab entry and directories exist or not.

verify the entry in oratab file

oracle:testdb@odap2 ~> grep mytest /etc/oratab
mytestdb:/u01/app/oracle/product/11.2.0.3/dbhome_1:N    # line added by Agent

oracle:testdb@odap2 ~> ls -l /u01/app/oracle/diag/rdbms/mytestprd/mytestdb/trace
total 160
-rw-r----- 1 oracle asmadmin 10037 Sep  7 10:20 alert_mytestdb.log
-rw-r----- 1 oracle asmadmin   893 Sep  7 10:20 mytestdb_asmb_31403.trc
-rw-r----- 1 oracle asmadmin    64 Sep  7 10:20 mytestdb_asmb_31403.trm
-rw-r----- 1 oracle asmadmin  1062 Sep  7 10:20 mytestdb_cjq0_31545.trc
....

oracle:testdb@odap2 ~> ls -l /u01/app/oracle/admin/mytestprd/adump
total 32
-rw-r----- 1 oracle asmadmin  796 Sep  7 10:20 mytestdb_ora_31064_2.aud
-rw-r----- 1 oracle asmadmin 2600 Sep  7 10:20 mytestdb_ora_31421_1.aud
....

Sqlplus command also show the db is relocated to 2nd db node

SQL> col host_name for a20
select instance_name, host_name from v$instance;

SQL>
INSTANCE_NAME	 HOST_NAME
---------------- --------------------
mytestdb	     odap2

Let’s do some check under $ORACLE_HOME/dbs directory on node 2.

oracle:mytestdb@odap2 dbs> ls -l *mytestdb*
-rw-rw---- 1 oracle asmadmin 1544 Sep  7 10:20 hc_mytestdb.dat
-rw-r--r-- 1 oracle oinstall   67 Sep  7 10:19 initmytestdb.ora

oracle:mytestdb@odap2 dbs> cat initmytestdb.ora
SPFILE='+DATA/mytestprd/spfilemytestdb.ora'		# line added by Agent

Check out the same $ORACLE_HOME/dbs directory on node 1

oracle:mytestdb@odap1 dbs> ls -l *mytest*
-rw-rw---- 1 oracle asmadmin 1544 Sep  7 10:19 hc_mytestdb.dat
-rw-r----- 1 oracle oinstall   44 Sep  6 09:12 initmytestdb.ora
-rw-r----- 1 oracle oinstall 1536 Sep  6 09:05 orapwmytestdb

You can notice that the password file is missing on the node 2. So do the copy work from db node 1.

oracle:mytestdb@odap2 dbs> cd $ORACLE_HOME/dbs
oracle:mytestdb@odap2 dbs> scp -p oracle@odap1:$PWD/orapwmytestdb .
orapwmytestdb                                                                     100% 1536     1.5KB/s   00:00
oracle:mytestdb@odap2 dbs> ls -lart *mytest*
-rw-r----- 1 oracle oinstall 1536 Sep  6 09:05 orapwmytestdb
-rw-r--r-- 1 oracle oinstall   67 Sep  7 10:19 initmytestdb.ora
-rw-rw---- 1 oracle asmadmin 1544 Sep  7 10:20 hc_mytestdb.dat

Run the crsstat command, it also shows the right result.

oracle:mytestdb@odap2 dbs> crsstat
Resource Name            Resource Type    Target     State        Node            FC RC State Details
------------------------ ---------------- ---------- ------------ --------------- -- -- ------
ora.DATA.dg               diskgroup      C ONLINE     ONLINE       odap1       0  0
ora.DATA.dg               diskgroup      C ONLINE     ONLINE       odap2       0  0
ora.LISTENER.lsnr         Listener       C ONLINE     ONLINE       odap1       0  0
ora.LISTENER.lsnr         Listener       C ONLINE     ONLINE       odap2       0  0
ora.LISTENER_SCAN1.lsnr   SCAN Listener  L ONLINE     ONLINE       odap1       0  0
ora.LISTENER_SCAN2.lsnr   SCAN Listener  L ONLINE     ONLINE       odap2       0  0
ora.RECO.dg               diskgroup      C ONLINE     ONLINE       odap1       0  0
ora.RECO.dg               diskgroup      C ONLINE     ONLINE       odap2       0  0
ora.REDO.dg               diskgroup      C ONLINE     ONLINE       odap1       0  0
ora.REDO.dg               diskgroup      C ONLINE     ONLINE       odap2       0  0
ora.asm                   ASM            C ONLINE     ONLINE       odap1       0  0 Started
ora.asm                   ASM            C ONLINE     ONLINE       odap2       0  0 Started
ora.cvu                   cvu            L ONLINE     ONLINE       odap2       0  0
ora.dbm.db                database       L ONLINE     ONLINE       odap1       0  0 Open
ora.odap1.vip             Cluster VIP    L ONLINE     ONLINE       odap1       0  0
ora.odap2.vip             Cluster VIP    L ONLINE     ONLINE       odap2       0  0
ora.gsd                   Gbl Svc Daemon C OFFLINE    OFFLINE      (odap1)     0  0
ora.gsd                   Gbl Svc Daemon C OFFLINE    OFFLINE      (odap2)     0  0
ora.mytestprd.db          database       L ONLINE     ONLINE       odap2       0  0 Open
ora.mytestprd.mytest.svc  service        L ONLINE     ONLINE       odap2       0  0
ora.net1.network          Network (VIP)  C ONLINE     ONLINE       odap1       0  0
ora.net1.network          Network (VIP)  C ONLINE     ONLINE       odap2       0  0
ora.oc4j                  OC4J           L ONLINE     ONLINE       odap2       0  0
ora.ons                   Ora Notif Svc  C ONLINE     ONLINE       odap1       0  0
ora.ons                   Ora Notif Svc  C ONLINE     ONLINE       odap2       0  0
ora.reco.acfsvol.acfs     acfs           C ONLINE     ONLINE       odap1       0  0 mounted on /cloudfs
ora.reco.acfsvol.acfs     acfs           C ONLINE     ONLINE       odap2       0  0 mounted on /cloudfs
ora.registry.acfs         registry       C ONLINE     ONLINE       odap1       0  0
ora.registry.acfs         registry       C ONLINE     ONLINE       odap2       0  0
ora.scan1.vip             SCAN VIP       L ONLINE     ONLINE       odap1       0  0
ora.scan2.vip             SCAN VIP       L ONLINE     ONLINE       odap2       0  0
ora.testdb.db             database       L ONLINE     ONLINE       odap2       0  0 Open
ora.testwz.db             database       L ONLINE     ONLINE       odap1       0  0 Open

After the db is relocated to node 2, I want to do some testings to try out some interesting scenarios

Scenario 1. While db is running on node 2, what’s happened if I try to bring it up on node 1?
Of course, everyone know it should not work for single-instance db. But just want to see what is the result.

SYS@mytestdb> startup nomount
ORA-00304: requested INSTANCE_NUMBER is busy

As I expected, the db can not start. Otherwise, who is going to buy Oracle RAC license!

Scenario 2. what happened if I stop the db from node 2 and start the db on node 1 using sqlplus?

On node 1
oracle:mytestdb@odap1 dbs> srvctl stop database -d mytestdb
oracle:mytestdb@odap1 dbs> sqlplus / as sysdba
SQL*Plus: Release 11.2.0.3.0 Production on Sat Sep 7 10:44:56 2013
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to an idle instance.
SYS@mytestdb> startup nomount
ORACLE instance started.

Total System Global Area 4275781632 bytes
Fixed Size 2235208 bytes
Variable Size 822084792 bytes
Database Buffers 3422552064 bytes
Redo Buffers 28909568 bytes
SYS@mytestdb> alter database mount;
Database altered.

SYS@mytestdb> alter database open;
Database altered.

SYS@mytestdb> col host_name for a20
select instance_name, host_name from v$instance;

INSTANCE_NAME HOST_NAME
—————- ——————–
mytestdb odap1

It’s an interesting result as I am expecting it can not start on the node 1. So what’s happened if shutdown the db and start the db using srvctl on db node 1.

SYS@mytestdb> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SYS@mytestdb> quit

oracle:mytestdb@odap1 dbs> srvctl start database -d mytestprd
oracle:mytestdb@odap1 dbs> srvctl status database -d mytestprd
Instance mytestdb is running on node odap2
oracle:mytestdb@odap1 dbs> srvctl status service -d mytestprd -s mytest
Service mytest is running on instance(s) mytestdb

Not surprise. The db and service were started on db node 2 as expected.

Scenario 3. Based on the scenario 2 with db startup on node 1, can I start service on node 1 as well?

oracle:mytestdb@odap1 dbs> sqlplus / as sysdba
SQL*Plus: Release 11.2.0.3.0 Production on Sat Sep 7 10:59:20 2013
Copyright (c) 1982, 2011, Oracle. All rights reserved.

Connected to an idle instance.
SYS@mytestdb> startup
ORACLE instance started.

Total System Global Area 4275781632 bytes
Fixed Size 2235208 bytes
Variable Size 822084792 bytes
Database Buffers 3422552064 bytes
Redo Buffers 28909568 bytes
Database mounted.
Database opened.
SYS@mytestdb> quit

oracle:mytestdb@odap1 dbs> srvctl status database -d mytestprd
Instance mytestdb is not running on node odap2
oracle:mytestdb@odap1 dbs> srvctl status service -d mytestprd -s mytest
Service mytest is not running.
oracle:mytestdb@odap1 dbs> srvctl start service -d mytestprd -s mytest
PRCD-1084 : Failed to start service mytest
PRCR-1079 : Failed to start resource ora.mytestprd.mytest.svc
CRS-5017: The resource action “ora.mytestprd.db start” encountered the following error:
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Process ID: 0
Session ID: 0 Serial number: 0
. For details refer to “(:CLSN00107:)” in “/u01/app/11.2.0.3/grid/log/odap2/agent/crsd/oraagent_oracle/oraagent_oracle.log”.

CRS-2674: Start of ‘ora.mytestprd.db’ on ‘odap2’ failed
CRS-2632: There are no more servers to try to place resource ‘ora.mytestprd.mytest.svc’ on that would satisfy its placement policy

oracle:mytestdb@odap1 dbs> ls -l /u01/app/11.2.0.3/grid/log/odap2/agent/crsd/oraagent_oracle/oraagent_oracle.log
ls: /u01/app/11.2.0.3/grid/log/odap2/agent/crsd/oraagent_oracle/oraagent_oracle.log: No such file or directory

There is no odap2 directory under /u01/app/11.2.0.3/grid/log on node 1. It is on db node 2.
oracle:mytestdb@odap1 dbs> ls -l /u01/app/11.2.0.3/grid/log
total 12
drwxr-xr-x 2 grid oinstall 4096 Apr 5 22:29 crs
drwxr-xr-t 25 root oinstall 4096 Jul 1 12:10 odap1
drwxrwxr-x 5 grid asmadmin 4096 Jul 1 12:44 diag

oracle:mytestdb@odap1 dbs> crsstat | egrep 'Target|mytest'
Resource Name             Resource Type    Target State    Node            FC RC State Details
ora.mytestprd.db          database       L ONLINE OFFLINE  (odap2)     0  0 Instance Shutdown
ora.mytestprd.mytest.svc  service        L ONLINE OFFLINE  (odap2)     0  0

Therefore, from the above test case, once configured the db starting up on a different node. You can bring up the db from either database nodes using sqlplus. However, if you are using service name, you have to start db from the server you associate db with by the “srvctl modify database” command. In general, this trick can be used in non-RAC environment on ODA, Exadata and any cluster that is using the same GI across multiple nodes.

Advertisements

Change DNS on Exadata

phone

At least at the time I wrote this blog, there is no oracle support document showing how to change DNS on Exadata. So it might be a good idea to show how to do it. Similar to my previous post, Change Time Zone Configuration on Exadata, changing DNS also involves the changes in the four components on Exadata.

  • DB nodes
  • Cell nodes
  • IB Switches
  • Ethernet Switches

The following example, we assume the current DNS servers are using the following two IPs, 192.168.10.12 and 192.168.10.13 and we would like to change nameserver to 192.168.10.14 and 192.168.10.15

Step 1. Change at InfiniBand Switches

1. Logon to the first IB switch as root user.
ssh root@enkx3sw-ib2.enkitec.com

2. Edit file /etc/resolv.conf

cp -p /etc/resolv.conf /etc/resolv.conf.yyyymmdd
vi /etc/resolv.conf

Change the line of
nameserver 192.168.10.12
nameserver 192.168.10.13
to
nameserver 192.168.10.14
nameserver 192.168.10.15
3. Verify the change
Note: Interestingly, there is no nslookup program at CentOS on InfiniBand Switch. So have to use ping a hostname to see whether it can translate hostname to an IP.

[root@enkx3sw-ib2 etc]# cat /etc/redhat-release
CentOS release 5.2 (Final)

[root@enkx3sw-ib2 etc]# nslookup enkitec.com
-bash: nslookup: command not found

[root@enkx3sw-ib2 etc]# ping enkitec.com
PING enkitec.com (192.168.100.19) 56(84) bytes of data.
64 bytes from esert-cloud.enkitec.com (192.168.100.19): icmp_seq=1 ttl=64 time=7.99 ms
64 bytes from esert-cloud.enkitec.com (192.168.100.19): icmp_seq=2 ttl=64 time=5.99 ms
64 bytes from esert-cloud.enkitec.com (192.168.100.19): icmp_seq=3 ttl=64 time=4.99 ms
^C

4. Goto the rest of IB switches to make the similar change

Step 2. Changes at db nodes
1. Logon to db node 1 as root user
ssh root@enkx3db01.enkitec.com

2. Modify /etc/resolv.conf

cp -p /etc/resolv.conf /etc/resolv.conf.yyyymmdd
vi /etc/resolv.conf

Do the similar changes at IB switches

3. Use nslookup command to verify the change

[root@enkx3db01 ~]# nslookup enkx3cel01
Server:		192.168.10.14
Address:	192.168.10.14#53

Name:	enkx3cel01.enkitec.com
Address: 192.168.8.234

4. Repeat the same process to the rest of db nodes

Step 3. Changes at cell nodes and their ILOM’s
When make changes to cell nodes, make sure working on one cell node at a time and ensure all disks are re-synced before proceeding to the next cell node.

1. Make a backup of /opt/oracle.cellos/cell.conf

cd /opt/oracle.cellos
cp -p cell.conf cell.conf.yyyymmdd

2. Shutdown all cell services by running the following command
cellcli -e alter cell shutdown services all

3. Execute ipconf to change DNS entry. Make sure only change DNS entry, not others.

[root@dm01cel01 oracle.cellos]# /opt/oracle.cellos/ipconf
Logging started to /var/log/cellos/ipconf.log
Interface ib0 is Linked.  hca: mlx4_0
Interface ib1 is Linked.  hca: mlx4_0
Interface eth0 is Linked.  driver/mac: igb/00:21:28:f8:6c:de
Interface eth1 is ... Unlinked.  driver/mac: igb/00:21:28:f8:6c:df
Interface eth2 is ... Unlinked.  driver/mac: igb/00:21:28:f8:6c:e0
Interface eth3 is ... Unlinked.  driver/mac: igb/00:21:28:f8:6c:e1

Network interfaces
Name     State      IP address      Netmask         Gateway         Net type     Hostname
ib0      Linked
ib1      Linked
eth0     Linked
eth1     Unlinked
eth2     Unlinked
eth3     Unlinked
Warning. Some network interface(s) are disconnected. Check cables and swicthes and retry
Do you want to retry (y/n) [y]: n

The current nameserver(s): 192.168.10.12 192.168.10.13
Do you want to change it (y/n) [n]: y
Nameserver: 192.168.10.14
Add more nameservers (y/n) [n]: y
Nameserver: 192.168.10.15
Add more nameservers (y/n) [n]: n
The current timezone: America/Chicago
Do you want to change it (y/n) [n]:
The current NTP server(s): 192.168.10.15
Do you want to change it (y/n) [n]:

Network interfaces
Name     State      IP address      Netmask         Gateway         Net type     Hostname
eth0     Linked     192.168.100.73  255.255.255.0   192.168.100.254 Management   enkx3cel01.enkitec.com
eth1     Unlinked
eth2     Unlinked
eth3     Unlinked
bondib0  ib0,ib1    172.30.1.3      255.255.255.0                   Private      enkx3cel01-priv.enkitec.com
Select interface name to configure or press Enter to continue:

Select canonical hostname from the list below
1: enkx3cel01.enkitec.com
2: enkx3cel01-priv.enkitec.com
Canonical fully qualified domain name [1]:

Select default gateway interface from the list below
1: eth0
Default gateway interface [1]:

Canonical hostname: enkx3cel01.enkitec.com
Nameservers: 192.168.10.14 192.168.10.15
Timezone: America/Chicago
NTP servers: 192.168.10.15
Default gateway device: eth0
Network interfaces
Name     State      IP address      Netmask         Gateway         Net type     Hostname
eth0     Linked     192.168.100.73  255.255.255.0   192.168.100.254 Management   enkx3cel01.enkitec.com
eth1     Unlinked
eth2     Unlinked
eth3     Unlinked
bondib0  ib0,ib1    172.30.1.3      255.255.255.0                   Private      enkx3cel01-priv.enkitec.com
Is this correct (y/n) [y]:

Do you want to configure basic ILOM settings (y/n) [y]:
Loading basic configuration settings from ILOM ...
ILOM Fully qualified hostname [enkx3cel01-ilom.enkitec.com]:
ILOM IP address [192.168.100.78]:
ILOM Netmask [255.255.255.0]:
ILOM Gateway or none [192.168.100.254]:
ILOM Nameserver or none [192.168.10.12]: 192.168.10.14
ILOM Use NTP Servers (enabled/disabled) [enabled]:
ILOM First NTP server. Fully qualified hostname or ip address or none [192.168.10.15]:
ILOM Second NTP server. Fully qualified hostname or ip address or none [none]:

Basic ILOM configuration settings:
Hostname             : enkx3cel01-ilom.enkitec.com
IP Address           : 192.168.100.78
Netmask              : 255.255.255.0
Gateway              : 192.168.100.254
DNS servers          : 192.168.10.14
Use NTP servers      : enabled
First NTP server     : 192.168.10.15
Second NTP server    : none
Timezone (read-only) : America/Chicago

Is this correct (y/n) [y]: y
Connected. Use ^D to exit.
-> set /SP/clients/dns nameserver=192.168.10.14
Set 'nameserver' to '192.168.10.14'

-> Session closed
Disconnected

Info. Run /opt/oracle.cellos/validations/init.d/saveconfig
Info. Custom changes have been detected in /etc/resolv.conf
Info. Original file will be saved in /etc/resolv.conf.backupbyExadata

Warning. You modified DNS name server.
         Ensure you also update the Infiniband Switch DNS server
         if the same DNS server was also used by the Infiniband switch.

4. Compare the differences between the following two files:

diff /opt/oracle.cellos/cell.conf /opt/oracle.cellos/cell.conf.yyyymmdd

You will see a lot of difference. But if look at it more closely, they are actually the same one, just move things around the file.

5. Restart all cell services by running the following command
cellcli -e alter cell restart services all

6. Verify cell processes are up by running the following
cellcli -e list cell detail

The last three lines should be in running state

	 cellsrvStatus:     	 running
	 msStatus:          	 running
	 rsStatus:          	 running

7. Regularly run the following command to make sure griddisk status changes from SYNCING to ONLINE

[root@enkx3cel01 ~]# cellcli -e list griddisk attributes name,asmmodestatus
	 DATA_CD_00_enkx3cel01   	 SYNCING
	 DATA_CD_01_enkx3cel01   	 SYNCING
	 DATA_CD_02_enkx3cel01   	 SYNCING
	 DATA_CD_03_enkx3cel01   	 SYNCING
	 DATA_CD_04_enkx3cel01   	 SYNCING
	 DATA_CD_05_enkx3cel01   	 SYNCING
	 DATA_CD_06_enkx3cel01   	 SYNCING
	 DATA_CD_07_enkx3cel01   	 SYNCING
	 DATA_CD_08_enkx3cel01   	 SYNCING
	 DATA_CD_09_enkx3cel01   	 SYNCING
	 DATA_CD_10_enkx3cel01   	 SYNCING
	 DATA_CD_11_enkx3cel01   	 SYNCING
	 DBFS_DG_CD_02_enkx3cel01	 ONLINE
	 DBFS_DG_CD_03_enkx3cel01	 ONLINE
	 DBFS_DG_CD_04_enkx3cel01	 ONLINE
	 DBFS_DG_CD_05_enkx3cel01	 ONLINE
	 DBFS_DG_CD_06_enkx3cel01	 ONLINE
	 DBFS_DG_CD_07_enkx3cel01	 ONLINE
	 DBFS_DG_CD_08_enkx3cel01	 ONLINE
	 DBFS_DG_CD_09_enkx3cel01	 ONLINE
	 DBFS_DG_CD_10_enkx3cel01	 ONLINE
	 DBFS_DG_CD_11_enkx3cel01	 ONLINE
	 RECO_CD_00_enkx3cel01   	 SYNCING
	 RECO_CD_01_enkx3cel01   	 SYNCING
	 RECO_CD_02_enkx3cel01   	 SYNCING
	 RECO_CD_03_enkx3cel01   	 SYNCING
	 RECO_CD_04_enkx3cel01   	 SYNCING
	 RECO_CD_05_enkx3cel01   	 SYNCING
	 RECO_CD_06_enkx3cel01   	 SYNCING
	 RECO_CD_07_enkx3cel01   	 SYNCING
	 RECO_CD_08_enkx3cel01   	 SYNCING
	 RECO_CD_09_enkx3cel01   	 SYNCING
	 RECO_CD_10_enkx3cel01   	 SYNCING
	 RECO_CD_11_enkx3cel01   	 SYNCING

8. This step is optional, only uses when iptables command still shows using old DNS entries.

[root@enkx3cel01 oracle.cellos]# iptables -L -n
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:5042
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:5042 flags:0x17/0x02
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:3260
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:3260 flags:0x17/0x02
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:22
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:22 flags:0x17/0x02
ACCEPT     udp  --  192.168.100.240      0.0.0.0/0           udp spt:123
ACCEPT     tcp  --  192.168.10.14        0.0.0.0/0           tcp spt:53
ACCEPT     udp  --  192.168.10.14        0.0.0.0/0           udp spt:53
ACCEPT     tcp  --  192.168.10.15        0.0.0.0/0           tcp spt:53
ACCEPT     udp  --  192.168.10.15        0.0.0.0/0           udp spt:53
REJECT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpts:1024:65535 flags:0x17/0x02 reject-with icmp-port-unreachable
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpts:1024:65535
REJECT     tcp  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-port-unreachable
REJECT     udp  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-port-unreachable
ACCEPT     udp  --  192.168.100.78       0.0.0.0/0           udp dpt:162
ACCEPT     udp  --  192.168.100.78       0.0.0.0/0           udp spt:623 dpts:1024:65535
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:22 flags:0x17/0x02
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:22
ACCEPT     udp  --  192.168.100.240      0.0.0.0/0           udp spt:123
ACCEPT     tcp  --  192.168.10.14        0.0.0.0/0           tcp spt:53
ACCEPT     udp  --  192.168.10.14        0.0.0.0/0           udp spt:53
ACCEPT     tcp  --  192.168.10.15        0.0.0.0/0           tcp spt:53
ACCEPT     udp  --  192.168.10.15        0.0.0.0/0           udp spt:53
REJECT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpts:1024:65535 flags:0x17/0x02 reject-with icmp-port-unreachable
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpts:1024:65535
REJECT     tcp  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-port-unreachable
REJECT     udp  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-port-unreachable
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0
REJECT     udp  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-port-unreachable
REJECT     tcp  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-port-unreachable

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

If you see there are still entries for old DNS, like 192.168.10.12 or 192.168.10.13, then you need to restart cellwall service. Cellwall implements firewall services on each cell using IPTables.

service cellwall restart

9. Verify the result using nslookup command.

Change Time Zone Configuration on Exadata

time

If you have an Exadata system that is located in west coast, and your company then changes mind and decide to move it to east coast. The chance is high you need to change timezone on the Exadata system. There is an Oracle support document about the changing timezone, HOW TO CHANGE TIME ZONE SETTINGS ON AN ORACLE SUN DATABASE MACHINE ENVIRONMENT (Doc ID 1099573.1). However, this document is not up to date and have some minor errors. So I thought it might be a good idea to document the procedures to change timezone. In addition to the steps in the document, I also add a few more steps.

I wish changing timezone on Exadata can be as easy as click a button on a GUI and everything uses the correct timezone automatically. Unfortunately this is not the case on Exadata. If you have done the work to change default passwords on Exadata, you have alreday known the pain to go through every hardware component to change password. Similarly like changing default password on Exadata, changing timezone also involves many steps to change the following all major four hardware componets as follows:

  • DB nodes
  • Cell nodes
  • IB Switches
  • Ethernet Switches

Important note: The oracle document does not mention any way to do the rolling change on db or cell node one by one. Therefore, it requires a down time for all databases on the Exadata. So if it is in production, plan ahead. Interestingly, the change of DNS on Exadata can be done in a rolling fashion and I will discuss it in the future post.

Here are the steps:

Step 1. Stop CRS on all DB nodes

1. Logon to db node 1 as root user
2. Stopy CRS by running the following command
/u01/app/11.2.0.3/grid/bin/crsctl stop crs

During the shutdown process of CRS, run the following command regularly to check the number of oracle processes. It should reduce to 0 when CRS is stopped
ps -ef|grep d.bin|grep -v grep|wc -l

3. After db node 1 completes, repeat the same stop crs process on the rest of db nodes

Step 2. Stop processes on all CELL nodes

1. Logon to cell node 1 as root user

2. Run the following command to check current status

CellCLI> list cell detail
	 name:              	 enkx3cel01
	 bbuTempThreshold:  	 60
	 bbuChargeThreshold:	 800
	 bmcType:           	 IPMI
	 cellVersion:       	 OSS_11.2.3.2.1_LINUX.X64_130109
	 cpuCount:          	 12
	 diagHistoryDays:   	 7
	 fanCount:          	 8/8
	 fanStatus:         	 normal
	 flashCacheMode:    	 WriteBack
	 id:                	 1246FM501R
	 interconnectCount: 	 3
	 interconnect1:     	 bondib0
	 iormBoost:         	 0.0
	 ipaddress1:        	 192.168.12.3/24
	 kernelVersion:     	 2.6.32-400.11.1.el5uek
	 locatorLEDStatus:  	 off
	 makeModel:         	 Oracle Corporation SUN FIRE X4270 M3 SAS
	 metricHistoryDays: 	 7
	 notificationMethod:	 snmp,mail
	 notificationPolicy:	 critical,warning,clear
	 offloadEfficiency: 	 405.7
	 powerCount:        	 2/2
	 powerStatus:       	 normal
	 releaseVersion:    	 11.2.3.2.1
	 releaseTrackingBug:	 14522699
	 smtpFrom:          	 "Enkitec X3"
	 smtpFromAddr:      	 enkitec_x3@enkitec.com
	 smtpPort:          	 25
	 smtpServer:        	 192.168.10.15
	 smtpToAddr:        	 exadata_dba@enkitec.com
	 smtpUser:
	 smtpUseSSL:        	 FALSE
	 snmpSubscriber:    	 host=enkx3db02.enkitec.com,port=3878,community=public
	                    	 host=enkx3db01.enkitec.com,port=3878,community=public
	 status:            	 online
	 temperatureReading:	 21.0
	 temperatureStatus: 	 normal
	 upTime:            	 0 days, 5:28
	 cellsrvStatus:     	 running
	 msStatus:          	 running
	 rsStatus:          	 running

Pay attention to the last 3 lines of status for the processes

3. Shutdown the services
cellcli -e alter cell shutdown services all

4. Repeat the above shutdown services processes for the rest cell nodes.

Step 3. Change timezone on all CELL nodes
1. Logon to cell node 1 as root user

2. Make a backup copy of /opt/oracle.cellos/cell.conf, /etc/sysconfig/clock and /etc/localtime files. Change yyyymmdd to the datestamp you like.

cd /opt/oracle.cellos
cp -p cell.conf cell.conf.yyyymmdd
cd /etc/sysconfig
cp -p clock clock.yyyymmdd
cd /etc
cp -p localtime localtime.yyyymmdd

A sample of cell.conf file

[root@enkx3cel01 ~]# cat /opt/oracle.cellos/cell.conf
$VAR1 = {
          'Internal' => {
                          'Interface infiniband prefix' => 'ib',
                          'Interface ethernet prefix' => 'eth'
                        },
          'Hostname' => 'enkx3cel01.enkitec.com',
          'Timezone' => 'America/Chicago',
          'Ntp drift' => '/var/lib/ntp/drift',
          'Interfaces' => [
                            {
                              'State' => 1,
                              'Name' => 'ib0',
                              'Master' => 'bondib0'
                            },
                            {
                              'State' => 1,
                              'Name' => 'ib1',
                              'Master' => 'bondib0'
                            },
                            {
                              'Hostname' => 'enkx3cel01.enkitec.com',
                              'IP address' => '192.168.8.234',
                              'Net type' => 'Management',
                              'Netmask' => '255.255.252.0',
                              'State' => 1,
                              'Name' => 'eth0',
                              'Gateway' => '192.168.10.1'
                            },
                            {
                              'State' => 0,
                              'Name' => 'eth1'
                            },
                            {
                              'State' => 0,
                              'Name' => 'eth2'
                            },
                            {
                              'State' => 0,
                              'Name' => 'eth3'
                            },
                            {
                              'Hostname' => 'enkx3cel01-priv.enkitec.com',
                              'IP address' => '192.168.12.3',
                              'Net type' => 'Private',
                              'Netmask' => '255.255.255.0',
                              'Slaves' => [
                                            'ib0',
                                            'ib1'
                                          ],
                              'State' => 1,
                              'Name' => 'bondib0'
                            }
                          ],
          'Version' => '11.2.3.2.0',
          'Ntp servers' => [
                             '192.168.10.15'
                           ],
          'Nameservers' => [
                             '192.168.10.19',
                             '192.168.10.15'
                           ],
          'Unlinked interfaces' => [],
          'Node type' => 'cell',
          'Default gateway device' => 'eth0',
          'ilom' => {
                      'ILOM Nameserver' => '192.168.10.19',
                      'ILOM Netmask' => '255.255.252.0',
                      'ILOM IP address' => '192.168.8.239',
                      'ILOM Timezone' => 'America/Chicago',
                      'ILOM IP discovery (static/dhcp)' => 'static',
                      'ILOM Search' => 'enkitec.com',
                      'ILOM Second NTP server' => '0.0.0.0',
                      'ILOM Short Hostname' => 'enkx3cel01-ilom',
                      'ILOM Fully qualified hostname' => 'enkx3cel01-ilom.enkitec.com',
                      'ILOM First NTP server' => '192.168.10.15',
                      'ILOM Use NTP Servers' => 'enabled',
                      'ILOM Gateway' => '192.168.10.1'
                    }
        };

A sample of clock file

[root@enkx3cel01 ~]# cat /etc/sysconfig/clock
ZONE="America/Chicago"
UTC=true
ARC=false

A sample of localtime file
Note: this file is binary file, you need to use strings command to check

[root@enkx3cel01 ~]# strings /etc/localtime
TZif2
Dp/~Y
&p1gv
p3GX
p5':
CdopD/
EDQpE
NpM|y
0pO\[
pQ<=
ypb-
cg[pd
eG=pe
Jp|~u
,p~^W
TZif2
Cdop
EDQp
cg[p
eG=p
CST6CDT,M3.2.0,M11.1.0

Pay attention to the last line. It should report same timezone referenced on the previous files.

3. Run date command to check the value for current timezone

[root@enkx3cel01 ~]# date
Sun Aug 25 17:16:01 CDT 2013

4. Use ipconf to generate new cell.conf, clock and localtime file
/opt/oracle.cellos/ipconf

Important Note:

  • There will be some questions regarding other components like network interfaces which shouldn’t be modified.
  • Continue until getting into the Time Zone questions. Each Country is identified by a number and once it is selected, will be another set of numbers for the different time zones.
  • (Very Important) Do NOT change ILOM settings. Process will complete generating the new files.

If not all cell services are not shutdown, you will see the following error

[enkcel01:root] /opt/oracle.cellos/ipconf
Logging started to /var/log/cellos/ipconf.log
Error. All CELL services must be stopped prior to using ipconf

Below is a partial output of ipconf execution. As I didn’t capture the output, so I use the content from Oracle Support document. There is one error in the Oracle Support Document. The country code for US is 233, not 230.

[root@dmorlcel01 oracle.cellos]# ipconf
Logging started to /var/log/cellos/ipconf.log
Interface eth0 is Linked.
Interface eth1 is ... Unlinked.
Interface eth2 is ... Unlinked.
Interface eth3 is ... Unlinked.
Interface ib0 is Linked.
Interface ib1 is Linked.

Network interfaces
Name  State      IP address      Netmask         Gateway         Hostname
eth0  Linked
eth1  Unlinked
eth2  Unlinked
eth3  Unlinked
ib0   Linked
ib1   Linked
Warning. Some network interface(s) are disconnected. Check cables and swicthes and retry
Do you want to retry (y/n) [y]: n

The current nameserver(s): 138.2.202.15 130.35.249.52 130.35.249.41
Do you want to change it (y/n) [n]:
The current timezone: America/New_York

The current timezone: Antarctica/McMurdo
Do you want to change it (y/n) [n]: y

Setting up local time...

1) Andorra
2) United Arab Emirates
3) Afghanistan
4) Antigua & Barbuda
5) Anguilla
6) Albania
7) Armenia
8) Netherlands Antilles
9) Angola
10) Antarctica
11) Argentina
12) Samoa (American)
13) Austria
14) Australia
15) Aruba
16) Aaland Islands
Select country by number, [n]ext, [l]ast:

The number for United States is 233    (230) wrong

Selected country: United States (US). Now choose a zone

1) America/New_York
2) America/Detroit
3) America/Kentucky/Louisville
4) America/Kentucky/Monticello
5) America/Indiana/Indianapolis
6) America/Indiana/Knox
7) America/Indiana/Winamac
8) America/Indiana/Marengo
9) America/Indiana/Vevay
10) America/Chicago
11) America/Indiana/Vincennes
12) America/Indiana/Petersburg
13) America/Menominee
14) America/North_Dakota/Center
15) America/North_Dakota/New_Salem
16) America/Denver
Select zone by number, [n]ext: 1
Selected timezone: America/New_York
Is this correct (y/n) [y]:

Continue to answer the next questions without making additional changes:

Do you want to configure basic ILOM settings (y/n) [y]: n
Info. Run /opt/oracle.cellos/validations/init.d/saveconfig
/opt/oracle.cellos/validations/init.d/saveconfig started at 2010_05_12_10_28
Copy cell configs from /opt/oracle/cell/cellsrv/deploy/config to /opt/oracle.cellos/iso/lastGoodConfig/cell/cellsrv/deploy/config
[INFO] Copying ssh host settings from //etc/ssh to /opt/oracle.cellos/iso/lastGoodConfig/etc/ssh ...
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel)

5. After ipconf finishes, verify the changes

cd /opt/oracle.cellos
diff cell.conf cell.conf.yyyymmdd
cd /etc/sysconfig
diff  clock clock.yyyymmdd
cd /etc
strings localtime

At first look, you might notice new cell.conf seems dramatically different than the saved backup copy of cell.conf.yyyymmdd. If you look at more detail, you will notice they actually are the same one and the new one just move some parts to other location of the same document.

6. If everything looks fine, reboot the current cell node.

7. After reboot, type command
date
It should show the correct timezone.
Check Reviewing $ADR_BASE/diag/asm/cell//alert.log, the time when processes were restated should match the current and correct time.

8. Repeat the same process for the rest of cell nodes.

Step 4. Change timezone on all DB nodes
1. Logon to db node 1 as root user

2. Make sure all CRS stack is down
ps -ef|grep d.bin|grep -v grep|wc -l
It should return 0

3. Make a backup of /etc/localtime and /etc/sysconfig/clock files

cd /etc/sysconfig
cp -p clock clock.yyyymmdd
cd /etc
cp -p localtime localtime.yyyymmdd

4. Copy clock and localtime file from cell node 1

 scp root@:/etc/localtime /etc/localtime
 scp root@:/etc/sysconfig/clock /etc/sysconfig/clock

5. Change CRS settings to avoid starting it after the server reboot
/u01/app/11.2.0.3/grid/bin/crsctl disable crs

6. Reboot the current db node

7. Validate date has been changed using date command

8. Change CRS settings back to start after reboot
/u01/app/11.2.0.3/grid/bin/crsctl enable crs

9. Start CRS on the current db node
/u01/app/11.2.0.3/grid/bin/crsctl start crs

10. After it fully starts, repeat the same timezone change process on the rest of db nodes.

Step 5. Change timezone on all IB Switches
1. Logon to IB switch 1 as root user

2. Oracle support document says verify the version using nm2version command. The nm2version command is actually deprecated and should use version command instead.

[root@enkx3sw-ib2 ~]# version
SUN DCS 36p version: 1.3.3-2
Build time: Apr  4 2011 11:15:19
SP board info:
Manufacturing Date: 2012.09.04
Serial Number: "NCDA11112"
Hardware Revision: 0x0007
Firmware Revision: 0x0000
BIOS version: SUN0R100
BIOS date: 06/22/2010

As the software version is 1.3.3-2, higher than 1.1.3­2 or higher, the administration of the switch can be done through ILOM.

3. Logon to IB switch ILOM as root user.
http://enkx3sw-ib2.enkitec.com

4. After logon, Click Configuration tab, then click Clock tab
Make sure check field for Synchronize Time Using NTP, is enabled and There is an ip in the Server 1 field.

5. Oracle document miss this step. Then click TimeZone tab on the right of Clock tab. Choose the correct Timezone, then click Save button.

6. Repeat the same process for the rest IB Switches

Step 6. Change timezone on Cisco Ethernet Switch
There is another error in the Oracle support document. For X3 Exadata, it is using ssh, not telnet, to connect to Ethernet Switch. The user name is not root, but admin.

ssh admin@enkx3sw-ip.enkitec.com

Connected to enkx3sw-ip
Escape character is '^]'.
User Access Verification
Password:
enkx3sw-ip>enable
Password:
enkx3sw-ip#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
enkx3sw-ip(config)#clock timezone EST -­5
enkx3sw-ip(config)#clock summer­-time EDT recurring
enkx3sw-ip(config)#end

enkx3sw-ip#write memory
Building configuration...
Compressed configuration from 6421 bytes to 2041 bytes[OK]
enkx3sw-ip>show clock
17:30:23.681 EDT Sun Aug 25 2013
enkx3sw-ip#

This is the end of changing timezone on Exadata. In the next post, I will show the steps to change DNS on Exadata.