Change Time Zone Configuration on Exadata

time

If you have an Exadata system that is located in west coast, and your company then changes mind and decide to move it to east coast. The chance is high you need to change timezone on the Exadata system. There is an Oracle support document about the changing timezone, HOW TO CHANGE TIME ZONE SETTINGS ON AN ORACLE SUN DATABASE MACHINE ENVIRONMENT (Doc ID 1099573.1). However, this document is not up to date and have some minor errors. So I thought it might be a good idea to document the procedures to change timezone. In addition to the steps in the document, I also add a few more steps.

I wish changing timezone on Exadata can be as easy as click a button on a GUI and everything uses the correct timezone automatically. Unfortunately this is not the case on Exadata. If you have done the work to change default passwords on Exadata, you have alreday known the pain to go through every hardware component to change password. Similarly like changing default password on Exadata, changing timezone also involves many steps to change the following all major four hardware componets as follows:

  • DB nodes
  • Cell nodes
  • IB Switches
  • Ethernet Switches

Important note: The oracle document does not mention any way to do the rolling change on db or cell node one by one. Therefore, it requires a down time for all databases on the Exadata. So if it is in production, plan ahead. Interestingly, the change of DNS on Exadata can be done in a rolling fashion and I will discuss it in the future post.

Here are the steps:

Step 1. Stop CRS on all DB nodes

1. Logon to db node 1 as root user
2. Stopy CRS by running the following command
/u01/app/11.2.0.3/grid/bin/crsctl stop crs

During the shutdown process of CRS, run the following command regularly to check the number of oracle processes. It should reduce to 0 when CRS is stopped
ps -ef|grep d.bin|grep -v grep|wc -l

3. After db node 1 completes, repeat the same stop crs process on the rest of db nodes

Step 2. Stop processes on all CELL nodes

1. Logon to cell node 1 as root user

2. Run the following command to check current status

CellCLI> list cell detail
	 name:              	 enkx3cel01
	 bbuTempThreshold:  	 60
	 bbuChargeThreshold:	 800
	 bmcType:           	 IPMI
	 cellVersion:       	 OSS_11.2.3.2.1_LINUX.X64_130109
	 cpuCount:          	 12
	 diagHistoryDays:   	 7
	 fanCount:          	 8/8
	 fanStatus:         	 normal
	 flashCacheMode:    	 WriteBack
	 id:                	 1246FM501R
	 interconnectCount: 	 3
	 interconnect1:     	 bondib0
	 iormBoost:         	 0.0
	 ipaddress1:        	 192.168.12.3/24
	 kernelVersion:     	 2.6.32-400.11.1.el5uek
	 locatorLEDStatus:  	 off
	 makeModel:         	 Oracle Corporation SUN FIRE X4270 M3 SAS
	 metricHistoryDays: 	 7
	 notificationMethod:	 snmp,mail
	 notificationPolicy:	 critical,warning,clear
	 offloadEfficiency: 	 405.7
	 powerCount:        	 2/2
	 powerStatus:       	 normal
	 releaseVersion:    	 11.2.3.2.1
	 releaseTrackingBug:	 14522699
	 smtpFrom:          	 "Enkitec X3"
	 smtpFromAddr:      	 enkitec_x3@enkitec.com
	 smtpPort:          	 25
	 smtpServer:        	 192.168.10.15
	 smtpToAddr:        	 exadata_dba@enkitec.com
	 smtpUser:
	 smtpUseSSL:        	 FALSE
	 snmpSubscriber:    	 host=enkx3db02.enkitec.com,port=3878,community=public
	                    	 host=enkx3db01.enkitec.com,port=3878,community=public
	 status:            	 online
	 temperatureReading:	 21.0
	 temperatureStatus: 	 normal
	 upTime:            	 0 days, 5:28
	 cellsrvStatus:     	 running
	 msStatus:          	 running
	 rsStatus:          	 running

Pay attention to the last 3 lines of status for the processes

3. Shutdown the services
cellcli -e alter cell shutdown services all

4. Repeat the above shutdown services processes for the rest cell nodes.

Step 3. Change timezone on all CELL nodes
1. Logon to cell node 1 as root user

2. Make a backup copy of /opt/oracle.cellos/cell.conf, /etc/sysconfig/clock and /etc/localtime files. Change yyyymmdd to the datestamp you like.

cd /opt/oracle.cellos
cp -p cell.conf cell.conf.yyyymmdd
cd /etc/sysconfig
cp -p clock clock.yyyymmdd
cd /etc
cp -p localtime localtime.yyyymmdd

A sample of cell.conf file

[root@enkx3cel01 ~]# cat /opt/oracle.cellos/cell.conf
$VAR1 = {
          'Internal' => {
                          'Interface infiniband prefix' => 'ib',
                          'Interface ethernet prefix' => 'eth'
                        },
          'Hostname' => 'enkx3cel01.enkitec.com',
          'Timezone' => 'America/Chicago',
          'Ntp drift' => '/var/lib/ntp/drift',
          'Interfaces' => [
                            {
                              'State' => 1,
                              'Name' => 'ib0',
                              'Master' => 'bondib0'
                            },
                            {
                              'State' => 1,
                              'Name' => 'ib1',
                              'Master' => 'bondib0'
                            },
                            {
                              'Hostname' => 'enkx3cel01.enkitec.com',
                              'IP address' => '192.168.8.234',
                              'Net type' => 'Management',
                              'Netmask' => '255.255.252.0',
                              'State' => 1,
                              'Name' => 'eth0',
                              'Gateway' => '192.168.10.1'
                            },
                            {
                              'State' => 0,
                              'Name' => 'eth1'
                            },
                            {
                              'State' => 0,
                              'Name' => 'eth2'
                            },
                            {
                              'State' => 0,
                              'Name' => 'eth3'
                            },
                            {
                              'Hostname' => 'enkx3cel01-priv.enkitec.com',
                              'IP address' => '192.168.12.3',
                              'Net type' => 'Private',
                              'Netmask' => '255.255.255.0',
                              'Slaves' => [
                                            'ib0',
                                            'ib1'
                                          ],
                              'State' => 1,
                              'Name' => 'bondib0'
                            }
                          ],
          'Version' => '11.2.3.2.0',
          'Ntp servers' => [
                             '192.168.10.15'
                           ],
          'Nameservers' => [
                             '192.168.10.19',
                             '192.168.10.15'
                           ],
          'Unlinked interfaces' => [],
          'Node type' => 'cell',
          'Default gateway device' => 'eth0',
          'ilom' => {
                      'ILOM Nameserver' => '192.168.10.19',
                      'ILOM Netmask' => '255.255.252.0',
                      'ILOM IP address' => '192.168.8.239',
                      'ILOM Timezone' => 'America/Chicago',
                      'ILOM IP discovery (static/dhcp)' => 'static',
                      'ILOM Search' => 'enkitec.com',
                      'ILOM Second NTP server' => '0.0.0.0',
                      'ILOM Short Hostname' => 'enkx3cel01-ilom',
                      'ILOM Fully qualified hostname' => 'enkx3cel01-ilom.enkitec.com',
                      'ILOM First NTP server' => '192.168.10.15',
                      'ILOM Use NTP Servers' => 'enabled',
                      'ILOM Gateway' => '192.168.10.1'
                    }
        };

A sample of clock file

[root@enkx3cel01 ~]# cat /etc/sysconfig/clock
ZONE="America/Chicago"
UTC=true
ARC=false

A sample of localtime file
Note: this file is binary file, you need to use strings command to check

[root@enkx3cel01 ~]# strings /etc/localtime
TZif2
Dp/~Y
&p1gv
p3GX
p5':
CdopD/
EDQpE
NpM|y
0pO\[
pQ<=
ypb-
cg[pd
eG=pe
Jp|~u
,p~^W
TZif2
Cdop
EDQp
cg[p
eG=p
CST6CDT,M3.2.0,M11.1.0

Pay attention to the last line. It should report same timezone referenced on the previous files.

3. Run date command to check the value for current timezone

[root@enkx3cel01 ~]# date
Sun Aug 25 17:16:01 CDT 2013

4. Use ipconf to generate new cell.conf, clock and localtime file
/opt/oracle.cellos/ipconf

Important Note:

  • There will be some questions regarding other components like network interfaces which shouldn’t be modified.
  • Continue until getting into the Time Zone questions. Each Country is identified by a number and once it is selected, will be another set of numbers for the different time zones.
  • (Very Important) Do NOT change ILOM settings. Process will complete generating the new files.

If not all cell services are not shutdown, you will see the following error

[enkcel01:root] /opt/oracle.cellos/ipconf
Logging started to /var/log/cellos/ipconf.log
Error. All CELL services must be stopped prior to using ipconf

Below is a partial output of ipconf execution. As I didn’t capture the output, so I use the content from Oracle Support document. There is one error in the Oracle Support Document. The country code for US is 233, not 230.

[root@dmorlcel01 oracle.cellos]# ipconf
Logging started to /var/log/cellos/ipconf.log
Interface eth0 is Linked.
Interface eth1 is ... Unlinked.
Interface eth2 is ... Unlinked.
Interface eth3 is ... Unlinked.
Interface ib0 is Linked.
Interface ib1 is Linked.

Network interfaces
Name  State      IP address      Netmask         Gateway         Hostname
eth0  Linked
eth1  Unlinked
eth2  Unlinked
eth3  Unlinked
ib0   Linked
ib1   Linked
Warning. Some network interface(s) are disconnected. Check cables and swicthes and retry
Do you want to retry (y/n) [y]: n

The current nameserver(s): 138.2.202.15 130.35.249.52 130.35.249.41
Do you want to change it (y/n) [n]:
The current timezone: America/New_York

The current timezone: Antarctica/McMurdo
Do you want to change it (y/n) [n]: y

Setting up local time...

1) Andorra
2) United Arab Emirates
3) Afghanistan
4) Antigua & Barbuda
5) Anguilla
6) Albania
7) Armenia
8) Netherlands Antilles
9) Angola
10) Antarctica
11) Argentina
12) Samoa (American)
13) Austria
14) Australia
15) Aruba
16) Aaland Islands
Select country by number, [n]ext, [l]ast:

The number for United States is 233    (230) wrong

Selected country: United States (US). Now choose a zone

1) America/New_York
2) America/Detroit
3) America/Kentucky/Louisville
4) America/Kentucky/Monticello
5) America/Indiana/Indianapolis
6) America/Indiana/Knox
7) America/Indiana/Winamac
8) America/Indiana/Marengo
9) America/Indiana/Vevay
10) America/Chicago
11) America/Indiana/Vincennes
12) America/Indiana/Petersburg
13) America/Menominee
14) America/North_Dakota/Center
15) America/North_Dakota/New_Salem
16) America/Denver
Select zone by number, [n]ext: 1
Selected timezone: America/New_York
Is this correct (y/n) [y]:

Continue to answer the next questions without making additional changes:

Do you want to configure basic ILOM settings (y/n) [y]: n
Info. Run /opt/oracle.cellos/validations/init.d/saveconfig
/opt/oracle.cellos/validations/init.d/saveconfig started at 2010_05_12_10_28
Copy cell configs from /opt/oracle/cell/cellsrv/deploy/config to /opt/oracle.cellos/iso/lastGoodConfig/cell/cellsrv/deploy/config
[INFO] Copying ssh host settings from //etc/ssh to /opt/oracle.cellos/iso/lastGoodConfig/etc/ssh ...
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel)

5. After ipconf finishes, verify the changes

cd /opt/oracle.cellos
diff cell.conf cell.conf.yyyymmdd
cd /etc/sysconfig
diff  clock clock.yyyymmdd
cd /etc
strings localtime

At first look, you might notice new cell.conf seems dramatically different than the saved backup copy of cell.conf.yyyymmdd. If you look at more detail, you will notice they actually are the same one and the new one just move some parts to other location of the same document.

6. If everything looks fine, reboot the current cell node.

7. After reboot, type command
date
It should show the correct timezone.
Check Reviewing $ADR_BASE/diag/asm/cell//alert.log, the time when processes were restated should match the current and correct time.

8. Repeat the same process for the rest of cell nodes.

Step 4. Change timezone on all DB nodes
1. Logon to db node 1 as root user

2. Make sure all CRS stack is down
ps -ef|grep d.bin|grep -v grep|wc -l
It should return 0

3. Make a backup of /etc/localtime and /etc/sysconfig/clock files

cd /etc/sysconfig
cp -p clock clock.yyyymmdd
cd /etc
cp -p localtime localtime.yyyymmdd

4. Copy clock and localtime file from cell node 1

 scp root@:/etc/localtime /etc/localtime
 scp root@:/etc/sysconfig/clock /etc/sysconfig/clock

5. Change CRS settings to avoid starting it after the server reboot
/u01/app/11.2.0.3/grid/bin/crsctl disable crs

6. Reboot the current db node

7. Validate date has been changed using date command

8. Change CRS settings back to start after reboot
/u01/app/11.2.0.3/grid/bin/crsctl enable crs

9. Start CRS on the current db node
/u01/app/11.2.0.3/grid/bin/crsctl start crs

10. After it fully starts, repeat the same timezone change process on the rest of db nodes.

Step 5. Change timezone on all IB Switches
1. Logon to IB switch 1 as root user

2. Oracle support document says verify the version using nm2version command. The nm2version command is actually deprecated and should use version command instead.

[root@enkx3sw-ib2 ~]# version
SUN DCS 36p version: 1.3.3-2
Build time: Apr  4 2011 11:15:19
SP board info:
Manufacturing Date: 2012.09.04
Serial Number: "NCDA11112"
Hardware Revision: 0x0007
Firmware Revision: 0x0000
BIOS version: SUN0R100
BIOS date: 06/22/2010

As the software version is 1.3.3-2, higher than 1.1.3­2 or higher, the administration of the switch can be done through ILOM.

3. Logon to IB switch ILOM as root user.
http://enkx3sw-ib2.enkitec.com

4. After logon, Click Configuration tab, then click Clock tab
Make sure check field for Synchronize Time Using NTP, is enabled and There is an ip in the Server 1 field.

5. Oracle document miss this step. Then click TimeZone tab on the right of Clock tab. Choose the correct Timezone, then click Save button.

6. Repeat the same process for the rest IB Switches

Step 6. Change timezone on Cisco Ethernet Switch
There is another error in the Oracle support document. For X3 Exadata, it is using ssh, not telnet, to connect to Ethernet Switch. The user name is not root, but admin.

ssh admin@enkx3sw-ip.enkitec.com

Connected to enkx3sw-ip
Escape character is '^]'.
User Access Verification
Password:
enkx3sw-ip>enable
Password:
enkx3sw-ip#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
enkx3sw-ip(config)#clock timezone EST -­5
enkx3sw-ip(config)#clock summer­-time EDT recurring
enkx3sw-ip(config)#end

enkx3sw-ip#write memory
Building configuration...
Compressed configuration from 6421 bytes to 2041 bytes[OK]
enkx3sw-ip>show clock
17:30:23.681 EDT Sun Aug 25 2013
enkx3sw-ip#

This is the end of changing timezone on Exadata. In the next post, I will show the steps to change DNS on Exadata.

Advertisements

3 thoughts on “Change Time Zone Configuration on Exadata

  1. Pingback: Change DNS on Exadata | My Big Data World

    • Hi, David,
      Thanks a lot for your information. This blog was written based on the work I did about three years ago on a X3 Exadata. At the time of writing of the blog, the original Oracle Doc 1099573.1 did not contain this s_crsconfig__env.txt file and I was not aware of this file at the time. I did check out the original Oracle Doc I saved and the modification date of Doc 1099573.1 was May 23, 2012. Thank you for letting me know this new update and I will keep in mind if I need to do the same work again in the future.

      Thanks,

      Weidong

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s