Change DNS on Exadata

phone

At least at the time I wrote this blog, there is no oracle support document showing how to change DNS on Exadata. So it might be a good idea to show how to do it. Similar to my previous post, Change Time Zone Configuration on Exadata, changing DNS also involves the changes in the four components on Exadata.

  • DB nodes
  • Cell nodes
  • IB Switches
  • Ethernet Switches

The following example, we assume the current DNS servers are using the following two IPs, 192.168.10.12 and 192.168.10.13 and we would like to change nameserver to 192.168.10.14 and 192.168.10.15

Step 1. Change at InfiniBand Switches

1. Logon to the first IB switch as root user.
ssh root@enkx3sw-ib2.enkitec.com

2. Edit file /etc/resolv.conf

cp -p /etc/resolv.conf /etc/resolv.conf.yyyymmdd
vi /etc/resolv.conf

Change the line of
nameserver 192.168.10.12
nameserver 192.168.10.13
to
nameserver 192.168.10.14
nameserver 192.168.10.15
3. Verify the change
Note: Interestingly, there is no nslookup program at CentOS on InfiniBand Switch. So have to use ping a hostname to see whether it can translate hostname to an IP.

[root@enkx3sw-ib2 etc]# cat /etc/redhat-release
CentOS release 5.2 (Final)

[root@enkx3sw-ib2 etc]# nslookup enkitec.com
-bash: nslookup: command not found

[root@enkx3sw-ib2 etc]# ping enkitec.com
PING enkitec.com (192.168.100.19) 56(84) bytes of data.
64 bytes from esert-cloud.enkitec.com (192.168.100.19): icmp_seq=1 ttl=64 time=7.99 ms
64 bytes from esert-cloud.enkitec.com (192.168.100.19): icmp_seq=2 ttl=64 time=5.99 ms
64 bytes from esert-cloud.enkitec.com (192.168.100.19): icmp_seq=3 ttl=64 time=4.99 ms
^C

4. Goto the rest of IB switches to make the similar change

Step 2. Changes at db nodes
1. Logon to db node 1 as root user
ssh root@enkx3db01.enkitec.com

2. Modify /etc/resolv.conf

cp -p /etc/resolv.conf /etc/resolv.conf.yyyymmdd
vi /etc/resolv.conf

Do the similar changes at IB switches

3. Use nslookup command to verify the change

[root@enkx3db01 ~]# nslookup enkx3cel01
Server:		192.168.10.14
Address:	192.168.10.14#53

Name:	enkx3cel01.enkitec.com
Address: 192.168.8.234

4. Repeat the same process to the rest of db nodes

Step 3. Changes at cell nodes and their ILOM’s
When make changes to cell nodes, make sure working on one cell node at a time and ensure all disks are re-synced before proceeding to the next cell node.

1. Make a backup of /opt/oracle.cellos/cell.conf

cd /opt/oracle.cellos
cp -p cell.conf cell.conf.yyyymmdd

2. Shutdown all cell services by running the following command
cellcli -e alter cell shutdown services all

3. Execute ipconf to change DNS entry. Make sure only change DNS entry, not others.

[root@dm01cel01 oracle.cellos]# /opt/oracle.cellos/ipconf
Logging started to /var/log/cellos/ipconf.log
Interface ib0 is Linked.  hca: mlx4_0
Interface ib1 is Linked.  hca: mlx4_0
Interface eth0 is Linked.  driver/mac: igb/00:21:28:f8:6c:de
Interface eth1 is ... Unlinked.  driver/mac: igb/00:21:28:f8:6c:df
Interface eth2 is ... Unlinked.  driver/mac: igb/00:21:28:f8:6c:e0
Interface eth3 is ... Unlinked.  driver/mac: igb/00:21:28:f8:6c:e1

Network interfaces
Name     State      IP address      Netmask         Gateway         Net type     Hostname
ib0      Linked
ib1      Linked
eth0     Linked
eth1     Unlinked
eth2     Unlinked
eth3     Unlinked
Warning. Some network interface(s) are disconnected. Check cables and swicthes and retry
Do you want to retry (y/n) [y]: n

The current nameserver(s): 192.168.10.12 192.168.10.13
Do you want to change it (y/n) [n]: y
Nameserver: 192.168.10.14
Add more nameservers (y/n) [n]: y
Nameserver: 192.168.10.15
Add more nameservers (y/n) [n]: n
The current timezone: America/Chicago
Do you want to change it (y/n) [n]:
The current NTP server(s): 192.168.10.15
Do you want to change it (y/n) [n]:

Network interfaces
Name     State      IP address      Netmask         Gateway         Net type     Hostname
eth0     Linked     192.168.100.73  255.255.255.0   192.168.100.254 Management   enkx3cel01.enkitec.com
eth1     Unlinked
eth2     Unlinked
eth3     Unlinked
bondib0  ib0,ib1    172.30.1.3      255.255.255.0                   Private      enkx3cel01-priv.enkitec.com
Select interface name to configure or press Enter to continue:

Select canonical hostname from the list below
1: enkx3cel01.enkitec.com
2: enkx3cel01-priv.enkitec.com
Canonical fully qualified domain name [1]:

Select default gateway interface from the list below
1: eth0
Default gateway interface [1]:

Canonical hostname: enkx3cel01.enkitec.com
Nameservers: 192.168.10.14 192.168.10.15
Timezone: America/Chicago
NTP servers: 192.168.10.15
Default gateway device: eth0
Network interfaces
Name     State      IP address      Netmask         Gateway         Net type     Hostname
eth0     Linked     192.168.100.73  255.255.255.0   192.168.100.254 Management   enkx3cel01.enkitec.com
eth1     Unlinked
eth2     Unlinked
eth3     Unlinked
bondib0  ib0,ib1    172.30.1.3      255.255.255.0                   Private      enkx3cel01-priv.enkitec.com
Is this correct (y/n) [y]:

Do you want to configure basic ILOM settings (y/n) [y]:
Loading basic configuration settings from ILOM ...
ILOM Fully qualified hostname [enkx3cel01-ilom.enkitec.com]:
ILOM IP address [192.168.100.78]:
ILOM Netmask [255.255.255.0]:
ILOM Gateway or none [192.168.100.254]:
ILOM Nameserver or none [192.168.10.12]: 192.168.10.14
ILOM Use NTP Servers (enabled/disabled) [enabled]:
ILOM First NTP server. Fully qualified hostname or ip address or none [192.168.10.15]:
ILOM Second NTP server. Fully qualified hostname or ip address or none [none]:

Basic ILOM configuration settings:
Hostname             : enkx3cel01-ilom.enkitec.com
IP Address           : 192.168.100.78
Netmask              : 255.255.255.0
Gateway              : 192.168.100.254
DNS servers          : 192.168.10.14
Use NTP servers      : enabled
First NTP server     : 192.168.10.15
Second NTP server    : none
Timezone (read-only) : America/Chicago

Is this correct (y/n) [y]: y
Connected. Use ^D to exit.
-> set /SP/clients/dns nameserver=192.168.10.14
Set 'nameserver' to '192.168.10.14'

-> Session closed
Disconnected

Info. Run /opt/oracle.cellos/validations/init.d/saveconfig
Info. Custom changes have been detected in /etc/resolv.conf
Info. Original file will be saved in /etc/resolv.conf.backupbyExadata

Warning. You modified DNS name server.
         Ensure you also update the Infiniband Switch DNS server
         if the same DNS server was also used by the Infiniband switch.

4. Compare the differences between the following two files:

diff /opt/oracle.cellos/cell.conf /opt/oracle.cellos/cell.conf.yyyymmdd

You will see a lot of difference. But if look at it more closely, they are actually the same one, just move things around the file.

5. Restart all cell services by running the following command
cellcli -e alter cell restart services all

6. Verify cell processes are up by running the following
cellcli -e list cell detail

The last three lines should be in running state

	 cellsrvStatus:     	 running
	 msStatus:          	 running
	 rsStatus:          	 running

7. Regularly run the following command to make sure griddisk status changes from SYNCING to ONLINE

[root@enkx3cel01 ~]# cellcli -e list griddisk attributes name,asmmodestatus
	 DATA_CD_00_enkx3cel01   	 SYNCING
	 DATA_CD_01_enkx3cel01   	 SYNCING
	 DATA_CD_02_enkx3cel01   	 SYNCING
	 DATA_CD_03_enkx3cel01   	 SYNCING
	 DATA_CD_04_enkx3cel01   	 SYNCING
	 DATA_CD_05_enkx3cel01   	 SYNCING
	 DATA_CD_06_enkx3cel01   	 SYNCING
	 DATA_CD_07_enkx3cel01   	 SYNCING
	 DATA_CD_08_enkx3cel01   	 SYNCING
	 DATA_CD_09_enkx3cel01   	 SYNCING
	 DATA_CD_10_enkx3cel01   	 SYNCING
	 DATA_CD_11_enkx3cel01   	 SYNCING
	 DBFS_DG_CD_02_enkx3cel01	 ONLINE
	 DBFS_DG_CD_03_enkx3cel01	 ONLINE
	 DBFS_DG_CD_04_enkx3cel01	 ONLINE
	 DBFS_DG_CD_05_enkx3cel01	 ONLINE
	 DBFS_DG_CD_06_enkx3cel01	 ONLINE
	 DBFS_DG_CD_07_enkx3cel01	 ONLINE
	 DBFS_DG_CD_08_enkx3cel01	 ONLINE
	 DBFS_DG_CD_09_enkx3cel01	 ONLINE
	 DBFS_DG_CD_10_enkx3cel01	 ONLINE
	 DBFS_DG_CD_11_enkx3cel01	 ONLINE
	 RECO_CD_00_enkx3cel01   	 SYNCING
	 RECO_CD_01_enkx3cel01   	 SYNCING
	 RECO_CD_02_enkx3cel01   	 SYNCING
	 RECO_CD_03_enkx3cel01   	 SYNCING
	 RECO_CD_04_enkx3cel01   	 SYNCING
	 RECO_CD_05_enkx3cel01   	 SYNCING
	 RECO_CD_06_enkx3cel01   	 SYNCING
	 RECO_CD_07_enkx3cel01   	 SYNCING
	 RECO_CD_08_enkx3cel01   	 SYNCING
	 RECO_CD_09_enkx3cel01   	 SYNCING
	 RECO_CD_10_enkx3cel01   	 SYNCING
	 RECO_CD_11_enkx3cel01   	 SYNCING

8. This step is optional, only uses when iptables command still shows using old DNS entries.

[root@enkx3cel01 oracle.cellos]# iptables -L -n
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:5042
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:5042 flags:0x17/0x02
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:3260
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:3260 flags:0x17/0x02
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:22
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:22 flags:0x17/0x02
ACCEPT     udp  --  192.168.100.240      0.0.0.0/0           udp spt:123
ACCEPT     tcp  --  192.168.10.14        0.0.0.0/0           tcp spt:53
ACCEPT     udp  --  192.168.10.14        0.0.0.0/0           udp spt:53
ACCEPT     tcp  --  192.168.10.15        0.0.0.0/0           tcp spt:53
ACCEPT     udp  --  192.168.10.15        0.0.0.0/0           udp spt:53
REJECT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpts:1024:65535 flags:0x17/0x02 reject-with icmp-port-unreachable
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpts:1024:65535
REJECT     tcp  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-port-unreachable
REJECT     udp  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-port-unreachable
ACCEPT     udp  --  192.168.100.78       0.0.0.0/0           udp dpt:162
ACCEPT     udp  --  192.168.100.78       0.0.0.0/0           udp spt:623 dpts:1024:65535
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:22 flags:0x17/0x02
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:22
ACCEPT     udp  --  192.168.100.240      0.0.0.0/0           udp spt:123
ACCEPT     tcp  --  192.168.10.14        0.0.0.0/0           tcp spt:53
ACCEPT     udp  --  192.168.10.14        0.0.0.0/0           udp spt:53
ACCEPT     tcp  --  192.168.10.15        0.0.0.0/0           tcp spt:53
ACCEPT     udp  --  192.168.10.15        0.0.0.0/0           udp spt:53
REJECT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpts:1024:65535 flags:0x17/0x02 reject-with icmp-port-unreachable
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpts:1024:65535
REJECT     tcp  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-port-unreachable
REJECT     udp  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-port-unreachable
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0
REJECT     udp  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-port-unreachable
REJECT     tcp  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-port-unreachable

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

If you see there are still entries for old DNS, like 192.168.10.12 or 192.168.10.13, then you need to restart cellwall service. Cellwall implements firewall services on each cell using IPTables.

service cellwall restart

9. Verify the result using nslookup command.

Advertisements

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s