Reimage ODA

The idea to reimage ODA is to refresh the environment without the need to jump from one by one to reach the last available version, or even rescue the system from S.O. failure/crash. The process to do a reimage can be check in the official documentation but unfortunately can be very tricky because the information (the order and steps) are not 100% clear. The idea is to show you how to reimage using version 18 (18.3 in this example), that represents the last available.

In resume the process is executed in the order:

  1. ILOM: Boot the ISO
  2. Prepare to create the appliance
  3. Upload GI and DB base version to the repository
  4. Linux: Create the appliance
  5. Firmware and patch
  6. Create Oracle Homes and the databases
  7. Finish and clean the install

The environment in this scenario is:

  • ODA HA: Oda HA version with two nodes and with SSD disk available.
  • Reimage with 18 version: 18.3 version at the moment that reimages was done, but can use any 18.x version
  • Databases: 11, 12, 18 for Oracle Home and available versions

Before even start the reimage you have two steps. The first is to verify the hardware against failures. Disks, memory, CPU, MB, whatever error that exists need to be clean before the start. If no, you can hit error during the process. To check about hw error, you can log in the ILOM in the browser and verify for some incidents/errors (this vary from version to version of ODA, but usually the errors are visible in the first page of ILOM). Or you can log in in the console for ILOM and execute the command “show faulty”. Whatever the mode, check in both nodes. If you have a connection with the system execute smartctl –a /dev/DISK and query about errors and look in the column “Total uncorrected errors”. Another option is check if the disk is in warning state using the command smartctl -q errorsonly -H -l xerror /dev/DISK; if you have the report will be similar to “SMART Health Status: WARNING: ascq=0x97 [asc=b, ascq=97]”. For the errors, open SR and wait for the maintenance from Oracle HW team.

The second is save/backup every config that you need. If you did not lose the system backup from /etc folder it is a good start. Another hint is to execute “ifconfig” and “route” to have one output from IP’s that you have and the routing table (vlan list it is important too if used). For Oracle files, backup tnsnames.ora, listener.ora, and sqlnet.ora for every OH or GI that you have. These files are an example, depend on every system and needs. Adjust if you need something additional.

ILOM

The process to reimage start putting the ISO in the ilom and booting both nodes with that. After the boot, the install runs automatically, and you have a Linux to create the appliance.

Before starting the reimage one explanation since the machine was already in use it needs to cleanup the disks. The first option is cleanup before the reimage, using the software from the currently running linux/oda/oak. Or running the cleanup after the reimage finish and before creating the appliance (the option that I will use here).

The steps to reimage from ILOM are done in this order:

  1. Login ILOM both nodes using the browser
  2. Navigate to “Host Management” menu and choose “Host Control” submenu.
    1. Select the option “CDROM” in the list and after click in SAVE.
  3. Navigate to “Remote Control” menu and choose “Redirection” submenu.
    1. Click on the option “Launch Remote Control Console”
    2. You can receive some confirmation to execute some java (depend on how your machine has java configured) or ask to agree about the certificate. Always click yes and agree or continue.
  4. In the newly opened window select the menu “KVMS” and after the submenu “Storage”.
  5. In the “Storage Devices” do:
    1. Click on the “Add”
    2. Select the ISO file and after click in the option “Select”
    3. After, verify if the ISO appeared in the “Path” and the “Device Type” is “ISO Image” in the Storage Devices menu. Select the line with the ISO and after click in the option “Connect”.
    4. After you click in the “Connect” the information marked in red will appear in the screen and the menu now will appear as “Disconnect”. You just need to click in the option “OK”.
    5. DO NOT CLONE the Console.
  6. Go back to the Browser and click in the menu “Host Management” and after in the submenu “Power Control”
    1. Select the option “Power Cycle” and after click in the “Save” option.
    2. Maybe you will receive on a question if you are sure about the reboot. Just click Ok.
  7. You will see in the console:
    1. Reboot
    2. The process will boot the ISO
    3. Install the new ODA Image
    4. In the end, you have the Linux console

All the steps above you can see in the Image gallery below. All the images are linked with the step mentioned.

This slideshow requires JavaScript.

Remember to do this in both nodes. You can do this both at the same time. The installation time depends more from the network because involve uploading the ISO from machine to ODA. The postscript part during the installation tends to take a lot of time.

Please does not close the Remote Console because this kills the ISO mounting device and the installation will crash. There is no problem if the browser login receives timeout, the installation will continue.

One important thing from this part is that does not update the ILOM SP or even other firmware for HW. This will be executed in the next steps.

Recently Oracle changed the ODA base page and removed the links to ISO files. But the patch 27604623 contains the Oracle Database Appliance Bare-metal ISO Image to download. Just to be clear, I am reimaging for baremetal (not OVM).

Prepare to create the appliance

After you “reimage” you need to configure and create the appliance. But before we need to execute some steps. Now, to connect in the Linux console we have two options: same java console opened to do the reimage, or, ssh to ilom ip and start the console (start /SP/console). I will use the last option because allow to work (copy and paste commands) more smoothly. Example (root/welcome1 are user/password that is the default for reimaged ODA):

Oracle(R) Integrated Lights Out Manager

Version 3.2.9.23 r116695

Copyright (c) 2017, Oracle and/or its affiliates. All rights reserved.

Warning: HTTPS certificate is set to factory default.

Hostname: odas1-ilom

-> start /SP/console
Are you sure you want to start /SP/console (y/n)? y

Serial console started.  To stop, type ESC (


Oracle Linux Server release 6.10
Kernel 4.1.12-124.18.6.el6uek.x86_64 on an x86_64

test0 login: root
[root@test0 ~]#

Cleanup

As told before, we need to clean the headers for ODA. I start checking the headers with command “cleanup.pl -checkHeader”:

[root@test0 ~]# /opt/oracle/oak/onecmd/cleanup.pl -checkHeader
Command may take few minutes...
OS Disk               Disk Type    OAK Header   ASM Header(p1)       ASM Header(p2)
 /dev/sda                HDD        Found           Found                Found
 /dev/sdb                HDD        Found           Found                Found
 /dev/sdaa               HDD        Found           Found                Found
 /dev/sdab               HDD        Found           Found                Found
 /dev/sdac               HDD        Found           Found                Found
 /dev/sdad               HDD        Found           Found                Found
 /dev/sdae               HDD        Found           Found                Found
 /dev/sdaf               HDD        Found           Found                Found
 /dev/sdag               HDD        Found           Found                Found
 /dev/sdah               HDD        Found          Erased               Erased
 /dev/sdai               HDD        Found           Found                Found
 /dev/sdaj               HDD        Found           Found                Found
 /dev/sdak               HDD        Found           Found                Found
 /dev/sdal               HDD        Found           Found                Found
 /dev/sdam               HDD        Found           Found                Found
 /dev/sdan               HDD        Found           Found                Found
 /dev/sdao               SSD        Found           Found       p2 not created
 /dev/sdap               SSD        Found           Found       p2 not created
 /dev/sdaq               SSD        Found           Found       p2 not created
 /dev/sdar               SSD        Found           Found       p2 not created
 /dev/sdas               SSD        Found           Found       p2 not created
 /dev/sdat               SSD        Found           Found       p2 not created
 /dev/sdau               SSD        Found           Found       p2 not created
 /dev/sdav               SSD        Found           Found       p2 not created
[root@test0 ~]#

Look above that some ASM Header was found. To clean we execute “/opt/oracle/oak/onecmd/cleanup.pl”  with option “-erasedata”. The cleanup script executes more than just erase disk headers, it cleans all the configurations (hostname, ips, users) and because of this needs to be executed in both nodes from HA env. This equalizes all nodes with the same config and avoid an error when you create the appliance. Example:

[root@test0 ~]# /opt/oracle/oak/onecmd/cleanup.pl -erasedata
INFO: *******************************************************************
INFO: ** Starting process to cleanup provisioned host oak1             **
INFO: *******************************************************************
WARNING: Secure Erase is an irrecoverable process. All data on the disk
WARNING: will be erased, and cannot be recovered by any means. On X3-2,
WARNING: X4-2, and X5-2 HA, the secure erase process can take more than
WARNING: 10 hours. If you need this data, then take a complete backup
WARNING: before proceeding.
Do you want to continue (yes/no) : yes
INFO:
....
....
INFO: Executing </opt/oracle/oak/bin/odaeraser.py -a -f -v >
Start erasing disks on the system
On some platforms, this will take several hours to finish, please wait
Do you want to continue (yes|no) [yes] ? yes
Number of disks are processing: 0

Disk     Vendor     Model                    Erase method         Status       Time(seconds)
e0_pd_00 HGST       H7280A520SUN8.0T         SCSI crypto erase    success               0
e0_pd_01 HGST       H7280A520SUN8.0T         SCSI crypto erase    success               0
e0_pd_02 HGST       H7280A520SUN8.0T         SCSI crypto erase    success               0
e0_pd_03 HGST       H7280A520SUN8.0T         SCSI crypto erase    success               0
e0_pd_04 HGST       H7280A520SUN8.0T         SCSI crypto erase    success               0
e0_pd_05 HGST       H7280A520SUN8.0T         SCSI crypto erase    success               0
e0_pd_06 HGST       H7280A520SUN8.0T         SCSI crypto erase    success               0
e0_pd_07 HGST       H7280A520SUN8.0T         SCSI crypto erase    success               0
e0_pd_08 HGST       H7280A520SUN8.0T         SCSI crypto erase    success               0
e0_pd_09 HGST       H7280A520SUN8.0T         SCSI cryp[  777.976350] reboot: Restarting system
...
...

After node reboot you can check the disk readers (look that the cleanup will change the hostname to “oak”):

[root@oak2 ~]# /opt/oracle/oak/onecmd/cleanup.pl -checkHeader
Command may take few minutes...
OS Disk               Disk Type    OAK Header   ASM Header(p1)       ASM Header(p2)
 /dev/sda                HDD       Erased          Erased              UnKnown
 /dev/sdb                HDD       Erased          Erased              UnKnown
 /dev/sdaa               HDD       Erased          Erased              UnKnown
 /dev/sdab               HDD       Erased          Erased              UnKnown
 /dev/sdac               HDD       Erased          Erased              UnKnown
 /dev/sdad               HDD       Erased          Erased              UnKnown
 /dev/sdae               HDD       Erased          Erased              UnKnown
 /dev/sdaf               HDD       Erased          Erased              UnKnown
 /dev/sdag               HDD       Erased          Erased              UnKnown
 /dev/sdah               HDD       Erased          Erased              UnKnown
 /dev/sdai               HDD       Erased          Erased              UnKnown
 /dev/sdaj               HDD       Erased          Erased              UnKnown
 /dev/sdak               HDD       Erased          Erased              UnKnown
 /dev/sdal               HDD       Erased          Erased              UnKnown
 /dev/sdam               HDD       Erased          Erased              UnKnown
 /dev/sdan               HDD       Erased          Erased              UnKnown
 /dev/sdao               SSD       Erased          Erased              UnKnown
 /dev/sdap               SSD       Erased          Erased              UnKnown
 /dev/sdaq               SSD       Erased          Erased              UnKnown
 /dev/sdar               SSD       Erased          Erased              UnKnown
 /dev/sdas               SSD       Erased          Erased              UnKnown
 /dev/sdat               SSD       Erased          Erased              UnKnown
 /dev/sdau               SSD       Erased          Erased              UnKnown
 /dev/sdav               SSD       Erased          Erased              UnKnown
[root@oak2 ~]#

First network

After the cleanup we can create the basic network. To do that (need to do in both nodes), just execute the script “/opt/oracle/dcs/bin/odacli” with parameter “configure-firstnet”. Look:

[root@oak1 ~]# /opt/oracle/dcs/bin/odacli configure-firstnet
Select the Interface to configure the network on (bond0 bond1) [bond0]:
Configure DHCP on bond0 (yes/no) [no]:
INFO: You have chosen Static configuration
Use VLAN on bond0 (yes/no) [no]:yes
Configure VLAN on bond0, input VLAN ID [2 - 4094] 2999
INFO: using network interface bond0.2999
Enter the IP address to configure : 200.200.67.100
Enter the Netmask address to configure : 255.255.255.0
Enter the Gateway address to configure[200.200.67.1] :
INFO: Restarting the network
Shutting down interface bond0:  [  OK  ]
Shutting down interface bond1:  [  OK  ]
Shutting down interface ib0:  [  OK  ]
Shutting down interface ib1:  [  OK  ]
Shutting down interface ibbond0:  [  OK  ]
Shutting down loopback interface:  [  OK  ]
Bringing up loopback interface:  [  OK  ]
Bringing up interface bond0:  [  OK  ]
Bringing up interface bond1:  [  OK  ]
Bringing up interface ibbond0:  Determining if ip address 192.168.16.24 is already in use for device ibbond0...
[  OK  ]
Bringing up interface bond0.2999:  Determining if ip address 200.200.67.100 is already in use for device bond0.2999...
[  OK  ]
INFO: Restarting the DCS agent
initdcsagent stop/waiting
initdcsagent start/running, process 23750
[root@oak1 ~]#

And in the second node:

[root@oak2 ~]# /opt/oracle/dcs/bin/odacli configure-firstnet
Select the Interface to configure the network on (bond0 bond1) [bond0]:
Configure DHCP on bond0 (yes/no) [no]:
INFO: You have chosen Static configuration
Use VLAN on bond0 (yes/no) [no]:yes
Configure VLAN on bond0, input VLAN ID [2 - 4094] 2999
INFO: using network interface bond0.2999
Enter the IP address to configure : 200.200.67.101
Enter the Netmask address to configure : 255.255.255.0
Enter the Gateway address to configure[200.200.67.1] :
INFO: Restarting the network
Shutting down interface bond0:  [  OK  ]
Shutting down interface bond1:  [  OK  ]
Shutting down interface ib0:  [  OK  ]
Shutting down interface ib1:  [  OK  ]
Shutting down interface ibbond0:  [  OK  ]
Shutting down loopback interface:  [  OK  ]
Bringing up loopback interface:  [  OK  ]
Bringing up interface bond0:  [  OK  ]
Bringing up interface bond1:  [  OK  ]
Bringing up interface ibbond0:  Determining if ip address 192.168.16.25 is already in use for device ibbond0...
[  OK  ]
Bringing up interface bond0.2999:  Determining if ip address 200.200.67.101 is already in use for device bond0.2999...
[  OK  ]
INFO: Restarting the DCS agent
initdcsagent stop/waiting
initdcsagent start/running, process 10775
[root@oak2 ~]#

The most important is to define correctly the vlan (if used, as in this case). You can rerun the command if needed. After that, you can exit the ILOM console and do ssh to the ODA node.

Upload GI and DB base version to the repository

After you configure the first network you need to upload the files that will fill the internal ODA repository and allow the “create appliance script” do initiate the ASM and base Oracle Home. There are some several steps, but you will execute in just one node.

Again, Oracle recently changed the ODA base page (https://support.oracle.com/epmos/faces/DocContentDisplay?id=888888.1) and removed the links to files with clone version for OH and GI. But the patch numbers are patch 27604593 for GI and patch 27604558 for RDBMS. Both are for baremetal image, if you are using VM installation, they are others (but you can find then patch search inside Oracle MOS.

The first step is to upload the files “p27604558_183000_Linux-x86-64.zip” and “p27604593_183000_Linux-x86-64.zip” to the first node and unzip. Remember that the file name will change if you are reimage using another version. Here I used in “/tmp”

[root@oak1 ~]# mkdir /tmp/deploy
[root@oak1 ~]# cd /tmp/deploy
[root@oak1 deploy]# ###### EXECUTE THE SCP OR COPY FILES TO THE FOLDER########
[root@oak1 deploy]# unzip -qa p27604558_183000_Linux-x86-64.zip
[root@oak1 deploy]# unzip -qa p27604593_183000_Linux-x86-64.zip
replace README.txt? [y]es, [n]o, [A]ll, [N]one, [r]ename: A
[root@oak1 deploy]#
[root@oak1 deploy]#
[root@oak1 deploy]# ls -l
total 20559868
-rw-r--r-- 1 root root 4322872013 Dec 18 08:24 odacli-dcs-18.3.0.0.0-180905-DB-18.0.0.0.zip
-rwxr-xr-x 1 root root 6193475789 Dec 18 08:03 odacli-dcs-18.3.0.0.0-181205-GI-18.3.0.0.zip
-rw-r--r-- 1 root root 4322873299 May 22 08:37 p27604558_183000_Linux-x86-64.zip
-rw-r--r-- 1 root root 6193477129 May 22 08:35 p27604593_183000_Linux-x86-64.zip
-rw-r--r-- 1 root root       2044 Dec 18 08:05 README.txt
[root@oak1 deploy]#

After that, we can upload these file to the internal ODA repository:

[root@oak1 deploy]# /opt/oracle/dcs/bin/odacli update-repository -f /tmp/deploy/odacli-dcs-18.3.0.0.0-181205-GI-18.3.0.0.zip,/tmp/deploy/odacli-dcs-18.3.0.0.0-180905-DB-18.0.0.0.zip
{
  "jobId" : "b66269aa-e007-4304-8074-7ba7e9c8c5c2",
  "status" : "Created",
  "message" : "/tmp/deploy/odacli-dcs-18.3.0.0.0-181205-GI-18.3.0.0.zip,/tmp/deploy/odacli-dcs-18.3.0.0.0-180905-DB-18.0.0.0.zip",
  "reports" : [ ],
  "createTimestamp" : "May 22, 2019 08:51:14 AM UTC",
  "resourceList" : [ ],
  "description" : "Repository Update",
  "updatedTime" : "May 22, 2019 08:51:14 AM UTC"
}
[root@oak1 deploy]#

Look that one job will be created, and you can check the progress for the job with the “/opt/oracle/dcs/bin/odacli describe-job -i” passing as parameter the jobId:

[root@oak1 deploy]# /opt/oracle/dcs/bin/odacli describe-job -i b66269aa-e007-4304-8074-7ba7e9c8c5c2

Job details
----------------------------------------------------------------
                     ID:  b66269aa-e007-4304-8074-7ba7e9c8c5c2
            Description:  Repository Update
                 Status:  Running
                Created:  May 22, 2019 8:51:14 AM UTC
                Message:  /tmp/deploy/odacli-dcs-18.3.0.0.0-181205-GI-18.3.0.0.zip,/tmp/deploy/odacli-dcs-18.3.0.0.0-180905-DB-18.0.0.0.zip

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Check AvailableSpace                     May 22, 2019 8:51:14 AM UTC         May 22, 2019 8:51:14 AM UTC         Success
Setting up ssh equivalance               May 22, 2019 8:51:15 AM UTC         May 22, 2019 8:51:15 AM UTC         Success
Copy BundleFile                          May 22, 2019 8:51:15 AM UTC         May 22, 2019 8:51:15 AM UTC         Running

[root@oak1 deploy]#
[root@oak1 deploy]# /opt/oracle/dcs/bin/odacli describe-job -i b66269aa-e007-4304-8074-7ba7e9c8c5c2

Job details
----------------------------------------------------------------
                     ID:  b66269aa-e007-4304-8074-7ba7e9c8c5c2
            Description:  Repository Update
                 Status:  Success
                Created:  May 22, 2019 8:51:14 AM UTC
                Message:  /tmp/deploy/odacli-dcs-18.3.0.0.0-181205-GI-18.3.0.0.zip,/tmp/deploy/odacli-dcs-18.3.0.0.0-180905-DB-18.0.0.0.zip

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Check AvailableSpace                     May 22, 2019 8:51:14 AM UTC         May 22, 2019 8:51:14 AM UTC         Success
Setting up ssh equivalance               May 22, 2019 8:51:15 AM UTC         May 22, 2019 8:51:15 AM UTC         Success
Copy BundleFile                          May 22, 2019 8:51:15 AM UTC         May 22, 2019 8:52:21 AM UTC         Success
Validating CopiedFile                    May 22, 2019 8:52:24 AM UTC         May 22, 2019 8:52:48 AM UTC         Success
Unzip bundle                             May 22, 2019 8:52:48 AM UTC         May 22, 2019 8:54:17 AM UTC         Success
Unzip bundle                             May 22, 2019 8:54:17 AM UTC         May 22, 2019 8:55:47 AM UTC         Success
Delete PatchBundles                      May 22, 2019 8:55:47 AM UTC         May 22, 2019 8:55:48 AM UTC         Success
Removing ssh keys                        May 22, 2019 8:55:48 AM UTC         May 22, 2019 8:55:49 AM UTC         Success

[root@oak1 deploy]#

Create appliance

After you clean the disk headers, prepare the first network and upload the files in the repository you can create the appliance. This means that you will configure the operational system, groups, users, folders, GI, ASM, and Oracle database. There are two ways to create, one is using the web interface (this replace the old java app that creates the XML file), and the other is from CLI. Unfortunately, until I wrote this, if you use VLAN you can’t create the appliance with the web interface, just with CLI.

The CLI uses the script “/opt/oracle/dcs/bin/odacli create-appliance” that receive a JSON file as a parameter. There is not some much information the parameter for this JSON file, just the readme with some examples and one web page with other examples. But in both, no real good explanation to each option, look: https://docs.oracle.com/en/engineered-systems/oracle-database-appliance/18.3/cmtxd/create-appliance-using-json-file.html#GUID-42250FD2-EA91-4457-9ED7-CA3A2A863B40

Fortunately, most of the options are self-explained, other parts I will explain below. First, the full JSON file that I used:

[root@oak1 ~]# cat odas.json
{
  "instance" : {
    "name" : "odas",
    "instanceBaseName" : "odas",
    "systemPassword" : null,
    "dbEdition" : "EE",
    "timeZone" : "Europe/Luxembourg",
    "ntpServers" : [ "200.200.13.125" ],
    "dnsServers" : [ "200.200.1.32", "200.200.1.33" ],
    "domainName" : "xxxx.xxxx.xxx",
    "isRoleSeparated" : true,
    "osUserGroup" : {
      "groups" : [ {
        "groupId" : 1001,
        "groupName" : "oinstall",
        "groupRole" : "oinstall"
      }, {
        "groupId" : 1003,
        "groupName" : "dbaoper",
        "groupRole" : "dbaoper"
      }, {
        "groupId" : 1002,
        "groupName" : "dba",
        "groupRole" : "dba"
      }, {
        "groupId" : 1006,
        "groupName" : "asmadmin",
        "groupRole" : "asmadmin"
      }, {
        "groupId" : 1005,
        "groupName" : "asmoper",
        "groupRole" : "asmoper"
      }, {
        "groupId" : 1004,
        "groupName" : "asmdba",
        "groupRole" : "asmdba"
      } ],
      "users" : [ {
        "userId" : 1001,
        "userName" : "oracle",
        "userRole" : "oracleUser"
      }, {
        "userId" : 1000,
        "userName" : "grid",
        "userRole" : "gridUser"
      } ]
    },
    "objectStoreCredentials" : null
  },
  "nodes" : [ {
    "nodeNumber" : "0",
    "nodeName" : "odas1",
    "localDisk" : null,
    "network" : [ {
      "name" : null,
      "nicName" : "bond0.2999",
      "vlanId" : null,
      "ipAddress" : "200.200.67.100",
      "subNetMask" : "255.255.255.0",
      "gateway" : "200.200.67.1",
      "interfaceType" : null,
      "networkType" : [ "Public" ],
      "isDefaultNetwork" : true
    } ],
    "ilom" : null
  }, {
    "nodeNumber" : "1",
    "nodeName" : "odas2",
    "localDisk" : null,
    "network" : [ {
      "name" : null,
      "nicName" : "bond0.2999",
      "vlanId" : null,
      "ipAddress" : "200.200.67.101",
      "subNetMask" : "255.255.255.0",
      "gateway" : "200.200.67.1",
      "interfaceType" : null,
      "networkType" : [ "Public" ],
      "isDefaultNetwork" : true
    } ],
    "ilom" : null
  } ],
  "grid" : {
    "diskGroup" : [ {
      "diskGroupName" : "DATA",
      "redundancy" : "NORMAL",
      "disks" : null,
      "diskPercentage" : 90
    }, {
      "diskGroupName" : "RECO",
      "redundancy" : "NORMAL",
      "disks" : null,
      "diskPercentage" : 10
    }, {
      "diskGroupName" : "REDO",
      "redundancy" : "HIGH",
      "disks" : null,
      "diskPercentage" : null
    }, {
      "diskGroupName" : "FLASH",
      "redundancy" : "NORMAL",
      "disks" : null,
      "diskPercentage" : null
    } ],
    "scan" : {
      "scanName" : "odas-scan",
      "ipAddresses" : [ "200.200.67.104", "200.200.67.105" ]
    },
    "vip" : [ {
      "nodeNumber" : "0",
      "vipName" : "odas1-vip",
      "ipAddress" : "200.200.67.102"
    }, {
      "nodeNumber" : "1",
      "vipName" : "odas2-vip",
      "ipAddress" : "200.200.67.103"
    } ],
    "language" : "en",
    "enableAFD" : "TRUE",
    "gridVersion" : null
  },
  "database" : {
    "dbName" : "mfod",
    "databaseUniqueName" : "mfod_ora",
    "dbVersion" : "18.3.0.0.180717",
    "dbHomeId" : null,
    "instanceOnly" : false,
    "registerOnly" : null,
    "isCdb" : true,
    "pdBName" : "pdb1",
    "pdbAdminuserName" : "pdbuser",
    "enableTDE" : false,
    "adminPassword" : "WWeeeeee99##",
    "dbType" : "RAC",
    "dbTargetNodeNumber" : null,
    "dbClass" : "OLTP",
    "dbShape" : "odb1",
    "dbStorage" : "ASM",
    "dbRedundancy" : null,
    "dbCharacterSet" : {
      "characterSet" : "AL32UTF8",
      "nlsCharacterset" : "AL16UTF16",
      "dbTerritory" : "AMERICA",
      "dbLanguage" : "AMERICAN"
    },
    "dbConsoleEnable" : false,
    "backupConfigId" : null,
    "backupConfigName" : null,
    "dbOnFlashStorage" : true,
    "dbEdition" : "EE",
    "rmanBkupPassword" : null,
    "dbDomainName" : null,
    "enableFlashCache" : null,
    "level0BackupDay" : null
  },
  "asr" : null
}
[root@oak1 ~]#

The file starts with a basic config where you define the name parameter and the instanceBaseName that will define some basic internal identification parameters. One example is the time zone.

The first that needs attention is isRoleSeparated and osUserGroup. If you choose the first as false, you don’t need the specify grid users. Otherwise, is needed to specify it.

For nodes option, you specify the hostname, ips, and gateway. Be careful with the nickname that is the same name that you configured the first network. Some steps above. The ilom parameter, since I am not changing now (and I think that in reimage scenario you do not change at this moment), remains null.

The grid parameter defines the disk division. You can choose arbitrary values for diskPercentage and disks that you will select for each one if needed. For disk, left undefined that ODA will divide automatically and accordingly. The key point here is that even, by default, FLASH and REDO diskgroups are created with default values (check that I even defined disk percentage for each one), if you forgot to specify, they are not created and the disk will not be used. I already wrote about one problem in this particular part and you can read here: http://www.fernandosimon.com/blog/oda-json-and-flash/

The database parameter defines the information for database creation. You have the basic like database name, unique name, cdb (or no), pdb name. The most tricky here are dbVersion (that need to be the same from that you uploaded in the repository), adminPassword (that will be used for root password after create the appliance), dbShape (that is the same – you choose one – for ODA available shapes), and dbStorage (that you choose between ASM and ACFS, 11 just support ACFS). Again, for ASM and ACFS I already wrote about that here: http://www.fernandosimon.com/blog/oda-acfs-and-asm-dilemma/

Important hint, if you want to create a database in another version that 18c you need to upload the files/clone images in the repository before.

To create just execute:

[root@oak1 ~]# /opt/oracle/dcs/bin/odacli create-appliance -r odas.json
{
  "instance" : {
    "name" : "odas",
    "instanceBaseName" : "odas",
    "systemPassword" : null,
    "dbEdition" : "EE",
    "timeZone" : "Europe/Luxembourg",
    "ntpServers" : [ "200.200.13.125" ],
    "dnsServers" : [ "200.200.1.32", "200.200.1.33" ],
    "domainName" : "xxxx.xxxx.xxx",
    "isRoleSeparated" : true,
    "osUserGroup" : {
      "groups" : [ {
        "groupId" : 1001,
        "groupName" : "oinstall",
        "groupRole" : "oinstall"
      }, {
        "groupId" : 1003,
        "groupName" : "dbaoper",
        "groupRole" : "dbaoper"
      }, {
        "groupId" : 1002,
        "groupName" : "dba",
        "groupRole" : "dba"
      }, {
        "groupId" : 1006,
        "groupName" : "asmadmin",
        "groupRole" : "asmadmin"
      }, {
        "groupId" : 1005,
        "groupName" : "asmoper",
        "groupRole" : "asmoper"
      }, {
        "groupId" : 1004,
        "groupName" : "asmdba",
        "groupRole" : "asmdba"
      } ],
      "users" : [ {
        "userId" : 1001,
        "userName" : "oracle",
        "userRole" : "oracleUser"
      }, {
        "userId" : 1000,
        "userName" : "grid",
        "userRole" : "gridUser"
      } ]
    },
    "objectStoreCredentials" : null
  },
  "nodes" : [ {
    "nodeNumber" : "0",
    "nodeName" : "odas1",
    "localDisk" : null,
    "network" : [ {
      "name" : null,
      "nicName" : "bond0.2999",
      "vlanId" : null,
      "ipAddress" : "200.200.67.100",
      "subNetMask" : "255.255.255.0",
      "gateway" : "200.200.67.1",
      "interfaceType" : null,
      "networkType" : [ "Public" ],
      "isDefaultNetwork" : true
    } ],
    "ilom" : null
  }, {
    "nodeNumber" : "1",
    "nodeName" : "odas2",
    "localDisk" : null,
    "network" : [ {
      "name" : null,
      "nicName" : "bond0.2999",
      "vlanId" : null,
      "ipAddress" : "200.200.67.101",
      "subNetMask" : "255.255.255.0",
      "gateway" : "200.200.67.1",
      "interfaceType" : null,
      "networkType" : [ "Public" ],
      "isDefaultNetwork" : true
    } ],
    "ilom" : null
  } ],
  "grid" : {
    "diskGroup" : [ {
      "diskGroupName" : "DATA",
      "redundancy" : "NORMAL",
      "disks" : null,
      "diskPercentage" : 90
    }, {
      "diskGroupName" : "RECO",
      "redundancy" : "NORMAL",
      "disks" : null,
      "diskPercentage" : 10
    }, {
      "diskGroupName" : "REDO",
      "redundancy" : "HIGH",
      "disks" : null,
      "diskPercentage" : null
    }, {
      "diskGroupName" : "FLASH",
      "redundancy" : "NORMAL",
      "disks" : null,
      "diskPercentage" : null
    } ],
    "scan" : {
      "scanName" : "odas-scan",
      "ipAddresses" : [ "200.200.67.104", "200.200.67.105" ]
    },
    "vip" : [ {
      "nodeNumber" : "0",
      "vipName" : "odas1-vip",
      "ipAddress" : "200.200.67.102"
    }, {
      "nodeNumber" : "1",
      "vipName" : "odas2-vip",
      "ipAddress" : "200.200.67.103"
    } ],
    "language" : "en",
    "enableAFD" : "TRUE",
    "gridVersion" : null
  },
  "database" : {
    "dbName" : "mfod",
    "databaseUniqueName" : "mfod_ora",
    "dbVersion" : "18.3.0.0.180717",
    "dbHomeId" : null,
    "instanceOnly" : false,
    "registerOnly" : null,
    "isCdb" : true,
    "pdBName" : "pdb1",
    "pdbAdminuserName" : "pdbuser",
    "enableTDE" : false,
    "adminPassword" : "WWeeeeee99##",
    "dbType" : "RAC",
    "dbTargetNodeNumber" : null,
    "dbClass" : "OLTP",
    "dbShape" : "odb1",
    "dbStorage" : "ASM",
    "dbRedundancy" : null,
    "dbCharacterSet" : {
      "characterSet" : "AL32UTF8",
      "nlsCharacterset" : "AL16UTF16",
      "dbTerritory" : "AMERICA",
      "dbLanguage" : "AMERICAN"
    },
    "dbConsoleEnable" : false,
    "backupConfigId" : null,
    "backupConfigName" : null,
    "dbOnFlashStorage" : true,
    "dbEdition" : "EE",
    "rmanBkupPassword" : null,
    "dbDomainName" : null,
    "enableFlashCache" : null,
    "level0BackupDay" : null
  },
  "asr" : null
}
{
  "jobId" : "f07961fa-4192-489b-9a55-53165a012376",
  "status" : "Created",
  "message" : null,
  "reports" : [ ],
  "createTimestamp" : "May 22, 2019 09:03:36 AM UTC",
  "resourceList" : [ ],
  "description" : "Provisioning service creation",
  "updatedTime" : "May 22, 2019 09:03:36 AM UTC"
}
[root@oak1 ~]#

Look that you will receive one jobId if everything is fine. To follow just describe the job:

[root@oak1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i f07961fa-4192-489b-9a55-53165a012376

Job details
----------------------------------------------------------------
                     ID:  f07961fa-4192-489b-9a55-53165a012376
            Description:  Provisioning service creation
                 Status:  Running
                Created:  May 22, 2019 11:03:36 AM CEST
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
networks updation                        May 22, 2019 11:03:37 AM CEST       May 22, 2019 11:03:42 AM CEST       Success
updating network                         May 22, 2019 11:03:37 AM CEST       May 22, 2019 11:03:42 AM CEST       Success
Setting up Vlan                          May 22, 2019 11:03:38 AM CEST       May 22, 2019 11:03:42 AM CEST       Success
networks updation                        May 22, 2019 11:03:42 AM CEST       May 22, 2019 11:03:47 AM CEST       Success
updating network                         May 22, 2019 11:03:42 AM CEST       May 22, 2019 11:03:47 AM CEST       Success
Setting up Vlan                          May 22, 2019 11:03:43 AM CEST       May 22, 2019 11:03:47 AM CEST       Success
OS usergroup 'asmdba'creation            May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:47 AM CEST       Success
OS usergroup 'asmoper'creation           May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:47 AM CEST       Success
OS usergroup 'asmadmin'creation          May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:47 AM CEST       Success
OS usergroup 'dba'creation               May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:47 AM CEST       Success
OS usergroup 'dbaoper'creation           May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:47 AM CEST       Success
OS usergroup 'oinstall'creation          May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:47 AM CEST       Success
OS user 'grid'creation                   May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:48 AM CEST       Success
OS user 'oracle'creation                 May 22, 2019 11:03:48 AM CEST       May 22, 2019 11:03:48 AM CEST       Success
SSH equivalance setup                    May 22, 2019 11:03:48 AM CEST       May 22, 2019 11:03:48 AM CEST       Success
Grid home creation                       May 22, 2019 11:03:49 AM CEST       May 22, 2019 11:03:49 AM CEST       Running
Creating GI home directories             May 22, 2019 11:03:49 AM CEST       May 22, 2019 11:03:49 AM CEST       Success
Cloning Gi home                          May 22, 2019 11:03:49 AM CEST       May 22, 2019 11:03:49 AM CEST       Running

[root@oak1 ~]#

And after some time:

[root@oak1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i f07961fa-4192-489b-9a55-53165a012376

Job details
----------------------------------------------------------------
                     ID:  f07961fa-4192-489b-9a55-53165a012376
            Description:  Provisioning service creation
                 Status:  Success
                Created:  May 22, 2019 11:03:36 AM CEST
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
networks updation                        May 22, 2019 11:03:37 AM CEST       May 22, 2019 11:03:42 AM CEST       Success
updating network                         May 22, 2019 11:03:37 AM CEST       May 22, 2019 11:03:42 AM CEST       Success
Setting up Vlan                          May 22, 2019 11:03:38 AM CEST       May 22, 2019 11:03:42 AM CEST       Success
networks updation                        May 22, 2019 11:03:42 AM CEST       May 22, 2019 11:03:47 AM CEST       Success
updating network                         May 22, 2019 11:03:42 AM CEST       May 22, 2019 11:03:47 AM CEST       Success
Setting up Vlan                          May 22, 2019 11:03:43 AM CEST       May 22, 2019 11:03:47 AM CEST       Success
OS usergroup 'asmdba'creation            May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:47 AM CEST       Success
OS usergroup 'asmoper'creation           May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:47 AM CEST       Success
OS usergroup 'asmadmin'creation          May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:47 AM CEST       Success
OS usergroup 'dba'creation               May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:47 AM CEST       Success
OS usergroup 'dbaoper'creation           May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:47 AM CEST       Success
OS usergroup 'oinstall'creation          May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:47 AM CEST       Success
OS user 'grid'creation                   May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:48 AM CEST       Success
OS user 'oracle'creation                 May 22, 2019 11:03:48 AM CEST       May 22, 2019 11:03:48 AM CEST       Success
SSH equivalance setup                    May 22, 2019 11:03:48 AM CEST       May 22, 2019 11:03:48 AM CEST       Success
Grid home creation                       May 22, 2019 11:03:49 AM CEST       May 22, 2019 11:08:00 AM CEST       Success
Creating GI home directories             May 22, 2019 11:03:49 AM CEST       May 22, 2019 11:03:49 AM CEST       Success
Cloning Gi home                          May 22, 2019 11:03:49 AM CEST       May 22, 2019 11:06:36 AM CEST       Success
Cloning Gi home                          May 22, 2019 11:06:36 AM CEST       May 22, 2019 11:07:58 AM CEST       Success
Updating GiHome version                  May 22, 2019 11:07:58 AM CEST       May 22, 2019 11:08:00 AM CEST       Success
Updating GiHome version                  May 22, 2019 11:07:58 AM CEST       May 22, 2019 11:08:00 AM CEST       Success
Storage discovery                        May 22, 2019 11:08:00 AM CEST       May 22, 2019 11:19:03 AM CEST       Success
Grid stack creation                      May 22, 2019 11:19:03 AM CEST       May 22, 2019 11:53:45 AM CEST       Success
Configuring GI                           May 22, 2019 11:19:03 AM CEST       May 22, 2019 11:20:08 AM CEST       Success
Running GI root scripts                  May 22, 2019 11:20:08 AM CEST       May 22, 2019 11:32:18 AM CEST       Success
Running GI config assistants             May 22, 2019 11:37:22 AM CEST       May 22, 2019 11:40:04 AM CEST       Success
Setting AUDIT SYSLOG LEVEL               May 22, 2019 11:50:06 AM CEST       May 22, 2019 11:50:06 AM CEST       Success
Post cluster OAKD configuration          May 22, 2019 11:53:45 AM CEST       May 22, 2019 11:59:11 AM CEST       Success
Disk group 'RECO'creation                May 22, 2019 11:59:20 AM CEST       May 22, 2019 11:59:38 AM CEST       Success
Disk group 'REDO'creation                May 22, 2019 11:59:38 AM CEST       May 22, 2019 11:59:47 AM CEST       Success
Disk group 'FLASH'creation               May 22, 2019 11:59:47 AM CEST       May 22, 2019 11:59:58 AM CEST       Success
Volume 'commonstore'creation             May 22, 2019 11:59:58 AM CEST       May 22, 2019 12:00:48 PM CEST       Success
ACFS File system 'DATA'creation          May 22, 2019 12:00:48 PM CEST       May 22, 2019 12:01:13 PM CEST       Success
Database home creation                   May 22, 2019 12:01:13 PM CEST       May 22, 2019 12:05:28 PM CEST       Success
Validating dbHome available space        May 22, 2019 12:01:13 PM CEST       May 22, 2019 12:01:13 PM CEST       Success
Validating dbHome available space        May 22, 2019 12:01:13 PM CEST       May 22, 2019 12:01:13 PM CEST       Success
Creating DbHome Directory                May 22, 2019 12:01:13 PM CEST       May 22, 2019 12:01:13 PM CEST       Success
Extract DB clones                        May 22, 2019 12:01:14 PM CEST       May 22, 2019 12:04:12 PM CEST       Success
Clone Db home                            May 22, 2019 12:04:12 PM CEST       May 22, 2019 12:05:11 PM CEST       Success
Enable DB options                        May 22, 2019 12:05:11 PM CEST       May 22, 2019 12:05:27 PM CEST       Success
Run Root DB scripts                      May 22, 2019 12:05:27 PM CEST       May 22, 2019 12:05:27 PM CEST       Success
Provisioning service creation            May 22, 2019 12:05:28 PM CEST       May 22, 2019 12:23:12 PM CEST       Success
Database Creation                        May 22, 2019 12:05:28 PM CEST       May 22, 2019 12:20:27 PM CEST       Success
Change permission for xdb wallet files   May 22, 2019 12:20:27 PM CEST       May 22, 2019 12:20:27 PM CEST       Success
Add Startup Trigger to Open all PDBS     May 22, 2019 12:20:27 PM CEST       May 22, 2019 12:20:28 PM CEST       Success
SqlPatch upgrade                         May 22, 2019 12:21:53 PM CEST       May 22, 2019 12:23:08 PM CEST       Success
Running dbms_stats init_package          May 22, 2019 12:23:08 PM CEST       May 22, 2019 12:23:11 PM CEST       Success
updating the Database version            May 22, 2019 12:23:11 PM CEST       May 22, 2019 12:23:12 PM CEST       Success
users tablespace creation                May 22, 2019 12:23:12 PM CEST       May 22, 2019 12:23:15 PM CEST       Success
Install TFA                              May 22, 2019 12:23:16 PM CEST       May 22, 2019 12:28:08 PM CEST       Success

[root@oak1 ~]#

You can follow the log too if you want to see more detailed (a lot) info. To do just go to directory /opt/oracle/dcs/log/ and check the file dcs-agent.log. Everything will be there, and it is really importing if some step fails. Here you can see one example of the output.

After the end of the job you can check the details for the appliance:

[root@oak1 ~]# /opt/oracle/dcs/bin/odacli describe-system

Appliance Information
----------------------------------------------------------------
                     ID: 6ce235cf-effc-4748-9d0f-ac246e9ee819
               Platform: X5-2-HA
        Data Disk Count: 24
         CPU Core Count: 36
                Created: May 22, 2019 11:03:36 AM CEST

System Information
----------------------------------------------------------------
                   Name: odas
            Domain Name: XXX.XXX.XXX
              Time Zone: Europe/Luxembourg
             DB Edition: EE
            DNS Servers: 200.200.1.32 200.200.1.33
            NTP Servers: 200.200.13.125

Disk Group Information
----------------------------------------------------------------
DG Name                   Redundancy                Percentage
------------------------- ------------------------- ------------
Data                      Normal                    90
Reco                      Normal                    10
Redo                      High                      100
Flash                     Normal                    100

[root@oak1 ~]#
[root@oak1 ~]# odaadmcli show diskgroup
DiskGroups
----------
DATA
FLASH
RECO
REDO
[root@oak1 ~]#
[root@oak1 ~]# su - grid
[grid@odas1 ~]$ asmcmd
ASMCMD> lsdg
State    Type    Rebal  Sector  Logical_Sector  Block       AU   Total_MB    Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  NORMAL  N         512             512   4096  4194304  108019712  108007188          6751232        50627978              0             Y  DATA/
MOUNTED  NORMAL  N         512             512   4096  4194304    1525760    1518672           381440          568616              0             N  FLASH/
MOUNTED  NORMAL  N         512             512   4096  4194304   11997184   11995736           749824         5622956              0             N  RECO/
MOUNTED  HIGH    N         512             512   4096  4194304     762880     749908           190720          186396              0             N  REDO/
ASMCMD> exit
[grid@odas1 ~]$
[grid@odas1 ~]$ logout
[root@oak1 ~]#
[root@oak1 ~]# /opt/oracle/dcs/bin/odacli list-dbstorages

ID                                       Type   DBUnique Name        Status
---------------------------------------- ------ -------------------- ----------
d26bf34f-9379-4872-839b-ae0af6926aae     Asm    mfod_ora             Configured
[root@oak1 ~]#
[root@oak1 ~]# /opt/oracle/dcs/bin/odacli describe-dbstorage -i d26bf34f-9379-4872-839b-ae0af6926aae
Database Storage details
----------------------------------------------------------------
                     ID: d26bf34f-9379-4872-839b-ae0af6926aae
                DB Name: mfod
          DBUnique Name: mfod_ora
         DB Resource ID: 938b616a-99ed-4466-8d76-820d3489517d
           Storage Type: ASM
                   DATA:
                         Location: +FLASH/mfod_ora
                       Used Space: 3.21GB
                       Free Space: 741.53GB
                   REDO:
                         Location: +REDO/mfod_ora
                       Used Space: 4.06GB
                       Free Space: 244.11GB
                   RECO:
                         Location: +RECO/mfod_ora
                       Used Space: 334.0MB
                       Free Space: 5.72TB
                  State: ResourceState(status=Configured)
                Created: May 22, 2019 11:03:36 AM CEST
            UpdatedTime: May 22, 2019 12:01:13 PM CEST
[root@oak1 ~]#

After that you reboot both nodes to have a fresh start with all the configuration up and running:

[root@oak1 ~]# reboot

Broadcast message from root@odas1
        (/dev/ttyS0) at 12:37 ...

The system is going down for reboot NOW!
[root@oak1 ~]#
        
Oracle Linux Server release 6.10
Kernel 4.1.12-124.18.6.el6uek.x86_64 on an x86_64

odas1 login: root
Password:
Last login: Wed May 22 11:10:13 from odap1.xxxx.xxxx.xxx
[root@odas1 ~]#

As an example, you can add vlan to ODA if needed (after the reboot):

[root@odas1 ~]# odaadmcli show vlan
        NAME                     ID    INTERFACE   CONFIG_TYPE IP_ADDRESS      NETMASK         GATEWAY         NODENUM
        Public-network           2999  bond0       Public      200.200.67.100   255.255.255.0   200.200.67.1     0
        Public-network           2999  bond0       Public      200.200.67.101   255.255.255.0   200.200.67.1     1
[root@odas1 ~]#
[root@odas1 ~]# cat /proc/net/vlan/config
VLAN Dev name    | VLAN ID
Name-Type: VLAN_NAME_TYPE_RAW_PLUS_VID_NO_PAD
bond0.2999     | 2999  | bond0
[root@odas1 ~]#
[root@odas1 ~]# odaadmcli create vlan VLAN_3001 -vlanid 3001 -if bond0 -node 0 -setuptype database -ip 200.200.131.58 -netmask 255.255.255.0 -gateway 200.200.131.1

Created Vlan : VLAN_3001
[root@odas1 ~]# odaadmcli create vlan VLAN_3010 -vlanid 3010 -if bond0 -node 0 -setuptype database -ip 200.190.3.58 -netmask 255.255.255.0 -gateway 200.190.3.1

Created Vlan : VLAN_3010
[root@odas1 ~]# odaadmcli create vlan VLAN_3030 -vlanid 3030 -if bond0 -node 0 -setuptype management -ip 200.200.192.156 -netmask 255.255.255.0 -gateway 200.200.192.1

Created Vlan : VLAN_3030
[root@odas1 ~]# odaadmcli create vlan VLAN_3002 -vlanid 3002 -if bond1 -node 0 -setuptype backup -ip 200.200.68.128 -netmask 255.255.255.0 -gateway 200.200.68.1

Created Vlan : VLAN_3002
[root@odas1 ~]# odaadmcli create vlan VLAN_3001 -vlanid 3001 -if bond0 -node 1 -setuptype database -ip 200.200.131.59 -netmask 255.255.255.0 -gateway 200.200.131.1

Created Vlan : VLAN_3001
[root@odas1 ~]# odaadmcli create vlan VLAN_3010 -vlanid 3010 -if bond0 -node 1 -setuptype database -ip 200.190.3.59 -netmask 255.255.255.0 -gateway 200.190.3.1

Created Vlan : VLAN_3010
[root@odas1 ~]# odaadmcli create vlan VLAN_3030 -vlanid 3030 -if bond0 -node 1 -setuptype management -ip 200.200.192.157 -netmask 255.255.255.0 -gateway 200.200.192.1

Created Vlan : VLAN_3030
[root@odas1 ~]# odaadmcli create vlan VLAN_3002 -vlanid 3002 -if bond1 -node 1 -setuptype backup -ip 200.200.68.129 -netmask 255.255.255.0 -gateway 200.200.68.1

Created Vlan : VLAN_3002
[root@odas1 ~]#
[root@odas1 ~]#
[root@odas1 ~]# odaadmcli show vlan
        NAME                     ID    INTERFACE   CONFIG_TYPE IP_ADDRESS      NETMASK         GATEWAY         NODENUM
        Public-network           2999  bond0       Public      200.200.67.100   255.255.255.0   200.200.67.1     0
        Public-network           2999  bond0       Public      200.200.67.101   255.255.255.0   200.200.67.1     1
        VLAN_3002                3002  bond1       backup      200.200.68.128   255.255.255.0   200.200.68.1     0
        VLAN_3002                3002  bond1       backup      200.200.68.129   255.255.255.0   200.200.68.1     1
        VLAN_3001                3001  bond0       database    200.200.131.58   255.255.255.0   200.200.131.1    0
        VLAN_3001                3001  bond0       database    200.200.131.59   255.255.255.0   200.200.131.1    1
        VLAN_3010                3010  bond0       database    200.190.3.58     255.255.255.0   200.190.3.1      0
        VLAN_3010                3010  bond0       database    200.190.3.59     255.255.255.0   200.190.3.1      1
        VLAN_3030                3030  bond0       management  200.200.192.156  255.255.255.0   200.200.192.1    0
        VLAN_3030                3030  bond0       management  200.200.192.157  255.255.255.0   200.200.192.1    1
[root@odas1 ~]#

Firmware and patch

As described in the documentation, the reimage process does not update firmware and we need to this after creating the appliance. This point ins controversy, because there is no information when need to do this, you can run before or after creating the appliance. Here I made after the create, but if you are in really old version and have a lot of old firmware running maybe you can run before the create to avoid issues.

The funny part is that the firmware’s are inside of one the default patch file. Again, you need to search in patch search page because Oracle removed from 88888.1 page these links (look the path number in the example below). So, the procedure to update firmware is the same to apply a patch in ODA and need to download the same patch version. As usual, the first step is to upload the files:

[root@odas1 ~]# cd /tmp/deploy/
[root@odas1 deploy]# 
[root@odas1 deploy]# unzip -qa p28864490_183000_Linux-x86-64_1of3.zip
[root@odas1 deploy]# unzip -qa p28864490_183000_Linux-x86-64_2of3.zip
[root@odas1 deploy]# unzip -qa p28864490_183000_Linux-x86-64_3of3.zip
[root@odas1 deploy]#
[root@odas1 deploy]# rm -rf p*zip
[root@odas1 deploy]#
[root@odas1 deploy]# ls -l
total 15441928
-rw-r--r-- 1 root root 2767978208 Dec 13 14:50 oda-sm-18.3.0.0.0-181205-server1of3.zip
-rwxr-xr-x 1 root root 6835610213 Dec 13 15:08 oda-sm-18.3.0.0.0-181205-server2of3.zip
-rwxr-xr-x 1 root root 6193475789 Dec 13 15:22 oda-sm-18.3.0.0.0-181205-server3of3.zip
-rw-r--r-- 1 root root       1155 Dec 13 15:22 README.txt
[root@odas1 deploy]#
[root@odas1 deploy]#

If you want to check what are you running, just execute odacli describe-component (look ilom version as an example):

[root@odas1 ~]# odacli describe-component
System Version
---------------
18.3.0.0.0

System node Name
---------------
odas1

Local System Version
---------------
18.3.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       18.3.0.0.0            up-to-date

GI                                        18.3.0.0.180717       up-to-date

DB                                        18.3.0.0.180717       up-to-date

DCSAGENT                                  18.3.0.0.0            up-to-date

ILOM                                      3.2.9.23.r116695      4.0.2.26.b.r125868

BIOS                                      30110000              30130500

OS                                        6.10                  up-to-date

FIRMWARECONTROLLER {
[ c0 ]                                    4.650.00-7176         up-to-date
[ c1,c2 ]                                 13.00.00.00           up-to-date
}

FIRMWAREEXPANDER                          0018                  up-to-date

FIRMWAREDISK {
[ c0d0,c0d1 ]                             A7E0                  up-to-date
[ c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,     PAG1                  up-to-date
c1d7,c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,
c1d14,c1d15,c2d0,c2d1,c2d2,c2d3,c2d4,
c2d5,c2d6,c2d7,c2d8,c2d9,c2d10,c2d11,
c2d12,c2d13,c2d14,c2d15 ]
[ c1d16,c1d17,c1d18,c1d19,c1d20,c1d21,    A29A                  up-to-date
c1d22,c1d23,c2d16,c2d17,c2d18,c2d19,
c2d20,c2d21,c2d22,c2d23 ]
}

System node Name
---------------
odas2

Local System Version
---------------
18.3.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       18.3.0.0.0            up-to-date

GI                                        18.3.0.0.180717       up-to-date

DB                                        18.3.0.0.180717       up-to-date

DCSAGENT                                  18.3.0.0.0            up-to-date

ILOM                                      3.2.9.23.r116695      4.0.2.26.b.r125868

BIOS                                      30110000              30130500

OS                                        6.10                  up-to-date

FIRMWARECONTROLLER {
[ c0 ]                                    4.650.00-7176         up-to-date
[ c1,c2 ]                                 13.00.00.00           up-to-date
}

FIRMWAREEXPANDER                          0018                  up-to-date

FIRMWAREDISK {
[ c0d0,c0d1 ]                             A7E0                  up-to-date
[ c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,     PAG1                  up-to-date
c1d7,c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,
c1d14,c1d15,c2d0,c2d1,c2d2,c2d3,c2d4,
c2d5,c2d6,c2d7,c2d8,c2d9,c2d10,c2d11,
c2d12,c2d13,c2d14,c2d15 ]
[ c1d16,c1d17,c1d18,c1d19,c1d20,c1d21,    A29A                  up-to-date
c1d22,c1d23,c2d16,c2d17,c2d18,c2d19,
c2d20,c2d21,c2d22,c2d23 ]
}


[root@odas1 ~]#

After that you can upload the files in the repository (look the path to the files):

[root@odas1 deploy]# odacli update-repository -f /tmp/deploy/oda-sm-18.3.0.0.0-181205-server1of3.zip,/tmp/deploy/oda-sm-18.3.0.0.0-181205-server2of3.zip,/tmp/deploy/oda-sm-18.3.0.0.0-181205-server3of3.zip
{
  "jobId" : "cdfd2e72-777b-4ccb-a670-17b88c7cc102",
  "status" : "Created",
  "message" : "/tmp/deploy/oda-sm-18.3.0.0.0-181205-server1of3.zip,/tmp/deploy/oda-sm-18.3.0.0.0-181205-server2of3.zip,/tmp/deploy/oda-sm-18.3.0.0.0-181205-server3of3.zip",
  "reports" : [ ],
  "createTimestamp" : "May 22, 2019 15:01:03 PM CEST",
  "resourceList" : [ ],
  "description" : "Repository Update",
  "updatedTime" : "May 22, 2019 15:01:03 PM CEST"
}
[root@odas1 deploy]#
[root@odas1 deploy]#
[root@odas1 deploy]# /opt/oracle/dcs/bin/odacli describe-job -i cdfd2e72-777b-4ccb-a670-17b88c7cc102

Job details
----------------------------------------------------------------
                     ID:  cdfd2e72-777b-4ccb-a670-17b88c7cc102
            Description:  Repository Update
                 Status:  Success
                Created:  May 22, 2019 3:01:03 PM CEST
                Message:  /tmp/deploy/oda-sm-18.3.0.0.0-181205-server1of3.zip,/tmp/deploy/oda-sm-18.3.0.0.0-181205-server2of3.zip,/tmp/deploy/oda-sm-18.3.0.0.0-181205-server3of3.zip

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Check AvailableSpace                     May 22, 2019 3:01:03 PM CEST        May 22, 2019 3:01:03 PM CEST        Success
Setting up ssh equivalance               May 22, 2019 3:01:03 PM CEST        May 22, 2019 3:01:03 PM CEST        Success
Copy BundleFile                          May 22, 2019 3:01:03 PM CEST        May 22, 2019 3:02:47 PM CEST        Success
Validating CopiedFile                    May 22, 2019 3:02:49 PM CEST        May 22, 2019 3:03:23 PM CEST        Success
Unzip bundle                             May 22, 2019 3:03:23 PM CEST        May 22, 2019 3:05:43 PM CEST        Success
Unzip bundle                             May 22, 2019 3:05:43 PM CEST        May 22, 2019 3:08:08 PM CEST        Success
Delete PatchBundles                      May 22, 2019 3:08:08 PM CEST        May 22, 2019 3:08:11 PM CEST        Success
Removing ssh keys                        May 22, 2019 3:08:11 PM CEST        May 22, 2019 3:08:26 PM CEST        Success

[root@odas1 deploy]#
[root@odas1 deploy]#

After the success to upload you first update the storage using odacli update-storage command to upload to desired version (the version are in the filename or check what you downloaded from MOS):

[root@odas1 ~]# odacli update-storage -v 18.3.0.0.0 --rolling
{
  "jobId" : "18d1e5f8-7c31-4ce7-8241-f7b2ed5673cd",
  "status" : "Created",
  "message" : "Success of Storage Update may trigger reboot of node after 4-5 minutes. Please wait till node restart",
  "reports" : [ ],
  "createTimestamp" : "May 22, 2019 15:25:24 PM CEST",
  "resourceList" : [ ],
  "description" : "Storage Firmware Patching",
  "updatedTime" : "May 22, 2019 15:25:24 PM CEST"
}
[root@odas1 ~]#
[root@odas1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i 18d1e5f8-7c31-4ce7-8241-f7b2ed5673cd

Job details
----------------------------------------------------------------
                     ID:  18d1e5f8-7c31-4ce7-8241-f7b2ed5673cd
            Description:  Storage Firmware Patching
                 Status:  Success
                Created:  May 22, 2019 3:25:24 PM CEST
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Applying Firmware Disk Patches           May 22, 2019 3:25:24 PM CEST        May 22, 2019 3:25:45 PM CEST        Success
Applying Firmware Controller Patches     May 22, 2019 3:25:45 PM CEST        May 22, 2019 3:25:58 PM CEST        Success

[root@odas1 ~]#

After you can update the server (running odacli update-server). Here, since you are “updating” to the same version that you are running, the steps related to yum does not execute and just firmware part runs:

[root@odas1 ~]# odacli update-server -v 18.3.0.0.0
{
  "jobId" : "8bc7cc15-a543-4e26-ade4-8dd578ea67ba",
  "status" : "Created",
  "message" : "Success of Server Update may trigger reboot of node after 4-5 minutes. Please wait till node restart",
  "reports" : [ ],
  "createTimestamp" : "May 22, 2019 15:29:15 PM CEST",
  "resourceList" : [ ],
  "description" : "Server Patching",
  "updatedTime" : "May 22, 2019 15:29:15 PM CEST"
}
[root@odas1 ~]# 
[root@odas1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i 8bc7cc15-a543-4e26-ade4-8dd578ea67ba

Job details
----------------------------------------------------------------
                     ID:  8bc7cc15-a543-4e26-ade4-8dd578ea67ba
            Description:  Server Patching
                 Status:  Running
                Created:  May 22, 2019 3:29:15 PM CEST
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Patch location validation                May 22, 2019 3:29:16 PM CEST        May 22, 2019 3:29:16 PM CEST        Success
Patch location validation                May 22, 2019 3:29:16 PM CEST        May 22, 2019 3:29:16 PM CEST        Success
dcs-controller upgrade                   May 22, 2019 3:29:19 PM CEST        May 22, 2019 3:29:19 PM CEST        Success
dcs-controller upgrade                   May 22, 2019 3:29:19 PM CEST        May 22, 2019 3:29:20 PM CEST        Success
Patch location validation                May 22, 2019 3:29:20 PM CEST        May 22, 2019 3:29:21 PM CEST        Success
Patch location validation                May 22, 2019 3:29:20 PM CEST        May 22, 2019 3:29:21 PM CEST        Success
dcs-cli upgrade                          May 22, 2019 3:29:21 PM CEST        May 22, 2019 3:29:21 PM CEST        Success
dcs-cli upgrade                          May 22, 2019 3:29:21 PM CEST        May 22, 2019 3:29:21 PM CEST        Success
Creating repositories using yum          May 22, 2019 3:29:29 PM CEST        May 22, 2019 3:29:36 PM CEST        Success
Creating repositories using yum          May 22, 2019 3:29:36 PM CEST        May 22, 2019 3:29:36 PM CEST        Success
Creating repositories using yum          May 22, 2019 3:29:36 PM CEST        May 22, 2019 3:29:36 PM CEST        Success
Creating repositories using yum          May 22, 2019 3:29:36 PM CEST        May 22, 2019 3:29:36 PM CEST        Success
Creating repositories using yum          May 22, 2019 3:29:36 PM CEST        May 22, 2019 3:29:36 PM CEST        Success
Creating repositories using yum          May 22, 2019 3:29:37 PM CEST        May 22, 2019 3:29:37 PM CEST        Success
Creating repositories using yum          May 22, 2019 3:29:37 PM CEST        May 22, 2019 3:29:37 PM CEST        Success
Updating YumPluginVersionLock rpm        May 22, 2019 3:29:37 PM CEST        May 22, 2019 3:29:37 PM CEST        Success
Applying OS Patches                      May 22, 2019 3:29:37 PM CEST        May 22, 2019 3:31:38 PM CEST        Success
Creating repositories using yum          May 22, 2019 3:31:38 PM CEST        May 22, 2019 3:31:39 PM CEST        Success
Applying HMP Patches                     May 22, 2019 3:31:39 PM CEST        May 22, 2019 3:31:39 PM CEST        Success
Patch location validation                May 22, 2019 3:31:40 PM CEST        May 22, 2019 3:31:40 PM CEST        Success
Patch location validation                May 22, 2019 3:31:40 PM CEST        May 22, 2019 3:31:40 PM CEST        Success
oda-hw-mgmt upgrade                      May 22, 2019 3:31:40 PM CEST        May 22, 2019 3:31:41 PM CEST        Success
oda-hw-mgmt upgrade                      May 22, 2019 3:31:41 PM CEST        May 22, 2019 3:31:41 PM CEST        Success
OSS Patching                             May 22, 2019 3:31:41 PM CEST        May 22, 2019 3:31:43 PM CEST        Success
Applying Firmware Disk Patches           May 22, 2019 3:32:02 PM CEST        May 22, 2019 3:32:15 PM CEST        Success
Applying Firmware Expander Patches       May 22, 2019 3:32:27 PM CEST        May 22, 2019 3:32:34 PM CEST        Success
Applying Firmware Controller Patches     May 22, 2019 3:32:47 PM CEST        May 22, 2019 3:32:55 PM CEST        Success
Checking Ilom patch Version              May 22, 2019 3:32:56 PM CEST        May 22, 2019 3:32:59 PM CEST        Success
Checking Ilom patch Version              May 22, 2019 3:32:59 PM CEST        May 22, 2019 3:33:01 PM CEST        Success
Patch location validation                May 22, 2019 3:33:01 PM CEST        May 22, 2019 3:33:02 PM CEST        Success
Patch location validation                May 22, 2019 3:33:01 PM CEST        May 22, 2019 3:33:02 PM CEST        Success
Save password in Wallet                  May 22, 2019 3:33:03 PM CEST        May 22, 2019 3:33:04 PM CEST        Success
Apply Ilom patch                         May 22, 2019 3:33:04 PM CEST        May 22, 2019 3:45:07 PM CEST        Success
Apply Ilom patch                         May 22, 2019 3:45:07 PM CEST        May 22, 2019 3:56:45 PM CEST        Success
Copying Flash Bios to Temp location      May 22, 2019 3:56:45 PM CEST        May 22, 2019 3:56:45 PM CEST        Success
Copying Flash Bios to Temp location      May 22, 2019 3:56:45 PM CEST        May 22, 2019 3:56:45 PM CEST        Success
Starting the clusterware                 May 22, 2019 3:58:41 PM CEST        May 22, 2019 4:00:09 PM CEST        Success
clusterware patch verification           May 22, 2019 4:00:09 PM CEST        May 22, 2019 4:00:10 PM CEST        Success
clusterware patch verification           May 22, 2019 4:00:09 PM CEST        May 22, 2019 4:00:10 PM CEST        Success
Patch location validation                May 22, 2019 4:00:10 PM CEST        May 22, 2019 4:00:10 PM CEST        Success
Patch location validation                May 22, 2019 4:00:10 PM CEST        May 22, 2019 4:00:10 PM CEST        Success
Opatch updation                          May 22, 2019 4:00:11 PM CEST        May 22, 2019 4:00:11 PM CEST        Success
Opatch updation                          May 22, 2019 4:00:11 PM CEST        May 22, 2019 4:00:11 PM CEST        Success
Patch conflict check                     May 22, 2019 4:00:11 PM CEST        May 22, 2019 4:00:11 PM CEST        Success
Patch conflict check                     May 22, 2019 4:00:11 PM CEST        May 22, 2019 4:00:11 PM CEST        Success
clusterware upgrade                      May 22, 2019 4:00:11 PM CEST        May 22, 2019 4:00:11 PM CEST        Success
clusterware upgrade                      May 22, 2019 4:00:11 PM CEST        May 22, 2019 4:00:11 PM CEST        Success
Updating GiHome version                  May 22, 2019 4:00:11 PM CEST        May 22, 2019 4:00:12 PM CEST        Success
Updating GiHome version                  May 22, 2019 4:00:11 PM CEST        May 22, 2019 4:00:12 PM CEST        Success
Update System version                    May 22, 2019 4:00:33 PM CEST        May 22, 2019 4:00:33 PM CEST        Success
Update System version                    May 22, 2019 4:00:33 PM CEST        May 22, 2019 4:00:33 PM CEST        Success
preRebootNode Actions                    May 22, 2019 4:00:33 PM CEST        May 22, 2019 4:00:33 PM CEST        Running

[root@odas1 ~]#

After, reboot both nodes!

As example check that ilom was updated:

[root@odas1 ~]# odacli describe-component
System Version
---------------
18.3.0.0.0

System node Name
---------------
odas1

Local System Version
---------------
18.3.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       18.3.0.0.0            up-to-date

GI                                        18.3.0.0.180717       up-to-date

DB                                        18.3.0.0.180717       up-to-date

DCSAGENT                                  18.3.0.0.0            up-to-date

ILOM                                      4.0.2.26.b.r125868    up-to-date

BIOS                                      30130500              up-to-date

OS                                        6.10                  up-to-date

FIRMWARECONTROLLER {
[ c0 ]                                    4.650.00-7176         up-to-date
[ c1,c2 ]                                 13.00.00.00           up-to-date
}

FIRMWAREEXPANDER                          0018                  up-to-date

FIRMWAREDISK {
[ c0d0,c0d1 ]                             A7E0                  up-to-date
[ c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,     PAG1                  up-to-date
c1d7,c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,
c1d14,c1d15,c2d0,c2d1,c2d2,c2d3,c2d4,
c2d5,c2d6,c2d7,c2d8,c2d9,c2d10,c2d11,
c2d12,c2d13,c2d14,c2d15 ]
[ c1d16,c1d17,c1d18,c1d19,c1d20,c1d21,    A29A                  up-to-date
c1d22,c1d23,c2d16,c2d17,c2d18,c2d19,
c2d20,c2d21,c2d22,c2d23 ]
}

System node Name
---------------
odas2

Local System Version
---------------
18.3.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       18.3.0.0.0            up-to-date

GI                                        18.3.0.0.180717       up-to-date

DB                                        18.3.0.0.180717       up-to-date

DCSAGENT                                  18.3.0.0.0            up-to-date

ILOM                                      4.0.2.26.b.r125868    up-to-date

BIOS                                      30130500              up-to-date

OS                                        6.10                  up-to-date

FIRMWARECONTROLLER {
[ c0 ]                                    4.650.00-7176         up-to-date
[ c1,c2 ]                                 13.00.00.00           up-to-date
}

FIRMWAREEXPANDER                          0018                  up-to-date

FIRMWAREDISK {
[ c0d0,c0d1 ]                             A7E0                  up-to-date
[ c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,     PAG1                  up-to-date
c1d7,c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,
c1d14,c1d15,c2d0,c2d1,c2d2,c2d3,c2d4,
c2d5,c2d6,c2d7,c2d8,c2d9,c2d10,c2d11,
c2d12,c2d13,c2d14,c2d15 ]
[ c1d16,c1d17,c1d18,c1d19,c1d20,c1d21,    A29A                  up-to-date
c1d22,c1d23,c2d16,c2d17,c2d18,c2d19,
c2d20,c2d21,c2d22,c2d23 ]
}


[root@odas1 ~]#
[root@odas1 ~]#

Create Oracle Homes and the databases

After update all the firmware’s you can upload the clone files for Oracle Homes versions that you want and create it. The procedure is the same than before, upload the files to (and just) ODA node 1 in somewhere and after that upload the repository and create the homes.

Upload the files and update the repository. Look below that I made one for each version:

[root@odas1 ~]# cd /tmp/deploy/
[root@odas1 deploy]#
[root@odas1 deploy]# unzip -qa p23494992_183000_Linux-x86-64.zip
[root@odas1 deploy]# rm p23494992_183000_Linux-x86-64.zip
rm: remove regular file `p23494992_183000_Linux-x86-64.zip'? y
[root@odas1 deploy]#
[root@odas1 deploy]# unzip -qa p23494997_183000_Linux-x86-64.zip
replace README.txt? [y]es, [n]o, [A]ll, [N]one, [r]ename: A
[root@odas1 deploy]# rm p23494997_183000_Linux-x86-64.zip
rm: remove regular file `p23494997_183000_Linux-x86-64.zip'? y
[root@odas1 deploy]# 
[root@odas1 deploy]# odacli update-repository -f /tmp/deploy/odacli-dcs-18.3.0.0.0-180905-DB-12.1.0.2.zip
{
  "jobId" : "b88c4922-1107-41f5-b7cd-e1648093051b",
  "status" : "Created",
  "message" : "/tmp/deploy/odacli-dcs-18.3.0.0.0-180905-DB-12.1.0.2.zip",
  "reports" : [ ],
  "createTimestamp" : "May 22, 2019 16:34:28 PM CEST",
  "resourceList" : [ ],
  "description" : "Repository Update",
  "updatedTime" : "May 22, 2019 16:34:28 PM CEST"
}
[root@odas1 deploy]#
[root@odas1 deploy]# /opt/oracle/dcs/bin/odacli describe-job -i b88c4922-1107-41f5-b7cd-e1648093051b

Job details
----------------------------------------------------------------
                     ID:  b88c4922-1107-41f5-b7cd-e1648093051b
            Description:  Repository Update
                 Status:  Success
                Created:  May 22, 2019 4:34:28 PM CEST
                Message:  /tmp/deploy/odacli-dcs-18.3.0.0.0-180905-DB-12.1.0.2.zip

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Check AvailableSpace                     May 22, 2019 4:34:28 PM CEST        May 22, 2019 4:34:28 PM CEST        Success
Setting up ssh equivalance               May 22, 2019 4:34:28 PM CEST        May 22, 2019 4:34:29 PM CEST        Success
Copy BundleFile                          May 22, 2019 4:34:29 PM CEST        May 22, 2019 4:34:54 PM CEST        Success
Validating CopiedFile                    May 22, 2019 4:34:54 PM CEST        May 22, 2019 4:35:07 PM CEST        Success
Unzip bundle                             May 22, 2019 4:35:07 PM CEST        May 22, 2019 4:35:42 PM CEST        Success
Unzip bundle                             May 22, 2019 4:35:46 PM CEST        May 22, 2019 4:36:23 PM CEST        Success
Delete PatchBundles                      May 22, 2019 4:36:24 PM CEST        May 22, 2019 4:36:25 PM CEST        Success
Removing ssh keys                        May 22, 2019 4:36:25 PM CEST        May 22, 2019 4:36:47 PM CEST        Success

[root@odas1 deploy]#
[root@odas1 deploy]# odacli update-repository -f /tmp/deploy/odacli-dcs-18.3.0.0.0-180905-DB-11.2.0.4.zip
{
  "jobId" : "6698aa6d-1a44-4b1c-bda1-1e691a41a133",
  "status" : "Created",
  "message" : "/tmp/deploy/odacli-dcs-18.3.0.0.0-180905-DB-11.2.0.4.zip",
  "reports" : [ ],
  "createTimestamp" : "May 22, 2019 16:38:53 PM CEST",
  "resourceList" : [ ],
  "description" : "Repository Update",
  "updatedTime" : "May 22, 2019 16:38:53 PM CEST"
}
[root@odas1 deploy]#
[root@odas1 deploy]# /opt/oracle/dcs/bin/odacli describe-job -i 6698aa6d-1a44-4b1c-bda1-1e691a41a133

Job details
----------------------------------------------------------------
                     ID:  6698aa6d-1a44-4b1c-bda1-1e691a41a133
            Description:  Repository Update
                 Status:  Success
                Created:  May 22, 2019 4:38:53 PM CEST
                Message:  /tmp/deploy/odacli-dcs-18.3.0.0.0-180905-DB-11.2.0.4.zip

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Check AvailableSpace                     May 22, 2019 4:38:53 PM CEST        May 22, 2019 4:38:53 PM CEST        Success
Setting up ssh equivalance               May 22, 2019 4:38:53 PM CEST        May 22, 2019 4:38:53 PM CEST        Success
Copy BundleFile                          May 22, 2019 4:38:54 PM CEST        May 22, 2019 4:39:04 PM CEST        Success
Validating CopiedFile                    May 22, 2019 4:39:06 PM CEST        May 22, 2019 4:39:14 PM CEST        Success
Unzip bundle                             May 22, 2019 4:39:14 PM CEST        May 22, 2019 4:39:32 PM CEST        Success
Unzip bundle                             May 22, 2019 4:39:32 PM CEST        May 22, 2019 4:39:54 PM CEST        Success
Delete PatchBundles                      May 22, 2019 4:39:54 PM CEST        May 22, 2019 4:39:55 PM CEST        Success
Removing ssh keys                        May 22, 2019 4:39:55 PM CEST        May 22, 2019 4:39:55 PM CEST        Success

[root@odas1 deploy]#
[root@odas1 deploy]#

After that, you can create the Oracle home for each version using the command odacli create-dbhome. 12c:

[root@odas1 deploy]#
[root@odas1 deploy]# odacli create-dbhome -v 12.1.0.2.180717

Job details
----------------------------------------------------------------
                     ID:  5ddcdd49-2d1a-404d-b4b8-7ab79c5f9707
            Description:  Database Home OraDB12102_home1 creation with version :12.1.0.2.180717
                 Status:  Created
                Created:  May 22, 2019 4:42:58 PM CEST
                Message:  Create Database Home

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------

[root@odas1 deploy]#
[root@odas1 deploy]# /opt/oracle/dcs/bin/odacli describe-job -i 5ddcdd49-2d1a-404d-b4b8-7ab79c5f9707

Job details
----------------------------------------------------------------
                     ID:  5ddcdd49-2d1a-404d-b4b8-7ab79c5f9707
            Description:  Database Home OraDB12102_home1 creation with version :12.1.0.2.180717
                 Status:  Success
                Created:  May 22, 2019 4:42:58 PM CEST
                Message:  Create Database Home

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Setting up ssh equivalance               May 22, 2019 4:42:59 PM CEST        May 22, 2019 4:42:59 PM CEST        Success
Validating dbHome available space        May 22, 2019 4:42:59 PM CEST        May 22, 2019 4:42:59 PM CEST        Success
Validating dbHome available space        May 22, 2019 4:42:59 PM CEST        May 22, 2019 4:42:59 PM CEST        Success
Creating DbHome Directory                May 22, 2019 4:42:59 PM CEST        May 22, 2019 4:42:59 PM CEST        Success
Extract DB clones                        May 22, 2019 4:42:59 PM CEST        May 22, 2019 4:46:22 PM CEST        Success
Clone Db home                            May 22, 2019 4:46:22 PM CEST        May 22, 2019 4:47:32 PM CEST        Success
Enable DB options                        May 22, 2019 4:47:32 PM CEST        May 22, 2019 4:47:41 PM CEST        Success
Run Root DB scripts                      May 22, 2019 4:47:41 PM CEST        May 22, 2019 4:47:42 PM CEST        Success
Removing ssh keys                        May 22, 2019 4:47:48 PM CEST        May 22, 2019 4:47:48 PM CEST        Success

[root@odas1 deploy]#
[root@odas1 deploy]#

And 11g:

[root@odas1 deploy]# odacli create-dbhome -v 11.2.0.4.180717

Job details
----------------------------------------------------------------
                     ID:  d4512833-3198-4384-b0d5-3588e3ec8cdb
            Description:  Database Home OraDB11204_home1 creation with version :11.2.0.4.180717
                 Status:  Created
                Created:  May 22, 2019 4:50:06 PM CEST
                Message:  Create Database Home

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------

[root@odas1 deploy]#
[root@odas1 deploy]# /opt/oracle/dcs/bin/odacli describe-job -i d4512833-3198-4384-b0d5-3588e3ec8cdb

Job details
----------------------------------------------------------------
                     ID:  d4512833-3198-4384-b0d5-3588e3ec8cdb
            Description:  Database Home OraDB11204_home1 creation with version :11.2.0.4.180717
                 Status:  Success
                Created:  May 22, 2019 4:50:06 PM CEST
                Message:  Create Database Home

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Setting up ssh equivalance               May 22, 2019 4:50:07 PM CEST        May 22, 2019 4:50:07 PM CEST        Success
Validating dbHome available space        May 22, 2019 4:50:07 PM CEST        May 22, 2019 4:50:07 PM CEST        Success
Validating dbHome available space        May 22, 2019 4:50:07 PM CEST        May 22, 2019 4:50:07 PM CEST        Success
Creating DbHome Directory                May 22, 2019 4:50:07 PM CEST        May 22, 2019 4:50:07 PM CEST        Success
Extract DB clones                        May 22, 2019 4:50:07 PM CEST        May 22, 2019 4:51:59 PM CEST        Success
Clone Db home                            May 22, 2019 4:52:00 PM CEST        May 22, 2019 4:53:06 PM CEST        Success
Enable DB options                        May 22, 2019 4:53:06 PM CEST        May 22, 2019 4:53:11 PM CEST        Success
Run Root DB scripts                      May 22, 2019 4:53:11 PM CEST        May 22, 2019 4:53:11 PM CEST        Success
Removing ssh keys                        May 22, 2019 4:53:16 PM CEST        May 22, 2019 4:53:17 PM CEST        Success

[root@odas1 deploy]#
[root@odas1 deploy]#

And after that you can use the odacli list-dbhomes to see the options for database creation:

[root@odas1 deploy]# odacli list-dbhomes

ID                                       Name                 DB Version                               Home Location                                 Status
---------------------------------------- -------------------- ---------------------------------------- --------------------------------------------- ----------
886021e5-9bbf-4a14-9b50-398ddd00bfd0     OraDB18000_home1     18.3.0.0.180717                          /u01/app/oracle/product/18.0.0.0/dbhome_1     Configured
d791001f-b812-49fd-94f5-b11c5f580a30     OraDB12102_home1     12.1.0.2.180717                          /u01/app/oracle/product/12.1.0.2/dbhome_1     Configured
d162ff30-dc0d-4c4b-810e-c8f513dc9f62     OraDB11204_home1     11.2.0.4.180717                          /u01/app/oracle/product/11.2.0.4/dbhome_1     Configured

[root@odas1 deploy]#

As an example, if you want to delete the database that was created during the creation of appliance just execute odacli delete-database:

[root@odas1 ~]# odacli delete-database --dbName mfod
{
  "jobId" : "69adb079-e5e4-4d64-8b20-4eee73c65491",
  "status" : "Running",
  "message" : null,
  "reports" : [ ],
  "createTimestamp" : "May 22, 2019 16:17:27 PM CEST",
  "resourceList" : [ ],
  "description" : "Database service deletion with db name: mfod with id : 938b616a-99ed-4466-8d76-820d3489517d",
  "updatedTime" : "May 22, 2019 16:17:27 PM CEST"
}
[root@odas1 ~]#

Finish and clean the install

After all the steps listed until now, you have your ODA reimaged, updated, and with 18c, 12c and 11g available version for database creation. To finish you can update some files as needed. Examples of important files are /etc/fstab, /etc/sysctl.conf, /etc/sysconfig/network-scripts/route* and /etc/sysconfig/network-scripts/rules* (both related with routing tables for vlans – if needed), and add the listener for all the vlans that you created (if needed of course).

To finish, reboot both nodes and change the root password.

 

The reference that can be consulted to help you to understand the reimage and deployment procedure:

https://docs.oracle.com/en/engineered-systems/oracle-database-appliance/18.3/cmtxl/oracle-database-appliance-x7-2-deployment-and-users-guide.pdf         

https://docs.oracle.com/en/engineered-systems/oracle-database-appliance/18.3/cmtxd/oracle-database-appliance-x5-2-x4-2-x3-2-deployment-and-users-guide.pdf

https://docs.oracle.com/en/engineered-systems/oracle-database-appliance/18.3/cmtxl/odacli-comparison-chart.html#GUID-567E862F-F4A4-4E1E-A485-BD1945BEC673

https://docs.oracle.com/en/engineered-systems/oracle-database-appliance/18.3/cmtxl/postinstallation-tasks-oda.html#GUID-A256DFBD-9335-41D2-9BD5-A1BA6AAC4C1E

https://docs.oracle.com/cd/E80799_01/doc.121/e80521/GUID-4EAD207D-41FF-4FD7-8804-DE1B2EB7BE90.htm#CMTXG-GUID-E0CC0E23-A339-45D8-AFA2-09732DC6BC18

https://docs.oracle.com/en/engineered-systems/oracle-database-appliance/18.3/cmtxl/reimaging-oda.html#GUID-6DEDCA82-94B7-4155-9FBD-3B1FDC232FCC

https://docs.oracle.com/en/engineered-systems/oracle-database-appliance/18.5/cmtxl/create-appliance-using-json-file.html#GUID-42250FD2-EA91-4457-9ED7-CA3A2A863B40

https://docs.oracle.com/cd/E75550_01/doc.121/e77147/GUID-F99D3337-E913-405D-BE3F-85C31103A26A.htm#CMTAR-GUID-CF1FECF0-722F-4B43-82AA-525EA169B3E8

 

Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community. Post protected by copyright.”

One thought on “Reimage ODA

  1. Pingback: Patch ODA from 18.3 to 19.8. Part 1 – 18.3 to 18.8 | Fernando Simon

Leave a Reply

Your email address will not be published. Required fields are marked *