Creating a RAC Flex Cluster using ansible-oracle

As of version 1.3 it is possible to create a RAC Flex Cluster using ansible-oracle. From a ansible-oracle configuration perspective there is not a huge difference from a normal ‘standard’ cluster, basically a few new parameters. There are other differences though, specifically in how you have to run the playbook and deal with the inventory configuration.
In a ‘standard’ cluster, you have your database cluster nodes and that’s it (basically). In a Flex Cluster configuration there are 2 types of nodes:

  • Hub nodes. These have access to the shared storage and will house your databases
  • Leaf nodes. These nodes are connected to the interconnect network, but not to the shared storage. So for instance, you could run your application on these nodes.

So given that, from an ansible-oracle perspective, it presents a few challenges. With a normal cluster, you could just run the playbook against all your cluster nodes as they are all the same.
Now, when building a Flex Cluster, there are a few things that should only be done on the hub nodes (configuring shared storage, install the database server and create the database(s)
And how do we do that? With a little Ansible inventory ninja-ism.

New Parameters

So the parameters that have been added are these.

  • oracle_gi_cluster_type. The type of cluster(standard/flex)
  • oracle_gi_gns_subdomain. The subdomain that has been delegated to the GNS.
  • oracle_gi_gns_vip. The VIP-adress of the GNS
  • hostgroup_hub: The default is  “{{ hostgroup }}-hub”.
  • hostgroup_leaf: The default is “{{ hostgroup }}-leaf”

I’m not going to go into any sort of detail regarding the Grid Naming Service (GNS), as others have written about it already, and the documentation is here, but GNS is basically a way to have the Grid Infrastructure Cluster provide names, ip-addresses and name resolution for the cluster nodes, instead of using a DNS. You can choose how much you want the GNS to do, but at a minimum you’d need a static ip-address for the GNS VIP.

Setting up the inventory

So this time we’ll call the hostgroup orarac-flex. We also need to define which hosts should be hub-nodes and which should be leaf-nodes

[orarac-flex-hub] <-- These are the hub-nodes
orarac03    
orarac04

[orarac-flex-leaf] <-- These are the leaf-nodes
orarac06

[orarac-flex:children] <-- This is the hostgroup that should be specified in the configuration (i.e hostgroup=orarac-flex)
orarac-flex-hub
orarac-flex-leaf

When setting up a standard cluster, the nodes making up the GI cluster and database cluster are usually the same, and during the installation the hostnames are picked up automatically by looping through the members of the hostgroup that you define with the parameter ‘hostgroup’. With a flex cluster, the members of the GI cluster are hubnodes + leaf nodes but the database cluster consists only of the hub-nodes and that is why the inventory definition (and subsequently the playbook) have to look a bit different.

---
- name: Host configuration
  hosts: orarac-flex   <- This runs on all nodes (maps to orarac-flex:children)
  user: ansible
  sudo: yes
  roles:
   - common
   - orahost

- name: Storage Configuration 
  hosts: orarac-flex-hub  <-- Only runs on the HUB nodes (maps to orarac-flex-hub)
  user: ansible
  sudo: yes
  roles:
   - orahost-storage

- name: Oracle Grid Infrastructure installation
  hosts: orarac-flex  <-- Runs on all nodes (maps to orarac-flex:children)
  user: ansible
  sudo: yes
  roles:
   - oraswgi-install

- name: ASM Configuration, Database Server Installation & Database Creation
  hosts: orarac-flex-hub <-- Only runs on the HUB nodes (maps to orarac-flex-hub)
  user: ansible
  sudo: yes
  roles:
   - oraasm-createdg
   - oraswdb-install
   - oradb-create

Note how we don’t not have any plays that will run only against the leaf nodes, but that could easily be added if you wanted to deploy an application to the leaf nodes after the cluster is configured.
But on the other hand, you’d probably do that in a different playbook.

Host_group configuration

This is what the configuration should look like. The variables in bold are the one’s that differentiates a Standard Cluster from a Flex Cluster, the rest is the same.

---
 hostgroup: orarac-flex
 role_separation: true
 device_persistence: udev
 configure_interconnect: true 
 configure_ssh: true 
 configure_host_disks: true 
 configure_cluster: true
################ Grid Infrastructure specifics ################
 oracle_install_option_gi: CRS_CONFIG 
 oracle_install_version_gi: 12.1.0.2
 oracle_gi_cluster_type: flex 
 oracle_gi_gns_subdomain: gns.discworld.lab
 oracle_gi_gns_vip: orarac-dc2-gns.gns.discworld.lab
 oracle_password: Oracle123
 oracle_scan: orarac-scan-dc2.discworld.lab
 oracle_vip: -vip
 oracle_scan_port: 1521
 oracle_asm_init_dg: crs 

 oracle_databases: 
 - home: rachome1 
   oracle_version_db: 12.1.0.2 
   oracle_edition: EE 
   oracle_db_name: racdba 
   oracle_db_passwd: Oracle123 
   oracle_db_type: RAC 
   is_container: "false" 
   pdb_prefix: pdb
   num_pdbs: 1
   is_racone: "false" 
   storage_type: ASM 
   service_name: racdb_serv 
   oracle_init_params: "open_cursors=300,processes=500" 
   oracle_db_mem_percent: 25 
   oracle_database_type: MULTIPURPOSE 
   redolog_size_in_mb: 100
   delete_db: false 

 oracle_dbf_dir_asm: "DATA" 
 oracle_reco_dir_asm: "FRA" 

 host_fs_layout: 
  u01: 
    {mntp: /u01, device: /dev/sdb, vgname: vgora, pvname: /dev/sdb1, lvname: lvora, fstype: ext4}
 
asm_diskgroups: 
  - crs
  - data
  - fra
 
asm_storage_layout: 
  crs:
    - {device: /dev/sdc, asmlabel: crs01}
  data:
    - {device: /dev/sdd, asmlabel: data01}
    - {device: /dev/sde, asmlabel: data02}
 fra:
    - {device: /dev/sdf, asmlabel: fra01}
    - {device: /dev/sdg, asmlabel: fra02}
    - {device: /dev/sdh, asmlabel: fra03}

Running the playbook

So the thing that we should pay attention to below is that the different plays will use the different parts of the inventory that we configured previously.

[miksan@ponderstibbons ansible-oracle]$ time ansible-playbook full-rac-flex-install.yml -i inventory/flex 
PLAY [Host configuration] ***************************************************** 
GATHERING FACTS *************************************************************** 
ok: [orarac03]
ok: [orarac04]
ok: [orarac06]

TASK: [common | Install EPEL Repo] ******************************************** 
ok: [orarac06]
ok: [orarac03]
ok: [orarac04]

TASK: [common | Get newest repo-file for OL6 (public-yum)] ******************** 
ok: [orarac04]
ok: [orarac03]
ok: [orarac06]

TASK: [common | Install common packages] ************************************** 
ok: [orarac04] => (item=screen,facter,procps,module-init-tools,ethtool,bc,bind-utils,nfs-utils,make,sysstat,openssh-clients,compat-libcap1,twm,collectl,rlwrap,tigervnc-server,ntp,expect,git,lvm2,xfsprogs,btrfs-progs)
ok: [orarac03] => (item=screen,facter,procps,module-init-tools,ethtool,bc,bind-utils,nfs-utils,make,sysstat,openssh-clients,compat-libcap1,twm,collectl,rlwrap,tigervnc-server,ntp,expect,git,lvm2,xfsprogs,btrfs-progs)
ok: [orarac06] => (item=screen,facter,procps,module-init-tools,ethtool,bc,bind-utils,nfs-utils,make,sysstat,openssh-clients,compat-libcap1,twm,collectl,rlwrap,tigervnc-server,ntp,expect,git,lvm2,xfsprogs,btrfs-progs)
................. SKIP ............
PLAY [Storage Configuration] ************************************************** 
GATHERING FACTS *************************************************************** 
ok: [orarac04]
ok: [orarac03]
TASK: [orahost-storage | ASMlib | Create device to label mappings for asm] **** 
skipping: [orarac04] => (item=crs)
skipping: [orarac04] => (item=data)
skipping: [orarac04] => (item=fra)
changed: [orarac03] => (item=crs)
changed: [orarac03] => (item=data)
changed: [orarac03] => (item=fra)
............ SKIP ......
PLAY [Oracle Grid Infrastructure installation] ******************************** 
....... SKIP ....
ok: [orarac06] => {
 "opatchls.stdout_lines": [
 "Oracle Interim Patch Installer version 12.1.0.1.3", 
 "Copyright (c) 2014, Oracle Corporation. All rights reserved.", 
 "", 
 "", 
 "Oracle Home : /u01/app/12.1.0.2/grid", 
 "Central Inventory : /u01/app/oraInventory", 
 " from : /u01/app/12.1.0.2/grid/oraInst.loc", 
 "OPatch version : 12.1.0.1.3", 
 "OUI version : 12.1.0.2.0", 
 "Log file location : /u01/app/12.1.0.2/grid/cfgtoollogs/opatch/opatch2014-11-26_13-05-48PM_1.log", 
 "", 
 "Lsinventory Output file location : /u01/app/12.1.0.2/grid/cfgtoollogs/opatch/lsinv/lsinventory2014-11-26_13-05-48PM.txt", 
 "", 
 "--------------------------------------------------------------------------------", 
 "Installed Top-level Products (1): ", 
 "", 
 "Oracle Grid Infrastructure 12c 12.1.0.2.0", 
 "There are 1 products installed in this Oracle Home.", 
 "", 
 "", 
 "There are no Interim patches installed in this Oracle Home.", 
 "", 
 "", 
 "Patch level status of Cluster nodes :", 
 "", 
 " Patching Level \t\t Nodes", 
 " -------------- \t\t -----", 
 " 0 \t\t orarac03,orarac04,orarac06", 
 "", 
 "--------------------------------------------------------------------------------", 
 "", 
 "OPatch succeeded."
 ]
}
TASK: [oraswgi-install | Check olsnodes (GI)] ********************************* 
changed: [orarac04]
changed: [orarac03]
changed: [orarac06]
TASK: [oraswgi-install | Check olsnodes (GI)] ********************************* 
ok: [orarac03] => {
 "olsnodes.stdout_lines": [
 "orarac03\t1\tActive\tHub\tUnpinned", 
 "orarac04\t2\tActive\tHub\tUnpinned", 
 "orarac06\t100\tActive\tLeaf\tUnpinned"
 ]
}
ok: [orarac06] => {
 "olsnodes.stdout_lines": [
 "orarac03\t1\tActive\tHub\tUnpinned", 
 "orarac04\t2\tActive\tHub\tUnpinned", 
 "orarac06\t100\tActive\tLeaf\tUnpinned"
 ]
}
ok: [orarac04] => {
 "olsnodes.stdout_lines": [
 "orarac03\t1\tActive\tHub\tUnpinned", 
 "orarac04\t2\tActive\tHub\tUnpinned", 
 "orarac06\t100\tActive\tLeaf\tUnpinned"
 ]
}
......... SKIP .....
TASK: [oradb-create | debug var=psout.stdout_lines] *************************** 
ok: [orarac03] => {
 "psout.stdout_lines": [
 "oracle 24346 1 0 13:23 ? 00:00:00 ora_pmon_racdba1", 
 "grid 41140 1 0 12:47 ? 00:00:00 asm_pmon_+ASM1", 
 "grid 54745 1 0 12:59 ? 00:00:00 mdb_pmon_-MGMTDB"
 ]
}
ok: [orarac04] => {
 "psout.stdout_lines": [
 "grid 33032 1 0 12:53 ? 00:00:00 asm_pmon_+ASM2", 
 "oracle 59452 1 0 13:23 ? 00:00:00 ora_pmon_racdba2"
 ]
}
PLAY RECAP ******************************************************************** 

orarac03 : ok=122 changed=77 unreachable=0 failed=0 
orarac04 : ok=113 changed=55 unreachable=0 failed=0 
orarac06 : ok=61 changed=36 unreachable=0 failed=0 

real 60m43.143s
user 0m13.592s
sys 0m8.279s

Ok, the playbook is finished and we got our database, but what kind of cluster did we end up with?

[miksan@ponderstibbons ansible-oracle]$ ansible orarac-flex -a "/u01/app/12.1.0.2/grid/bin/olsnodes -a -s -n -t" -i inventory/flex --limit orarac03 -s
orarac03 | success | rc=0 >>
orarac03 1 Active Hub Unpinned
orarac04 2 Active Hub Unpinned
orarac06 100 Active Leaf Unpinned

[miksan@ponderstibbons ansible-oracle]$ 
[miksan@ponderstibbons ansible-oracle]$ ansible orarac-flex -a "/u01/app/12.1.0.2/grid/bin/crsctl get node role config -all" -i inventory/flex --limit orarac03 -s
orarac03 | success | rc=0 >>
Node 'orarac03' configured role is 'hub'
Node 'orarac04' configured role is 'hub'
Node 'orarac06' configured role is 'leaf'

[miksan@ponderstibbons ansible-oracle]$ 
[miksan@ponderstibbons ansible-oracle]$ 
[miksan@ponderstibbons ansible-oracle]$ ansible orarac-flex-hub -m shell -a "source /home/grid/.profile_grid; export ORACLE_SID=+ASM1; asmcmd showclustermode" -i inventory/flex -s

orarac03 | success | rc=0 >>
ASM cluster : Flex mode enabled

orarac04 | success | rc=0 >>
ASM cluster : Flex mode enabled
[miksan@ponderstibbons ansible-oracle]$

And that’s it. One Flex Cluster configured and ready to use.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s