ORA-27106: system pages not available to allocate memory

If you have set use_large_pages=ONLY, server need to have enough huge pages to allocate the SGA in the number of pages available, if not, you might get this error while trying to start instance:

[oracle@DBNODE_A ~]$ srvctl start instance -d dbname -i INSTANCE_1
PRCR-1013 : Failed to start resource ora.dbname.db
PRCR-1064 : Failed to start resource ora.dbname.db on node DBNODE_A
CRS-5017: The resource action "ora.dbname.db start" encountered the following error: 
ORA-27106: system pages not available to allocate memory
Additional information: 6048
Additional information: 1
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/DBNODE_A/crs/trace/crsd_oraagent_oracle.trc".
CRS-2674: Start of 'ora.dbname.db' on 'DBNODE_A' failed
[oracle@DBNODE_A ~]$ 

How many pages available? 

[oracle@DBNODE_A trace]$  cat /sys/devices/system/node/node*/meminfo | fgrep Huge
Node 0 AnonHugePages:         0 kB
Node 0 ShmemHugePages:        0 kB
Node 0 HugePages_Total: 25600  
Node 0 HugePages_Free:   8705   <<--------------
Node 0 HugePages_Surp:      0
[oracle@DBNODE_A trace]$ cat /proc/meminfo | grep Hugepagesize
Hugepagesize:       2048 kB
[oracle@DBNODE_A trace]$ 

This is 2048kB x 8705 = 17G maximum


SQL> show parameter use_large_pages
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
use_large_pages                      string      ONLY
SQL> show parameter sga_

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
sga_max_size                         big integer 28G
sga_min_size                         big integer 0
sga_target                           big integer 28G
unified_audit_sga_queue_size         integer     1048576
SQL> 

In this system you have only enough pages to allocate 17G and you require 28G, not good.
How do you fix? You have 3 options:

• Change use_large_pages to TRUE or FALSE, this will or not use huge pages or allow the start even if there are not enough. Do you really want this? I don’t think so…
• Reduce SGA to the maximum number of pages free. Not really an option since it will have impact on performance.
• Increase the number of huge pages at server level.

The safest route is to increase the number of huge pages available. how?

1. Edit /etc/sysctl.conf file and specify the number of hugepages in the nr_hugepages parameter. The entry makes the parameter persist across reboots but does not come into effect until you run the 'sysctl -p' command described in the next step.

	# vi /etc/sysctl.conf
	vm.nr_hugepages = XXXXXXXXX

2. Execute 'sysctl -p' command to enable the hugepages parameter.

	# sysctl -p