Problem

How to tune IPC parameters for CFT multi-nodes or multi-instances ?

Resolution

* IPC resources used per CFT Instance in legacy mode or per CFT Node in multi-nodes mode:

- 3472615 bytes

- 33554432 bytes

2 sets of semaphores:

- 1 set of 4 semaphores

- 1 set of 1 semaphore

1 message queue per CFTxxx daemon processus

* IPC tuning parameters to consider:

msgmnb Maximum number of bytes on a single System V IPC message queue.

msgmni Maximum number of permitted system-wide System V IPC message queues (IDs) allowed.

shmall System-wide limits on the total number of pages of System V IPC shared memory.

shmmax Maximum size for a System V shared memory segment.

shmmni Number of System V shared memory segment identifiers in the system.

shmseg Maximum number of System V shared memory segments per process.

semmsl Maximum number of System V IPC semaphores per identifier.

semmns Number of System V IPC system-wide semaphores

semopm Maximum number of semaphore operations that can be performed per semop()

semmni Maximum number of System V IPC system-wide semaphore identifiers

* IPC tuning default settings based on RedHat Entreprise Linux 5 documentation (example):

msgmnb: 16384 bytes

msgmni: 16 identifiers

shmall: 20971524Kb pages

shmmax: 33554432 bytes

shmmni: 4096 identifiers

shmseg: 10 mapped segments per process

semmsl: 250 semaphores per set

semmns: 32000 semaphores

semopm: 100 semaphores operations per semop(2)

semmni: 128 identifiers

* A typical CFT configuration runs about 20 simultaneous CFT daemons. This number can be significantly larger as it depends on the actual CFT configurations.

* The value 20 is used in following example of IPC settings for 12 CFT instances.

1) CFT's needs:

Message queues:

- msgmni must fit the 12*20 = 240 messages for the corresponding 240 CFTxxx daemons.

- msgmax=8192

- msgmnb=16384 are suitable but larger values will not harm.

Shared memory:

- shmmni must fit the 12*2 = 24 shared memory sections needed by the 12 CFT instances. shmmni >= 24

- shmmax must be >= 33554432

the default values documented by RedHat 5 are suitable, i.e:

* shmall -> about 2Gb

* shmmax -> about 32Mb

* shmmni -> 4096

* shmseg -> 10

Semaphores:

The 12 CFT instances will use 12*2 = 24 semaphores sets for a total of 12*(4+1) = 60 semaphones.

The default RedHat 5 value should fit, unless some other applications are using a large number of semaphores on the same machine.

2) Parameters to check or to change

msgmnb >= 2*msgmax

msgmni >= (value without the 12 CFTs) + 240

shmall default of 2097152 is ok

shmmax default of 33554432 is ok

shmmni >= (value without the 12 CFTs) + 24

shmseg default of 10 should be ok

semmni >= (value without the 12 CFTs) + 24

3) Rule of thumb:

For 12 CFT instances:

- msgmni is significantly larger than 240 --> at least 500.

- shmmni is significantly larger than 24 --> at least 100.

- semmni is significantly larger than 24 --> at least 100.