How do I figure out the size my mailbox should be and make it so ?

Gateway is provided with a kfmcp utility which allows you to resize your mailbox.
The kfmcp utility is already introduced in the Gateway online documentation in Maintaing the mailbox.
kfmcp can be used to review the xfer.dat and xferb.dat (EDIINT protocols).
kfmcp -t xfer.dat will return the number of records currently in the mailbox

kfmcp -S xfer.dat  will return the following mailbox information.
-Version
-Maximum number records it can contain maxr. There is actually no control done on this therefore you can exceed that number.
-File allocation size in bytes alsz2.
-hhint
-hp1
-hp2
-Number of records.
kfmcp infile(s) outfile will copy the content of the infile(s) mailbox into the outfile if the outfile does not exist it will be created with the following parameters:
Allocation size 5000000 bytes (if all entries from the original files can fit in it or the appropriate size depending the number of entries you have in the infile. The file size is increased by a multiple of the allocation 5000000 bytes size).
kfmcp -M maxc alsz_Kb alsz2_Kb infile(s) outfile will copy the content of the infile(s) mailbox into the outfile if the outfile does not exist it will be created with the provided parameters.
kfmcp -x version  maxc alsz_b alsz2_b infile(s) outfile will copy the content of the infile(s) mailbox into the outfile if the outfile does not exist it will be created with the provided parameters. You have to specify the version here and careful the alsz_b is in bytes here as opposed the Kbytes in the -M version.

The mailbox size has to be anticipated to fit the desired max number of records it could contain at anytime in order to avoid the necessity of the p_sup resizing the mailbox dynamically which requires additional resources and can be disastrous on an already loaded server. Having a correctly sized mailbox also improves performances as the closer to the actual record number the mailbox will be the less fragmented the entries can get.
Mailbox size recommendations:
Version will be 11
Number of records in the mailbox
Maxc or (maxr) maximum number of records
alsz_b size of the file at creation in bytes is the maximum number of records you wish to have x 16 x 1024 = alsz_b
alsz2_b allocation size for dynamic size increase

Mailbox Maximum number of                     alsz_b                alsz2_b
               records in the mailbox
               before size increas maxc
Defaultxfer.dat 312                262144     5 000 000 5000000
Very low 1 000 262144 20 000 000 5000000
Low 10 000 262144 165 000 000 5000000
Medium 50 000 262144 820 000 000 5000000
High 100 000 262144 1 640 000 000 5000000

To rebuild the mailbox:
Stop Gateway
Run
ls -l xfer.dat
mv xfer.dat xfer.dat.orig
kfmcp -S xfer.dat
kfmcp -M <maxr from above> <size of old xfer.dat in kilobytes> <alsz2 from above/1024> xfer.dat.orig xfer.dat

Now an example:

$ls -l xfer.dat.orig
-rw-------   1 axway axway  5120000 Apr 28 09:48 xfer.dat.orig

$kfmcp -S xfer.dat.orig
file=xfer.dat.orig
version=11
maxr=262144
alsz2=5120000
hhint=1
hp1=0
hp2=4
records=113

$kfmcp -M 262144 5000 5000 xfer.dat.orig xfer.dat
will get cipher attributes from "xfer.dat.orig"
"xfer.dat.orig" is not ciphered
creating "xfer.dat"
opening existing file "xfer.dat.orig"
"xfer.dat.orig" is not ciphered

$kfmcp -S xfer.dat
file=xfer.dat
version=11
maxr=262144
alsz2=5120000
hhint=1
hp1=0
hp2=4
records=113

alsz describes the inital size of the mailbox. For most platforms, the maximum size supported by kfmcp is 2097151 KB.
alsz2 is the size of the extents that will added to the mailbox if it runs out of space. Keeping these large helps reduce fragmentation of the file. Keeping these small cuts down on wasted space and shortens the time needed to allocate the extent.