DL 380p Gen8
he ProLiant DL380P G8 can hold up to 512GB (16 x 32GB) 1600MHz for ECC Registered system memory and up to 768GB (24 x 32GB) 1866Mhz for Load Reduced Server Memory. Depending on the memory configuration and processor model, the memory speed may run at 1333MHz or 1600MHz registered DDR3 DIMMs
You start by identifying the RAM you are currently using in it. You need to match to it. How much ram do you need is based on what you are running under VMWare. You have a single CPU. I would spend the money on the second CPU instead of RAM. VMWare will turn them into vCPUs.
I have an HP DL320e Gen8 with dual 6 core Xeons. VMWare sees them as 24 vCPUs. You will find having more vCPUs allows more VMs to install and run. Next, I don't use non-HP ram in a HP server that came with memory unless you are going to remove it. I match memory on servers. HP memory is not expensive on Amazon. I have 96GB of RAM for my Ubuntu VMs and Cisco VIRL.