Backup Agent service increases memory usage when running VMware backups.
This is a known issue with the Virtual Disk API libraries provided by VMware, specifically the code for locking and unlocking disks. VMware is aware of the issue with their APIs and are working to fix this in a forthcoming major release. Until such time as this is resolved, it must be worked around.
To reduce the impact of the VMware library issue, it is recommended to split up backups which include large virtual machines into multiple jobs which do not run concurrently. In rare cases, installing the Backup Agent on an alternate workstation may produce better results.