Poor Performance and Pending Tasks in Satellite 6.1

We recently installed a new Satellite 6.1 server on VMWare to replace our older physical Satellite server. On our VMWare engineer’s recommendation we configure the VM with 2 cores and 8GB of RAM, a bit under what RedHat calls for. This is from the Red Hat Satellite 6.1 Installation Guide:

Red Hat Satellite requires a networked base system with the following minimum specifications:
64-bit architecture
The latest version of Red Hat Enterprise Linux 6 Server or 7 Server
A minimum of two CPU cores, but four CPU cores are recommended.
A minimum of 12 GB memory but ideally 16 GB of memory for each instance of Satellite. A minimum of 4 GB of swap space is recommended.

Looking at the system, it didn’t appear to be busy. But, tasks would sit in the Pending state and never complete. After a lot of work with Red Hat, we looked at the /etc/default/pulp_workers file:
# Configuration file for Pulp's Celery workers

# Define the number of worker nodes you wish to have here. This defaults to the number of processors
# that are detected on the system if left commented here.

If PULP_CONCURRENCY were commented out, then the number of worker processes would be set to the number of CPUs on startup, or 2 in our case. But, with it set to 1, there aren’t enough processes to take the work off the queue. Once we changed PULP_CONCURRENCY to 4, the system load increased and tasks started moving. Red Hat wasn’t sure how this is set at install time, but tuning the setting made a big difference.

We also increased the number of vCPUs to 4 and the RAM to 12GB, which dramatically improved performance. VCOPS will tell your VMWare administrator to cut back on resources because Satellite is idle almost all the time. But, you need to tune to the peak load times, when Satellite is synchronizing repositories, installing packages or running puppet tasks.

Our server runs at almost 100% idle, with almost no load, and about 7.5GB of ram used. While running repository synchronization, the CPU utilization goes to 100%, with a run queue of 8 to 15, and about 8.3GB of RAM used.

Leave a Reply

Your email address will not be published. Required fields are marked *