Virtualisation Tips & Tricks

While integrating Cloud Director with Aria Operations to provide customers with resource usage reports and billing. We noticed that the CPU load on the primary node in the Postgres DB cluster was increasing. It gradually increased and remained constant for days. It could even reach 85 Ghz CPU loads. When looking at the load with the TOP command, you can see an abnormal number of selects hanging in the air all the time.

The temporary solution to the problem is to restart the VCD service on all nodes.

systemctl restart vmware-vcd

The high CPU utilization is caused by excessive stats-activity-pool threads, stemming from an overload of activity-related records in the database tables. It is suspected that this is caused by Aria Operations.

When contacting VMware support, they suggested this solution.

Resolution

NOTE: The following will require services to be stopped on all Cells, and Data to be removed from the Database.

1. Take a backup of the Cloud Director database, for more details see Backup and Restore of VMware Cloud Director Appliance in the Cloud Director documentation. 
2. Stop VCD service on all the cells.
   • /opt/vmware/vcloud-director/bin/cell-management-tool cell -i $(service vmware-vcd pid cell) -s
3. Connect to the Database
   • sudo -i -u postgres psql vcloud
4. Run the below commands to clean up the activity tables.
   • truncate table activity;
   • truncate table scheduled_activity_jobs;
   • truncate table activity_pc_queue;
   • truncate table activity_pc_event_queue;
   • truncate table fifo_activity_queue;
   • truncate table task_activity_queue;
   • truncate table vc_activity_queue;
   • truncate table activity_stats_queue;
   • truncate table activity_vsm_listener_queue;
5. Start the vcd service on all the cells.
   • systemctl start vmware-vcd.service