Recovery Broken Historic Metrics Data

Metrics are not displaying anymore, but using API you can query and see them fine.

https://developer.vmware.com/docs/14143/vmware-cloud-director-api-programming-guide/GUID-2F8CD28B-70C7-45E7-9059-FC32400981E9.html

<Link rel="up" href="https://x.x.x.x/api/vApp/vm-29843161-9be9-4947-9445-d859b5767830" type="application/vnd.vmware.vcloud.vm+xml"/><Link rel="up" href="https://x.x.x.x/api/vApp/vm-29843161-9be9-4947-9445-d859b5767830" type="application/vnd.vmware.vcloud.vm+json"/><Metric name="cpu.usage.average" unit="PERCENT" value="1.11"/><Metric name="disk.used.latest.0" unit="KILOBYTE" value="5918504.0"/><Metric name="disk.used.latest.1" unit="KILOBYTE" value="5918504.0"/><Metric name="disk.used.latest.2" unit="KILOBYTE" value="5918504.0"/><Metric name="disk.provisioned.latest.0" unit="KILOBYTE" value="23151401"/><Metric name="disk.provisioned.latest.1" unit="KILOBYTE" value="23151401"/><Metric name="disk.provisioned.latest.2" unit="KILOBYTE" value="23151401"/><Metric name="cpu.usage.maximum" unit="PERCENT" value="1.11"/><Metric name="cpu.usagemhz.average" unit="MEGAHERTZ" value="25.0"/><Metric name="mem.usage.average" unit="PERCENT" value="0.99"/><Metric name="disk.read.average" unit="KILOBYTES_PER_SECOND" value="0.0"/><Metric name="disk.write.average" unit="KILOBYTES_PER_SECOND" value="1.0"/></CurrentUsage>

Easy – Short Way

Connect to primary DB cell and execute command below. Check after a few minutes if metrics are coming back or not. If the issue still persists, move to the next step.

/opt/vmware/vcloud-director/bin/cell-management-tool manage-config -n statsFeeder.metrics.publishing.enabled -v true

Hard – Long Way

Open an SSH to each of the Cloud Director cells and run the following command to shutdown VCD service:

/opt/vmware/vcloud-director/bin/cell-management-tool cell -i $(service vmware-vcd pid cell) -s

Once you gracefully shutdown all the service, you should backup VCD database from the primary cell, by running the following command:

/opt/vmware/appliance/bin/create-db-backup.sh vcloud

After backup, Connect to the database with the command:

su - postgres -c 'psql -d vcloud'

Once connected (you will get the prompt vcloud#), you should run the following sql queries to cleanup the database, you can run them one by one or copy and paste all of them at once:

delete from activity;
delete from scheduled_activity_jobs;
delete from fifo_activity_queue;
truncate table activity_pc_event_queue;
truncate table activity_stats_queue;
truncate table activity_vsm_listener_queue;
truncate table scheduled_activity_jobs;
delete from task;
update jobs set status = 3 where status = 1;
update last_jobs set status = 3 where status = 1;
delete from busy_object;
delete from task_activity_queue;
delete from failed_cells;

Once completed this step, log out of DB (you can use \q) and on the primary cell run the following command to start VCD service:

service vmware-vcd start

After about 5 minutes you should be able to log back into the Cloud Director UI and if you go to Monitor tab you should see vCenter reconnecting and resyncing the inventory.

After about 10-15 minutes Cloud Director and vCenter Server will have synced. Now we can start an additional VCD cell.

Repeat this step for each remaining VCD cell in the environment.

Finally wait about 15 minutes to see metrics started to populate!