perpetual draft

I've done some long running gnocchi tests with the file backend. Setup:

• devstack on single baremetal node 8 core, 16GB
• ceilometer dispatching to gnocchi
  • default pipeline.yaml, 5 second polling interval
  • 4 notification agents, 4 collectors
• gnocchi using mysql indexer, file storage
  • 8 gnocchi-metricd processes

Short version: Behavior is better than might have been expected, but has some issues that could work against using carbonara at scale where there are many different metrics.

The intent here is to see how gnocchi write and read behavior maintains after several days, not just initially. Note that this is intentionally anecdotal rather than heavy on the measurements because this gives us the context to do more targeted investigations later. This is gestalt style review of things.

This is using ceilometer dispatch code that is a bit out of date so there is more resource history than would be created with the most recent code. This reveals bugs that will surely come up for actual installs with large numbers of resources or resources that change often.

When running 20 tiny flavored instances, gnocchi-metricd can keep all eight cpus pegged processing incoming measures. In any given second there are around 60 metric dirs waiting to be processed. While the load on the machine is quite high (sometimes near 40) interactive response at the command line is still good and there doesn't appear to be any huge lag in processing. Sometimes measures arrive for an already in progress metric dir, but this is expected and handled behavior.

With this number of metrics, gnocchi-metricd is coping. Based on comparisons of how the system behaves when there are 5 instead of 20 instances, it is pretty clear that write performance will degrade when there aren't enough gnocchi-metricd processes to handle all the input in a timely fashion. Raising the number of metricd workers (to 32) appears to work out pretty well. There's sufficient IO going on that there are plenty of opportunities for any given process to be waiting on IO and thus be in position for the OS to choose to context switch.

Obviously performance would be better if the gnocchi-metricd was running on its own machine and not contending with ceilometer. This is easy enough to do but do we have a model in place for running gnocchi-metricd on multiple machines for the same storage backend? I guess this would mean a gnocchi-api server in some kind of load balancing setup, but then how to share the file backend?

I've not done testing with swift or ceph as the backend but I would guess that for swift the overhead of HTTP would be an important factor.

It really depends on how much lag can be tolerated and how much load can be tolerated while processing the backlog. My machine is currently blowing the fans on high with a load of 80. If I wasn't making it do that on purpose I'd be kind of concerned.

The good news is that retrieving data (the aggregated measures) is super fast. This is pretty much what you'd expect: they are pre-aggregated and the data files are small, the only processing that is going on for simple queries like:

curl -v -H "x-auth-token: $TOKEN" http://192.168.2.2/metric/v1/resource/instance/533f1d53-2af2-4862-8b05-5590a84cf427/metric/cpu_util/measures?aggregation=sum

is making the JSON.

Making aggregations across metrics takes a bit more work, but again, if the starting data is pre-aggregated then it's pretty quick.

Caveats

It would be useful to do this again with variations:

• the latest dispatcher code
• a different backend, probably swift
• a different set or size of resources so the impact of number of different metrics is more clear
• multi-node

Summary

As we've always known, how writes are to be effectively handled will depend a lot on the constraints of the installation. If I were running my own little cloud and wasn't worried about alarming for the sake of scaling or monitoring, then using the file backend with a small number of gnocchi-metricd processes would work quite well. I could retrieve info about various measures fairly quickly and do lightweight aggregations. I would almost certainly want to run gnocchi on its own node, separate from the ceilometer-* processes, so that gnocchi-metricd was not contending with other long running processes.

The medium sized cloud use case is still unclear but it seems likely that the large cloud comments below apply.

If I were running a large cloud or was especially concerned with up to date alarming there will be some hoops (some of them described above) I would need to go through to make the file (or any carbonara-based) backend effective. These hoops would mostly involve ways of getting the measures spread out more widely onto more cpus. Eventually you can image a situation where any given gnocchi-metricd process anywhere in the cloud is responsible for updating measures for just one metric.

You'll note that many of the issues described here go away if we lower the polling cycle to today's ceilometer default of 600 seconds. We need to stop considering that as normal. This is the 21st century. People have expectations of now, not ten minutes from now.