A Fedex package has arrived full of stuff to deal with. Actually, no, is just my contract again, proper copies.

Watching top and doing some straces on various python process it is clear some of the servers are sitting in quite expensive select loops. By design? Is libev an option?

Gerrit will be initially challenging to get used to.

Now going to chase and trace some code (doing this before reading all of this is probably foolhardy:

Looking at ceillometer-collector

code references, unless otherwise qualified, are in the ceilometer package

ceilometer-collector is a long running process defined via an entry-point configured in setup.cfg. The entry-point points to ceilometer.cli:collector_service.

That function calls service:prepare_service to establish logging and messaging (oslo: message bug stuff) and set some global config.

It then gets a ProcessLauncher, a long running parent with a pipe to an arbitrary number of children (in this case just one child unless config'd otherwise). Each child has a "greenthread" watching for a closed pipe.

The child creates a Launcher that actually launches the desired service (CollectorService). Launcher has a Services; one or more threads operating the service. eventlet is involved to provide a [backdoor] for inspecting and handling done.¹

The CollectorService is a subclass of Service. A Service hosts another ThreadGroup and deals with stopping and waiting. The CollectorService discovers a dispatcher and establishes an rpc_server which is listen for methods calls (on itself, the method is record_metering_data) from the message bus. It also optionally listens on UDP.

A dispatcher is the tool for recording data that has been "collected". Both the rpc_server and the UDP listener call record_metering_data on the dispatcher (currently this is either a file or to a variety of databases). The file dispatcher appears to be a simple proof of concept or a handy debugging tool. The database dispatcher does signature verification, date-time normalization and then sends the data to a storage implementation (record_metering_data, the method name, is called three subsequent times in different packages).

The storage implementation is loaded by a DriverManager identified by a database URI from config. DriverManager comes from stevedore. It eventually gives you an object on which methods can be called (e.g. record_metering_data for the third time).

The slqalchemy implementation has a Connection class which the record_metering_data method. At this time the models seem a bit inscrutable, will need to look at a real schema for clarity, but some model objects are assembled, added to the session and (due to context manager) committed.

After a piece of data is written the collector waits for more.

Here endeth today's lesson.