""" The following code replicates an HTTP client exiting upon receiving an unexpected response from the server, in this case a 401 status when a keystone token is either missing or invalid. The status is being evaluated before the entire HTTP response has been retrieved and the client exits, destroying the socket to the server. On the server side the server is unable to send its remaining data to the client and produces errors similar to: error: [Errno 32] Broken pipe There are two potential fixes for when this sort of thing is going, both on the client side: * Send a valid keystone token * Accept a 401 response as a valid indicator of a healthy server (You'll only ever get a 401 when _something_ intelligent is there). In addition an HTTP client (the thing sending the OPTIONS) should not treat a 401 response as a catastrophic error and raise an untrapped exception (and thus exit). Instead the socket should be closed gracefully: the conversation with the server was _valid_ it just wasn't what was desired. The problem only appears to show up when there are concurrent requests and the eventlet handling gets confused. You can stimulate the problem by running this script in parallel. for in in {1..20}; do echo $i; python /tmp/request.py & done The upshot of all this is that health check clients needs to be careful about behaving as reasonable HTTP clients. """ import socket import sys import time def do(nasty=True): # Open a request to the ceilometer api and make a minimal # OPTIONS request. clientsocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) clientsocket.connect(('localhost', 8777)) clientsocket.send('OPTIONS / HTTP/1.1\n') clientsocket.send('X-Auth-Token: amazing\n\n') # A place to accumulate the response cache = [] # Read one single byte atom = clientsocket.recv(1) # Loop reading one byte until we note the 401, then exit # violently without cleaning receiving all the response. while atom: if atom == '1' and cache[-2:] == ['4', '0']: print '#### GOT 401' if nasty: sys.exit(1) cache.append(atom) atom = clientsocket.recv(1) if __name__ == '__main__': for i in range(5): print '### TRY', i do()