RabbitMQ CLI tooling

[rjgildea]

We have a bunch of command line tools that use the JMX interface to get information from ActiveMQ (dlstbx.dlq_check, dlstbx.get_activemq_statistics, dlstbx.queue_monitor). This doesn't exist in RabbitMQ, so we need some alternative mechanism to get that sort of information

• dlstbx.dlq_{check,purge,reinject} (@d-j-hatton Extend dlq_* command line tools to work with RabbitMQ #96)
• dlstbx.get_activemq_statistics
• dlstbx.queue_monitor (Add RMQ version of queue monitor #58, Combined ActiveMQ/RabbitMQ cli queue_monitor #93)
• dlstbx.status_monitor (@ndevenish WIP)
• dlstbx.shutdown (New zocalo.shutdown command for stopping zocalo services python-zocalo#124)
• dlstbx.queue_drain (New zocalo.queue_drain command python-zocalo#127)
• dlstbx.pickup
• dlstbx.run_system_tests

relative task complexity: large

[rjgildea]

dlstbx.get_activemq_statistics - this is essentially two separate things:

1. CLI output that gives something like e.g.

$ dlstbx.get_activemq_statistics 
ActiveMQ connections: 348
Storage statistics:
   persistent:  6 %
   temporary :  0 %
   memory    :  0 %
Virtual machine memory statistics:
   heap: using 14857.8 MB of 27716.3 MB
   used memory outside of heap: 98.7 MB
Mimas held queue size: 0

2. Write statistics to RDD file for the monitor html to pick up
   I think 2. will be superseded by prometheus/grafana, so isn't needed. 1. could probably be provided for by rabbitmq diagnostics tools, so not sure how much it is really needed.

[rjgildea]

Rabbit-flavoured queue monitor in progress here.

[d-j-hatton]

FYI: I'm working on extending dlstbx.dlq_* to work with RabbitMQ at the moment

[stufisher]

Could these tools migrate into python-zocalo once they support rabbitmq? This was the primary blocking reason for them not being ported ~ 6 months ago. These tools are useful for the rest of us :)

Would be ideal if they supported activemq and rabbitmq as we dont yet have a rabbitmq migration plan

FYI: i have extracted system_tests, but not done anything about rabbitmq: DiamondLightSource/python-zocalo#120

I also took the basic tools and removed all the hardcoded paths / addresses in favour of using zocalo.configuration here:
https://github.com/stufisher/python-zocalo/tree/feature/activemq-cli-tools (so we can actually monitor our installation)

[d-j-hatton]

The dlq tools at least will support both activemq and rabbitmq

[stufisher]

Excellent, thanks

[d-j-hatton]

dlq tool extension PR #96

[stufisher]

@d-j-hatton excellent thanks, could those be pulled out into https://github.com/diamondlightsource/python-zocalo now? :)

[d-j-hatton]

Just preparing a PR for it now