I came, I coded, I debugged.

There are a few posts talking about Storm's metrics mechanism, among which you can find  Michael Noll's post ,  Jason Trost's post  and the  storm-metrics-statsd  github project, and last but not least (or is it?)   Storm's documentation . While all of the above provide a decent amount of information, and one is definitely encouraged to read them all before proceeding, it feels like in order to get the full picture one needs to combine them all, and even then a few bits and pieces are left missing. It is these missing bits I'll be rambling about in this post.

(Crossposting from Outbrain's techblog ) Using Storm for real time distributed computations has become a widely adopted approach, and today one can easily find more than a few posts on Storm's architecture, internals, and what have you (e.g., Storm wiki , Understanding the parallelism of a storm topology , Understanding storm internal message buffers , etc ). So you read all these posts and and got yourself a running Storm cluster. You even wrote a topology that does something you need, and managed to get it deployed. "How cool is this?", you think to yourself. "Extremely cool", you reply to yourself sipping the morning coffee. The next step would probably be writing some sort of a validation procedure, to make sure your distributed Storm computation does what you think it does, and does it well. Here at Outbrain we have these validation processes running hourly, making sure our realtime layer data is consistent with our batch layer data - which we c...