One of the hardest parts of writing distributed systems or tools is understanding the effect network latency has on its performance.  I have spent a better part of a decade working with tools and systems that were designed to work in a LAN and then having to try and stretch them to work across a WAN that spans the globe. This blog post outlines a handy way of setting up a latency simulation test network to test any distributed applications or tools.