2012-10-12 11:31:12.665756 D [68823:70214360935360] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 11:31:12.669620 T [68823:70214360455900] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> Sending -- #   action    fail attempt     
2012-10-12 11:31:12.669978 D [68823:70214360455900] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> Sent 35 bytes
2012-10-12 11:31:12.670423 D [68823:70214360935360] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 11:31:12.670504 D [68823:70214360935360] SimpleTCPServer -- 
****************** Received request
2012-10-12 11:31:12.670516 T [68823:70214360935360] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 11:31:12.670531 D [68823:70214360935360] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 11:31:12.670608 D [68823:70214360935360] SimpleTCPServer -- Server closed
2012-10-12 11:31:12.670620 D [68823:70214360935360] SimpleTCPServer -- thread killed
2012-10-12 11:31:12.674155 D [68823:70214360935360] SimpleTCPServer -- Server Restarted
2012-10-12 11:31:12.674851 D [68823:70214360910620] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 11:31:12.674881 T [68823:70214360455900] ResilientSocket::TCPClient ["localhost:2000"] -- <== #read Received
2012-10-12 11:31:12.674894 D [68823:70214360455900] (4.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- <== #read Received upto 4 bytes
2012-10-12 11:33:02.840566 D [69275:70343142288320] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 11:33:02.842261 D [69275:70343142288320] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 11:33:02.843041 T [69275:70343138171600] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> Sending -- #   action    fail attempt     
2012-10-12 11:33:02.843240 D [69275:70343138171600] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> Sent 35 bytes
2012-10-12 11:33:02.843364 D [69275:70343142288320] SimpleTCPServer -- 
****************** Received request
2012-10-12 11:33:02.843376 T [69275:70343142288320] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 11:33:02.843390 D [69275:70343142288320] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 11:33:02.843437 D [69275:70343142288320] SimpleTCPServer -- Server closed
2012-10-12 11:33:02.843447 D [69275:70343142288320] SimpleTCPServer -- thread killed
2012-10-12 11:33:02.847096 D [69275:70343142288320] SimpleTCPServer -- Server Restarted
2012-10-12 11:33:02.847243 D [69275:70343142264020] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 11:33:02.848480 D [69275:70343138171600] (5.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- <== #read 4 bytes -- NoMethodError: undefined method `length' for nil:NilClass
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:295:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:277:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:368:in `retry_on_connection_failure'
test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 11:36:02.992025 D [69955:70357499959240] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 11:36:02.993938 D [69955:70357499959240] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 11:36:02.994366 T [69955:70357499468500] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> Sending -- #   action    fail attempt     
2012-10-12 11:36:02.994649 D [69955:70357499959240] SimpleTCPServer -- 
****************** Received request
2012-10-12 11:36:02.994666 T [69955:70357499959240] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 11:36:02.994682 D [69955:70357499959240] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 11:36:02.994777 D [69955:70357499959240] SimpleTCPServer -- Server closed
2012-10-12 11:36:02.994789 D [69955:70357499959240] SimpleTCPServer -- thread killed
2012-10-12 11:36:02.998701 D [69955:70357499959240] SimpleTCPServer -- Server Restarted
2012-10-12 11:36:02.999357 D [69955:70357499937540] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 11:36:02.999425 D [69955:70357499468500] (5.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> Sent 35 bytes
2012-10-12 11:36:02.999509 T [69955:70357499468500] ResilientSocket::TCPClient ["localhost:2000"] -- <== #read -- nil
2012-10-12 11:36:02.999522 W [69955:70357499468500] ResilientSocket::TCPClient ["localhost:2000"] -- #read Connection lost while reading data
2012-10-12 11:36:02.999584 D [69955:70357499468500] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- <== #read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:299:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:277:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:372:in `retry_on_connection_failure'
test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 11:36:02.999652 W [69955:70357499468500] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure Connection failure: Connection lost while reading data. Retry: 1
2012-10-12 11:36:03.000658 D [69955:70357499937540] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 11:36:03.000881 T [69955:70357499468500] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> Sending -- #   action    fail attempt     
2012-10-12 11:36:03.001125 D [69955:70357499937540] SimpleTCPServer -- 
****************** Received request
2012-10-12 11:36:03.001141 T [69955:70357499937540] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-12 11:36:03.001157 D [69955:70357499937540] SimpleTCPServer -- Sending Reply
2012-10-12 11:36:03.001166 T [69955:70357499937540] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-12 11:36:03.001552 D [69955:70357499468500] (0.6ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> Sent 35 bytes
2012-10-12 11:36:03.001673 T [69955:70357499468500] ResilientSocket::TCPClient ["localhost:2000"] -- <== #read -- "\x16\x00\x00\x00"
2012-10-12 11:36:03.001684 D [69955:70357499468500] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- <== #read 4 bytes
2012-10-12 11:36:03.002016 T [69955:70357499468500] ResilientSocket::TCPClient ["localhost:2000"] -- <== #read -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-12 11:36:03.002031 D [69955:70357499468500] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- <== #read 18 bytes
2012-10-12 11:38:58.626505 D [70229:70113408706400] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 11:38:58.628315 T [70229:70113403558620] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 11:38:58.629926 D [70229:70113403558620] (1.6ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 11:38:58.630066 D [70229:70113408706400] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 11:38:58.630144 D [70229:70113408706400] SimpleTCPServer -- 
****************** Received request
2012-10-12 11:38:58.630156 T [70229:70113408706400] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 11:38:58.630171 D [70229:70113408706400] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 11:38:58.630254 D [70229:70113408706400] SimpleTCPServer -- Server closed
2012-10-12 11:38:58.630266 D [70229:70113408706400] SimpleTCPServer -- thread killed
2012-10-12 11:38:58.638421 D [70229:70113408706400] SimpleTCPServer -- Server Restarted
2012-10-12 11:38:58.638600 D [70229:70113408688760] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 11:38:58.638657 T [70229:70113403558620] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 11:38:58.638670 W [70229:70113403558620] ResilientSocket::TCPClient ["localhost:2000"] -- #read Connection lost while reading data
2012-10-12 11:38:58.638737 D [70229:70113403558620] (8.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== Reading 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:299:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:277:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:372:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 11:38:58.638864 W [70229:70113403558620] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure Connection failure: Connection lost while reading data. Retry: 1
2012-10-12 11:38:58.640292 T [70229:70113403558620] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 11:38:58.640343 D [70229:70113408688760] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 11:38:58.641368 D [70229:70113403558620] (1.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 11:38:58.641655 D [70229:70113408688760] SimpleTCPServer -- 
****************** Received request
2012-10-12 11:38:58.641695 T [70229:70113408688760] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-12 11:38:58.641713 D [70229:70113408688760] SimpleTCPServer -- Sending Reply
2012-10-12 11:38:58.641722 T [70229:70113408688760] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-12 11:38:58.641907 T [70229:70113403558620] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x16\x00\x00\x00"
2012-10-12 11:38:58.641922 D [70229:70113403558620] (0.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== Reading 4 bytes
2012-10-12 11:38:58.641989 T [70229:70113403558620] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-12 11:38:58.642000 D [70229:70113403558620] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== Reading 18 bytes
2012-10-12 11:38:58.644216 D [70229:70113407715360] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 11:38:58.644956 T [70229:70113403558620] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 11:38:58.645135 D [70229:70113403558620] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 11:38:58.645233 D [70229:70113407715360] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 11:38:58.645291 D [70229:70113407715360] SimpleTCPServer -- 
****************** Received request
2012-10-12 11:38:58.645302 T [70229:70113407715360] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 11:38:58.645316 D [70229:70113407715360] SimpleTCPServer -- Sending Reply
2012-10-12 11:38:58.645325 T [70229:70113407715360] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 11:38:58.645409 T [70229:70113403558620] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 11:38:58.645420 D [70229:70113403558620] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== Reading 4 bytes
2012-10-12 11:38:58.645468 T [70229:70113403558620] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 11:38:58.645478 D [70229:70113403558620] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== Reading 19 bytes
2012-10-12 11:38:58.646669 D [70229:70113403967460] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 11:38:58.647777 T [70229:70113403558620] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-12 11:38:58.647956 D [70229:70113403558620] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-12 11:38:58.648042 D [70229:70113403967460] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 11:38:58.648111 D [70229:70113403967460] SimpleTCPServer -- 
****************** Received request
2012-10-12 11:38:58.648123 T [70229:70113403967460] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-12 11:39:01.649115 W [70229:70113403558620] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-12 11:39:01.649555 D [70229:70113403558620] (3001.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== Reading 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:284:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:277:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:80:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:78:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 11:39:01.656271 D [70229:70113404090060] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 11:39:01.657693 T [70229:70113403558620] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 11:39:01.657761 D [70229:70113403558620] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 11:39:01.657847 D [70229:70113404090060] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 11:39:01.658871 D [70229:70113404090060] SimpleTCPServer -- 
****************** Received request
2012-10-12 11:39:01.658893 T [70229:70113404090060] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 11:39:01.658913 D [70229:70113404090060] SimpleTCPServer -- Sending Reply
2012-10-12 11:39:01.658924 T [70229:70113404090060] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 11:39:01.659651 T [70229:70113403558620] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 11:39:01.659670 D [70229:70113403558620] (1.9ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== Reading 4 bytes
2012-10-12 11:39:01.659729 T [70229:70113403558620] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 11:39:01.659740 D [70229:70113403558620] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== Reading 19 bytes
2012-10-12 11:39:01.663395 D [70229:70113404238260] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 11:39:01.664062 W [70229:70113403558620] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 11:39:02.165565 W [70229:70113403558620] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 11:39:02.667637 W [70229:70113403558620] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 11:39:03.168872 W [70229:70113403558620] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 11:39:03.670303 W [70229:70113403558620] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 11:39:04.171792 W [70229:70113403558620] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-12 11:39:04.672910 W [70229:70113403558620] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-12 11:39:05.174641 W [70229:70113403558620] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-12 11:39:05.676771 W [70229:70113403558620] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-12 11:39:06.178450 W [70229:70113403558620] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-12 11:39:06.680400 E [70229:70113403558620] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-12 11:39:06.680633 I [70229:70113403558620] (5017.6ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:453:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:431:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:424:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:223:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `each_with_index'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:193:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 11:39:06.682018 T [70229:70113403558620] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 11:39:06.682426 D [70229:70113404238260] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 11:39:06.682531 D [70229:70113403558620] (0.5ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 11:39:06.683374 D [70229:70113404238260] SimpleTCPServer -- 
****************** Received request
2012-10-12 11:39:06.683398 T [70229:70113404238260] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 11:39:06.683420 D [70229:70113404238260] SimpleTCPServer -- Sending Reply
2012-10-12 11:39:06.683435 T [70229:70113404238260] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 11:39:06.684300 T [70229:70113403558620] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 11:39:06.684322 D [70229:70113403558620] (1.8ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== Reading 4 bytes
2012-10-12 11:39:06.684387 T [70229:70113403558620] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 11:39:06.684399 D [70229:70113403558620] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== Reading 19 bytes
2012-10-12 11:39:06.686595 W [70229:70113403558620] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 11:39:06.787131 W [70229:70113403558620] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 11:39:06.887664 W [70229:70113403558620] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 11:39:06.989089 W [70229:70113403558620] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 11:39:07.090564 W [70229:70113403558620] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 11:39:07.192389 E [70229:70113403558620] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-12 11:39:07.192620 I [70229:70113403558620] (506.6ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:453:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:431:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:424:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:230:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:193:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 11:40:11.410947 D [70320:70194497435480] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 11:40:11.412292 T [70320:70194487843540] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 11:40:11.412358 D [70320:70194487843540] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 11:40:11.412468 D [70320:70194497435480] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 11:40:11.412539 D [70320:70194497435480] SimpleTCPServer -- 
****************** Received request
2012-10-12 11:40:11.412551 T [70320:70194497435480] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 11:40:11.412564 D [70320:70194497435480] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 11:40:11.412602 D [70320:70194497435480] SimpleTCPServer -- Server closed
2012-10-12 11:40:11.412612 D [70320:70194497435480] SimpleTCPServer -- thread killed
2012-10-12 11:40:11.420527 D [70320:70194497435480] SimpleTCPServer -- Server Restarted
2012-10-12 11:40:11.420835 T [70320:70194487843540] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 11:40:11.420890 W [70320:70194487843540] ResilientSocket::TCPClient ["localhost:2000"] -- #read Connection lost while reading data
2012-10-12 11:40:11.420983 D [70320:70194487843540] (8.6ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:299:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:277:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:372:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 11:40:11.421076 W [70320:70194487843540] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure Connection failure: Connection lost while reading data. Retry: 1
2012-10-12 11:40:11.421193 D [70320:70194488865660] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 11:40:11.422856 T [70320:70194487843540] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 11:40:11.423028 D [70320:70194488865660] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 11:40:11.423079 D [70320:70194487843540] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 11:40:11.423167 D [70320:70194488865660] SimpleTCPServer -- 
****************** Received request
2012-10-12 11:40:11.423181 T [70320:70194488865660] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-12 11:40:11.423197 D [70320:70194488865660] SimpleTCPServer -- Sending Reply
2012-10-12 11:40:11.423207 T [70320:70194488865660] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-12 11:40:11.423293 T [70320:70194487843540] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x16\x00\x00\x00"
2012-10-12 11:40:11.423304 D [70320:70194487843540] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 11:40:11.423351 T [70320:70194487843540] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-12 11:40:11.423361 D [70320:70194487843540] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 18 bytes
2012-10-12 11:40:11.424862 D [70320:70194488026880] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 11:40:11.426045 T [70320:70194487843540] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 11:40:11.426234 D [70320:70194488026880] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 11:40:11.426297 D [70320:70194488026880] SimpleTCPServer -- 
****************** Received request
2012-10-12 11:40:11.426308 T [70320:70194488026880] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 11:40:11.426322 D [70320:70194488026880] SimpleTCPServer -- Sending Reply
2012-10-12 11:40:11.426331 T [70320:70194488026880] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 11:40:11.426785 D [70320:70194487843540] (0.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 11:40:11.426851 T [70320:70194487843540] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 11:40:11.426862 D [70320:70194487843540] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 11:40:11.426907 T [70320:70194487843540] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 11:40:11.426916 D [70320:70194487843540] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 11:40:11.427926 D [70320:70194496087060] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 11:40:11.428475 T [70320:70194487843540] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-12 11:40:11.428659 D [70320:70194496087060] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 11:40:11.428726 D [70320:70194496087060] SimpleTCPServer -- 
****************** Received request
2012-10-12 11:40:11.428738 T [70320:70194496087060] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-12 11:40:11.428834 D [70320:70194487843540] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-12 11:40:14.429954 W [70320:70194487843540] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-12 11:40:14.430153 D [70320:70194487843540] (3001.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:284:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:277:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:80:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:78:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 11:40:14.431846 D [70320:70194496133060] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 11:40:14.432450 T [70320:70194487843540] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 11:40:14.432561 D [70320:70194487843540] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 11:40:14.432898 D [70320:70194496133060] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 11:40:14.432970 D [70320:70194496133060] SimpleTCPServer -- 
****************** Received request
2012-10-12 11:40:14.432981 T [70320:70194496133060] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 11:40:14.432997 D [70320:70194496133060] SimpleTCPServer -- Sending Reply
2012-10-12 11:40:14.433007 T [70320:70194496133060] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 11:40:14.433471 T [70320:70194487843540] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 11:40:14.433486 D [70320:70194487843540] (0.9ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 11:40:14.433535 T [70320:70194487843540] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 11:40:14.433545 D [70320:70194487843540] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 11:40:14.434552 D [70320:70194496266720] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 11:40:14.435173 W [70320:70194487843540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 11:40:14.936536 W [70320:70194487843540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 11:40:15.438569 W [70320:70194487843540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 11:40:15.941032 W [70320:70194487843540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 11:40:16.442417 W [70320:70194487843540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 11:40:16.943564 W [70320:70194487843540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-12 11:40:17.445145 W [70320:70194487843540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-12 11:40:17.946956 W [70320:70194487843540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-12 11:40:18.448744 W [70320:70194487843540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-12 11:40:18.949651 W [70320:70194487843540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-12 11:40:19.451141 E [70320:70194487843540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-12 11:40:19.451375 I [70320:70194487843540] (5016.9ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:453:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:431:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:424:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:223:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `each_with_index'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:193:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 11:40:19.455051 T [70320:70194487843540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 11:40:19.455131 D [70320:70194487843540] (0.0ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 11:40:19.455210 D [70320:70194496266720] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 11:40:19.455305 D [70320:70194496266720] SimpleTCPServer -- 
****************** Received request
2012-10-12 11:40:19.455321 T [70320:70194496266720] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 11:40:19.455341 D [70320:70194496266720] SimpleTCPServer -- Sending Reply
2012-10-12 11:40:19.455354 T [70320:70194496266720] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 11:40:19.456212 T [70320:70194487843540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 11:40:19.456257 D [70320:70194487843540] (1.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-12 11:40:19.456325 T [70320:70194487843540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 11:40:19.456339 D [70320:70194487843540] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-12 11:40:19.457904 W [70320:70194487843540] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 11:40:19.559735 W [70320:70194487843540] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 11:40:19.660427 W [70320:70194487843540] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 11:40:19.762103 W [70320:70194487843540] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 11:40:19.863972 W [70320:70194487843540] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 11:40:19.964894 E [70320:70194487843540] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-12 11:40:19.964992 I [70320:70194487843540] (507.6ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:453:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:431:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:424:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:230:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:193:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 13:55:32.622372 D [84877:70361036717920] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 13:55:32.624151 D [84877:70361036717920] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 13:55:32.624465 T [84877:70361035266780] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 13:55:32.624521 D [84877:70361035266780] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 13:55:32.624662 D [84877:70361036717920] SimpleTCPServer -- 
****************** Received request
2012-10-12 13:55:32.624675 T [84877:70361036717920] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 13:55:32.624691 D [84877:70361036717920] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 13:55:32.624730 D [84877:70361036717920] SimpleTCPServer -- Server closed
2012-10-12 13:55:32.624741 D [84877:70361036717920] SimpleTCPServer -- thread killed
2012-10-12 13:55:32.631466 D [84877:70361036717920] SimpleTCPServer -- Server Restarted
2012-10-12 13:55:32.631757 T [84877:70361035266780] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 13:55:32.631829 W [84877:70361035266780] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 13:55:32.631911 D [84877:70361035266780] (7.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:375:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 13:55:32.631962 E [84877:70361035266780] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure Connection failure: ResilientSocket::ConnectionFailure: Connection lost while reading data. Giving up after 0 retries
2012-10-12 13:55:32.633378 D [84877:70361039357600] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 13:55:32.634042 D [84877:70361039357600] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 13:55:32.634194 T [84877:70361035266780] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 13:55:32.634314 D [84877:70361039357600] SimpleTCPServer -- 
****************** Received request
2012-10-12 13:55:32.634326 T [84877:70361039357600] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 13:55:32.634341 D [84877:70361039357600] SimpleTCPServer -- Sending Reply
2012-10-12 13:55:32.634351 T [84877:70361039357600] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 13:55:32.635296 D [84877:70361035266780] (1.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 13:55:32.635375 T [84877:70361035266780] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 13:55:32.635480 W [84877:70361035266780] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 13:55:32.635551 D [84877:70361035266780] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:70:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 13:55:32.637369 D [84877:70361035138260] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 13:55:32.643733 D [84877:70361035138260] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 13:55:32.644169 T [84877:70361035266780] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-12 13:55:32.644412 D [84877:70361035138260] SimpleTCPServer -- 
****************** Received request
2012-10-12 13:55:32.644425 T [84877:70361035138260] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-12 13:55:32.644499 D [84877:70361035266780] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-12 13:55:35.645612 W [84877:70361035266780] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-12 13:55:35.645826 D [84877:70361035266780] (3001.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:283:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:80:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:78:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 13:55:35.651443 T [84877:70361035266780] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 13:55:35.651539 D [84877:70361035266780] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 13:55:35.652158 D [84877:70361035633260] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 13:55:35.652202 D [84877:70361035633260] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 13:55:35.652267 D [84877:70361035633260] SimpleTCPServer -- 
****************** Received request
2012-10-12 13:55:35.652278 T [84877:70361035633260] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 13:55:35.652294 D [84877:70361035633260] SimpleTCPServer -- Sending Reply
2012-10-12 13:55:35.652305 T [84877:70361035633260] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 13:55:35.654137 T [84877:70361035266780] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 13:55:35.654233 W [84877:70361035266780] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 13:55:35.654308 D [84877:70361035266780] (2.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:129:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 13:55:35.658219 D [84877:70361039571840] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 13:55:35.662197 W [84877:70361035266780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 13:55:36.163029 W [84877:70361035266780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 13:55:36.664782 W [84877:70361035266780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 13:55:37.167016 W [84877:70361035266780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 13:55:37.670257 W [84877:70361035266780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 13:55:38.171552 W [84877:70361035266780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-12 13:55:38.679117 W [84877:70361035266780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-12 13:55:39.180577 W [84877:70361035266780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-12 13:55:39.682198 W [84877:70361035266780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-12 13:55:40.183773 W [84877:70361035266780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-12 13:55:40.685553 E [84877:70361035266780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-12 13:55:40.685676 I [84877:70361035266780] (5027.6ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ArgumentError: wrong number of arguments (1 for 2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/exceptions.rb:26:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:463:in `new'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:463:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:441:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:434:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:222:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `each_with_index'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:192:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 13:55:40.688555 W [84877:70361035266780] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 13:55:40.790133 W [84877:70361035266780] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 13:55:40.891646 W [84877:70361035266780] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 13:55:40.993564 W [84877:70361035266780] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 13:55:41.094248 W [84877:70361035266780] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 13:55:41.195701 E [84877:70361035266780] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-12 13:55:41.195814 I [84877:70361035266780] (508.1ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ArgumentError: wrong number of arguments (1 for 2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/exceptions.rb:26:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:463:in `new'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:463:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:441:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:434:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:229:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:192:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 13:57:47.497268 D [85122:70239376023820] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 13:57:47.498726 T [85122:70239366896340] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 13:57:47.498817 D [85122:70239366896340] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 13:57:47.498950 D [85122:70239376023820] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 13:57:47.499026 D [85122:70239376023820] SimpleTCPServer -- 
****************** Received request
2012-10-12 13:57:47.499038 T [85122:70239376023820] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 13:57:47.499053 D [85122:70239376023820] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 13:57:47.499132 D [85122:70239376023820] SimpleTCPServer -- Server closed
2012-10-12 13:57:47.499146 D [85122:70239376023820] SimpleTCPServer -- thread killed
2012-10-12 13:57:47.505911 D [85122:70239376023820] SimpleTCPServer -- Server Restarted
2012-10-12 13:57:47.506144 D [85122:70239376015880] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 13:57:47.506363 T [85122:70239366896340] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 13:57:47.506431 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 13:57:47.506506 D [85122:70239366896340] (7.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:375:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 13:57:47.506552 E [85122:70239366896340] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure Connection failure: ResilientSocket::ConnectionFailure: Connection lost while reading data. Giving up after 0 retries
2012-10-12 13:57:47.507196 D [85122:70239367333620] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 13:57:47.507813 T [85122:70239366896340] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 13:57:47.509167 D [85122:70239366896340] (1.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 13:57:47.509449 D [85122:70239367333620] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 13:57:47.509666 D [85122:70239367333620] SimpleTCPServer -- 
****************** Received request
2012-10-12 13:57:47.509679 T [85122:70239367333620] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 13:57:47.509694 D [85122:70239367333620] SimpleTCPServer -- Sending Reply
2012-10-12 13:57:47.509703 T [85122:70239367333620] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 13:57:47.510016 T [85122:70239366896340] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 13:57:47.510217 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 13:57:47.510572 D [85122:70239366896340] (1.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:70:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 13:57:47.512079 D [85122:70239367064740] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 13:57:47.512815 T [85122:70239366896340] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-12 13:57:47.513005 D [85122:70239367064740] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 13:57:47.513076 D [85122:70239367064740] SimpleTCPServer -- 
****************** Received request
2012-10-12 13:57:47.513087 T [85122:70239367064740] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-12 13:57:47.513225 D [85122:70239366896340] (0.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-12 13:57:50.514859 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-12 13:57:50.515094 D [85122:70239366896340] (3001.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:283:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:80:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:78:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 13:57:50.519724 D [85122:70239367320740] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 13:57:50.526359 T [85122:70239366896340] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 13:57:50.526444 D [85122:70239366896340] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 13:57:50.530111 D [85122:70239367320740] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 13:57:50.530203 D [85122:70239367320740] SimpleTCPServer -- 
****************** Received request
2012-10-12 13:57:50.530217 T [85122:70239367320740] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 13:57:50.530232 D [85122:70239367320740] SimpleTCPServer -- Sending Reply
2012-10-12 13:57:50.530242 T [85122:70239367320740] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 13:57:50.530785 T [85122:70239366896340] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 13:57:50.530872 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 13:57:50.530943 D [85122:70239366896340] (4.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:129:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 13:57:50.533368 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 13:57:50.534291 D [85122:70239367590700] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 13:57:51.034581 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 13:57:51.535760 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 13:57:52.037236 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 13:57:52.538659 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 13:57:53.039838 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-12 13:57:53.540923 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-12 13:57:54.041787 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-12 13:57:54.543074 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-12 13:57:55.044695 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-12 13:57:55.546049 E [85122:70239366896340] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-12 13:57:55.546150 I [85122:70239366896340] (5014.0ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:463:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:441:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:434:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:222:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `each_with_index'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:192:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 13:57:55.547159 T [85122:70239366896340] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 13:57:55.547385 D [85122:70239367590700] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 13:57:55.547498 D [85122:70239367590700] SimpleTCPServer -- 
****************** Received request
2012-10-12 13:57:55.547511 T [85122:70239367590700] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 13:57:55.547527 D [85122:70239367590700] SimpleTCPServer -- Sending Reply
2012-10-12 13:57:55.547537 T [85122:70239367590700] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 13:57:55.548290 D [85122:70239366896340] (1.0ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 13:57:55.548436 T [85122:70239366896340] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 13:57:55.548506 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 13:57:55.548571 D [85122:70239366896340] (0.2ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:109:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 13:57:55.549912 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 13:57:55.650565 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 13:57:55.751993 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 13:57:55.853194 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 13:57:55.954789 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 13:57:56.055992 E [85122:70239366896340] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-12 13:57:56.056098 I [85122:70239366896340] (506.8ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:463:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:441:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:434:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:229:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:192:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 13:59:57.336644 D [85520:70245608590360] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 13:59:57.338475 D [85520:70245608590360] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 13:59:57.338768 T [85520:70245608020700] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 13:59:57.338908 D [85520:70245608590360] SimpleTCPServer -- 
****************** Received request
2012-10-12 13:59:57.338921 T [85520:70245608590360] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 13:59:57.338936 D [85520:70245608590360] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 13:59:57.339010 D [85520:70245608590360] SimpleTCPServer -- Server closed
2012-10-12 13:59:57.339022 D [85520:70245608590360] SimpleTCPServer -- thread killed
2012-10-12 13:59:57.345742 D [85520:70245608590360] SimpleTCPServer -- Server Restarted
2012-10-12 13:59:57.345875 D [85520:70245608020700] (7.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 13:59:57.345975 T [85520:70245608020700] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 13:59:57.346041 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 13:59:57.346113 D [85520:70245608020700] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:375:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 13:59:57.346151 E [85520:70245608020700] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure Connection failure: ResilientSocket::ConnectionFailure: Connection lost while reading data. Giving up after 0 retries
2012-10-12 13:59:57.348668 D [85520:70245607872100] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 13:59:57.349376 T [85520:70245608020700] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 13:59:57.349543 D [85520:70245608020700] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 13:59:57.349654 D [85520:70245607872100] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 13:59:57.349716 D [85520:70245607872100] SimpleTCPServer -- 
****************** Received request
2012-10-12 13:59:57.349728 T [85520:70245607872100] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 13:59:57.349742 D [85520:70245607872100] SimpleTCPServer -- Sending Reply
2012-10-12 13:59:57.349751 T [85520:70245607872100] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 13:59:57.350656 T [85520:70245608020700] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 13:59:57.351382 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 13:59:57.351455 D [85520:70245608020700] (1.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:70:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 13:59:57.353773 D [85520:70245612170760] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 13:59:57.354358 T [85520:70245608020700] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-12 13:59:57.354544 D [85520:70245608020700] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-12 13:59:57.354643 D [85520:70245612170760] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 13:59:57.354714 D [85520:70245612170760] SimpleTCPServer -- 
****************** Received request
2012-10-12 13:59:57.354726 T [85520:70245612170760] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-12 14:00:00.354848 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-12 14:00:00.355131 D [85520:70245608020700] (3000.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:283:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:80:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:78:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:00:00.356698 D [85520:70245612377960] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:00:00.358435 T [85520:70245608020700] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:00:00.358646 D [85520:70245612377960] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:00:00.358718 D [85520:70245612377960] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:00:00.358730 T [85520:70245612377960] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:00:00.358746 D [85520:70245612377960] SimpleTCPServer -- Sending Reply
2012-10-12 14:00:00.358757 T [85520:70245612377960] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:00:00.358887 D [85520:70245608020700] (0.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:00:00.358998 T [85520:70245608020700] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:00:00.359065 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:00:00.359134 D [85520:70245608020700] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:129:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:00:00.360998 D [85520:70245608302060] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:00:00.361455 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:00:00.862952 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:00:01.364109 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:00:01.865351 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:00:02.366790 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:00:02.868237 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-12 14:00:03.369736 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-12 14:00:03.871367 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-12 14:00:04.372900 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-12 14:00:04.874175 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-12 14:00:05.375156 E [85520:70245608020700] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-12 14:00:05.375266 I [85520:70245608020700] (5015.3ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:463:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:441:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:434:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:222:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `each_with_index'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:192:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:00:05.377611 T [85520:70245608020700] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:00:05.377778 D [85520:70245608020700] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:00:05.378293 D [85520:70245608302060] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:00:05.378438 D [85520:70245608302060] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:00:05.378452 T [85520:70245608302060] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:00:05.378469 D [85520:70245608302060] SimpleTCPServer -- Sending Reply
2012-10-12 14:00:05.378479 T [85520:70245608302060] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:00:05.379133 T [85520:70245608020700] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:00:05.379222 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:00:05.379299 D [85520:70245608020700] (1.5ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:109:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:00:05.381563 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:00:05.483024 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:00:05.583684 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:00:05.684715 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:00:05.785773 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:00:05.887253 E [85520:70245608020700] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-12 14:00:05.887361 I [85520:70245608020700] (506.4ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:463:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:441:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:434:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:229:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:192:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:00:58.925677 D [85573:70245948591400] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:00:58.926984 T [85573:70245943565020] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:00:58.927047 D [85573:70245943565020] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:00:58.927156 D [85573:70245948591400] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:00:58.927231 D [85573:70245948591400] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:00:58.927243 T [85573:70245948591400] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:00:58.927256 D [85573:70245948591400] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:00:58.927293 D [85573:70245948591400] SimpleTCPServer -- Server closed
2012-10-12 14:00:58.927304 D [85573:70245948591400] SimpleTCPServer -- thread killed
2012-10-12 14:00:58.933927 D [85573:70245948591400] SimpleTCPServer -- Server Restarted
2012-10-12 14:00:58.936784 T [85573:70245943565020] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:00:58.936838 W [85573:70245943565020] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:00:58.936919 D [85573:70245943565020] (9.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:375:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:00:58.936968 E [85573:70245943565020] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure Connection failure: ResilientSocket::ConnectionFailure: Connection lost while reading data. Giving up after 0 retries
2012-10-12 14:00:58.938703 D [85573:70245943469400] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:00:58.939924 T [85573:70245943565020] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:00:58.940012 D [85573:70245943469400] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:00:58.940066 D [85573:70245943565020] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:00:58.940387 D [85573:70245943469400] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:00:58.940452 T [85573:70245943469400] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:00:58.940469 D [85573:70245943469400] SimpleTCPServer -- Sending Reply
2012-10-12 14:00:58.940479 T [85573:70245943469400] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:00:58.940755 T [85573:70245943565020] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:00:58.940768 D [85573:70245943565020] (0.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:00:58.940926 T [85573:70245943565020] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:00:58.940937 D [85573:70245943565020] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:00:58.944427 D [85573:70245947694240] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:00:58.945108 T [85573:70245943565020] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-12 14:00:58.945204 D [85573:70245947694240] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:00:58.945244 D [85573:70245943565020] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-12 14:00:58.945554 D [85573:70245947694240] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:00:58.945583 T [85573:70245947694240] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-12 14:01:01.946075 W [85573:70245943565020] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-12 14:01:01.946566 D [85573:70245943565020] (3001.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:283:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:80:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:78:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:01:01.952805 D [85573:70245947913760] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:01:01.953566 T [85573:70245943565020] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:01:01.953631 D [85573:70245943565020] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:01:01.954688 D [85573:70245947913760] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:01:01.954780 D [85573:70245947913760] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:01:01.954796 T [85573:70245947913760] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:01:01.954815 D [85573:70245947913760] SimpleTCPServer -- Sending Reply
2012-10-12 14:01:01.954828 T [85573:70245947913760] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:01:01.956135 T [85573:70245943565020] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:01:01.956153 D [85573:70245943565020] (2.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:01:01.956263 T [85573:70245943565020] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:01:01.956275 D [85573:70245943565020] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:01:01.957435 D [85573:70245948039840] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:01:01.957848 W [85573:70245943565020] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:01:02.459119 W [85573:70245943565020] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:01:02.960275 W [85573:70245943565020] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:01:03.461303 W [85573:70245943565020] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:01:03.962874 W [85573:70245943565020] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:01:04.464053 W [85573:70245943565020] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-12 14:01:04.965555 W [85573:70245943565020] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-12 14:01:05.467057 W [85573:70245943565020] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-12 14:01:05.968483 W [85573:70245943565020] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-12 14:01:06.469947 W [85573:70245943565020] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-12 14:01:06.971429 E [85573:70245943565020] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-12 14:01:06.971538 I [85573:70245943565020] (5014.2ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:463:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:441:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:434:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:222:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `each_with_index'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:192:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:01:06.972617 T [85573:70245943565020] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:01:06.972708 D [85573:70245943565020] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:01:06.972997 D [85573:70245948039840] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:01:06.973074 D [85573:70245948039840] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:01:06.973088 T [85573:70245948039840] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:01:06.973106 D [85573:70245948039840] SimpleTCPServer -- Sending Reply
2012-10-12 14:01:06.973117 T [85573:70245948039840] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:01:06.973927 T [85573:70245943565020] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:01:06.973946 D [85573:70245943565020] (1.2ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:01:06.974277 T [85573:70245943565020] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:01:06.974293 D [85573:70245943565020] (0.3ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:01:06.976099 W [85573:70245943565020] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:01:07.077664 W [85573:70245943565020] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:01:07.178556 W [85573:70245943565020] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:01:07.280154 W [85573:70245943565020] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:01:07.382027 W [85573:70245943565020] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:01:07.482950 E [85573:70245943565020] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-12 14:01:07.483067 I [85573:70245943565020] (507.5ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:463:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:441:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:434:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:229:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:192:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:02:51.217714 D [85588:70292869545880] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:02:51.218814 D [85588:70292869545880] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:02:51.219411 T [85588:70292865243860] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:02:51.222055 D [85588:70292869545880] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:02:51.222076 T [85588:70292869545880] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:02:51.222092 D [85588:70292869545880] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:02:51.222168 D [85588:70292869545880] SimpleTCPServer -- Server closed
2012-10-12 14:02:51.222180 D [85588:70292869545880] SimpleTCPServer -- thread killed
2012-10-12 14:02:51.225599 D [85588:70292869545880] SimpleTCPServer -- Server Restarted
2012-10-12 14:02:51.226299 D [85588:70292869353320] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:02:51.226488 D [85588:70292865243860] (7.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:02:51.226732 T [85588:70292865243860] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:02:51.226849 W [85588:70292865243860] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:02:51.226915 D [85588:70292865243860] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:375:in `retry_on_connection_failure'
test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:02:51.226950 E [85588:70292865243860] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure Connection failure: ResilientSocket::ConnectionFailure: Connection lost while reading data. Giving up after 0 retries
2012-10-12 14:05:21.621842 D [85769:70150582291740] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:05:21.623955 D [85769:70150582291740] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:05:21.624319 T [85769:70150577674960] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:05:21.628660 D [85769:70150582291740] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:05:21.628680 T [85769:70150582291740] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:05:21.628696 D [85769:70150582291740] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:05:21.628748 D [85769:70150582291740] SimpleTCPServer -- Server closed
2012-10-12 14:05:21.628774 D [85769:70150582291740] SimpleTCPServer -- thread killed
2012-10-12 14:05:21.629394 D [85769:70150577674960] (5.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:05:21.629475 T [85769:70150577674960] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:05:21.629545 W [85769:70150577674960] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:05:21.629622 D [85769:70150577674960] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:375:in `retry_on_connection_failure'
test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:05:21.629664 E [85769:70150577674960] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure Connection failure: ResilientSocket::ConnectionFailure: Connection lost while reading data. Giving up after 0 retries
2012-10-12 14:07:43.285154 D [86670:70357407772840] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:07:43.287130 T [86670:70357407193820] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:07:43.287221 D [86670:70357407193820] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:07:43.287348 D [86670:70357407772840] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:07:43.287419 D [86670:70357407772840] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:07:43.287430 T [86670:70357407772840] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:07:43.287450 D [86670:70357407772840] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:07:43.287526 D [86670:70357407772840] SimpleTCPServer -- Server closed
2012-10-12 14:07:43.287538 D [86670:70357407772840] SimpleTCPServer -- thread killed
2012-10-12 14:07:43.291364 D [86670:70357407772840] SimpleTCPServer -- Server Restarted
2012-10-12 14:07:43.291673 T [86670:70357407193820] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:07:43.291735 W [86670:70357407193820] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:07:43.291804 D [86670:70357407193820] (4.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:375:in `retry_on_connection_failure'
test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:07:43.291847 E [86670:70357407193820] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure Connection failure: ResilientSocket::ConnectionFailure: Connection lost while reading data. Giving up after 0 retries
2012-10-12 14:08:46.381062 D [86758:70128331265960] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:08:46.382710 D [86758:70128331265960] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:08:46.383593 T [86758:70128326892240] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:08:46.383805 D [86758:70128326892240] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:08:46.383940 D [86758:70128331265960] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:08:46.383952 T [86758:70128331265960] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:08:46.383967 D [86758:70128331265960] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:08:46.384018 D [86758:70128331265960] SimpleTCPServer -- Server closed
2012-10-12 14:08:46.384031 D [86758:70128331265960] SimpleTCPServer -- thread killed
2012-10-12 14:08:46.387766 D [86758:70128331265960] SimpleTCPServer -- Server Restarted
2012-10-12 14:08:46.387879 T [86758:70128326892240] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:08:46.387940 W [86758:70128326892240] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:08:46.388008 D [86758:70128326892240] (4.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:375:in `retry_on_connection_failure'
test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:08:46.388048 E [86758:70128326892240] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure Connection failure: ResilientSocket::ConnectionFailure: Connection lost while reading data. Giving up after 0 retries
2012-10-12 14:09:14.286315 D [86843:70099927650540] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:09:14.287451 D [86843:70099927650540] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:09:14.287808 T [86843:70099927259860] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:09:14.288358 D [86843:70099927259860] (0.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:09:14.288578 D [86843:70099927650540] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:09:14.288590 T [86843:70099927650540] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:09:14.288605 D [86843:70099927650540] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:09:14.288642 D [86843:70099927650540] SimpleTCPServer -- Server closed
2012-10-12 14:09:14.288653 D [86843:70099927650540] SimpleTCPServer -- thread killed
2012-10-12 14:09:14.292351 D [86843:70099927650540] SimpleTCPServer -- Server Restarted
2012-10-12 14:09:14.292622 D [86843:70099927542900] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:09:14.292817 T [86843:70099927259860] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:09:14.292921 W [86843:70099927259860] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:09:14.293044 D [86843:70099927259860] (4.6ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:375:in `retry_on_connection_failure'
test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:09:14.293087 E [86843:70099927259860] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure Connection failure: ResilientSocket::ConnectionFailure: Connection lost while reading data. Giving up after 0 retries
2012-10-12 14:09:24.878192 D [86849:70108244836020] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:09:24.880230 T [86849:70108240370400] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:09:24.880666 D [86849:70108240370400] (0.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:09:24.884095 D [86849:70108244836020] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:09:24.884201 D [86849:70108244836020] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:09:24.884214 T [86849:70108244836020] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:09:24.884229 D [86849:70108244836020] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:09:24.884313 D [86849:70108244836020] SimpleTCPServer -- Server closed
2012-10-12 14:09:24.884324 D [86849:70108244836020] SimpleTCPServer -- thread killed
2012-10-12 14:09:24.890843 D [86849:70108244836020] SimpleTCPServer -- Server Restarted
2012-10-12 14:09:24.892852 D [86849:70108244760320] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:09:24.893589 T [86849:70108240370400] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:09:24.893722 W [86849:70108240370400] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:09:24.893810 D [86849:70108240370400] (13.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:375:in `retry_on_connection_failure'
test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:09:24.893858 E [86849:70108240370400] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure Connection failure: ResilientSocket::ConnectionFailure: Connection lost while reading data. Giving up after 0 retries
2012-10-12 14:11:27.790511 D [86865:70306103159280] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:11:27.793826 D [86865:70306103159280] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:11:27.794126 T [86865:70306094078680] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:11:27.794263 D [86865:70306103159280] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:11:27.794276 T [86865:70306103159280] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:11:27.794291 D [86865:70306103159280] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:11:27.794366 D [86865:70306103159280] SimpleTCPServer -- Server closed
2012-10-12 14:11:27.794377 D [86865:70306103159280] SimpleTCPServer -- thread killed
2012-10-12 14:11:27.802375 D [86865:70306103159280] SimpleTCPServer -- Server Restarted
2012-10-12 14:11:27.802881 D [86865:70306094078680] (8.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:11:27.803347 D [86865:70306095534140] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:11:27.803381 T [86865:70306094078680] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:11:27.803489 W [86865:70306094078680] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:11:27.803573 D [86865:70306094078680] (0.6ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:375:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:11:27.803615 E [86865:70306094078680] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure Connection failure: ResilientSocket::ConnectionFailure: Connection lost while reading data. Giving up after 0 retries
2012-10-12 14:11:27.809372 D [86865:70306103314900] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:11:27.813374 D [86865:70306103314900] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:11:27.813698 T [86865:70306094078680] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:11:27.813762 D [86865:70306094078680] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:11:27.814232 D [86865:70306103314900] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:11:27.814297 T [86865:70306103314900] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:11:27.814316 D [86865:70306103314900] SimpleTCPServer -- Sending Reply
2012-10-12 14:11:27.814326 T [86865:70306103314900] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:11:27.814898 T [86865:70306094078680] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:11:27.814933 D [86865:70306094078680] (1.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:11:27.815186 T [86865:70306094078680] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:11:27.815199 D [86865:70306094078680] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:11:27.816713 D [86865:70306094133000] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:11:27.818735 T [86865:70306094078680] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-12 14:11:27.822977 D [86865:70306094133000] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:11:27.823100 D [86865:70306094133000] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:11:27.823694 T [86865:70306094133000] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-12 14:11:27.833721 D [86865:70306094078680] (15.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-12 14:11:30.835458 W [86865:70306094078680] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-12 14:11:30.835661 D [86865:70306094078680] (3001.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:283:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:80:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:78:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:11:30.838256 D [86865:70306094583300] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:11:30.838996 T [86865:70306094078680] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:11:30.839196 D [86865:70306094583300] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:11:30.839266 D [86865:70306094583300] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:11:30.839277 T [86865:70306094583300] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:11:30.839292 D [86865:70306094583300] SimpleTCPServer -- Sending Reply
2012-10-12 14:11:30.839302 T [86865:70306094583300] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:11:30.839785 D [86865:70306094078680] (0.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:11:30.839852 T [86865:70306094078680] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:11:30.839864 D [86865:70306094078680] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:11:30.839909 T [86865:70306094078680] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:11:30.839918 D [86865:70306094078680] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:11:30.841117 D [86865:70306094786940] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:11:30.842242 W [86865:70306094078680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:11:31.343626 W [86865:70306094078680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:11:31.845427 W [86865:70306094078680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:11:32.346949 W [86865:70306094078680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:11:32.847690 W [86865:70306094078680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:11:33.349265 W [86865:70306094078680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-12 14:11:33.851288 W [86865:70306094078680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-12 14:11:34.352963 W [86865:70306094078680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-12 14:11:34.854255 W [86865:70306094078680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-12 14:11:35.356136 W [86865:70306094078680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-12 14:11:35.857864 E [86865:70306094078680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-12 14:11:35.858101 I [86865:70306094078680] (5017.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 connect attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:464:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:442:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:435:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:222:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `each_with_index'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:192:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:11:35.860561 T [86865:70306094078680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:11:35.860666 D [86865:70306094078680] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:11:35.860772 D [86865:70306094786940] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:11:35.861296 D [86865:70306094786940] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:11:35.861314 T [86865:70306094786940] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:11:35.861333 D [86865:70306094786940] SimpleTCPServer -- Sending Reply
2012-10-12 14:11:35.861347 T [86865:70306094786940] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:11:35.862155 T [86865:70306094078680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:11:35.862173 D [86865:70306094078680] (1.5ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:11:35.862240 T [86865:70306094078680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:11:35.862253 D [86865:70306094078680] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:11:35.867488 W [86865:70306094078680] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:11:35.968975 W [86865:70306094078680] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:11:36.070204 W [86865:70306094078680] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:11:36.171688 W [86865:70306094078680] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:11:36.273459 W [86865:70306094078680] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:11:36.374761 E [86865:70306094078680] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-12 14:11:36.374862 I [86865:70306094078680] (508.8ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 connect attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:464:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:442:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:435:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:229:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:192:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:12:21.434341 D [87041:70156580976300] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:12:21.435912 D [87041:70156580976300] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:12:21.437144 T [87041:70156571335380] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:12:21.437508 D [87041:70156580976300] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:12:21.437526 T [87041:70156580976300] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:12:21.437542 D [87041:70156580976300] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:12:21.437582 D [87041:70156580976300] SimpleTCPServer -- Server closed
2012-10-12 14:12:21.437593 D [87041:70156580976300] SimpleTCPServer -- thread killed
2012-10-12 14:12:21.437619 D [87041:70156571335380] (0.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:12:21.437698 T [87041:70156571335380] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:12:21.437737 W [87041:70156571335380] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:12:21.437811 D [87041:70156571335380] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:375:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:12:21.437851 E [87041:70156571335380] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure Connection failure: ResilientSocket::ConnectionFailure: Connection lost while reading data. Giving up after 0 retries
2012-10-12 14:12:21.440736 D [87041:70156580901360] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:12:21.441795 T [87041:70156571335380] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:12:21.441996 D [87041:70156571335380] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:12:21.442097 D [87041:70156580901360] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:12:21.442156 D [87041:70156580901360] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:12:21.442167 T [87041:70156580901360] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:12:21.442187 D [87041:70156580901360] SimpleTCPServer -- Sending Reply
2012-10-12 14:12:21.442198 T [87041:70156580901360] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:12:21.442282 T [87041:70156571335380] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:12:21.442294 D [87041:70156571335380] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:12:21.442342 T [87041:70156571335380] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:12:21.442352 D [87041:70156571335380] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:12:21.444538 D [87041:70156580806300] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:12:21.445110 T [87041:70156571335380] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-12 14:12:21.445305 D [87041:70156571335380] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-12 14:12:21.445407 D [87041:70156580806300] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:12:21.445472 D [87041:70156580806300] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:12:21.445483 T [87041:70156580806300] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-12 14:12:24.446129 W [87041:70156571335380] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-12 14:12:24.446703 D [87041:70156571335380] (3001.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:283:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:80:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:78:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:12:24.449137 D [87041:70156580768620] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:12:24.451354 T [87041:70156571335380] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:12:24.451461 D [87041:70156571335380] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:12:24.452238 D [87041:70156580768620] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:12:24.452411 D [87041:70156580768620] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:12:24.452472 T [87041:70156580768620] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:12:24.452540 D [87041:70156580768620] SimpleTCPServer -- Sending Reply
2012-10-12 14:12:24.452556 T [87041:70156580768620] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:12:24.453310 T [87041:70156571335380] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:12:24.453332 D [87041:70156571335380] (1.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:12:24.453396 T [87041:70156571335380] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:12:24.453409 D [87041:70156571335380] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:12:24.455185 D [87041:70156580700420] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:12:24.455633 W [87041:70156571335380] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:12:24.956565 W [87041:70156571335380] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:12:25.458081 W [87041:70156571335380] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:12:25.959763 W [87041:70156571335380] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:12:26.460852 W [87041:70156571335380] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:12:26.962428 W [87041:70156571335380] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-12 14:12:27.471306 W [87041:70156571335380] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-12 14:12:27.972661 W [87041:70156571335380] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-12 14:12:28.474509 W [87041:70156571335380] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-12 14:12:28.975726 W [87041:70156571335380] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-12 14:12:29.477227 E [87041:70156571335380] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-12 14:12:29.477356 I [87041:70156571335380] (5022.3ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 connection attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:464:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:442:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:435:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:222:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `each_with_index'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:192:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:12:29.479718 T [87041:70156571335380] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:12:29.480720 D [87041:70156571335380] (0.3ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:12:29.480899 D [87041:70156580700420] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:12:29.480989 D [87041:70156580700420] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:12:29.481004 T [87041:70156580700420] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:12:29.481025 D [87041:70156580700420] SimpleTCPServer -- Sending Reply
2012-10-12 14:12:29.481038 T [87041:70156580700420] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:12:29.481152 T [87041:70156571335380] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:12:29.481169 D [87041:70156571335380] (0.4ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:12:29.481233 T [87041:70156571335380] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:12:29.481247 D [87041:70156571335380] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:12:29.484306 W [87041:70156571335380] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:12:29.586019 W [87041:70156571335380] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:12:29.686868 W [87041:70156571335380] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:12:29.788227 W [87041:70156571335380] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:12:29.889702 W [87041:70156571335380] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:12:29.991171 E [87041:70156571335380] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-12 14:12:29.991270 I [87041:70156571335380] (507.5ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 connection attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:464:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:442:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:435:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:229:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:192:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:12:47.206333 D [87045:70150579411940] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:12:47.208072 D [87045:70150579411940] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:12:47.208415 T [87045:70150577674960] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:12:47.208562 D [87045:70150579411940] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:12:47.208591 T [87045:70150579411940] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:12:47.208608 D [87045:70150579411940] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:12:47.208648 D [87045:70150579411940] SimpleTCPServer -- Server closed
2012-10-12 14:12:47.208659 D [87045:70150579411940] SimpleTCPServer -- thread killed
2012-10-12 14:12:47.216235 D [87045:70150579411940] SimpleTCPServer -- Server Restarted
2012-10-12 14:12:47.216349 D [87045:70150577674960] (7.9ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:12:47.216455 T [87045:70150577674960] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:12:47.216516 W [87045:70150577674960] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:12:47.216592 D [87045:70150577674960] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:375:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:12:47.216631 E [87045:70150577674960] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure Connection failure: ResilientSocket::ConnectionFailure: Connection lost while reading data. Giving up after 0 retries
2012-10-12 14:12:47.219919 D [87045:70150577576460] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:12:47.221431 T [87045:70150577674960] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:12:47.221800 D [87045:70150577576460] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:12:47.221883 D [87045:70150577576460] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:12:47.221895 T [87045:70150577576460] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:12:47.221910 D [87045:70150577576460] SimpleTCPServer -- Sending Reply
2012-10-12 14:12:47.221920 T [87045:70150577576460] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:12:47.222005 D [87045:70150577674960] (0.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:12:47.222067 T [87045:70150577674960] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:12:47.222078 D [87045:70150577674960] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:12:47.222121 T [87045:70150577674960] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:12:47.222131 D [87045:70150577674960] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:12:47.223125 D [87045:70150577633100] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:12:47.223651 D [87045:70150577633100] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:12:47.223794 T [87045:70150577674960] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-12 14:12:47.223911 D [87045:70150577633100] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:12:47.223923 T [87045:70150577633100] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-12 14:12:47.223954 D [87045:70150577674960] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-12 14:12:50.224404 W [87045:70150577674960] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-12 14:12:50.224609 D [87045:70150577674960] (3000.6ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:283:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:80:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:78:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:12:50.226452 D [87045:70150582008980] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:12:50.231400 T [87045:70150577674960] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:12:50.231741 D [87045:70150577674960] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:12:50.231957 D [87045:70150582008980] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:12:50.232720 D [87045:70150582008980] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:12:50.232745 T [87045:70150582008980] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:12:50.232762 D [87045:70150582008980] SimpleTCPServer -- Sending Reply
2012-10-12 14:12:50.232772 T [87045:70150582008980] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:12:50.233556 T [87045:70150577674960] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:12:50.233575 D [87045:70150577674960] (1.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:12:50.233625 T [87045:70150577674960] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:12:50.233634 D [87045:70150577674960] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:12:50.234939 W [87045:70150577674960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:12:50.235026 D [87045:70150582211420] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:12:50.736529 W [87045:70150577674960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:12:51.238157 W [87045:70150577674960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:12:51.739723 W [87045:70150577674960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:12:52.241470 W [87045:70150577674960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:12:52.742897 W [87045:70150577674960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-12 14:12:53.243853 W [87045:70150577674960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-12 14:12:53.745439 W [87045:70150577674960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-12 14:12:54.246344 W [87045:70150577674960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-12 14:12:54.747525 W [87045:70150577674960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-12 14:12:55.248401 E [87045:70150577674960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-12 14:12:55.248547 I [87045:70150577674960] (5014.3ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 connection attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:464:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:442:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:435:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:222:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `each_with_index'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:192:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:12:55.251859 T [87045:70150577674960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:12:55.252120 D [87045:70150577674960] (0.2ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:12:55.252596 D [87045:70150582211420] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:12:55.252854 D [87045:70150582211420] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:12:55.252958 T [87045:70150582211420] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:12:55.252983 D [87045:70150582211420] SimpleTCPServer -- Sending Reply
2012-10-12 14:12:55.252995 T [87045:70150582211420] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:12:55.253947 T [87045:70150577674960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:12:55.254062 D [87045:70150577674960] (1.9ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:12:55.254260 T [87045:70150577674960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:12:55.254310 D [87045:70150577674960] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:12:55.256082 W [87045:70150577674960] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:12:55.357732 W [87045:70150577674960] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:12:55.459779 W [87045:70150577674960] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:12:55.561064 W [87045:70150577674960] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:12:55.662512 W [87045:70150577674960] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:12:55.763421 E [87045:70150577674960] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-12 14:12:55.763521 I [87045:70150577674960] (507.9ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 connection attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:464:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:442:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:435:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:229:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:192:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:13:06.764916 D [87048:70168462328720] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:13:06.769874 T [87048:70168457992920] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:13:06.771473 D [87048:70168462328720] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:13:06.771744 D [87048:70168457992920] (1.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:13:06.772295 D [87048:70168462328720] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:13:06.772312 T [87048:70168462328720] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:13:06.772328 D [87048:70168462328720] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:13:06.772371 D [87048:70168462328720] SimpleTCPServer -- Server closed
2012-10-12 14:13:06.772383 D [87048:70168462328720] SimpleTCPServer -- thread killed
2012-10-12 14:13:06.776563 D [87048:70168462328720] SimpleTCPServer -- Server Restarted
2012-10-12 14:13:06.776863 D [87048:70168462257660] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:13:06.777539 T [87048:70168457992920] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:13:06.777781 W [87048:70168457992920] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:13:06.777854 D [87048:70168457992920] (6.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:375:in `retry_on_connection_failure'
test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:13:06.777897 E [87048:70168457992920] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure Connection failure: ResilientSocket::ConnectionFailure: Connection lost while reading data. Giving up after 0 retries
2012-10-12 14:16:58.251849 D [88386:70249315751040] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:16:58.254831 D [88386:70249315751040] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:16:58.255271 T [88386:70249311591120] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:16:58.255588 D [88386:70249315751040] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:16:58.255605 T [88386:70249315751040] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:16:58.255620 D [88386:70249315751040] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:16:58.255787 D [88386:70249315751040] SimpleTCPServer -- Server closed
2012-10-12 14:16:58.255800 D [88386:70249315751040] SimpleTCPServer -- thread killed
2012-10-12 14:16:58.259102 D [88386:70249315751040] SimpleTCPServer -- Server Restarted
2012-10-12 14:16:58.259890 D [88386:70249312140180] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:16:58.260111 D [88386:70249311591120] (4.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:16:58.260367 T [88386:70249311591120] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:16:58.260486 W [88386:70249311591120] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:16:58.260553 D [88386:70249311591120] (0.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:374:in `retry_on_connection_failure'
test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:17:22.961775 D [88543:70100200206280] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:17:22.964023 T [88543:70100195695320] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:17:22.964294 D [88543:70100200206280] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:17:22.964657 D [88543:70100200206280] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:17:22.964692 T [88543:70100200206280] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:17:22.964707 D [88543:70100200206280] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:17:22.964748 D [88543:70100200206280] SimpleTCPServer -- Server closed
2012-10-12 14:17:22.964760 D [88543:70100200206280] SimpleTCPServer -- thread killed
2012-10-12 14:17:22.968853 D [88543:70100200206280] SimpleTCPServer -- Server Restarted
2012-10-12 14:17:22.968934 D [88543:70100195695320] (4.9ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:17:22.969029 T [88543:70100195695320] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:17:22.969086 W [88543:70100195695320] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:17:22.969155 D [88543:70100195695320] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:374:in `retry_on_connection_failure'
test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:17:46.416583 D [88706:70206697709800] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:17:46.417916 T [88706:70206697462480] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:17:46.418001 D [88706:70206697462480] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:17:46.418214 D [88706:70206697709800] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:17:46.418291 D [88706:70206697709800] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:17:46.418302 T [88706:70206697709800] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:17:46.418316 D [88706:70206697709800] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:17:46.418353 D [88706:70206697709800] SimpleTCPServer -- Server closed
2012-10-12 14:17:46.418364 D [88706:70206697709800] SimpleTCPServer -- thread killed
2012-10-12 14:17:46.421777 D [88706:70206697709800] SimpleTCPServer -- Server Restarted
2012-10-12 14:17:46.422033 D [88706:70206697683840] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:17:46.423022 T [88706:70206697462480] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:17:46.423100 W [88706:70206697462480] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:17:46.423174 D [88706:70206697462480] (5.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:301:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:277:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:375:in `retry_on_connection_failure'
test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:17:46.423216 W [88706:70206697462480] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure retry 1 due to ResilientSocket::ConnectionFailure: Connection lost while reading data
2012-10-12 14:17:46.424675 T [88706:70206697462480] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:17:46.425062 D [88706:70206697683840] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:17:46.425203 D [88706:70206697683840] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:17:46.425264 T [88706:70206697683840] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-12 14:17:46.425285 D [88706:70206697683840] SimpleTCPServer -- Sending Reply
2012-10-12 14:17:46.425295 T [88706:70206697683840] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-12 14:17:46.425800 D [88706:70206697462480] (1.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:17:46.425865 T [88706:70206697462480] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x16\x00\x00\x00"
2012-10-12 14:17:46.425876 D [88706:70206697462480] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:17:46.425921 T [88706:70206697462480] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-12 14:17:46.425930 D [88706:70206697462480] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 18 bytes
2012-10-12 14:17:51.002799 D [88708:70305213956240] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:17:51.004778 T [88708:70305209080540] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:17:51.005027 D [88708:70305213956240] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:17:51.005137 D [88708:70305213956240] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:17:51.005149 T [88708:70305213956240] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:17:51.005164 D [88708:70305213956240] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:17:51.005214 D [88708:70305213956240] SimpleTCPServer -- Server closed
2012-10-12 14:17:51.005228 D [88708:70305213956240] SimpleTCPServer -- thread killed
2012-10-12 14:17:51.011440 D [88708:70305213956240] SimpleTCPServer -- Server Restarted
2012-10-12 14:17:51.011698 D [88708:70305209080540] (6.9ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:17:51.011800 T [88708:70305209080540] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:17:51.011913 W [88708:70305209080540] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:17:51.012208 D [88708:70305209080540] (0.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:301:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:277:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:375:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:17:51.012249 W [88708:70305209080540] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure retry 1 due to ResilientSocket::ConnectionFailure: Connection lost while reading data
2012-10-12 14:17:51.013147 D [88708:70305213911080] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:17:51.014149 T [88708:70305209080540] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:17:51.014224 D [88708:70305209080540] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:17:51.014571 D [88708:70305213911080] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:17:51.014875 D [88708:70305213911080] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:17:51.014892 T [88708:70305213911080] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-12 14:17:51.014909 D [88708:70305213911080] SimpleTCPServer -- Sending Reply
2012-10-12 14:17:51.014919 T [88708:70305213911080] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-12 14:17:51.015404 T [88708:70305209080540] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x16\x00\x00\x00"
2012-10-12 14:17:51.015423 D [88708:70305209080540] (1.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:17:51.015470 T [88708:70305209080540] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-12 14:17:51.015480 D [88708:70305209080540] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 18 bytes
2012-10-12 14:17:51.016952 D [88708:70305213127260] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:17:51.017601 D [88708:70305213127260] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:17:51.017849 T [88708:70305209080540] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:17:51.018126 D [88708:70305213127260] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:17:51.018143 T [88708:70305213127260] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:17:51.018157 D [88708:70305213127260] SimpleTCPServer -- Sending Reply
2012-10-12 14:17:51.018166 T [88708:70305213127260] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:17:51.018599 D [88708:70305209080540] (0.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:17:51.018810 T [88708:70305209080540] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:17:51.018851 D [88708:70305209080540] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:17:51.019113 T [88708:70305209080540] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:17:51.019155 D [88708:70305209080540] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:17:51.020184 D [88708:70305209266600] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:17:51.020801 D [88708:70305209266600] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:17:51.021051 T [88708:70305209080540] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-12 14:17:51.021334 D [88708:70305209266600] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:17:51.021351 T [88708:70305209266600] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-12 14:17:51.021574 D [88708:70305209080540] (0.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-12 14:17:54.022063 W [88708:70305209080540] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-12 14:17:54.022483 D [88708:70305209080540] (3000.9ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:284:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:277:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:80:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:78:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:17:54.023995 D [88708:70305209631080] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:17:54.025113 T [88708:70305209080540] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:17:54.025180 D [88708:70305209080540] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:17:54.025446 D [88708:70305209631080] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:17:54.025811 D [88708:70305209631080] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:17:54.025828 T [88708:70305209631080] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:17:54.025847 D [88708:70305209631080] SimpleTCPServer -- Sending Reply
2012-10-12 14:17:54.025860 T [88708:70305209631080] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:17:54.026769 T [88708:70305209080540] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:17:54.026785 D [88708:70305209080540] (1.6ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:17:54.026848 T [88708:70305209080540] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:17:54.026861 D [88708:70305209080540] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:17:54.028992 D [88708:70305209702780] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:17:54.029341 W [88708:70305209080540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:17:54.530781 W [88708:70305209080540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:17:55.032342 W [88708:70305209080540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:17:55.533826 W [88708:70305209080540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:17:56.035345 W [88708:70305209080540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:17:56.537306 W [88708:70305209080540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-12 14:17:57.038959 W [88708:70305209080540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-12 14:17:57.540812 W [88708:70305209080540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-12 14:17:58.041884 W [88708:70305209080540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-12 14:17:58.543089 W [88708:70305209080540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-12 14:17:59.044557 E [88708:70305209080540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-12 14:17:59.044664 I [88708:70305209080540] (5016.0ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 connection attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:465:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:443:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:436:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:223:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `each_with_index'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:193:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:17:59.046144 T [88708:70305209080540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:17:59.046239 D [88708:70305209080540] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:17:59.046327 D [88708:70305209702780] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:17:59.046427 D [88708:70305209702780] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:17:59.046533 T [88708:70305209702780] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:17:59.046553 D [88708:70305209702780] SimpleTCPServer -- Sending Reply
2012-10-12 14:17:59.046564 T [88708:70305209702780] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:17:59.046967 T [88708:70305209080540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:17:59.046982 D [88708:70305209080540] (0.7ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:17:59.047043 T [88708:70305209080540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:17:59.047053 D [88708:70305209080540] (0.0ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:17:59.049316 W [88708:70305209080540] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:17:59.150736 W [88708:70305209080540] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:17:59.252711 W [88708:70305209080540] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:17:59.353501 W [88708:70305209080540] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:17:59.454479 W [88708:70305209080540] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:17:59.555710 E [88708:70305209080540] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-12 14:17:59.555817 I [88708:70305209080540] (506.9ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 connection attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:465:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:443:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:436:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:230:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:193:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:43:12.474024 D [94165:70323884299840] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:43:12.476565 T [94165:70323882121940] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:43:12.476667 D [94165:70323882121940] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:43:12.476788 D [94165:70323884299840] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:43:12.476863 D [94165:70323884299840] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:43:12.476926 T [94165:70323884299840] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:43:12.476942 D [94165:70323884299840] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:43:12.477020 D [94165:70323884299840] SimpleTCPServer -- Server closed
2012-10-12 14:43:12.477032 D [94165:70323884299840] SimpleTCPServer -- thread killed
2012-10-12 14:43:12.484377 D [94165:70323884299840] SimpleTCPServer -- Server Restarted
2012-10-12 14:43:12.485291 D [94165:70323890399120] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:43:12.485464 T [94165:70323882121940] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:43:12.485581 W [94165:70323882121940] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:43:12.485673 D [94165:70323882121940] (9.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:277:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:93:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:374:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:88:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:43:12.485723 W [94165:70323882121940] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure retry 1 due to ResilientSocket::ConnectionFailure: Connection lost while reading data
2012-10-12 14:43:12.486880 D [94165:70323890399120] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:43:12.487009 T [94165:70323882121940] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:43:12.487060 D [94165:70323882121940] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:43:12.487564 D [94165:70323890399120] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:43:12.487625 T [94165:70323890399120] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-12 14:43:12.487644 D [94165:70323890399120] SimpleTCPServer -- Sending Reply
2012-10-12 14:43:12.487654 T [94165:70323890399120] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-12 14:43:12.488604 T [94165:70323882121940] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x16\x00\x00\x00"
2012-10-12 14:43:12.488622 D [94165:70323882121940] (1.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:43:12.488674 T [94165:70323882121940] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-12 14:43:12.488684 D [94165:70323882121940] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 18 bytes
2012-10-12 14:43:12.489802 D [94165:70323882302980] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:43:12.490521 T [94165:70323882121940] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:43:12.490705 D [94165:70323882121940] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:43:12.490784 D [94165:70323882302980] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:43:12.490840 D [94165:70323882302980] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:43:12.490851 T [94165:70323882302980] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:43:12.490866 D [94165:70323882302980] SimpleTCPServer -- Sending Reply
2012-10-12 14:43:12.490876 T [94165:70323882302980] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:43:12.491443 T [94165:70323882121940] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:43:12.491511 D [94165:70323882121940] (0.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:43:12.491563 T [94165:70323882121940] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:43:12.491573 D [94165:70323882121940] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:43:12.492561 D [94165:70323890321500] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:43:12.493161 T [94165:70323882121940] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-12 14:43:12.493349 D [94165:70323890321500] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:43:12.493418 D [94165:70323890321500] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:43:12.493429 T [94165:70323890321500] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-12 14:43:12.493525 D [94165:70323882121940] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-12 14:43:15.494680 W [94165:70323882121940] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-12 14:43:15.494880 D [94165:70323882121940] (3001.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:283:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:277:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:81:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:79:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:43:15.497046 D [94165:70323882738660] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:43:15.498752 T [94165:70323882121940] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:43:15.498858 D [94165:70323882121940] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:43:15.498985 D [94165:70323882738660] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:43:15.499066 D [94165:70323882738660] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:43:15.499081 T [94165:70323882738660] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:43:15.499101 D [94165:70323882738660] SimpleTCPServer -- Sending Reply
2012-10-12 14:43:15.499113 T [94165:70323882738660] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:43:15.500541 T [94165:70323882121940] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:43:15.500562 D [94165:70323882121940] (1.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:43:15.500618 T [94165:70323882121940] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:43:15.500629 D [94165:70323882121940] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:43:15.502199 W [94165:70323882121940] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:43:15.502366 D [94165:70323882804880] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:43:16.002937 W [94165:70323882121940] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:43:16.503736 W [94165:70323882121940] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:43:17.005312 W [94165:70323882121940] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:43:17.506295 W [94165:70323882121940] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:43:18.007136 W [94165:70323882121940] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-12 14:43:18.508810 W [94165:70323882121940] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-12 14:43:19.009673 W [94165:70323882121940] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-12 14:43:19.511106 W [94165:70323882121940] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-12 14:43:20.012974 W [94165:70323882121940] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-12 14:43:20.513609 E [94165:70323882121940] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-12 14:43:20.513726 I [94165:70323882121940] (5012.5ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 connection attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:488:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:466:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:459:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:223:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `each_with_index'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:193:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:102:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:102:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:43:20.515084 T [94165:70323882121940] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:43:20.515187 D [94165:70323882121940] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:43:20.515369 D [94165:70323882804880] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:43:20.515454 D [94165:70323882804880] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:43:20.515465 T [94165:70323882804880] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:43:20.515482 D [94165:70323882804880] SimpleTCPServer -- Sending Reply
2012-10-12 14:43:20.515492 T [94165:70323882804880] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:43:20.515611 T [94165:70323882121940] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:43:20.515625 D [94165:70323882121940] (0.4ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:43:20.515687 T [94165:70323882121940] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:43:20.515698 D [94165:70323882121940] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:43:20.517371 W [94165:70323882121940] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:43:20.655235 W [94165:70323882121940] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:43:20.756773 W [94165:70323882121940] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:43:20.858358 W [94165:70323882121940] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:43:20.959889 W [94165:70323882121940] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:43:21.061374 E [94165:70323882121940] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-12 14:43:21.061477 I [94165:70323882121940] (544.8ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 connection attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:488:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:466:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:459:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:230:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:193:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:47:30.639564 D [94994:70108233573700] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:47:30.641858 T [94994:70108231981780] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:47:30.642145 D [94994:70108233573700] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:47:30.642282 D [94994:70108233573700] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:47:30.642295 T [94994:70108233573700] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:47:30.642309 D [94994:70108233573700] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:47:30.642347 D [94994:70108233573700] SimpleTCPServer -- Server closed
2012-10-12 14:47:30.642358 D [94994:70108233573700] SimpleTCPServer -- thread killed
2012-10-12 14:47:30.648926 D [94994:70108233573700] SimpleTCPServer -- Server Restarted
2012-10-12 14:47:30.649261 D [94994:70108233535660] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:47:30.649620 D [94994:70108231981780] (7.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:47:30.650168 T [94994:70108231981780] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:47:30.650293 W [94994:70108231981780] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:47:30.650369 D [94994:70108231981780] (0.6ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:277:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:108:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:374:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:103:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:47:30.650412 W [94994:70108231981780] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure retry 1 due to ResilientSocket::ConnectionFailure: Connection lost while reading data
2012-10-12 14:47:30.651661 T [94994:70108231981780] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:47:30.651815 D [94994:70108231981780] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:47:30.652074 D [94994:70108233535660] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:47:30.652332 D [94994:70108233535660] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:47:30.652348 T [94994:70108233535660] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-12 14:47:30.652365 D [94994:70108233535660] SimpleTCPServer -- Sending Reply
2012-10-12 14:47:30.652374 T [94994:70108233535660] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-12 14:47:30.652716 T [94994:70108231981780] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x16\x00\x00\x00"
2012-10-12 14:47:30.652730 D [94994:70108231981780] (0.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:47:30.653880 T [94994:70108231981780] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-12 14:47:30.653922 D [94994:70108231981780] (1.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 18 bytes
2012-10-12 14:47:30.656512 D [94994:70108232155040] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:47:30.660349 T [94994:70108231981780] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:47:30.660853 D [94994:70108232155040] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:47:30.661285 D [94994:70108231981780] (0.9ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:47:30.661405 D [94994:70108232155040] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:47:30.661418 T [94994:70108232155040] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:47:30.661433 D [94994:70108232155040] SimpleTCPServer -- Sending Reply
2012-10-12 14:47:30.661443 T [94994:70108232155040] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:47:30.662214 T [94994:70108231981780] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:47:30.662237 D [94994:70108231981780] (0.9ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:47:30.662580 T [94994:70108231981780] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:47:30.662655 D [94994:70108231981780] (0.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:47:30.664436 D [94994:70108232388920] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:47:30.665291 T [94994:70108231981780] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-12 14:47:30.665465 D [94994:70108231981780] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-12 14:47:30.665750 D [94994:70108232388920] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:47:30.665933 D [94994:70108232388920] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:47:30.665945 T [94994:70108232388920] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-12 14:47:33.666721 W [94994:70108231981780] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-12 14:47:33.666999 D [94994:70108231981780] (3001.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:283:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:277:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:94:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:92:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:47:34.167029 D [94994:70108232388920] SimpleTCPServer -- Sending Reply
2012-10-12 14:47:34.167075 T [94994:70108232388920] SimpleTCPServer -- Reply -- {"result"=>"sleep"}
2012-10-12 14:47:34.168004 T [94994:70108231981780] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:47:34.168024 D [94994:70108231981780] (500.6ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:47:34.168084 T [94994:70108231981780] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00sleep\x00\x00"
2012-10-12 14:47:34.168094 D [94994:70108231981780] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:47:34.170533 D [94994:70108232676360] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:47:34.175392 T [94994:70108231981780] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-12 14:47:34.175482 D [94994:70108231981780] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-12 14:47:34.176048 D [94994:70108232676360] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:47:34.176128 D [94994:70108232676360] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:47:34.176140 T [94994:70108232676360] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-12 14:47:37.175990 W [94994:70108231981780] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-12 14:47:37.177756 D [94994:70108231981780] (3002.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:283:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:277:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:82:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:80:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:47:37.179521 D [94994:70108232772540] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:47:37.180807 T [94994:70108231981780] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:47:37.180862 D [94994:70108231981780] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:47:37.181190 D [94994:70108232772540] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:47:37.181258 D [94994:70108232772540] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:47:37.181269 T [94994:70108232772540] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:47:37.181333 D [94994:70108232772540] SimpleTCPServer -- Sending Reply
2012-10-12 14:47:37.181346 T [94994:70108232772540] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:47:37.181675 T [94994:70108231981780] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:47:37.181688 D [94994:70108231981780] (0.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:47:37.181734 T [94994:70108231981780] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:47:37.181744 D [94994:70108231981780] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:47:37.184015 W [94994:70108231981780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:47:37.184104 D [94994:70108240285200] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:47:37.685441 W [94994:70108231981780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:47:38.187993 W [94994:70108231981780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:47:38.688803 W [94994:70108231981780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:47:39.189904 W [94994:70108231981780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:47:39.690852 W [94994:70108231981780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-12 14:47:40.195817 W [94994:70108231981780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-12 14:47:40.697215 W [94994:70108231981780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-12 14:47:41.198276 W [94994:70108231981780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-12 14:47:41.700573 W [94994:70108231981780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-12 14:47:42.201527 E [94994:70108231981780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-12 14:47:42.201622 I [94994:70108231981780] (5018.2ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 connection attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:488:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:466:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:459:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:223:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `each_with_index'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:193:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:117:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:117:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:47:42.202396 D [94994:70108240285200] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:47:42.202746 T [94994:70108231981780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:47:42.203014 D [94994:70108231981780] (0.3ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:47:42.203448 D [94994:70108240285200] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:47:42.203460 T [94994:70108240285200] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:47:42.203476 D [94994:70108240285200] SimpleTCPServer -- Sending Reply
2012-10-12 14:47:42.203485 T [94994:70108240285200] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:47:42.203942 T [94994:70108231981780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:47:42.203958 D [94994:70108231981780] (0.6ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:47:42.204005 T [94994:70108231981780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:47:42.204014 D [94994:70108231981780] (0.0ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:47:42.205687 W [94994:70108231981780] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:47:42.308097 W [94994:70108231981780] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:47:42.409508 W [94994:70108231981780] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:47:42.510895 W [94994:70108231981780] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:47:42.612308 W [94994:70108231981780] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:47:42.715472 E [94994:70108231981780] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-12 14:47:42.715572 I [94994:70108231981780] (510.3ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 connection attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:488:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:466:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:459:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:230:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:193:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:48:31.729954 D [97106:70173941791180] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:48:31.731147 T [97106:70173939948240] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:48:31.731501 D [97106:70173941791180] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:48:31.731808 D [97106:70173941791180] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:48:31.731825 T [97106:70173941791180] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:48:31.731840 D [97106:70173941791180] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:48:31.731878 D [97106:70173941791180] SimpleTCPServer -- Server closed
2012-10-12 14:48:31.731889 D [97106:70173941791180] SimpleTCPServer -- thread killed
2012-10-12 14:48:31.738691 D [97106:70173941791180] SimpleTCPServer -- Server Restarted
2012-10-12 14:48:31.738974 D [97106:70173941766720] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:48:31.739753 D [97106:70173939948240] (8.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:48:31.740163 T [97106:70173939948240] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:48:31.740235 W [97106:70173939948240] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:48:31.740313 D [97106:70173939948240] (0.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:277:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:111:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:374:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:106:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:48:31.740356 W [97106:70173939948240] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure retry 1 due to ResilientSocket::ConnectionFailure: Connection lost while reading data
2012-10-12 14:48:31.751830 T [97106:70173939948240] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:48:31.751909 D [97106:70173941766720] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:48:31.752237 D [97106:70173939948240] (0.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:48:31.752387 D [97106:70173941766720] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:48:31.752399 T [97106:70173941766720] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-12 14:48:31.752415 D [97106:70173941766720] SimpleTCPServer -- Sending Reply
2012-10-12 14:48:31.752424 T [97106:70173941766720] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-12 14:48:31.753148 T [97106:70173939948240] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x16\x00\x00\x00"
2012-10-12 14:48:31.753183 D [97106:70173939948240] (0.9ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:48:31.753513 T [97106:70173939948240] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-12 14:48:31.753545 D [97106:70173939948240] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 18 bytes
2012-10-12 14:48:31.754365 D [97106:70173940135240] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:48:31.755206 D [97106:70173940135240] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:48:31.755471 T [97106:70173939948240] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:48:31.755718 D [97106:70173939948240] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:48:31.755916 D [97106:70173940135240] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:48:31.755930 T [97106:70173940135240] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:48:31.755943 D [97106:70173940135240] SimpleTCPServer -- Sending Reply
2012-10-12 14:48:31.755952 T [97106:70173940135240] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:48:31.756525 T [97106:70173939948240] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:48:31.756568 D [97106:70173939948240] (0.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:48:31.756787 T [97106:70173939948240] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:48:31.756826 D [97106:70173939948240] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:48:31.757574 D [97106:70173940366260] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:48:31.758393 T [97106:70173939948240] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-12 14:48:31.758580 D [97106:70173940366260] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:48:31.758652 D [97106:70173940366260] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:48:31.758664 T [97106:70173940366260] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-12 14:48:31.759098 D [97106:70173939948240] (0.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-12 14:48:34.761285 W [97106:70173939948240] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-12 14:48:34.761427 D [97106:70173939948240] (3002.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:283:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:277:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:96:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:94:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:48:35.258792 D [97106:70173940366260] SimpleTCPServer -- Sending Reply
2012-10-12 14:48:35.258836 T [97106:70173940366260] SimpleTCPServer -- Reply -- {"result"=>"sleep"}
2012-10-12 14:48:35.259229 T [97106:70173939948240] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:48:35.259247 D [97106:70173939948240] (497.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:48:35.259306 T [97106:70173939948240] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00sleep\x00\x00"
2012-10-12 14:48:35.259316 D [97106:70173939948240] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:48:35.260984 D [97106:70173948383600] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:48:35.261595 T [97106:70173939948240] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-12 14:48:35.261655 D [97106:70173939948240] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-12 14:48:35.262155 D [97106:70173948383600] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:48:35.262323 D [97106:70173948383600] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:48:35.262335 T [97106:70173948383600] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-12 14:48:38.261696 W [97106:70173939948240] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-12 14:48:38.261831 D [97106:70173939948240] (3000.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:283:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:277:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:84:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:82:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:48:38.262831 D [97106:70173948469700] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:48:38.263686 T [97106:70173939948240] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:48:38.263928 D [97106:70173948469700] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:48:38.264121 D [97106:70173948469700] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:48:38.264159 T [97106:70173948469700] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:48:38.264179 D [97106:70173948469700] SimpleTCPServer -- Sending Reply
2012-10-12 14:48:38.264189 T [97106:70173948469700] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:48:38.264282 D [97106:70173939948240] (0.6ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:48:38.264344 T [97106:70173939948240] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:48:38.264356 D [97106:70173939948240] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:48:38.264401 T [97106:70173939948240] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:48:38.264410 D [97106:70173939948240] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:48:38.265736 D [97106:70173948593420] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:48:38.266428 W [97106:70173939948240] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:48:38.767814 W [97106:70173939948240] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:48:39.268839 W [97106:70173939948240] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:48:39.776071 W [97106:70173939948240] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:48:40.277552 W [97106:70173939948240] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:48:40.778748 W [97106:70173939948240] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-12 14:48:41.280273 W [97106:70173939948240] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-12 14:48:41.781468 W [97106:70173939948240] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-12 14:48:42.285373 W [97106:70173939948240] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-12 14:48:42.786964 W [97106:70173939948240] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-12 14:48:43.289119 E [97106:70173939948240] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-12 14:48:43.289217 I [97106:70173939948240] (5023.5ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 connection attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:488:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:466:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:459:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:223:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `each_with_index'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:193:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:120:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:120:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:48:43.290104 D [97106:70173948593420] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:48:43.290358 T [97106:70173939948240] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:48:43.290436 D [97106:70173939948240] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:48:43.290686 D [97106:70173948593420] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:48:43.290699 T [97106:70173948593420] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:48:43.290715 D [97106:70173948593420] SimpleTCPServer -- Sending Reply
2012-10-12 14:48:43.290725 T [97106:70173948593420] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:48:43.291305 T [97106:70173939948240] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:48:43.291322 D [97106:70173939948240] (0.9ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:48:43.291369 T [97106:70173939948240] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:48:43.291378 D [97106:70173939948240] (0.0ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:48:43.293150 W [97106:70173939948240] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:48:43.394532 W [97106:70173939948240] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:48:43.495883 W [97106:70173939948240] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:48:43.597049 W [97106:70173939948240] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:48:43.697836 W [97106:70173939948240] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:48:43.798624 E [97106:70173939948240] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-12 14:48:43.798733 I [97106:70173939948240] (506.6ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 connection attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:488:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:466:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:459:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:230:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:193:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:52:58.335515 D [239:70142215520920] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:52:58.337509 T [239:70142205844180] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:52:58.337630 D [239:70142215520920] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:52:58.338005 D [239:70142215520920] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:52:58.338022 T [239:70142215520920] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:52:58.338038 D [239:70142215520920] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:52:58.338077 D [239:70142215520920] SimpleTCPServer -- Server closed
2012-10-12 14:52:58.338088 D [239:70142215520920] SimpleTCPServer -- thread killed
2012-10-12 14:52:58.344566 D [239:70142215520920] SimpleTCPServer -- Server Restarted
2012-10-12 14:52:58.344663 D [239:70142205844180] (7.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:52:58.344905 D [239:70142215496480] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:52:58.344984 T [239:70142205844180] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:52:58.345050 W [239:70142205844180] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:52:58.345124 D [239:70142205844180] (0.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:309:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:286:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:111:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:383:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:106:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:52:58.345165 W [239:70142205844180] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure retry 1 due to ResilientSocket::ConnectionFailure: Connection lost while reading data
2012-10-12 14:52:58.346839 T [239:70142205844180] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:52:58.346887 D [239:70142205844180] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:52:58.346946 D [239:70142215496480] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:52:58.347007 D [239:70142215496480] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:52:58.347017 T [239:70142215496480] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-12 14:52:58.347033 D [239:70142215496480] SimpleTCPServer -- Sending Reply
2012-10-12 14:52:58.347043 T [239:70142215496480] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-12 14:52:58.347330 T [239:70142205844180] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x16\x00\x00\x00"
2012-10-12 14:52:58.347343 D [239:70142205844180] (0.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:52:58.347451 T [239:70142205844180] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-12 14:52:58.347462 D [239:70142205844180] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 18 bytes
2012-10-12 14:52:58.348660 D [239:70142205777800] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:52:58.349927 T [239:70142205844180] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:52:58.350116 D [239:70142205777800] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:52:58.350178 D [239:70142205777800] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:52:58.350190 T [239:70142205777800] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:52:58.350203 D [239:70142205777800] SimpleTCPServer -- Sending Reply
2012-10-12 14:52:58.350212 T [239:70142205777800] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:52:58.350660 D [239:70142205844180] (0.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:52:58.350726 T [239:70142205844180] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:52:58.350737 D [239:70142205844180] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:52:58.350781 T [239:70142205844180] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:52:58.350790 D [239:70142205844180] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:52:58.351779 D [239:70142206261960] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:52:58.352283 T [239:70142205844180] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-12 14:52:58.352463 D [239:70142205844180] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-12 14:52:58.352669 D [239:70142206261960] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:52:58.352773 D [239:70142206261960] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:52:58.352785 T [239:70142206261960] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-12 14:53:01.353532 W [239:70142205844180] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-12 14:53:01.353977 D [239:70142205844180] (3001.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:292:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:286:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:96:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:94:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:53:01.853382 D [239:70142206261960] SimpleTCPServer -- Sending Reply
2012-10-12 14:53:01.853502 T [239:70142206261960] SimpleTCPServer -- Reply -- {"result"=>"sleep"}
2012-10-12 14:53:01.854624 T [239:70142205844180] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:53:01.854673 D [239:70142205844180] (499.9ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:53:01.854922 T [239:70142205844180] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00sleep\x00\x00"
2012-10-12 14:53:01.854937 D [239:70142205844180] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:53:01.856442 D [239:70142214186200] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:53:01.858312 T [239:70142205844180] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-12 14:53:01.858449 D [239:70142214186200] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:53:01.858532 D [239:70142214186200] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:53:01.858548 T [239:70142214186200] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-12 14:53:01.859036 D [239:70142205844180] (0.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-12 14:53:04.859687 W [239:70142205844180] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-12 14:53:04.859887 D [239:70142205844180] (3000.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:292:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:286:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:84:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:82:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:53:04.860921 D [239:70142206477960] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:53:04.861859 T [239:70142205844180] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:53:04.861939 D [239:70142205844180] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:53:04.862326 D [239:70142206477960] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:53:04.862431 D [239:70142206477960] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:53:04.862443 T [239:70142206477960] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:53:04.862458 D [239:70142206477960] SimpleTCPServer -- Sending Reply
2012-10-12 14:53:04.862469 T [239:70142206477960] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:53:04.862559 T [239:70142205844180] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:53:04.862571 D [239:70142205844180] (0.6ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:53:04.862620 T [239:70142205844180] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:53:04.862630 D [239:70142205844180] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:53:04.863112 D [239:70142206576860] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:53:04.864503 W [239:70142205844180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:53:05.365307 W [239:70142205844180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:53:05.866380 W [239:70142205844180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:53:06.367896 W [239:70142205844180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:53:06.868838 W [239:70142205844180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:53:07.370262 W [239:70142205844180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-12 14:53:07.871536 W [239:70142205844180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-12 14:53:08.372337 W [239:70142205844180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-12 14:53:08.873712 W [239:70142205844180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-12 14:53:09.376788 W [239:70142205844180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-12 14:53:09.878309 E [239:70142205844180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-12 14:53:09.878413 I [239:70142205844180] (5015.4ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 connection attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:497:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:475:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:468:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:224:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:222:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:222:in `each_with_index'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:222:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:193:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:120:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:120:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:53:09.879772 T [239:70142205844180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:53:09.879858 D [239:70142205844180] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:53:09.880218 D [239:70142206576860] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:53:09.880289 D [239:70142206576860] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:53:09.880301 T [239:70142206576860] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:53:09.880317 D [239:70142206576860] SimpleTCPServer -- Sending Reply
2012-10-12 14:53:09.880326 T [239:70142206576860] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:53:09.880416 T [239:70142205844180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:53:09.880428 D [239:70142205844180] (0.5ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:53:09.880476 T [239:70142205844180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:53:09.880495 D [239:70142205844180] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:53:09.882439 W [239:70142205844180] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:53:09.982960 W [239:70142205844180] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:53:10.083708 W [239:70142205844180] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:53:10.184564 W [239:70142205844180] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:53:10.286062 W [239:70142205844180] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:53:10.387621 E [239:70142205844180] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-12 14:53:10.387730 I [239:70142205844180] (506.8ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 connection attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:497:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:475:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:468:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:231:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:193:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:53:50.651088 D [858:70288244856220] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:53:50.652372 D [858:70288244856220] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:53:50.653161 T [858:70288243120860] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:53:50.653428 D [858:70288244856220] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:53:50.653445 T [858:70288244856220] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:53:50.653460 D [858:70288244856220] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:53:50.653508 D [858:70288244856220] SimpleTCPServer -- Server closed
2012-10-12 14:53:50.653519 D [858:70288244856220] SimpleTCPServer -- thread killed
2012-10-12 14:53:50.659966 D [858:70288244856220] SimpleTCPServer -- Server Restarted
2012-10-12 14:53:50.660109 D [858:70288243120860] (6.9ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:53:50.662204 D [858:70288244828780] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:53:50.662469 T [858:70288243120860] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:53:50.662571 W [858:70288243120860] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:53:50.662654 D [858:70288243120860] (2.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:311:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:288:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:111:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:385:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:106:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:53:50.662701 W [858:70288243120860] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure retry 1 due to ResilientSocket::ConnectionFailure: Connection lost while reading data
2012-10-12 14:53:50.664698 T [858:70288243120860] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:53:50.664864 D [858:70288243120860] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:53:50.665090 D [858:70288244828780] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:53:50.665162 D [858:70288244828780] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:53:50.665174 T [858:70288244828780] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-12 14:53:50.665192 D [858:70288244828780] SimpleTCPServer -- Sending Reply
2012-10-12 14:53:50.665202 T [858:70288244828780] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-12 14:53:50.665798 T [858:70288243120860] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x16\x00\x00\x00"
2012-10-12 14:53:50.665875 D [858:70288243120860] (1.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:53:50.665930 T [858:70288243120860] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-12 14:53:50.665940 D [858:70288243120860] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 18 bytes
2012-10-12 14:53:50.666964 D [858:70288247236200] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:53:50.668036 T [858:70288243120860] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:53:50.668230 D [858:70288243120860] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:53:50.668318 D [858:70288247236200] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:53:50.668453 D [858:70288247236200] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:53:50.668465 T [858:70288247236200] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:53:50.668479 D [858:70288247236200] SimpleTCPServer -- Sending Reply
2012-10-12 14:53:50.668488 T [858:70288247236200] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:53:50.669109 T [858:70288243120860] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:53:50.669124 D [858:70288243120860] (0.9ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:53:50.669171 T [858:70288243120860] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:53:50.669181 D [858:70288243120860] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:53:50.670105 D [858:70288247468880] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:53:50.670739 T [858:70288243120860] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-12 14:53:50.670894 D [858:70288243120860] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-12 14:53:50.671112 D [858:70288247468880] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:53:50.671273 D [858:70288247468880] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:53:50.671305 T [858:70288247468880] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-12 14:53:53.672061 W [858:70288243120860] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-12 14:53:53.672205 D [858:70288243120860] (3001.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:294:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:288:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:96:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:94:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:53:54.172418 D [858:70288247468880] SimpleTCPServer -- Sending Reply
2012-10-12 14:53:54.172462 T [858:70288247468880] SimpleTCPServer -- Reply -- {"result"=>"sleep"}
2012-10-12 14:53:54.172820 T [858:70288243120860] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:53:54.172837 D [858:70288243120860] (500.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:53:54.172938 T [858:70288243120860] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00sleep\x00\x00"
2012-10-12 14:53:54.172949 D [858:70288243120860] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:53:54.174331 D [858:70288247514780] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:53:54.174858 T [858:70288243120860] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-12 14:53:54.174912 D [858:70288243120860] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-12 14:53:54.174986 D [858:70288247514780] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:53:54.175247 D [858:70288247514780] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:53:54.175260 T [858:70288247514780] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-12 14:53:57.175425 W [858:70288243120860] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-12 14:53:57.175561 D [858:70288243120860] (3000.6ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:294:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:288:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:84:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:82:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:53:57.177189 D [858:70288247634140] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:53:57.178115 T [858:70288243120860] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:53:57.178193 D [858:70288247634140] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:53:57.178225 D [858:70288243120860] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:53:57.178624 D [858:70288247634140] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:53:57.178641 T [858:70288247634140] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:53:57.178656 D [858:70288247634140] SimpleTCPServer -- Sending Reply
2012-10-12 14:53:57.178666 T [858:70288247634140] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:53:57.178830 T [858:70288243120860] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:53:57.178843 D [858:70288243120860] (0.6ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:53:57.178992 T [858:70288243120860] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:53:57.179004 D [858:70288243120860] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:53:57.180905 D [858:70288247730020] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:53:57.182770 W [858:70288243120860] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:53:57.685100 W [858:70288243120860] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:53:58.186733 W [858:70288243120860] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:53:58.687896 W [858:70288243120860] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:53:59.189396 W [858:70288243120860] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:53:59.690486 W [858:70288243120860] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-12 14:54:00.191841 W [858:70288243120860] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-12 14:54:00.693355 W [858:70288243120860] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-12 14:54:01.194884 W [858:70288243120860] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-12 14:54:01.696336 W [858:70288243120860] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-12 14:54:02.197314 E [858:70288243120860] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-12 14:54:02.197425 I [858:70288243120860] (5017.5ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 connection attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:499:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:477:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:470:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:226:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:224:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:224:in `each_with_index'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:224:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:193:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:120:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:120:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:54:02.198577 T [858:70288243120860] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:54:02.198646 D [858:70288243120860] (0.0ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:54:02.199132 D [858:70288247730020] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:54:02.199210 D [858:70288247730020] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:54:02.199223 T [858:70288247730020] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:54:02.199240 D [858:70288247730020] SimpleTCPServer -- Sending Reply
2012-10-12 14:54:02.199251 T [858:70288247730020] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:54:02.199767 T [858:70288243120860] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:54:02.199783 D [858:70288243120860] (1.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:54:02.200240 T [858:70288243120860] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:54:02.200269 D [858:70288243120860] (0.4ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:54:02.201489 W [858:70288243120860] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:54:02.301972 W [858:70288243120860] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:54:02.402974 W [858:70288243120860] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:54:02.503963 W [858:70288243120860] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:54:02.605835 W [858:70288243120860] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:54:02.707261 E [858:70288243120860] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-12 14:54:02.707366 I [858:70288243120860] (506.7ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 connection attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:499:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:477:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:470:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:233:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:193:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-15 07:37:11.882640 D [6080:70236440923760] SimpleTCPServer -- Waiting for a client to connect
2012-10-15 07:37:11.886870 D [6080:70236440923760] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-15 07:37:11.887341 T [6080:70236439272160] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-15 07:37:11.887517 D [6080:70236440923760] SimpleTCPServer -- 
****************** Received request
2012-10-15 07:37:11.887536 T [6080:70236440923760] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-15 07:37:11.887551 D [6080:70236440923760] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-15 07:37:11.887637 D [6080:70236440923760] SimpleTCPServer -- Server closed
2012-10-15 07:37:11.887648 D [6080:70236440923760] SimpleTCPServer -- thread killed
2012-10-15 07:37:11.894188 D [6080:70236440923760] SimpleTCPServer -- Server Restarted
2012-10-15 07:37:11.894328 D [6080:70236439272160] (7.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-15 07:37:11.894428 T [6080:70236439272160] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-15 07:37:11.894492 W [6080:70236439272160] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-15 07:37:11.894565 D [6080:70236439272160] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:327:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:304:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:111:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:382:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:106:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-15 07:37:11.894606 W [6080:70236439272160] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure retry 1 due to ResilientSocket::ConnectionFailure: Connection lost while reading data
2012-10-15 07:37:11.894683 D [6080:70236440899940] SimpleTCPServer -- Waiting for a client to connect
2012-10-15 07:37:11.895222 T [6080:70236439272160] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-15 07:37:11.895272 D [6080:70236439272160] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-15 07:37:11.895329 D [6080:70236440899940] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-15 07:37:11.895393 D [6080:70236440899940] SimpleTCPServer -- 
****************** Received request
2012-10-15 07:37:11.895404 T [6080:70236440899940] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-15 07:37:11.895419 D [6080:70236440899940] SimpleTCPServer -- Sending Reply
2012-10-15 07:37:11.895435 T [6080:70236440899940] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-15 07:37:11.895535 T [6080:70236439272160] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x16\x00\x00\x00"
2012-10-15 07:37:11.895547 D [6080:70236439272160] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-15 07:37:11.895592 T [6080:70236439272160] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-15 07:37:11.895601 D [6080:70236439272160] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 18 bytes
2012-10-15 07:37:11.899527 D [6080:70236439117340] SimpleTCPServer -- Waiting for a client to connect
2012-10-15 07:37:11.900053 D [6080:70236439117340] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-15 07:37:11.900706 T [6080:70236439272160] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-15 07:37:11.903822 D [6080:70236439117340] SimpleTCPServer -- 
****************** Received request
2012-10-15 07:37:11.903932 T [6080:70236439117340] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-15 07:37:11.903951 D [6080:70236439117340] SimpleTCPServer -- Sending Reply
2012-10-15 07:37:11.903961 T [6080:70236439117340] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-15 07:37:11.904974 D [6080:70236439272160] (4.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-15 07:37:11.905324 T [6080:70236439272160] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-15 07:37:11.905467 D [6080:70236439272160] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-15 07:37:11.905896 T [6080:70236439272160] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-15 07:37:11.905931 D [6080:70236439272160] (0.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-15 07:37:11.907122 D [6080:70236443709680] SimpleTCPServer -- Waiting for a client to connect
2012-10-15 07:37:11.907787 D [6080:70236443709680] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-15 07:37:11.908217 T [6080:70236439272160] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-15 07:37:11.908463 D [6080:70236443709680] SimpleTCPServer -- 
****************** Received request
2012-10-15 07:37:11.908479 T [6080:70236443709680] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-15 07:37:11.908874 D [6080:70236439272160] (0.6ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-15 07:37:14.909606 W [6080:70236439272160] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-15 07:37:14.909862 D [6080:70236439272160] (3000.9ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:310:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:304:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:96:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:94:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-15 07:37:15.409233 D [6080:70236443709680] SimpleTCPServer -- Sending Reply
2012-10-15 07:37:15.409298 T [6080:70236443709680] SimpleTCPServer -- Reply -- {"result"=>"sleep"}
2012-10-15 07:37:15.409637 T [6080:70236439272160] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-15 07:37:15.409655 D [6080:70236439272160] (499.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-15 07:37:15.409722 T [6080:70236439272160] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00sleep\x00\x00"
2012-10-15 07:37:15.409733 D [6080:70236439272160] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-15 07:37:15.412004 D [6080:70236439550780] SimpleTCPServer -- Waiting for a client to connect
2012-10-15 07:37:15.412533 T [6080:70236439272160] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-15 07:37:15.412589 D [6080:70236439272160] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-15 07:37:15.412955 D [6080:70236439550780] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-15 07:37:15.413026 D [6080:70236439550780] SimpleTCPServer -- 
****************** Received request
2012-10-15 07:37:15.413038 T [6080:70236439550780] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-15 07:37:18.413371 W [6080:70236439272160] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-15 07:37:18.413511 D [6080:70236439272160] (3000.9ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:310:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:304:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:84:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:82:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-15 07:37:18.415104 D [6080:70236439634160] SimpleTCPServer -- Waiting for a client to connect
2012-10-15 07:37:18.416085 T [6080:70236439272160] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-15 07:37:18.416264 D [6080:70236439634160] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-15 07:37:18.416332 D [6080:70236439634160] SimpleTCPServer -- 
****************** Received request
2012-10-15 07:37:18.416343 T [6080:70236439634160] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-15 07:37:18.416358 D [6080:70236439634160] SimpleTCPServer -- Sending Reply
2012-10-15 07:37:18.416367 T [6080:70236439634160] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-15 07:37:18.416803 D [6080:70236439272160] (0.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-15 07:37:18.416870 T [6080:70236439272160] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-15 07:37:18.416882 D [6080:70236439272160] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-15 07:37:18.416928 T [6080:70236439272160] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-15 07:37:18.416938 D [6080:70236439272160] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-15 07:37:18.418151 D [6080:70236439762580] SimpleTCPServer -- Waiting for a client to connect
2012-10-15 07:37:18.418602 W [6080:70236439272160] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-15 07:37:18.920069 W [6080:70236439272160] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-15 07:37:19.422183 W [6080:70236439272160] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-15 07:37:19.922988 W [6080:70236439272160] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-15 07:37:20.423619 W [6080:70236439272160] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-15 07:37:20.924851 W [6080:70236439272160] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-15 07:37:21.425925 W [6080:70236439272160] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-15 07:37:21.927070 W [6080:70236439272160] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-15 07:37:22.428558 W [6080:70236439272160] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-15 07:37:22.929588 W [6080:70236439272160] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-15 07:37:23.430719 E [6080:70236439272160] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-15 07:37:23.430818 I [6080:70236439272160] (5012.7ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 connection attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:496:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:474:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:467:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:234:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:232:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:232:in `each_with_index'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:232:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:201:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:120:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:120:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-15 07:37:23.433321 T [6080:70236439272160] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-15 07:37:23.433530 D [6080:70236439272160] (0.2ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-15 07:37:23.433707 D [6080:70236439762580] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-15 07:37:23.433821 D [6080:70236439762580] SimpleTCPServer -- 
****************** Received request
2012-10-15 07:37:23.433834 T [6080:70236439762580] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-15 07:37:23.433850 D [6080:70236439762580] SimpleTCPServer -- Sending Reply
2012-10-15 07:37:23.433860 T [6080:70236439762580] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-15 07:37:23.434631 T [6080:70236439272160] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-15 07:37:23.434648 D [6080:70236439272160] (1.0ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-15 07:37:23.434699 T [6080:70236439272160] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-15 07:37:23.434709 D [6080:70236439272160] (0.0ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-15 07:37:23.436235 W [6080:70236439272160] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-15 07:37:23.536689 W [6080:70236439272160] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-15 07:37:23.637537 W [6080:70236439272160] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-15 07:37:23.738516 W [6080:70236439272160] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-15 07:37:23.839666 W [6080:70236439272160] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-15 07:37:23.941269 E [6080:70236439272160] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-15 07:37:23.941403 I [6080:70236439272160] (506.0ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 connection attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:496:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:474:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:467:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:241:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:201:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-15 07:43:57.049787 D [6347:70199664563820] SimpleTCPServer -- Waiting for a client to connect
2012-10-15 07:43:57.051712 T [6347:70199663614680] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-15 07:43:57.051898 D [6347:70199663614680] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-15 07:43:57.052128 D [6347:70199664563820] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-15 07:43:57.052203 D [6347:70199664563820] SimpleTCPServer -- 
****************** Received request
2012-10-15 07:43:57.052214 T [6347:70199664563820] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-15 07:43:57.052229 D [6347:70199664563820] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-15 07:43:57.052269 D [6347:70199664563820] SimpleTCPServer -- Server closed
2012-10-15 07:43:57.052280 D [6347:70199664563820] SimpleTCPServer -- thread killed
2012-10-15 07:43:57.058958 D [6347:70199664563820] SimpleTCPServer -- Server Restarted
2012-10-15 07:43:57.059144 T [6347:70199663614680] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-15 07:43:57.059215 W [6347:70199663614680] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-15 07:43:57.059295 D [6347:70199663614680] (7.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:327:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:304:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:111:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:382:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:106:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-15 07:43:57.059346 W [6347:70199663614680] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure retry 1 due to ResilientSocket::ConnectionFailure: Connection lost while reading data
2012-10-15 07:43:57.059637 D [6347:70199668961080] SimpleTCPServer -- Waiting for a client to connect
2012-10-15 07:43:57.060770 T [6347:70199663614680] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-15 07:43:57.060822 D [6347:70199663614680] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-15 07:43:57.061008 D [6347:70199668961080] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-15 07:43:57.061325 D [6347:70199668961080] SimpleTCPServer -- 
****************** Received request
2012-10-15 07:43:57.061337 T [6347:70199668961080] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-15 07:43:57.061354 D [6347:70199668961080] SimpleTCPServer -- Sending Reply
2012-10-15 07:43:57.061364 T [6347:70199668961080] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-15 07:43:57.062390 T [6347:70199663614680] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x16\x00\x00\x00"
2012-10-15 07:43:57.062407 D [6347:70199663614680] (1.6ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-15 07:43:57.063858 T [6347:70199663614680] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-15 07:43:57.063875 D [6347:70199663614680] (1.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 18 bytes
2012-10-15 07:43:57.064968 D [6347:70199667644740] SimpleTCPServer -- Waiting for a client to connect
2012-10-15 07:43:57.065537 T [6347:70199663614680] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-15 07:43:57.065776 D [6347:70199663614680] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-15 07:43:57.065988 D [6347:70199667644740] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-15 07:43:57.066050 D [6347:70199667644740] SimpleTCPServer -- 
****************** Received request
2012-10-15 07:43:57.066061 T [6347:70199667644740] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-15 07:43:57.066076 D [6347:70199667644740] SimpleTCPServer -- Sending Reply
2012-10-15 07:43:57.066086 T [6347:70199667644740] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-15 07:43:57.066271 T [6347:70199663614680] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-15 07:43:57.066300 D [6347:70199663614680] (0.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-15 07:43:57.066352 T [6347:70199663614680] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-15 07:43:57.066362 D [6347:70199663614680] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-15 07:43:57.067852 D [6347:70199667925500] SimpleTCPServer -- Waiting for a client to connect
2012-10-15 07:43:57.068355 T [6347:70199663614680] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-15 07:43:57.068540 D [6347:70199667925500] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-15 07:43:57.068606 D [6347:70199667925500] SimpleTCPServer -- 
****************** Received request
2012-10-15 07:43:57.068617 T [6347:70199667925500] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-15 07:43:57.068713 D [6347:70199663614680] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-15 07:44:00.069863 W [6347:70199663614680] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-15 07:44:00.070061 D [6347:70199663614680] (3001.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:310:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:304:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:96:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:94:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-15 07:44:00.569653 D [6347:70199667925500] SimpleTCPServer -- Sending Reply
2012-10-15 07:44:00.569707 T [6347:70199667925500] SimpleTCPServer -- Reply -- {"result"=>"sleep"}
2012-10-15 07:44:00.570147 T [6347:70199663614680] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-15 07:44:00.570172 D [6347:70199663614680] (499.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-15 07:44:00.570345 T [6347:70199663614680] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00sleep\x00\x00"
2012-10-15 07:44:00.570359 D [6347:70199663614680] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-15 07:44:00.572329 D [6347:70199664083580] SimpleTCPServer -- Waiting for a client to connect
2012-10-15 07:44:00.573595 T [6347:70199663614680] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-15 07:44:00.573659 D [6347:70199663614680] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-15 07:44:00.573743 D [6347:70199664083580] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-15 07:44:00.573852 D [6347:70199664083580] SimpleTCPServer -- 
****************** Received request
2012-10-15 07:44:00.573865 T [6347:70199664083580] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-15 07:44:03.574774 W [6347:70199663614680] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-15 07:44:03.575109 D [6347:70199663614680] (3001.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:310:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:304:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:84:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:82:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-15 07:44:03.578264 D [6347:70199664181960] SimpleTCPServer -- Waiting for a client to connect
2012-10-15 07:44:03.579630 T [6347:70199663614680] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-15 07:44:03.579731 D [6347:70199664181960] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-15 07:44:03.580061 D [6347:70199664181960] SimpleTCPServer -- 
****************** Received request
2012-10-15 07:44:03.580078 T [6347:70199664181960] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-15 07:44:03.580104 D [6347:70199664181960] SimpleTCPServer -- Sending Reply
2012-10-15 07:44:03.580118 T [6347:70199664181960] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-15 07:44:03.580175 D [6347:70199663614680] (0.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-15 07:44:03.580900 T [6347:70199663614680] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-15 07:44:03.580947 D [6347:70199663614680] (0.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-15 07:44:03.581022 T [6347:70199663614680] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-15 07:44:03.581035 D [6347:70199663614680] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-15 07:44:03.582281 D [6347:70199664283240] SimpleTCPServer -- Waiting for a client to connect
2012-10-15 07:44:03.582747 W [6347:70199663614680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-15 07:44:04.083719 W [6347:70199663614680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-15 07:44:04.585135 W [6347:70199663614680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-15 07:44:05.086385 W [6347:70199663614680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-15 07:44:05.587839 W [6347:70199663614680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-15 07:44:06.089707 W [6347:70199663614680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-15 07:44:06.591624 W [6347:70199663614680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-15 07:44:07.093142 W [6347:70199663614680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-15 07:44:07.594050 W [6347:70199663614680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-15 07:44:08.096043 W [6347:70199663614680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-15 07:44:08.597585 E [6347:70199663614680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-15 07:44:08.597719 I [6347:70199663614680] (5015.6ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 connection attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:496:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:474:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:467:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:234:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:232:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:232:in `each_with_index'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:232:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:201:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:120:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:120:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-15 07:44:08.598803 D [6347:70199664283240] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-15 07:44:08.599193 T [6347:70199663614680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-15 07:44:08.599285 D [6347:70199663614680] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-15 07:44:08.599688 D [6347:70199664283240] SimpleTCPServer -- 
****************** Received request
2012-10-15 07:44:08.599702 T [6347:70199664283240] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-15 07:44:08.599720 D [6347:70199664283240] SimpleTCPServer -- Sending Reply
2012-10-15 07:44:08.599731 T [6347:70199664283240] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-15 07:44:08.600173 T [6347:70199663614680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-15 07:44:08.600192 D [6347:70199663614680] (0.9ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-15 07:44:08.600275 T [6347:70199663614680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-15 07:44:08.600326 D [6347:70199663614680] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-15 07:44:08.601916 W [6347:70199663614680] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-15 07:44:08.703389 W [6347:70199663614680] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-15 07:44:08.805279 W [6347:70199663614680] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-15 07:44:08.906189 W [6347:70199663614680] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-15 07:44:09.007756 W [6347:70199663614680] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-15 07:44:09.108625 E [6347:70199663614680] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-15 07:44:09.108746 I [6347:70199663614680] (507.6ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 connection attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:496:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:474:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:467:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:241:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:201:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-16 07:47:00.026936 D [34409:70134011837160] SimpleTCPServer -- Waiting for a client to connect
2012-10-16 07:47:00.028116 D [34409:70134011837160] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-16 07:47:00.028458 T [34409:70134010174160] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-16 07:47:00.028830 D [34409:70134011837160] SimpleTCPServer -- 
****************** Received request
2012-10-16 07:47:00.028843 T [34409:70134011837160] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-16 07:47:00.028857 D [34409:70134011837160] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-16 07:47:00.029000 D [34409:70134011837160] SimpleTCPServer -- Server closed
2012-10-16 07:47:00.029014 D [34409:70134011837160] SimpleTCPServer -- thread killed
2012-10-16 07:47:00.037700 D [34409:70134011837160] SimpleTCPServer -- Server Restarted
2012-10-16 07:47:00.037816 D [34409:70134010174160] (9.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-16 07:47:00.037919 T [34409:70134010174160] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-16 07:47:00.037985 W [34409:70134010174160] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-16 07:47:00.038060 D [34409:70134010174160] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:380:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:355:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:111:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:435:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:106:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-16 07:47:00.038101 W [34409:70134010174160] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure retry 1 due to ResilientSocket::ConnectionFailure: Connection lost while reading data
2012-10-16 07:47:00.038178 D [34409:70134011814020] SimpleTCPServer -- Waiting for a client to connect
2012-10-16 07:47:00.038827 T [34409:70134010174160] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-16 07:47:00.038876 D [34409:70134010174160] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-16 07:47:00.038934 D [34409:70134011814020] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-16 07:47:00.038996 D [34409:70134011814020] SimpleTCPServer -- 
****************** Received request
2012-10-16 07:47:00.039007 T [34409:70134011814020] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-16 07:47:00.039022 D [34409:70134011814020] SimpleTCPServer -- Sending Reply
2012-10-16 07:47:00.039032 T [34409:70134011814020] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-16 07:47:00.039146 T [34409:70134010174160] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x16\x00\x00\x00"
2012-10-16 07:47:00.039158 D [34409:70134010174160] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-16 07:47:00.039202 T [34409:70134010174160] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-16 07:47:00.039211 D [34409:70134010174160] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 18 bytes
2012-10-16 07:47:00.040007 D [34409:70134019057520] SimpleTCPServer -- Waiting for a client to connect
2012-10-16 07:47:00.045779 T [34409:70134010174160] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-16 07:47:00.046036 D [34409:70134019057520] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-16 07:47:00.046289 D [34409:70134010174160] (0.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-16 07:47:00.046518 D [34409:70134019057520] SimpleTCPServer -- 
****************** Received request
2012-10-16 07:47:00.046535 T [34409:70134019057520] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-16 07:47:00.046595 D [34409:70134019057520] SimpleTCPServer -- Sending Reply
2012-10-16 07:47:00.046606 T [34409:70134019057520] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-16 07:47:00.047105 T [34409:70134010174160] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-16 07:47:00.047152 D [34409:70134010174160] (0.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-16 07:47:00.047375 T [34409:70134010174160] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-16 07:47:00.047415 D [34409:70134010174160] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-16 07:47:00.048912 D [34409:70134010686180] SimpleTCPServer -- Waiting for a client to connect
2012-10-16 07:47:00.049559 D [34409:70134010686180] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-16 07:47:00.049842 T [34409:70134010174160] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-16 07:47:00.050038 D [34409:70134010174160] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-16 07:47:00.050262 D [34409:70134010686180] SimpleTCPServer -- 
****************** Received request
2012-10-16 07:47:00.050280 T [34409:70134010686180] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-16 07:47:03.050331 W [34409:70134010174160] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-16 07:47:03.050472 D [34409:70134010174160] (3000.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:362:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:355:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:96:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:94:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-16 07:47:03.551374 D [34409:70134010686180] SimpleTCPServer -- Sending Reply
2012-10-16 07:47:03.551415 T [34409:70134010686180] SimpleTCPServer -- Reply -- {"result"=>"sleep"}
2012-10-16 07:47:03.551784 T [34409:70134010174160] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-16 07:47:03.551800 D [34409:70134010174160] (501.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-16 07:47:03.551863 T [34409:70134010174160] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00sleep\x00\x00"
2012-10-16 07:47:03.551873 D [34409:70134010174160] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-16 07:47:03.552945 D [34409:70134010849320] SimpleTCPServer -- Waiting for a client to connect
2012-10-16 07:47:03.553832 T [34409:70134010174160] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-16 07:47:03.553915 D [34409:70134010174160] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-16 07:47:03.553994 D [34409:70134010849320] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-16 07:47:03.554058 D [34409:70134010849320] SimpleTCPServer -- 
****************** Received request
2012-10-16 07:47:03.554069 T [34409:70134010849320] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-16 07:47:06.555060 W [34409:70134010174160] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-16 07:47:06.555256 D [34409:70134010174160] (3001.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:362:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:355:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:84:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:82:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-16 07:47:06.556628 D [34409:70134018411500] SimpleTCPServer -- Waiting for a client to connect
2012-10-16 07:47:06.557245 T [34409:70134010174160] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-16 07:47:06.557421 D [34409:70134010174160] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-16 07:47:06.557496 D [34409:70134018411500] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-16 07:47:06.557563 D [34409:70134018411500] SimpleTCPServer -- 
****************** Received request
2012-10-16 07:47:06.557575 T [34409:70134018411500] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-16 07:47:06.557590 D [34409:70134018411500] SimpleTCPServer -- Sending Reply
2012-10-16 07:47:06.557599 T [34409:70134018411500] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-16 07:47:06.557780 T [34409:70134010174160] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-16 07:47:06.557794 D [34409:70134010174160] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-16 07:47:06.557844 T [34409:70134010174160] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-16 07:47:06.557854 D [34409:70134010174160] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-16 07:47:06.559757 W [34409:70134010174160] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-16 07:47:06.661390 W [34409:70134010174160] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-16 07:47:06.762851 W [34409:70134010174160] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-16 07:47:06.863705 W [34409:70134010174160] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-16 07:47:06.965009 W [34409:70134010174160] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-16 07:47:07.065542 E [34409:70134010174160] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-16 07:47:07.065666 I [34409:70134010174160] (506.3ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 connection attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:549:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:527:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:520:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:291:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-16 07:47:44.614457 D [34462:70327369397740] SimpleTCPServer -- Waiting for a client to connect
2012-10-16 07:47:44.616287 T [34462:70327367588560] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-16 07:47:44.616379 D [34462:70327367588560] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-16 07:47:44.616502 D [34462:70327369397740] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-16 07:47:44.616575 D [34462:70327369397740] SimpleTCPServer -- 
****************** Received request
2012-10-16 07:47:44.616586 T [34462:70327369397740] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-16 07:47:44.616601 D [34462:70327369397740] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-16 07:47:44.616677 D [34462:70327369397740] SimpleTCPServer -- Server closed
2012-10-16 07:47:44.616689 D [34462:70327369397740] SimpleTCPServer -- thread killed
2012-10-16 07:47:44.623762 D [34462:70327369397740] SimpleTCPServer -- Server Restarted
2012-10-16 07:47:44.624391 D [34462:70327369374380] SimpleTCPServer -- Waiting for a client to connect
2012-10-16 07:47:44.624425 T [34462:70327367588560] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-16 07:47:44.624492 W [34462:70327367588560] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-16 07:47:44.624566 D [34462:70327367588560] (8.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:380:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:355:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:111:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:435:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:106:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-16 07:47:44.624612 W [34462:70327367588560] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure retry 1 due to ResilientSocket::ConnectionFailure: Connection lost while reading data
2012-10-16 07:47:44.625973 T [34462:70327367588560] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-16 07:47:44.626024 D [34462:70327367588560] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-16 07:47:44.626278 D [34462:70327369374380] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-16 07:47:44.626430 D [34462:70327369374380] SimpleTCPServer -- 
****************** Received request
2012-10-16 07:47:44.626445 T [34462:70327369374380] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-16 07:47:44.626461 D [34462:70327369374380] SimpleTCPServer -- Sending Reply
2012-10-16 07:47:44.626472 T [34462:70327369374380] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-16 07:47:44.626852 T [34462:70327367588560] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x16\x00\x00\x00"
2012-10-16 07:47:44.626869 D [34462:70327367588560] (0.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-16 07:47:44.627079 T [34462:70327367588560] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-16 07:47:44.627093 D [34462:70327367588560] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 18 bytes
2012-10-16 07:47:44.628521 D [34462:70327367756760] SimpleTCPServer -- Waiting for a client to connect
2012-10-16 07:47:44.629159 T [34462:70327367588560] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-16 07:47:44.629339 D [34462:70327367588560] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-16 07:47:44.629425 D [34462:70327367756760] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-16 07:47:44.629490 D [34462:70327367756760] SimpleTCPServer -- 
****************** Received request
2012-10-16 07:47:44.629501 T [34462:70327367756760] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-16 07:47:44.629515 D [34462:70327367756760] SimpleTCPServer -- Sending Reply
2012-10-16 07:47:44.629524 T [34462:70327367756760] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-16 07:47:44.630009 T [34462:70327367588560] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-16 07:47:44.630022 D [34462:70327367588560] (0.6ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-16 07:47:44.630079 T [34462:70327367588560] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-16 07:47:44.630089 D [34462:70327367588560] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-16 07:47:44.631109 D [34462:70327371695280] SimpleTCPServer -- Waiting for a client to connect
2012-10-16 07:47:44.631595 T [34462:70327367588560] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-16 07:47:44.631760 D [34462:70327367588560] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-16 07:47:44.631833 D [34462:70327371695280] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-16 07:47:44.631901 D [34462:70327371695280] SimpleTCPServer -- 
****************** Received request
2012-10-16 07:47:44.631913 T [34462:70327371695280] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-16 07:47:47.632371 W [34462:70327367588560] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-16 07:47:47.632512 D [34462:70327367588560] (3000.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:362:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:355:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:96:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:94:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-16 07:47:48.132961 D [34462:70327371695280] SimpleTCPServer -- Sending Reply
2012-10-16 07:47:48.133006 T [34462:70327371695280] SimpleTCPServer -- Reply -- {"result"=>"sleep"}
2012-10-16 07:47:48.133387 T [34462:70327367588560] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-16 07:47:48.133406 D [34462:70327367588560] (500.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-16 07:47:48.133643 T [34462:70327367588560] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00sleep\x00\x00"
2012-10-16 07:47:48.133657 D [34462:70327367588560] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-16 07:47:48.134539 D [34462:70327368127000] SimpleTCPServer -- Waiting for a client to connect
2012-10-16 07:47:48.135229 T [34462:70327367588560] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-16 07:47:48.135284 D [34462:70327367588560] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-16 07:47:48.135602 D [34462:70327368127000] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-16 07:47:48.135831 D [34462:70327368127000] SimpleTCPServer -- 
****************** Received request
2012-10-16 07:47:48.135848 T [34462:70327368127000] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-16 07:47:51.136424 W [34462:70327367588560] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-16 07:47:51.136591 D [34462:70327367588560] (3001.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:362:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:355:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:84:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:82:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-16 07:47:51.138615 D [34462:70327368231040] SimpleTCPServer -- Waiting for a client to connect
2012-10-16 07:47:51.139269 T [34462:70327367588560] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-16 07:47:51.139473 D [34462:70327367588560] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-16 07:47:51.139703 D [34462:70327368231040] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-16 07:47:51.139796 D [34462:70327368231040] SimpleTCPServer -- 
****************** Received request
2012-10-16 07:47:51.139811 T [34462:70327368231040] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-16 07:47:51.139830 D [34462:70327368231040] SimpleTCPServer -- Sending Reply
2012-10-16 07:47:51.139843 T [34462:70327368231040] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-16 07:47:51.140042 T [34462:70327367588560] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-16 07:47:51.140059 D [34462:70327367588560] (0.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-16 07:47:51.140124 T [34462:70327367588560] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-16 07:47:51.140138 D [34462:70327367588560] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-16 07:47:51.140774 D [34462:70327368321140] SimpleTCPServer -- Waiting for a client to connect
2012-10-16 07:47:51.142059 W [34462:70327367588560] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-16 07:47:51.643275 W [34462:70327367588560] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-16 07:47:52.144094 W [34462:70327367588560] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-16 07:47:52.644753 W [34462:70327367588560] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-16 07:47:53.146052 W [34462:70327367588560] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-16 07:47:53.647850 W [34462:70327367588560] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-16 07:47:54.149082 W [34462:70327367588560] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-16 07:47:54.650500 W [34462:70327367588560] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-16 07:47:55.151021 W [34462:70327367588560] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-16 07:47:55.651566 W [34462:70327367588560] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-16 07:47:56.152912 E [34462:70327367588560] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-16 07:47:56.153009 I [34462:70327367588560] (5012.3ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 connection attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:549:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:527:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:520:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:260:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:258:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:258:in `find'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:258:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:120:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:120:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-16 07:47:56.154270 W [34462:70327367588560] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-16 07:47:56.255735 W [34462:70327367588560] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-16 07:47:56.357020 W [34462:70327367588560] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-16 07:47:56.458490 W [34462:70327367588560] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-16 07:47:56.560174 W [34462:70327367588560] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-16 07:47:56.661676 E [34462:70327367588560] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-16 07:47:56.661783 I [34462:70327367588560] (508.3ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 connection attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:549:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:527:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:520:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:291:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-16 07:49:06.904982 D [34591:70204618210940] SimpleTCPServer -- Waiting for a client to connect
2012-10-16 07:49:06.906350 T [34591:70204608699100] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-16 07:49:06.907045 D [34591:70204608699100] (0.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-16 07:49:06.907619 D [34591:70204618210940] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-16 07:49:06.907720 D [34591:70204618210940] SimpleTCPServer -- 
****************** Received request
2012-10-16 07:49:06.907731 T [34591:70204618210940] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-16 07:49:06.907746 D [34591:70204618210940] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-16 07:49:06.907786 D [34591:70204618210940] SimpleTCPServer -- Server closed
2012-10-16 07:49:06.907797 D [34591:70204618210940] SimpleTCPServer -- thread killed
2012-10-16 07:49:06.914000 D [34591:70204618210940] SimpleTCPServer -- Server Restarted
2012-10-16 07:49:06.914533 D [34591:70204618162680] SimpleTCPServer -- Waiting for a client to connect
2012-10-16 07:49:06.915245 T [34591:70204608699100] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-16 07:49:06.915361 W [34591:70204608699100] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-16 07:49:06.915463 D [34591:70204608699100] (8.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:382:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:357:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:111:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:437:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:106:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-16 07:49:06.915556 W [34591:70204608699100] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure retry 1 due to ResilientSocket::ConnectionFailure: Connection lost while reading data
2012-10-16 07:49:06.916616 T [34591:70204608699100] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-16 07:49:06.916767 D [34591:70204608699100] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-16 07:49:06.916838 D [34591:70204618162680] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-16 07:49:06.916906 D [34591:70204618162680] SimpleTCPServer -- 
****************** Received request
2012-10-16 07:49:06.916917 T [34591:70204618162680] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-16 07:49:06.916932 D [34591:70204618162680] SimpleTCPServer -- Sending Reply
2012-10-16 07:49:06.916942 T [34591:70204618162680] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-16 07:49:06.917467 T [34591:70204608699100] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x16\x00\x00\x00"
2012-10-16 07:49:06.917483 D [34591:70204608699100] (0.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-16 07:49:06.918057 T [34591:70204608699100] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-16 07:49:06.918073 D [34591:70204608699100] (0.6ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 18 bytes
2012-10-16 07:49:06.919555 D [34591:70204616966440] SimpleTCPServer -- Waiting for a client to connect
2012-10-16 07:49:06.920231 T [34591:70204608699100] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-16 07:49:06.920421 D [34591:70204608699100] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-16 07:49:06.920497 D [34591:70204616966440] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-16 07:49:06.920562 D [34591:70204616966440] SimpleTCPServer -- 
****************** Received request
2012-10-16 07:49:06.920573 T [34591:70204616966440] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-16 07:49:06.920588 D [34591:70204616966440] SimpleTCPServer -- Sending Reply
2012-10-16 07:49:06.920598 T [34591:70204616966440] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-16 07:49:06.921233 T [34591:70204608699100] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-16 07:49:06.921253 D [34591:70204608699100] (0.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-16 07:49:06.921356 T [34591:70204608699100] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-16 07:49:06.921369 D [34591:70204608699100] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-16 07:49:06.922640 T [34591:70204608699100] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-16 07:49:06.922795 D [34591:70204608699100] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-16 07:49:06.922896 D [34591:70204617202820] SimpleTCPServer -- Waiting for a client to connect
2012-10-16 07:49:06.922930 D [34591:70204617202820] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-16 07:49:06.923312 D [34591:70204617202820] SimpleTCPServer -- 
****************** Received request
2012-10-16 07:49:06.923331 T [34591:70204617202820] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-16 07:49:09.923631 W [34591:70204608699100] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-16 07:49:09.923788 D [34591:70204608699100] (3000.9ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:364:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:357:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:96:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:94:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-16 07:49:10.424406 D [34591:70204617202820] SimpleTCPServer -- Sending Reply
2012-10-16 07:49:10.424501 T [34591:70204617202820] SimpleTCPServer -- Reply -- {"result"=>"sleep"}
2012-10-16 07:49:10.425908 T [34591:70204608699100] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-16 07:49:10.425948 D [34591:70204608699100] (501.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-16 07:49:10.426521 T [34591:70204608699100] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00sleep\x00\x00"
2012-10-16 07:49:10.426536 D [34591:70204608699100] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-16 07:49:10.428045 D [34591:70204609082020] SimpleTCPServer -- Waiting for a client to connect
2012-10-16 07:49:10.429144 T [34591:70204608699100] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-16 07:49:10.429272 D [34591:70204608699100] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-16 07:49:10.429379 D [34591:70204609082020] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-16 07:49:10.429468 D [34591:70204609082020] SimpleTCPServer -- 
****************** Received request
2012-10-16 07:49:10.429484 T [34591:70204609082020] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-16 07:49:13.430343 W [34591:70204608699100] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-16 07:49:13.430480 D [34591:70204608699100] (3001.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:364:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:357:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:84:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:82:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-16 07:49:13.431699 D [34591:70204609168960] SimpleTCPServer -- Waiting for a client to connect
2012-10-16 07:49:13.432513 T [34591:70204608699100] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-16 07:49:13.432752 D [34591:70204609168960] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-16 07:49:13.432995 D [34591:70204609168960] SimpleTCPServer -- 
****************** Received request
2012-10-16 07:49:13.433008 T [34591:70204609168960] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-16 07:49:13.433024 D [34591:70204609168960] SimpleTCPServer -- Sending Reply
2012-10-16 07:49:13.433034 T [34591:70204609168960] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-16 07:49:13.433194 D [34591:70204608699100] (0.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-16 07:49:13.433265 T [34591:70204608699100] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-16 07:49:13.433277 D [34591:70204608699100] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-16 07:49:13.433323 T [34591:70204608699100] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-16 07:49:13.433333 D [34591:70204608699100] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-16 07:49:13.434258 D [34591:70204609280500] SimpleTCPServer -- Waiting for a client to connect
2012-10-16 07:49:13.434815 W [34591:70204608699100] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-16 07:49:13.936124 W [34591:70204608699100] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-16 07:49:14.437104 W [34591:70204608699100] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-16 07:49:14.937920 W [34591:70204608699100] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-16 07:49:15.439379 W [34591:70204608699100] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-16 07:49:15.940876 W [34591:70204608699100] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-16 07:49:16.442695 W [34591:70204608699100] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-16 07:49:16.944960 W [34591:70204608699100] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-16 07:49:17.445749 W [34591:70204608699100] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-16 07:49:17.947217 W [34591:70204608699100] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-16 07:49:18.448658 E [34591:70204608699100] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-16 07:49:18.448756 I [34591:70204608699100] (5014.7ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 connection attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:551:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:529:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:522:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:260:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:258:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:258:in `find'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:258:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:120:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:120:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-16 07:49:18.449642 T [34591:70204608699100] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-16 07:49:18.449712 D [34591:70204609280500] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-16 07:49:18.449745 D [34591:70204608699100] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-16 07:49:18.450091 D [34591:70204609280500] SimpleTCPServer -- 
****************** Received request
2012-10-16 07:49:18.450107 T [34591:70204609280500] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-16 07:49:18.450124 D [34591:70204609280500] SimpleTCPServer -- Sending Reply
2012-10-16 07:49:18.450135 T [34591:70204609280500] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-16 07:49:18.450558 T [34591:70204608699100] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-16 07:49:18.450575 D [34591:70204608699100] (0.8ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-16 07:49:18.450783 T [34591:70204608699100] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-16 07:49:18.450800 D [34591:70204608699100] (0.2ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-16 07:49:18.451815 W [34591:70204608699100] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-16 07:49:18.553051 W [34591:70204608699100] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-16 07:49:18.654938 W [34591:70204608699100] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-16 07:49:18.756613 W [34591:70204608699100] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-16 07:49:18.858166 W [34591:70204608699100] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-16 07:49:18.959609 E [34591:70204608699100] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-16 07:49:18.959707 I [34591:70204608699100] (508.4ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 connection attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:551:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:529:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:522:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:293:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-17 19:45:49.072575 D [59354:70121144190760] SimpleTCPServer -- Waiting for a client to connect
2012-10-17 19:45:49.076153 T [59354:70121142049500] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-17 19:45:49.076338 D [59354:70121142049500] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-17 19:45:49.077034 D [59354:70121144190760] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-17 19:45:49.077252 D [59354:70121144190760] SimpleTCPServer -- 
****************** Received request
2012-10-17 19:45:49.077468 T [59354:70121144190760] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-17 19:45:49.077485 D [59354:70121144190760] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-17 19:45:49.077809 D [59354:70121144190760] SimpleTCPServer -- Server closed
2012-10-17 19:45:49.078356 D [59354:70121144190760] SimpleTCPServer -- thread killed
2012-10-17 19:45:49.089703 D [59354:70121144190760] SimpleTCPServer -- Server Restarted
2012-10-17 19:45:49.089936 D [59354:70121144161840] SimpleTCPServer -- Waiting for a client to connect
2012-10-17 19:45:49.090364 T [59354:70121142049500] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-17 19:45:49.090458 W [59354:70121142049500] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-17 19:45:49.090544 D [59354:70121142049500] (14.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:398:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:367:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:111:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:454:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:106:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-17 19:45:49.090596 W [59354:70121142049500] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure retry 1 due to ResilientSocket::ConnectionFailure: Connection lost while reading data
2012-10-17 19:45:49.091493 T [59354:70121142049500] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-17 19:45:49.091540 D [59354:70121142049500] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-17 19:45:49.092823 D [59354:70121144161840] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-17 19:45:49.092895 D [59354:70121144161840] SimpleTCPServer -- 
****************** Received request
2012-10-17 19:45:49.092906 T [59354:70121144161840] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-17 19:45:49.092921 D [59354:70121144161840] SimpleTCPServer -- Sending Reply
2012-10-17 19:45:49.092931 T [59354:70121144161840] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-17 19:45:49.093373 T [59354:70121142049500] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x16\x00\x00\x00"
2012-10-17 19:45:49.093390 D [59354:70121142049500] (1.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-17 19:45:49.093440 T [59354:70121142049500] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-17 19:45:49.093450 D [59354:70121142049500] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 18 bytes
2012-10-17 19:45:49.094593 D [59354:70121142132220] SimpleTCPServer -- Waiting for a client to connect
2012-10-17 19:45:49.096199 T [59354:70121142049500] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-17 19:45:49.096308 D [59354:70121142049500] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-17 19:45:49.096569 D [59354:70121142132220] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-17 19:45:49.096739 D [59354:70121142132220] SimpleTCPServer -- 
****************** Received request
2012-10-17 19:45:49.096756 T [59354:70121142132220] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-17 19:45:49.096774 D [59354:70121142132220] SimpleTCPServer -- Sending Reply
2012-10-17 19:45:49.096784 T [59354:70121142132220] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-17 19:45:49.096964 T [59354:70121142049500] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-17 19:45:49.096977 D [59354:70121142049500] (0.6ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-17 19:45:49.097195 T [59354:70121142049500] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-17 19:45:49.097207 D [59354:70121142049500] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-17 19:45:49.098422 D [59354:70121146153160] SimpleTCPServer -- Waiting for a client to connect
2012-10-17 19:45:49.099109 T [59354:70121142049500] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-17 19:45:49.099162 D [59354:70121142049500] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-17 19:45:49.099228 D [59354:70121146153160] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-17 19:45:49.099292 D [59354:70121146153160] SimpleTCPServer -- 
****************** Received request
2012-10-17 19:45:49.099304 T [59354:70121146153160] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-17 19:45:52.100079 W [59354:70121142049500] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-17 19:45:52.100887 D [59354:70121142049500] (3001.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:385:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:367:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:96:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:94:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-17 19:45:52.104817 D [59354:70121142646180] SimpleTCPServer -- Waiting for a client to connect
2012-10-17 19:45:52.105867 T [59354:70121142049500] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-17 19:45:52.105966 D [59354:70121142646180] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-17 19:45:52.106067 D [59354:70121142646180] SimpleTCPServer -- 
****************** Received request
2012-10-17 19:45:52.106082 T [59354:70121142646180] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-17 19:45:52.106598 D [59354:70121142049500] (0.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-17 19:45:55.107584 W [59354:70121142049500] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-17 19:45:55.107763 D [59354:70121142049500] (3001.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:385:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:367:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:84:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:82:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-17 19:45:55.110409 D [59354:70121146222080] SimpleTCPServer -- Waiting for a client to connect
2012-10-17 19:45:55.111034 T [59354:70121142049500] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-17 19:45:55.111132 D [59354:70121146222080] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-17 19:45:55.111200 D [59354:70121142049500] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-17 19:45:55.111648 D [59354:70121146222080] SimpleTCPServer -- 
****************** Received request
2012-10-17 19:45:55.111669 T [59354:70121146222080] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-17 19:45:55.111689 D [59354:70121146222080] SimpleTCPServer -- Sending Reply
2012-10-17 19:45:55.111704 T [59354:70121146222080] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-17 19:45:55.112272 T [59354:70121142049500] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-17 19:45:55.112293 D [59354:70121142049500] (1.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-17 19:45:55.112583 T [59354:70121142049500] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-17 19:45:55.112599 D [59354:70121142049500] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-17 19:45:55.113388 D [59354:70121142792720] SimpleTCPServer -- Waiting for a client to connect
2012-10-17 19:45:55.113916 W [59354:70121142049500] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-17 19:45:55.615142 W [59354:70121142049500] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-17 19:45:56.115746 W [59354:70121142049500] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-17 19:45:56.617124 W [59354:70121142049500] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-17 19:45:57.118485 W [59354:70121142049500] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-17 19:45:57.619870 W [59354:70121142049500] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-17 19:45:58.120670 W [59354:70121142049500] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-17 19:45:58.621532 W [59354:70121142049500] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-17 19:45:59.123117 W [59354:70121142049500] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-17 19:45:59.623886 W [59354:70121142049500] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-17 19:46:00.125314 E [59354:70121142049500] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-17 19:46:00.125565 I [59354:70121142049500] (5012.3ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 connection attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:568:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:546:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:539:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:270:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:268:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:268:in `find'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:268:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:231:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:120:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:120:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-17 19:46:00.126622 T [59354:70121142049500] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-17 19:46:00.126869 D [59354:70121142792720] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-17 19:46:00.126947 D [59354:70121142049500] (0.3ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-17 19:46:00.127343 D [59354:70121142792720] SimpleTCPServer -- 
****************** Received request
2012-10-17 19:46:00.127366 T [59354:70121142792720] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-17 19:46:00.127387 D [59354:70121142792720] SimpleTCPServer -- Sending Reply
2012-10-17 19:46:00.127402 T [59354:70121142792720] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-17 19:46:00.127920 T [59354:70121142049500] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-17 19:46:00.127938 D [59354:70121142049500] (1.0ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-17 19:46:00.128223 T [59354:70121142049500] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-17 19:46:00.128246 D [59354:70121142049500] (0.3ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-17 19:46:00.129611 W [59354:70121142049500] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-17 19:46:00.231172 W [59354:70121142049500] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-17 19:46:00.332015 W [59354:70121142049500] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-17 19:46:00.433589 W [59354:70121142049500] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-17 19:46:00.534981 W [59354:70121142049500] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-17 19:46:00.636227 E [59354:70121142049500] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-17 19:46:00.636353 I [59354:70121142049500] (507.4ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 connection attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:568:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:546:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:539:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:303:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:231:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-17 19:50:59.926268 D [60050:70324755972180] SimpleTCPServer -- Waiting for a client to connect
2012-10-17 19:50:59.927990 T [60050:70324754537180] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-17 19:50:59.928085 D [60050:70324755972180] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-17 19:50:59.928334 D [60050:70324754537180] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-17 19:50:59.928650 D [60050:70324755972180] SimpleTCPServer -- 
****************** Received request
2012-10-17 19:50:59.928667 T [60050:70324755972180] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-17 19:50:59.928681 D [60050:70324755972180] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-17 19:50:59.928720 D [60050:70324755972180] SimpleTCPServer -- Server closed
2012-10-17 19:50:59.928732 D [60050:70324755972180] SimpleTCPServer -- thread killed
2012-10-17 19:50:59.935414 D [60050:70324755972180] SimpleTCPServer -- Server Restarted
2012-10-17 19:50:59.935708 D [60050:70324755937160] SimpleTCPServer -- Waiting for a client to connect
2012-10-17 19:50:59.936112 T [60050:70324754537180] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-17 19:50:59.936185 W [60050:70324754537180] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-17 19:50:59.936257 D [60050:70324754537180] (7.9ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:398:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:367:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:111:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:454:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:106:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-17 19:50:59.936302 W [60050:70324754537180] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure retry 1 due to ResilientSocket::ConnectionFailure: Connection lost while reading data
2012-10-17 19:50:59.937043 T [60050:70324754537180] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-17 19:50:59.937194 D [60050:70324754537180] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-17 19:50:59.937266 D [60050:70324755937160] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-17 19:50:59.937333 D [60050:70324755937160] SimpleTCPServer -- 
****************** Received request
2012-10-17 19:50:59.937344 T [60050:70324755937160] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-17 19:50:59.937359 D [60050:70324755937160] SimpleTCPServer -- Sending Reply
2012-10-17 19:50:59.937368 T [60050:70324755937160] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-17 19:50:59.937895 T [60050:70324754537180] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x16\x00\x00\x00"
2012-10-17 19:50:59.937911 D [60050:70324754537180] (0.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-17 19:50:59.938128 T [60050:70324754537180] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-17 19:50:59.938142 D [60050:70324754537180] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 18 bytes
2012-10-17 19:50:59.939203 D [60050:70324754660800] SimpleTCPServer -- Waiting for a client to connect
2012-10-17 19:50:59.939762 T [60050:70324754537180] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-17 19:50:59.939937 D [60050:70324754537180] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-17 19:50:59.940011 D [60050:70324754660800] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-17 19:50:59.940073 D [60050:70324754660800] SimpleTCPServer -- 
****************** Received request
2012-10-17 19:50:59.940085 T [60050:70324754660800] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-17 19:50:59.940098 D [60050:70324754660800] SimpleTCPServer -- Sending Reply
2012-10-17 19:50:59.940107 T [60050:70324754660800] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-17 19:50:59.940635 T [60050:70324754537180] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-17 19:50:59.940650 D [60050:70324754537180] (0.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-17 19:50:59.940881 T [60050:70324754537180] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-17 19:50:59.940898 D [60050:70324754537180] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-17 19:50:59.941885 D [60050:70324758584280] SimpleTCPServer -- Waiting for a client to connect
2012-10-17 19:50:59.942449 T [60050:70324754537180] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-17 19:50:59.942627 D [60050:70324754537180] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-17 19:50:59.942800 D [60050:70324758584280] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-17 19:50:59.942970 D [60050:70324758584280] SimpleTCPServer -- 
****************** Received request
2012-10-17 19:50:59.942986 T [60050:70324758584280] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-17 19:51:02.943847 W [60050:70324754537180] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-17 19:51:02.943999 D [60050:70324754537180] (3001.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:385:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:367:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:96:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:94:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-17 19:51:02.945894 D [60050:70324754881640] SimpleTCPServer -- Waiting for a client to connect
2012-10-17 19:51:02.946788 T [60050:70324754537180] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-17 19:51:02.946873 D [60050:70324754537180] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-17 19:51:02.946973 D [60050:70324754881640] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-17 19:51:02.947043 D [60050:70324754881640] SimpleTCPServer -- 
****************** Received request
2012-10-17 19:51:02.947055 T [60050:70324754881640] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-17 19:51:05.947722 W [60050:70324754537180] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-17 19:51:05.947942 D [60050:70324754537180] (3001.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:385:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:367:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:84:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:82:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-17 19:51:05.950339 T [60050:70324754537180] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-17 19:51:05.950590 D [60050:70324754537180] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-17 19:51:05.950680 D [60050:70324754998320] SimpleTCPServer -- Waiting for a client to connect
2012-10-17 19:51:05.950731 D [60050:70324754998320] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-17 19:51:05.951129 D [60050:70324754998320] SimpleTCPServer -- 
****************** Received request
2012-10-17 19:51:05.951151 T [60050:70324754998320] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-17 19:51:05.951171 D [60050:70324754998320] SimpleTCPServer -- Sending Reply
2012-10-17 19:51:05.951187 T [60050:70324754998320] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-17 19:51:05.951747 T [60050:70324754537180] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-17 19:51:05.951767 D [60050:70324754537180] (1.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-17 19:51:05.952060 T [60050:70324754537180] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-17 19:51:05.952079 D [60050:70324754537180] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-17 19:51:05.953674 W [60050:70324754537180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-17 19:51:05.953860 D [60050:70324758682380] SimpleTCPServer -- Waiting for a client to connect
2012-10-17 19:51:06.454697 W [60050:70324754537180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-17 19:51:06.956132 W [60050:70324754537180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-17 19:51:07.457843 W [60050:70324754537180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-17 19:51:07.959737 W [60050:70324754537180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-17 19:51:08.461259 W [60050:70324754537180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-17 19:51:08.963110 W [60050:70324754537180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-17 19:51:09.464720 W [60050:70324754537180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-17 19:51:09.966201 W [60050:70324754537180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-17 19:51:10.467671 W [60050:70324754537180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-17 19:51:10.969257 E [60050:70324754537180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-17 19:51:10.969387 I [60050:70324754537180] (5016.8ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 connection attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:568:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:546:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:539:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:270:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:268:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:268:in `find'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:268:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:231:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:120:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:120:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-17 19:51:10.970305 T [60050:70324754537180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-17 19:51:10.970412 D [60050:70324758682380] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-17 19:51:10.970454 D [60050:70324754537180] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-17 19:51:10.970914 D [60050:70324758682380] SimpleTCPServer -- 
****************** Received request
2012-10-17 19:51:10.970935 T [60050:70324758682380] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-17 19:51:10.970957 D [60050:70324758682380] SimpleTCPServer -- Sending Reply
2012-10-17 19:51:10.970972 T [60050:70324758682380] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-17 19:51:10.971087 T [60050:70324754537180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-17 19:51:10.971104 D [60050:70324754537180] (0.6ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-17 19:51:10.971167 T [60050:70324754537180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-17 19:51:10.971181 D [60050:70324754537180] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-17 19:51:10.973218 W [60050:70324754537180] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-17 19:51:11.074978 W [60050:70324754537180] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-17 19:51:11.176026 W [60050:70324754537180] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-17 19:51:11.276844 W [60050:70324754537180] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-17 19:51:11.377920 W [60050:70324754537180] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-17 19:51:11.479144 E [60050:70324754537180] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-17 19:51:11.479290 I [60050:70324754537180] (507.1ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 connection attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:568:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:546:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:539:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:303:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:231:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:43:59.029688 D [64578:70253071871000] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:43:59.030796 D [64578:70253071871000] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:43:59.031256 T [64578:70253069687520] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-18 08:43:59.031621 D [64578:70253069687520] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-18 08:43:59.031871 D [64578:70253071871000] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:43:59.031884 T [64578:70253071871000] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-18 08:43:59.031900 D [64578:70253071871000] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-18 08:43:59.032004 D [64578:70253071871000] SimpleTCPServer -- Server closed
2012-10-18 08:43:59.032016 D [64578:70253071871000] SimpleTCPServer -- thread killed
2012-10-18 08:43:59.038690 D [64578:70253071871000] SimpleTCPServer -- Server Restarted
2012-10-18 08:43:59.038836 D [64578:70253071841020] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:43:59.038869 T [64578:70253069687520] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-18 08:43:59.038935 W [64578:70253069687520] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-18 08:43:59.039013 D [64578:70253069687520] (7.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:398:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:367:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:121:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:454:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:116:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:43:59.039060 W [64578:70253069687520] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure retry 1 due to ResilientSocket::ConnectionFailure: Connection lost while reading data
2012-10-18 08:43:59.039720 T [64578:70253069687520] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-18 08:43:59.039772 D [64578:70253069687520] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-18 08:43:59.039837 D [64578:70253071841020] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:43:59.039902 D [64578:70253071841020] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:43:59.039913 T [64578:70253071841020] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-18 08:43:59.039928 D [64578:70253071841020] SimpleTCPServer -- Sending Reply
2012-10-18 08:43:59.039939 T [64578:70253071841020] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-18 08:43:59.040061 T [64578:70253069687520] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x16\x00\x00\x00"
2012-10-18 08:43:59.040073 D [64578:70253069687520] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-18 08:43:59.040118 T [64578:70253069687520] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-18 08:43:59.040128 D [64578:70253069687520] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 18 bytes
2012-10-18 08:43:59.040943 D [64578:70253071923820] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:43:59.041285 D [64578:70253071923820] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:43:59.041438 T [64578:70253069687520] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-18 08:43:59.041516 D [64578:70253069687520] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-18 08:43:59.041622 D [64578:70253071923820] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:43:59.041634 T [64578:70253071923820] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-18 08:43:59.041648 D [64578:70253071923820] SimpleTCPServer -- Sending Reply
2012-10-18 08:43:59.041657 T [64578:70253071923820] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-18 08:43:59.043627 T [64578:70253069687520] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-18 08:43:59.043644 D [64578:70253069687520] (2.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-18 08:43:59.043697 T [64578:70253069687520] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-18 08:43:59.043707 D [64578:70253069687520] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-18 08:43:59.048426 D [64578:70253070280260] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:43:59.049262 D [64578:70253070280260] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:43:59.049548 T [64578:70253069687520] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-18 08:43:59.049885 D [64578:70253070280260] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:43:59.049903 T [64578:70253070280260] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-18 08:43:59.050121 D [64578:70253069687520] (0.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-18 08:44:02.051525 W [64578:70253069687520] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-18 08:44:02.051690 D [64578:70253069687520] (3001.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:385:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:367:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:108:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:106:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:44:02.053372 D [64578:70253070364140] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:44:02.054093 T [64578:70253069687520] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-18 08:44:02.054170 D [64578:70253069687520] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-18 08:44:02.054246 D [64578:70253070364140] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:44:02.054335 D [64578:70253070364140] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:44:02.054351 T [64578:70253070364140] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-18 08:44:02.054370 D [64578:70253070364140] SimpleTCPServer -- Sending Reply
2012-10-18 08:44:02.054383 T [64578:70253070364140] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-18 08:44:02.055189 T [64578:70253069687520] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-18 08:44:02.055210 D [64578:70253069687520] (1.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-18 08:44:02.055498 T [64578:70253069687520] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-18 08:44:02.055517 D [64578:70253069687520] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-18 08:44:02.056356 D [64578:70253070434580] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:44:02.056697 W [64578:70253069687520] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-18 08:44:02.558165 W [64578:70253069687520] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-18 08:44:03.059245 W [64578:70253069687520] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-18 08:44:03.560787 W [64578:70253069687520] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-18 08:44:04.062318 W [64578:70253069687520] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-18 08:44:04.563063 W [64578:70253069687520] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-18 08:44:05.064530 W [64578:70253069687520] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-18 08:44:05.565753 W [64578:70253069687520] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-18 08:44:06.067273 W [64578:70253069687520] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-18 08:44:06.568310 W [64578:70253069687520] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-18 08:44:07.069929 E [64578:70253069687520] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-18 08:44:07.070063 I [64578:70253069687520] (5014.0ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 connection attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:568:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:546:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:539:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:270:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:268:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:268:in `find'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:268:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:231:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:130:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:130:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:44:07.070971 T [64578:70253069687520] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-18 08:44:07.071106 D [64578:70253069687520] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-18 08:44:07.071214 D [64578:70253070434580] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:44:07.071305 D [64578:70253070434580] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:44:07.071320 T [64578:70253070434580] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-18 08:44:07.071340 D [64578:70253070434580] SimpleTCPServer -- Sending Reply
2012-10-18 08:44:07.071352 T [64578:70253070434580] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-18 08:44:07.072165 T [64578:70253069687520] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-18 08:44:07.072187 D [64578:70253069687520] (1.0ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-18 08:44:07.072370 T [64578:70253069687520] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-18 08:44:07.072385 D [64578:70253069687520] (0.2ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-18 08:44:07.073583 D [64578:70253070852220] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:44:07.075290 W [64578:70253069687520] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-18 08:44:07.176666 W [64578:70253069687520] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-18 08:44:07.277678 W [64578:70253069687520] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-18 08:44:07.379158 W [64578:70253069687520] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-18 08:44:07.484569 W [64578:70253069687520] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-18 08:44:07.586172 E [64578:70253069687520] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-18 08:44:07.586285 I [64578:70253069687520] (511.5ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 connection attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:568:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:546:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:539:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:303:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:231:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:44:51.888537 D [64702:70094429670960] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:44:51.889809 T [64702:70094428527320] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-18 08:44:51.889892 D [64702:70094428527320] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-18 08:44:51.890398 D [64702:70094429670960] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:44:51.890636 D [64702:70094429670960] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:44:51.890652 T [64702:70094429670960] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-18 08:44:51.890667 D [64702:70094429670960] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-18 08:44:51.890709 D [64702:70094429670960] SimpleTCPServer -- Server closed
2012-10-18 08:44:51.890720 D [64702:70094429670960] SimpleTCPServer -- thread killed
2012-10-18 08:44:51.890769 T [64702:70094428527320] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-18 08:44:51.890808 W [64702:70094428527320] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-18 08:44:51.890879 D [64702:70094428527320] (0.9ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:398:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:367:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:122:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:454:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:117:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:44:51.890921 W [64702:70094428527320] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure retry 1 due to ResilientSocket::ConnectionFailure: Connection lost while reading data
2012-10-18 08:44:51.897022 D [64702:70094429670960] SimpleTCPServer -- Server Restarted
2012-10-18 08:44:51.897298 D [64702:70094429607140] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:44:51.897869 T [64702:70094428527320] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-18 08:44:51.897921 D [64702:70094428527320] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-18 08:44:51.897993 D [64702:70094429607140] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:44:51.898255 D [64702:70094429607140] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:44:51.898267 T [64702:70094429607140] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-18 08:44:51.898282 D [64702:70094429607140] SimpleTCPServer -- Sending Reply
2012-10-18 08:44:51.898291 T [64702:70094429607140] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-18 08:44:51.898655 T [64702:70094428527320] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x16\x00\x00\x00"
2012-10-18 08:44:51.898668 D [64702:70094428527320] (0.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-18 08:44:51.898716 T [64702:70094428527320] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-18 08:44:51.898726 D [64702:70094428527320] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 18 bytes
2012-10-18 08:44:51.900116 D [64702:70094428481400] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:44:51.900645 T [64702:70094428527320] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-18 08:44:51.900698 D [64702:70094428527320] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-18 08:44:51.900785 D [64702:70094428481400] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:44:51.901021 D [64702:70094428481400] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:44:51.901035 T [64702:70094428481400] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-18 08:44:51.901049 D [64702:70094428481400] SimpleTCPServer -- Sending Reply
2012-10-18 08:44:51.901058 T [64702:70094428481400] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-18 08:44:51.901764 T [64702:70094428527320] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-18 08:44:51.901781 D [64702:70094428527320] (1.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-18 08:44:51.901918 T [64702:70094428527320] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-18 08:44:51.901929 D [64702:70094428527320] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-18 08:44:51.902703 D [64702:70094432803760] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:44:51.903410 T [64702:70094428527320] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-18 08:44:51.903596 D [64702:70094428527320] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-18 08:44:51.903670 D [64702:70094432803760] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:44:51.903742 D [64702:70094432803760] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:44:51.903754 T [64702:70094432803760] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-18 08:44:54.904813 W [64702:70094428527320] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-18 08:44:54.904983 D [64702:70094428527320] (3001.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:385:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:367:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:109:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:107:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:44:54.906964 D [64702:70094433114700] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:44:54.907727 T [64702:70094428527320] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-18 08:44:54.908037 D [64702:70094428527320] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-18 08:44:54.908254 D [64702:70094433114700] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:44:54.908351 D [64702:70094433114700] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:44:54.908366 T [64702:70094433114700] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-18 08:44:54.908386 D [64702:70094433114700] SimpleTCPServer -- Sending Reply
2012-10-18 08:44:54.908398 T [64702:70094433114700] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-18 08:44:54.909000 T [64702:70094428527320] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-18 08:44:54.909021 D [64702:70094428527320] (0.9ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-18 08:44:54.909307 T [64702:70094428527320] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-18 08:44:54.909332 D [64702:70094428527320] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-18 08:44:54.910187 D [64702:70094433181180] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:44:54.910480 W [64702:70094428527320] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-18 08:44:55.412281 W [64702:70094428527320] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-18 08:44:55.914163 W [64702:70094428527320] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-18 08:44:56.416016 W [64702:70094428527320] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-18 08:44:56.916835 W [64702:70094428527320] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-18 08:44:57.418314 W [64702:70094428527320] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-18 08:44:57.920110 W [64702:70094428527320] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-18 08:44:58.421643 W [64702:70094428527320] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-18 08:44:58.923530 W [64702:70094428527320] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-18 08:44:59.424842 W [64702:70094428527320] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-18 08:44:59.926729 E [64702:70094428527320] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-18 08:44:59.926969 I [64702:70094428527320] (5017.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 connection attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:568:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:546:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:539:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:270:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:268:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:268:in `find'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:268:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:231:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:131:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:131:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:44:59.929373 T [64702:70094428527320] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-18 08:44:59.929479 D [64702:70094428527320] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-18 08:44:59.929806 D [64702:70094433181180] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:44:59.929896 D [64702:70094433181180] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:44:59.929912 T [64702:70094433181180] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-18 08:44:59.929954 D [64702:70094433181180] SimpleTCPServer -- Sending Reply
2012-10-18 08:44:59.929969 T [64702:70094433181180] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-18 08:44:59.930609 T [64702:70094428527320] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-18 08:44:59.930627 D [64702:70094428527320] (1.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-18 08:44:59.930693 T [64702:70094428527320] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-18 08:44:59.930706 D [64702:70094428527320] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-18 08:44:59.931850 D [64702:70094433490780] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:44:59.932857 T [64702:70094428527320] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-18 08:44:59.932955 D [64702:70094433490780] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:44:59.933022 D [64702:70094428527320] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-18 08:44:59.933458 D [64702:70094433490780] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:44:59.933479 T [64702:70094433490780] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-18 08:45:02.933985 W [64702:70094428527320] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-18 08:45:02.934156 D [64702:70094428527320] (3001.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:385:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:367:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:69:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:67:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:45:02.936063 W [64702:70094428527320] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-18 08:45:03.037499 W [64702:70094428527320] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-18 08:45:03.139020 W [64702:70094428527320] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-18 08:45:03.240346 W [64702:70094428527320] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-18 08:45:03.341952 W [64702:70094428527320] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-18 08:45:03.443976 E [64702:70094428527320] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-18 08:45:03.444207 I [64702:70094428527320] (509.0ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 connection attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:568:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:546:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:539:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:303:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:231:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:46:35.803716 D [65539:70305956743420] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:46:35.804809 D [65539:70305956743420] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:46:35.805458 T [65539:70305955666640] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-18 08:46:35.805958 D [65539:70305955666640] (0.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-18 08:46:35.806116 D [65539:70305956743420] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:46:35.806128 T [65539:70305956743420] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-18 08:46:35.806143 D [65539:70305956743420] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-18 08:46:35.806181 D [65539:70305956743420] SimpleTCPServer -- Server closed
2012-10-18 08:46:35.806246 D [65539:70305956743420] SimpleTCPServer -- thread killed
2012-10-18 08:46:35.812750 D [65539:70305956743420] SimpleTCPServer -- Server Restarted
2012-10-18 08:46:35.813058 D [65539:70305956685600] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:46:35.813720 T [65539:70305955666640] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-18 08:46:35.813795 W [65539:70305955666640] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-18 08:46:35.813871 D [65539:70305955666640] (7.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:398:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:367:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:122:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:454:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:117:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:46:35.813917 W [65539:70305955666640] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure retry 1 due to ResilientSocket::ConnectionFailure: Connection lost while reading data
2012-10-18 08:46:35.814605 D [65539:70305956685600] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:46:35.814821 T [65539:70305955666640] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-18 08:46:35.816593 D [65539:70305955666640] (1.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-18 08:46:35.817092 D [65539:70305956685600] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:46:35.817109 T [65539:70305956685600] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-18 08:46:35.817127 D [65539:70305956685600] SimpleTCPServer -- Sending Reply
2012-10-18 08:46:35.817136 T [65539:70305956685600] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-18 08:46:35.817564 T [65539:70305955666640] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x16\x00\x00\x00"
2012-10-18 08:46:35.817582 D [65539:70305955666640] (0.9ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-18 08:46:35.817819 T [65539:70305955666640] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-18 08:46:35.817834 D [65539:70305955666640] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 18 bytes
2012-10-18 08:46:35.819147 D [65539:70305959673700] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:46:35.819712 T [65539:70305955666640] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-18 08:46:35.819768 D [65539:70305955666640] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-18 08:46:35.819824 D [65539:70305959673700] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:46:35.819888 D [65539:70305959673700] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:46:35.819900 T [65539:70305959673700] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-18 08:46:35.819913 D [65539:70305959673700] SimpleTCPServer -- Sending Reply
2012-10-18 08:46:35.819922 T [65539:70305959673700] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-18 08:46:35.820541 T [65539:70305955666640] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-18 08:46:35.820557 D [65539:70305955666640] (0.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-18 08:46:35.820806 T [65539:70305955666640] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-18 08:46:35.820821 D [65539:70305955666640] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-18 08:46:35.821756 D [65539:70305959942640] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:46:35.822348 T [65539:70305955666640] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-18 08:46:35.822522 D [65539:70305955666640] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-18 08:46:35.822596 D [65539:70305959942640] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:46:35.822666 D [65539:70305959942640] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:46:35.822678 T [65539:70305959942640] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-18 08:46:38.823085 W [65539:70305955666640] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-18 08:46:38.823225 D [65539:70305955666640] (3000.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:385:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:367:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:109:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:107:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:46:38.825692 D [65539:70305960152640] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:46:38.826890 T [65539:70305955666640] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-18 08:46:38.826965 D [65539:70305955666640] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-18 08:46:38.827036 D [65539:70305960152640] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:46:38.827099 D [65539:70305960152640] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:46:38.827110 T [65539:70305960152640] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-18 08:46:38.827126 D [65539:70305960152640] SimpleTCPServer -- Sending Reply
2012-10-18 08:46:38.827135 T [65539:70305960152640] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-18 08:46:38.827242 T [65539:70305955666640] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-18 08:46:38.827255 D [65539:70305955666640] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-18 08:46:38.827349 T [65539:70305955666640] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-18 08:46:38.827360 D [65539:70305955666640] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-18 08:46:38.828894 D [65539:70305960216820] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:46:38.829367 W [65539:70305955666640] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-18 08:46:39.330545 W [65539:70305955666640] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-18 08:46:39.832439 W [65539:70305955666640] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-18 08:46:40.333774 W [65539:70305955666640] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-18 08:46:40.835093 W [65539:70305955666640] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-18 08:46:41.336273 W [65539:70305955666640] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-18 08:46:41.837796 W [65539:70305955666640] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-18 08:46:42.339238 W [65539:70305955666640] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-18 08:46:42.840711 W [65539:70305955666640] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-18 08:46:43.342547 W [65539:70305955666640] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-18 08:46:43.844237 E [65539:70305955666640] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-18 08:46:43.844364 I [65539:70305955666640] (5015.5ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 connection attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:568:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:546:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:539:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:270:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:268:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:268:in `find'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:268:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:231:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:131:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:131:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:46:43.845286 T [65539:70305955666640] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-18 08:46:43.845572 D [65539:70305955666640] (0.2ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-18 08:46:43.845670 D [65539:70305960216820] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:46:43.845759 D [65539:70305960216820] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:46:43.845774 T [65539:70305960216820] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-18 08:46:43.845793 D [65539:70305960216820] SimpleTCPServer -- Sending Reply
2012-10-18 08:46:43.845806 T [65539:70305960216820] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-18 08:46:43.846578 T [65539:70305955666640] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-18 08:46:43.846601 D [65539:70305955666640] (1.0ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-18 08:46:43.846665 T [65539:70305955666640] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-18 08:46:43.846678 D [65539:70305955666640] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-18 08:46:43.847975 D [65539:70305955992700] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:46:43.848722 T [65539:70305955666640] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-18 08:46:43.848820 D [65539:70305955992700] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:46:43.848860 D [65539:70305955666640] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-18 08:46:43.849313 D [65539:70305955992700] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:46:43.849334 T [65539:70305955992700] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-18 08:46:46.850132 W [65539:70305955666640] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-18 08:46:46.850436 D [65539:70305955666640] (3001.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:385:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:367:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:69:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:67:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:46:46.852706 W [65539:70305955666640] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-18 08:46:46.954200 W [65539:70305955666640] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-18 08:46:47.055639 W [65539:70305955666640] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-18 08:46:47.157184 W [65539:70305955666640] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-18 08:46:47.258502 W [65539:70305955666640] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-18 08:46:47.359987 E [65539:70305955666640] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-18 08:46:47.360112 I [65539:70305955666640] (508.3ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 connection attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:568:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:546:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:539:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:303:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:231:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:47:42.204454 D [66132:70104697124720] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:47:42.205747 T [66132:70104687794900] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-18 08:47:42.205837 D [66132:70104687794900] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-18 08:47:42.206366 D [66132:70104697124720] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:47:42.206600 D [66132:70104697124720] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:47:42.206616 T [66132:70104697124720] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-18 08:47:42.206630 D [66132:70104697124720] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-18 08:47:42.206672 D [66132:70104697124720] SimpleTCPServer -- Server closed
2012-10-18 08:47:42.206683 D [66132:70104697124720] SimpleTCPServer -- thread killed
2012-10-18 08:47:42.207100 T [66132:70104687794900] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-18 08:47:42.207170 W [66132:70104687794900] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-18 08:47:42.207241 D [66132:70104687794900] (1.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:398:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:367:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:123:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:454:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:118:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:47:42.207283 W [66132:70104687794900] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure retry 1 due to ResilientSocket::ConnectionFailure: Connection lost while reading data
2012-10-18 08:47:42.213578 D [66132:70104697124720] SimpleTCPServer -- Server Restarted
2012-10-18 08:47:42.213858 D [66132:70104688908480] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:47:42.214319 T [66132:70104687794900] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-18 08:47:42.214372 D [66132:70104687794900] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-18 08:47:42.214567 D [66132:70104688908480] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:47:42.215055 D [66132:70104688908480] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:47:42.215071 T [66132:70104688908480] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-18 08:47:42.215088 D [66132:70104688908480] SimpleTCPServer -- Sending Reply
2012-10-18 08:47:42.215099 T [66132:70104688908480] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-18 08:47:42.215479 T [66132:70104687794900] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x16\x00\x00\x00"
2012-10-18 08:47:42.215495 D [66132:70104687794900] (1.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-18 08:47:42.215940 T [66132:70104687794900] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-18 08:47:42.215956 D [66132:70104687794900] (0.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 18 bytes
2012-10-18 08:47:42.219633 D [66132:70104687717200] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:47:42.220200 T [66132:70104687794900] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-18 08:47:42.220391 D [66132:70104687794900] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-18 08:47:42.220466 D [66132:70104687717200] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:47:42.220536 D [66132:70104687717200] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:47:42.220549 T [66132:70104687717200] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-18 08:47:42.220564 D [66132:70104687717200] SimpleTCPServer -- Sending Reply
2012-10-18 08:47:42.220573 T [66132:70104687717200] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-18 08:47:42.221111 T [66132:70104687794900] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-18 08:47:42.221127 D [66132:70104687794900] (0.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-18 08:47:42.221359 T [66132:70104687794900] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-18 08:47:42.221376 D [66132:70104687794900] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-18 08:47:42.222171 D [66132:70104688124280] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:47:42.222627 T [66132:70104687794900] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-18 08:47:42.222811 D [66132:70104687794900] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-18 08:47:42.222886 D [66132:70104688124280] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:47:42.222957 D [66132:70104688124280] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:47:42.222970 T [66132:70104688124280] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-18 08:47:45.224022 W [66132:70104687794900] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-18 08:47:45.224193 D [66132:70104687794900] (3001.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:385:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:367:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:110:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:108:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:47:45.226092 D [66132:70104688254420] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:47:45.226693 T [66132:70104687794900] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-18 08:47:45.226813 D [66132:70104687794900] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-18 08:47:45.226914 D [66132:70104688254420] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:47:45.227005 D [66132:70104688254420] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:47:45.227020 T [66132:70104688254420] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-18 08:47:45.227040 D [66132:70104688254420] SimpleTCPServer -- Sending Reply
2012-10-18 08:47:45.227052 T [66132:70104688254420] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-18 08:47:45.227847 T [66132:70104687794900] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-18 08:47:45.227868 D [66132:70104687794900] (1.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-18 08:47:45.228148 T [66132:70104687794900] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-18 08:47:45.228167 D [66132:70104687794900] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-18 08:47:45.229119 D [66132:70104696113360] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:47:45.229596 W [66132:70104687794900] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-18 08:47:45.730237 W [66132:70104687794900] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-18 08:47:46.231808 W [66132:70104687794900] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-18 08:47:46.734287 W [66132:70104687794900] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-18 08:47:47.235535 W [66132:70104687794900] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-18 08:47:47.737094 W [66132:70104687794900] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-18 08:47:48.237960 W [66132:70104687794900] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-18 08:47:48.739486 W [66132:70104687794900] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-18 08:47:49.241070 W [66132:70104687794900] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-18 08:47:49.742650 W [66132:70104687794900] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-18 08:47:50.244448 E [66132:70104687794900] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-18 08:47:50.244572 I [66132:70104687794900] (5015.9ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 connection attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:568:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:546:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:539:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:270:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:268:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:268:in `find'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:268:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:231:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:132:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:132:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:47:50.245975 T [66132:70104687794900] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-18 08:47:50.246279 D [66132:70104696113360] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:47:50.246372 D [66132:70104696113360] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:47:50.246387 T [66132:70104696113360] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-18 08:47:50.246408 D [66132:70104696113360] SimpleTCPServer -- Sending Reply
2012-10-18 08:47:50.246421 T [66132:70104696113360] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-18 08:47:50.246507 D [66132:70104687794900] (0.5ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-18 08:47:50.246593 T [66132:70104687794900] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-18 08:47:50.246609 D [66132:70104687794900] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-18 08:47:50.246671 T [66132:70104687794900] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-18 08:47:50.246684 D [66132:70104687794900] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-18 08:47:50.249165 D [66132:70104696447660] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:47:50.249680 T [66132:70104687794900] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-18 08:47:50.249777 D [66132:70104696447660] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:47:50.250076 D [66132:70104687794900] (0.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-18 08:47:50.250337 D [66132:70104696447660] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:47:50.250355 T [66132:70104696447660] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-18 08:47:53.251291 W [66132:70104687794900] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-18 08:47:53.251494 D [66132:70104687794900] (3001.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:385:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:367:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:69:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:67:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:47:53.255461 W [66132:70104687794900] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-18 08:47:53.356892 W [66132:70104687794900] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-18 08:47:53.457913 W [66132:70104687794900] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-18 08:47:53.559390 W [66132:70104687794900] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-18 08:47:53.660964 W [66132:70104687794900] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-18 08:47:53.762132 E [66132:70104687794900] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-18 08:47:53.762278 I [66132:70104687794900] (508.0ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 connection attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:568:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:546:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:539:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:303:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:231:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:48:04.061883 D [66360:70236377435000] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:48:04.064448 T [66360:70236372163280] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-18 08:48:04.064683 D [66360:70236377435000] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:48:04.064722 D [66360:70236372163280] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-18 08:48:04.065081 D [66360:70236377435000] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:48:04.065097 T [66360:70236377435000] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-18 08:48:04.065113 D [66360:70236377435000] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-18 08:48:04.065154 D [66360:70236377435000] SimpleTCPServer -- Server closed
2012-10-18 08:48:04.065167 D [66360:70236377435000] SimpleTCPServer -- thread killed
2012-10-18 08:48:04.072542 D [66360:70236377435000] SimpleTCPServer -- Server Restarted
2012-10-18 08:48:04.072673 T [66360:70236372163280] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-18 08:48:04.072742 W [66360:70236372163280] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-18 08:48:04.072816 D [66360:70236372163280] (8.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:398:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:367:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:123:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:454:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:118:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:48:04.072864 W [66360:70236372163280] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure retry 1 due to ResilientSocket::ConnectionFailure: Connection lost while reading data
2012-10-18 08:48:04.073378 D [66360:70236377370260] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:48:04.073807 T [66360:70236372163280] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-18 08:48:04.073884 D [66360:70236377370260] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:48:04.073988 D [66360:70236377370260] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:48:04.074001 T [66360:70236377370260] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-18 08:48:04.074019 D [66360:70236377370260] SimpleTCPServer -- Sending Reply
2012-10-18 08:48:04.074028 T [66360:70236377370260] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-18 08:48:04.074121 D [66360:70236372163280] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-18 08:48:04.074183 T [66360:70236372163280] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x16\x00\x00\x00"
2012-10-18 08:48:04.074194 D [66360:70236372163280] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-18 08:48:04.074380 T [66360:70236372163280] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-18 08:48:04.074392 D [66360:70236372163280] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 18 bytes
2012-10-18 08:48:04.076500 D [66360:70236372088740] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:48:04.077993 T [66360:70236372163280] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-18 08:48:04.078479 D [66360:70236372163280] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-18 08:48:04.079009 D [66360:70236372088740] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:48:04.079082 D [66360:70236372088740] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:48:04.079094 T [66360:70236372088740] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-18 08:48:04.079109 D [66360:70236372088740] SimpleTCPServer -- Sending Reply
2012-10-18 08:48:04.079119 T [66360:70236372088740] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-18 08:48:04.079626 T [66360:70236372163280] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-18 08:48:04.079643 D [66360:70236372163280] (1.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-18 08:48:04.079727 T [66360:70236372163280] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-18 08:48:04.079738 D [66360:70236372163280] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-18 08:48:04.081259 D [66360:70236372490500] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:48:04.081787 T [66360:70236372163280] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-18 08:48:04.081971 D [66360:70236372163280] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-18 08:48:04.082058 D [66360:70236372490500] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:48:04.082130 D [66360:70236372490500] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:48:04.082142 T [66360:70236372490500] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-18 08:48:07.083247 W [66360:70236372163280] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-18 08:48:07.083425 D [66360:70236372163280] (3001.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:385:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:367:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:110:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:108:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:48:07.084942 D [66360:70236376298460] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:48:07.085569 T [66360:70236372163280] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-18 08:48:07.085657 D [66360:70236376298460] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:48:07.085693 D [66360:70236372163280] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-18 08:48:07.086102 D [66360:70236376298460] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:48:07.086120 T [66360:70236376298460] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-18 08:48:07.086138 D [66360:70236376298460] SimpleTCPServer -- Sending Reply
2012-10-18 08:48:07.086151 T [66360:70236376298460] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-18 08:48:07.086633 T [66360:70236372163280] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-18 08:48:07.086652 D [66360:70236372163280] (0.9ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-18 08:48:07.086890 T [66360:70236372163280] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-18 08:48:07.086906 D [66360:70236372163280] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-18 08:48:07.087622 D [66360:70236376369440] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:48:07.088146 W [66360:70236372163280] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-18 08:48:07.589541 W [66360:70236372163280] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-18 08:48:08.090986 W [66360:70236372163280] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-18 08:48:08.592819 W [66360:70236372163280] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-18 08:48:09.094149 W [66360:70236372163280] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-18 08:48:09.595739 W [66360:70236372163280] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-18 08:48:10.097496 W [66360:70236372163280] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-18 08:48:10.599053 W [66360:70236372163280] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-18 08:48:11.100590 W [66360:70236372163280] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-18 08:48:11.601546 W [66360:70236372163280] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-18 08:48:12.103355 E [66360:70236372163280] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-18 08:48:12.103624 I [66360:70236372163280] (5016.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 connection attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:568:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:546:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:539:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:270:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:268:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:268:in `find'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:268:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:231:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:132:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:132:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:48:12.104960 T [66360:70236372163280] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-18 08:48:12.105060 D [66360:70236372163280] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-18 08:48:12.105200 D [66360:70236376369440] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:48:12.105285 D [66360:70236376369440] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:48:12.105300 T [66360:70236376369440] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-18 08:48:12.105319 D [66360:70236376369440] SimpleTCPServer -- Sending Reply
2012-10-18 08:48:12.105331 T [66360:70236376369440] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-18 08:48:12.105476 T [66360:70236372163280] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-18 08:48:12.105493 D [66360:70236372163280] (0.4ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-18 08:48:12.105555 T [66360:70236372163280] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-18 08:48:12.105568 D [66360:70236372163280] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-18 08:48:12.107917 D [66360:70236372721420] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:48:12.108717 T [66360:70236372163280] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-18 08:48:12.108843 D [66360:70236372163280] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-18 08:48:12.108958 D [66360:70236372721420] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:48:12.109054 D [66360:70236372721420] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:48:12.109070 T [66360:70236372721420] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-18 08:48:15.109205 W [66360:70236372163280] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-18 08:48:15.109371 D [66360:70236372163280] (3000.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:385:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:367:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:69:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:67:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:48:15.111431 W [66360:70236372163280] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-18 08:48:15.212954 W [66360:70236372163280] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-18 08:48:15.313786 W [66360:70236372163280] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-18 08:48:15.415372 W [66360:70236372163280] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-18 08:48:15.517012 W [66360:70236372163280] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-18 08:48:15.618286 E [66360:70236372163280] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-18 08:48:15.618519 I [66360:70236372163280] (507.8ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 connection attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:568:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:546:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:539:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:303:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:231:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:48:51.283718 D [66633:70106379145140] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:48:51.285122 D [66633:70106379145140] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:48:51.285558 T [66633:70106369710800] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-18 08:48:51.285774 D [66633:70106369710800] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-18 08:48:51.285897 D [66633:70106379145140] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:48:51.285909 T [66633:70106379145140] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-18 08:48:51.285923 D [66633:70106379145140] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-18 08:48:51.286011 D [66633:70106379145140] SimpleTCPServer -- Server closed
2012-10-18 08:48:51.286022 D [66633:70106379145140] SimpleTCPServer -- thread killed
2012-10-18 08:48:51.292077 D [66633:70106379145140] SimpleTCPServer -- Server Restarted
2012-10-18 08:48:51.292381 D [66633:70106379088140] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:48:51.292948 T [66633:70106369710800] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-18 08:48:51.293023 W [66633:70106369710800] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-18 08:48:51.293095 D [66633:70106369710800] (7.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:399:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:368:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:123:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:455:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:118:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:48:51.293140 W [66633:70106369710800] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure retry 1 due to ResilientSocket::ConnectionFailure: Connection lost while reading data
2012-10-18 08:48:51.293930 T [66633:70106369710800] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-18 08:48:51.294081 D [66633:70106379088140] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:48:51.294216 D [66633:70106369710800] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-18 08:48:51.294392 D [66633:70106379088140] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:48:51.294407 T [66633:70106379088140] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-18 08:48:51.294423 D [66633:70106379088140] SimpleTCPServer -- Sending Reply
2012-10-18 08:48:51.294434 T [66633:70106379088140] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-18 08:48:51.294610 T [66633:70106369710800] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x16\x00\x00\x00"
2012-10-18 08:48:51.294623 D [66633:70106369710800] (0.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-18 08:48:51.294672 T [66633:70106369710800] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-18 08:48:51.294682 D [66633:70106369710800] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 18 bytes
2012-10-18 08:48:51.295836 D [66633:70106369883160] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:48:51.297407 T [66633:70106369710800] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-18 08:48:51.297605 D [66633:70106369883160] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:48:51.297750 D [66633:70106369710800] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-18 08:48:51.297841 D [66633:70106369883160] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:48:51.297853 T [66633:70106369883160] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-18 08:48:51.297867 D [66633:70106369883160] SimpleTCPServer -- Sending Reply
2012-10-18 08:48:51.297877 T [66633:70106369883160] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-18 08:48:51.298343 T [66633:70106369710800] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-18 08:48:51.298359 D [66633:70106369710800] (0.6ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-18 08:48:51.298582 T [66633:70106369710800] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-18 08:48:51.298596 D [66633:70106369710800] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-18 08:48:51.299282 D [66633:70106370126800] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:48:51.299885 T [66633:70106369710800] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-18 08:48:51.300079 D [66633:70106369710800] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-18 08:48:51.300260 D [66633:70106370126800] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:48:51.300411 D [66633:70106370126800] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:48:51.300427 T [66633:70106370126800] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-18 08:48:54.301436 W [66633:70106369710800] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-18 08:48:54.301608 D [66633:70106369710800] (3001.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:386:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:368:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:110:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:108:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:48:54.303639 D [66633:70106370292080] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:48:54.304651 T [66633:70106369710800] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-18 08:48:54.304851 D [66633:70106369710800] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-18 08:48:54.305074 D [66633:70106370292080] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:48:54.305328 D [66633:70106370292080] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:48:54.305350 T [66633:70106370292080] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-18 08:48:54.305371 D [66633:70106370292080] SimpleTCPServer -- Sending Reply
2012-10-18 08:48:54.305384 T [66633:70106370292080] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-18 08:48:54.305505 T [66633:70106369710800] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-18 08:48:54.305520 D [66633:70106369710800] (0.6ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-18 08:48:54.305584 T [66633:70106369710800] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-18 08:48:54.305596 D [66633:70106369710800] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-18 08:48:54.307070 D [66633:70106370392120] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:48:54.307807 W [66633:70106369710800] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-18 08:48:54.808909 W [66633:70106369710800] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-18 08:48:55.310358 W [66633:70106369710800] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-18 08:48:55.811291 W [66633:70106369710800] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-18 08:48:56.311997 W [66633:70106369710800] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-18 08:48:56.813460 W [66633:70106369710800] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-18 08:48:57.314919 W [66633:70106369710800] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-18 08:48:57.816406 W [66633:70106369710800] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-18 08:48:58.317962 W [66633:70106369710800] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-18 08:48:58.819405 W [66633:70106369710800] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-18 08:48:59.320951 E [66633:70106369710800] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-18 08:48:59.321063 I [66633:70106369710800] (5014.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 connection attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:569:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:547:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:540:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:271:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:269:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:269:in `find'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:269:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:232:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:132:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:132:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:48:59.322213 T [66633:70106369710800] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-18 08:48:59.322298 D [66633:70106369710800] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-18 08:48:59.322408 D [66633:70106370392120] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:48:59.322472 D [66633:70106370392120] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:48:59.322484 T [66633:70106370392120] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-18 08:48:59.322499 D [66633:70106370392120] SimpleTCPServer -- Sending Reply
2012-10-18 08:48:59.322508 T [66633:70106370392120] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-18 08:48:59.323119 T [66633:70106369710800] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-18 08:48:59.323135 D [66633:70106369710800] (0.8ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-18 08:48:59.323183 T [66633:70106369710800] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-18 08:48:59.323193 D [66633:70106369710800] (0.0ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-18 08:48:59.324390 D [66633:70106370701580] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:48:59.325066 T [66633:70106369710800] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-18 08:48:59.325142 D [66633:70106369710800] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-18 08:48:59.325238 D [66633:70106370701580] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:48:59.325303 D [66633:70106370701580] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:48:59.325314 T [66633:70106370701580] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-18 08:49:02.325481 W [66633:70106369710800] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-18 08:49:02.325796 D [66633:70106369710800] (3000.6ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:386:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:368:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:69:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:67:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:49:02.828692 D [66633:70106370701580] SimpleTCPServer -- Sending Reply
2012-10-18 08:49:02.828736 T [66633:70106370701580] SimpleTCPServer -- Reply -- {"result"=>"sleep"}
2012-10-18 08:49:02.836938 T [66633:70106369710800] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-18 08:49:02.836963 D [66633:70106369710800] (510.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-18 08:49:02.837026 T [66633:70106369710800] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00sleep\x00\x00"
2012-10-18 08:49:02.837036 D [66633:70106369710800] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-18 08:49:02.845207 W [66633:70106369710800] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-18 08:49:02.946301 W [66633:70106369710800] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-18 08:49:03.047821 W [66633:70106369710800] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-18 08:49:03.148957 W [66633:70106369710800] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-18 08:49:03.250395 W [66633:70106369710800] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-18 08:49:03.351568 E [66633:70106369710800] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-18 08:49:03.351671 I [66633:70106369710800] (507.7ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 connection attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:569:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:547:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:540:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:304:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:232:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:50:01.049431 D [66723:70170602509900] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:50:01.051623 T [66723:70170601282260] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-18 08:50:01.051827 D [66723:70170601282260] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-18 08:50:01.052030 D [66723:70170602509900] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:50:01.052258 D [66723:70170602509900] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:50:01.052273 T [66723:70170602509900] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-18 08:50:01.052288 D [66723:70170602509900] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-18 08:50:01.052329 D [66723:70170602509900] SimpleTCPServer -- Server closed
2012-10-18 08:50:01.052340 D [66723:70170602509900] SimpleTCPServer -- thread killed
2012-10-18 08:50:01.059150 D [66723:70170602509900] SimpleTCPServer -- Server Restarted
2012-10-18 08:50:01.059471 D [66723:70170602411460] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:50:01.059602 T [66723:70170601282260] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-18 08:50:01.059733 W [66723:70170601282260] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-18 08:50:01.059810 D [66723:70170601282260] (7.9ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:399:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:368:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:124:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:455:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:119:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:50:01.059858 W [66723:70170601282260] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure retry 1 due to ResilientSocket::ConnectionFailure: Connection lost while reading data
2012-10-18 08:50:01.062553 D [66723:70170602411460] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:50:01.062723 T [66723:70170601282260] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-18 08:50:01.062777 D [66723:70170601282260] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-18 08:50:01.062917 D [66723:70170602411460] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:50:01.062930 T [66723:70170602411460] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-18 08:50:01.062946 D [66723:70170602411460] SimpleTCPServer -- Sending Reply
2012-10-18 08:50:01.062955 T [66723:70170602411460] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-18 08:50:01.063611 T [66723:70170601282260] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x16\x00\x00\x00"
2012-10-18 08:50:01.063641 D [66723:70170601282260] (0.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-18 08:50:01.063870 T [66723:70170601282260] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-18 08:50:01.063884 D [66723:70170601282260] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 18 bytes
2012-10-18 08:50:01.064698 D [66723:70170601208120] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:50:01.065419 T [66723:70170601282260] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-18 08:50:01.065595 D [66723:70170601282260] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-18 08:50:01.065671 D [66723:70170601208120] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:50:01.065733 D [66723:70170601208120] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:50:01.065745 T [66723:70170601208120] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-18 08:50:01.065759 D [66723:70170601208120] SimpleTCPServer -- Sending Reply
2012-10-18 08:50:01.065768 T [66723:70170601208120] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-18 08:50:01.066098 T [66723:70170601282260] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-18 08:50:01.066111 D [66723:70170601282260] (0.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-18 08:50:01.066584 T [66723:70170601282260] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-18 08:50:01.066616 D [66723:70170601282260] (0.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-18 08:50:01.067510 D [66723:70170605538900] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:50:01.068212 T [66723:70170601282260] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-18 08:50:01.068387 D [66723:70170601282260] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-18 08:50:01.068562 D [66723:70170605538900] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:50:01.068736 D [66723:70170605538900] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:50:01.068752 T [66723:70170605538900] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-18 08:50:04.069525 W [66723:70170601282260] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-18 08:50:04.069660 D [66723:70170601282260] (3001.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:386:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:368:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:111:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:109:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:50:04.071331 D [66723:70170605877500] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:50:04.072173 T [66723:70170601282260] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-18 08:50:04.072250 D [66723:70170601282260] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-18 08:50:04.072344 D [66723:70170605877500] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:50:04.072406 D [66723:70170605877500] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:50:04.072417 T [66723:70170605877500] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-18 08:50:04.072432 D [66723:70170605877500] SimpleTCPServer -- Sending Reply
2012-10-18 08:50:04.072442 T [66723:70170605877500] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-18 08:50:04.073686 T [66723:70170601282260] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-18 08:50:04.073704 D [66723:70170601282260] (1.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-18 08:50:04.073753 T [66723:70170601282260] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-18 08:50:04.073762 D [66723:70170601282260] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-18 08:50:04.074966 D [66723:70170605942860] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:50:04.075701 W [66723:70170601282260] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-18 08:50:04.577167 W [66723:70170601282260] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-18 08:50:05.078942 W [66723:70170601282260] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-18 08:50:05.580570 W [66723:70170601282260] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-18 08:50:06.081568 W [66723:70170601282260] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-18 08:50:06.582275 W [66723:70170601282260] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-18 08:50:07.083166 W [66723:70170601282260] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-18 08:50:07.584658 W [66723:70170601282260] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-18 08:50:08.086104 W [66723:70170601282260] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-18 08:50:08.587934 W [66723:70170601282260] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-18 08:50:09.088597 E [66723:70170601282260] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-18 08:50:09.088723 I [66723:70170601282260] (5013.9ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 connection attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:569:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:547:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:540:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:271:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:269:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:269:in `find'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:269:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:232:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:133:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:133:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:50:09.089613 T [66723:70170601282260] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-18 08:50:09.089688 D [66723:70170601282260] (0.0ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-18 08:50:09.089769 D [66723:70170605942860] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:50:09.090053 D [66723:70170605942860] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:50:09.090067 T [66723:70170605942860] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-18 08:50:09.090084 D [66723:70170605942860] SimpleTCPServer -- Sending Reply
2012-10-18 08:50:09.090096 T [66723:70170605942860] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-18 08:50:09.090737 T [66723:70170601282260] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-18 08:50:09.090753 D [66723:70170601282260] (1.0ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-18 08:50:09.090811 T [66723:70170601282260] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-18 08:50:09.090823 D [66723:70170601282260] (0.0ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-18 08:50:09.092425 D [66723:70170606257080] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:50:09.093451 T [66723:70170601282260] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-18 08:50:09.093536 D [66723:70170601282260] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-18 08:50:09.093640 D [66723:70170606257080] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:50:09.093710 D [66723:70170606257080] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:50:09.093723 T [66723:70170606257080] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-18 08:50:12.094649 W [66723:70170601282260] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-18 08:50:12.094844 D [66723:70170601282260] (3001.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:386:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:368:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:69:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:67:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:50:12.594354 D [66723:70170606257080] SimpleTCPServer -- Sending Reply
2012-10-18 08:50:12.594426 T [66723:70170606257080] SimpleTCPServer -- Reply -- {"result"=>"sleep"}
2012-10-18 08:50:12.595178 T [66723:70170601282260] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-18 08:50:12.595208 D [66723:70170601282260] (500.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-18 08:50:12.595512 T [66723:70170601282260] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00sleep\x00\x00"
2012-10-18 08:50:12.595531 D [66723:70170601282260] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-18 08:50:12.597331 W [66723:70170601282260] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-18 08:50:12.698897 W [66723:70170601282260] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-18 08:50:12.799955 W [66723:70170601282260] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-18 08:50:12.901031 W [66723:70170601282260] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-18 08:50:13.001650 W [66723:70170601282260] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-18 08:50:13.103510 E [66723:70170601282260] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-18 08:50:13.103776 I [66723:70170601282260] (507.5ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 connection attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:569:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:547:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:540:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:304:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:232:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:51:40.756755 D [68105:70337837182460] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:51:40.758380 T [68105:70337832377040] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-18 08:51:40.758447 D [68105:70337832377040] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-18 08:51:40.758786 D [68105:70337837182460] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:51:40.758871 D [68105:70337837182460] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:51:40.758932 T [68105:70337837182460] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-18 08:51:40.758948 D [68105:70337837182460] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-18 08:51:40.759027 D [68105:70337837182460] SimpleTCPServer -- Server closed
2012-10-18 08:51:40.759039 D [68105:70337837182460] SimpleTCPServer -- thread killed
2012-10-18 08:51:40.765021 D [68105:70337837182460] SimpleTCPServer -- Server Restarted
2012-10-18 08:51:40.765168 D [68105:70337837128320] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:51:40.765200 T [68105:70337832377040] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-18 08:51:40.765266 W [68105:70337832377040] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-18 08:51:40.765340 D [68105:70337832377040] (6.9ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:399:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:368:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:126:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:455:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:121:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:51:40.765387 W [68105:70337832377040] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure retry 1 due to ResilientSocket::ConnectionFailure: Connection lost while reading data
2012-10-18 08:51:40.766596 D [68105:70337837128320] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:51:40.766721 T [68105:70337832377040] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-18 08:51:40.766771 D [68105:70337832377040] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-18 08:51:40.766859 D [68105:70337837128320] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:51:40.766871 T [68105:70337837128320] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-18 08:51:40.766885 D [68105:70337837128320] SimpleTCPServer -- Sending Reply
2012-10-18 08:51:40.766901 T [68105:70337837128320] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-18 08:51:40.767143 T [68105:70337832377040] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x16\x00\x00\x00"
2012-10-18 08:51:40.767156 D [68105:70337832377040] (0.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-18 08:51:40.767204 T [68105:70337832377040] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-18 08:51:40.767214 D [68105:70337832377040] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 18 bytes
2012-10-18 08:51:40.768258 D [68105:70337832285060] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:51:40.769581 D [68105:70337832285060] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:51:40.769788 T [68105:70337832377040] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-18 08:51:40.769841 D [68105:70337832377040] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-18 08:51:40.770221 D [68105:70337832285060] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:51:40.770238 T [68105:70337832285060] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-18 08:51:40.770252 D [68105:70337832285060] SimpleTCPServer -- Sending Reply
2012-10-18 08:51:40.770263 T [68105:70337832285060] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-18 08:51:40.770662 T [68105:70337832377040] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-18 08:51:40.770677 D [68105:70337832377040] (0.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-18 08:51:40.770904 T [68105:70337832377040] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-18 08:51:40.770920 D [68105:70337832377040] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-18 08:51:40.771763 D [68105:70337836715880] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:51:40.772289 T [68105:70337832377040] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-18 08:51:40.772472 D [68105:70337832377040] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-18 08:51:40.772546 D [68105:70337836715880] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:51:40.772616 D [68105:70337836715880] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:51:40.772628 T [68105:70337836715880] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-18 08:51:43.773721 W [68105:70337832377040] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-18 08:51:43.773892 D [68105:70337832377040] (3001.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:386:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:368:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:111:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:109:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:51:43.775276 D [68105:70337832668540] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:51:43.776196 T [68105:70337832377040] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-18 08:51:43.776297 D [68105:70337832668540] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:51:43.776582 D [68105:70337832377040] (0.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-18 08:51:43.776801 D [68105:70337832668540] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:51:43.776821 T [68105:70337832668540] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-18 08:51:43.776840 D [68105:70337832668540] SimpleTCPServer -- Sending Reply
2012-10-18 08:51:43.776853 T [68105:70337832668540] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-18 08:51:43.777420 T [68105:70337832377040] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-18 08:51:43.777440 D [68105:70337832377040] (0.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-18 08:51:43.777744 T [68105:70337832377040] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-18 08:51:43.777765 D [68105:70337832377040] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-18 08:51:43.778515 D [68105:70337832750080] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:51:43.779749 W [68105:70337832377040] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-18 08:51:44.281306 W [68105:70337832377040] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-18 08:51:44.782311 W [68105:70337832377040] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-18 08:51:45.283180 W [68105:70337832377040] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-18 08:51:45.784806 W [68105:70337832377040] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-18 08:51:46.286289 W [68105:70337832377040] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-18 08:51:46.787780 W [68105:70337832377040] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-18 08:51:47.288504 W [68105:70337832377040] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-18 08:51:47.790359 W [68105:70337832377040] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-18 08:51:48.292239 W [68105:70337832377040] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-18 08:51:48.793361 E [68105:70337832377040] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-18 08:51:48.793497 I [68105:70337832377040] (5015.2ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 connection attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:569:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:547:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:540:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:271:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:269:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:269:in `find'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:269:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:232:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:135:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:135:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:51:48.794279 T [68105:70337832377040] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-18 08:51:48.794340 D [68105:70337832377040] (0.0ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-18 08:51:48.794402 D [68105:70337832750080] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:51:48.794473 D [68105:70337832750080] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:51:48.794485 T [68105:70337832750080] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-18 08:51:48.794500 D [68105:70337832750080] SimpleTCPServer -- Sending Reply
2012-10-18 08:51:48.794509 T [68105:70337832750080] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-18 08:51:48.794691 T [68105:70337832377040] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-18 08:51:48.794704 D [68105:70337832377040] (0.3ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-18 08:51:48.794755 T [68105:70337832377040] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-18 08:51:48.794765 D [68105:70337832377040] (0.0ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-18 08:51:48.795679 D [68105:70337833033740] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 08:51:48.796841 T [68105:70337832377040] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-18 08:51:48.796992 D [68105:70337832377040] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-18 08:51:48.797064 D [68105:70337833033740] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 08:51:48.797131 D [68105:70337833033740] SimpleTCPServer -- 
****************** Received request
2012-10-18 08:51:48.797142 T [68105:70337833033740] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-18 08:51:51.797646 W [68105:70337832377040] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-18 08:51:51.797955 D [68105:70337832377040] (3000.9ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:386:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:368:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:69:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:67:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 08:51:52.298419 D [68105:70337833033740] SimpleTCPServer -- Sending Reply
2012-10-18 08:51:52.298515 T [68105:70337833033740] SimpleTCPServer -- Reply -- {"result"=>"sleep"}
2012-10-18 08:51:52.299475 T [68105:70337832377040] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-18 08:51:52.299521 D [68105:70337832377040] (500.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-18 08:51:52.300138 T [68105:70337832377040] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00sleep\x00\x00"
2012-10-18 08:51:52.300176 D [68105:70337832377040] (0.6ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-18 08:51:52.303033 W [68105:70337832377040] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-18 08:51:52.404724 W [68105:70337832377040] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-18 08:51:52.506281 W [68105:70337832377040] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-18 08:51:52.607779 W [68105:70337832377040] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-18 08:51:52.709360 W [68105:70337832377040] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-18 08:51:52.811199 E [68105:70337832377040] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-18 08:51:52.811436 I [68105:70337832377040] (509.8ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 connection attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:569:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:547:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:540:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:304:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:232:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 09:01:19.583056 D [68565:70112398237920] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 09:01:19.584386 D [68565:70112398237920] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 09:01:19.584769 T [68565:70112396925660] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-18 09:01:19.585058 D [68565:70112398237920] SimpleTCPServer -- 
****************** Received request
2012-10-18 09:01:19.585074 T [68565:70112398237920] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-18 09:01:19.585089 D [68565:70112398237920] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-18 09:01:19.585218 D [68565:70112398237920] SimpleTCPServer -- Server closed
2012-10-18 09:01:19.585230 D [68565:70112398237920] SimpleTCPServer -- thread killed
2012-10-18 09:01:19.592187 D [68565:70112398237920] SimpleTCPServer -- Server Restarted
2012-10-18 09:01:19.592919 D [68565:70112398185500] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 09:01:19.593057 D [68565:70112396925660] (8.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-18 09:01:19.593266 T [68565:70112396925660] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-18 09:01:19.593335 W [68565:70112396925660] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-18 09:01:19.593409 D [68565:70112396925660] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:404:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:373:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:126:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:460:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:121:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 09:01:19.593450 W [68565:70112396925660] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure retry 1 due to ResilientSocket::ConnectionFailure: Connection lost while reading data
2012-10-18 09:01:19.594201 T [68565:70112396925660] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-18 09:01:19.594352 D [68565:70112398185500] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 09:01:19.594479 D [68565:70112396925660] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-18 09:01:19.594558 D [68565:70112398185500] SimpleTCPServer -- 
****************** Received request
2012-10-18 09:01:19.594570 T [68565:70112398185500] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-18 09:01:19.594585 D [68565:70112398185500] SimpleTCPServer -- Sending Reply
2012-10-18 09:01:19.594594 T [68565:70112398185500] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-18 09:01:19.595096 T [68565:70112396925660] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x16\x00\x00\x00"
2012-10-18 09:01:19.595136 D [68565:70112396925660] (0.6ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-18 09:01:19.595348 T [68565:70112396925660] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-18 09:01:19.595387 D [68565:70112396925660] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 18 bytes
2012-10-18 09:01:19.596233 D [68565:70112401042020] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 09:01:19.596886 T [68565:70112396925660] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-18 09:01:19.597064 D [68565:70112396925660] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-18 09:01:19.597137 D [68565:70112401042020] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 09:01:19.597373 D [68565:70112401042020] SimpleTCPServer -- 
****************** Received request
2012-10-18 09:01:19.597389 T [68565:70112401042020] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-18 09:01:19.597403 D [68565:70112401042020] SimpleTCPServer -- Sending Reply
2012-10-18 09:01:19.597413 T [68565:70112401042020] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-18 09:01:19.597497 T [68565:70112396925660] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-18 09:01:19.597508 D [68565:70112396925660] (0.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-18 09:01:19.597556 T [68565:70112396925660] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-18 09:01:19.597565 D [68565:70112396925660] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-18 09:01:19.598638 D [68565:70112401293600] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 09:01:19.599256 T [68565:70112396925660] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-18 09:01:19.599428 D [68565:70112396925660] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-18 09:01:19.599501 D [68565:70112401293600] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 09:01:19.599569 D [68565:70112401293600] SimpleTCPServer -- 
****************** Received request
2012-10-18 09:01:19.599580 T [68565:70112401293600] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-18 09:01:22.600651 W [68565:70112396925660] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-18 09:01:22.600865 D [68565:70112396925660] (3001.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:391:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:373:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:111:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:109:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 09:01:22.602856 D [68565:70112401541720] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 09:01:22.603803 T [68565:70112396925660] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-18 09:01:22.603874 D [68565:70112396925660] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-18 09:01:22.604196 D [68565:70112401541720] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 09:01:22.604428 D [68565:70112401541720] SimpleTCPServer -- 
****************** Received request
2012-10-18 09:01:22.604448 T [68565:70112401541720] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-18 09:01:22.604469 D [68565:70112401541720] SimpleTCPServer -- Sending Reply
2012-10-18 09:01:22.604484 T [68565:70112401541720] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-18 09:01:22.605038 T [68565:70112396925660] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-18 09:01:22.605058 D [68565:70112396925660] (1.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-18 09:01:22.605352 T [68565:70112396925660] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-18 09:01:22.605371 D [68565:70112396925660] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-18 09:01:22.606043 D [68565:70112397073460] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 09:01:22.607235 W [68565:70112396925660] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-18 09:01:23.109535 W [68565:70112396925660] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-18 09:01:23.611021 W [68565:70112396925660] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-18 09:01:24.112470 W [68565:70112396925660] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-18 09:01:24.613913 W [68565:70112396925660] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-18 09:01:25.115481 W [68565:70112396925660] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-18 09:01:25.616961 W [68565:70112396925660] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-18 09:01:26.118560 W [68565:70112396925660] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-18 09:01:26.620401 W [68565:70112396925660] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-18 09:01:27.122269 W [68565:70112396925660] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-18 09:01:27.623741 E [68565:70112396925660] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-18 09:01:27.623842 I [68565:70112396925660] (5017.9ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 connection attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:574:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:552:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:545:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:271:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:269:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:269:in `find'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:269:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:232:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:135:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:135:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 09:01:27.624863 T [68565:70112396925660] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-18 09:01:27.624912 D [68565:70112396925660] (0.0ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-18 09:01:27.625180 D [68565:70112397073460] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 09:01:27.625345 D [68565:70112397073460] SimpleTCPServer -- 
****************** Received request
2012-10-18 09:01:27.625361 T [68565:70112397073460] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-18 09:01:27.625377 D [68565:70112397073460] SimpleTCPServer -- Sending Reply
2012-10-18 09:01:27.625388 T [68565:70112397073460] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-18 09:01:27.625793 T [68565:70112396925660] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-18 09:01:27.625810 D [68565:70112396925660] (0.9ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-18 09:01:27.625857 T [68565:70112396925660] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-18 09:01:27.625867 D [68565:70112396925660] (0.0ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-18 09:01:27.628328 D [68565:70112397381360] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 09:01:27.628988 T [68565:70112396925660] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-18 09:01:27.629063 D [68565:70112396925660] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-18 09:01:27.629135 D [68565:70112397381360] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 09:01:27.629202 D [68565:70112397381360] SimpleTCPServer -- 
****************** Received request
2012-10-18 09:01:27.629215 T [68565:70112397381360] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-18 09:01:30.630166 W [68565:70112396925660] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-18 09:01:30.630347 D [68565:70112396925660] (3001.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:391:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:373:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:69:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:67:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 09:01:31.130239 D [68565:70112397381360] SimpleTCPServer -- Sending Reply
2012-10-18 09:01:31.130291 T [68565:70112397381360] SimpleTCPServer -- Reply -- {"result"=>"sleep"}
2012-10-18 09:01:31.130759 T [68565:70112396925660] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-18 09:01:31.130779 D [68565:70112396925660] (500.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-18 09:01:31.130856 T [68565:70112396925660] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00sleep\x00\x00"
2012-10-18 09:01:31.130869 D [68565:70112396925660] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-18 09:01:31.132794 W [68565:70112396925660] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-18 09:01:31.233645 W [68565:70112396925660] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-18 09:01:31.334674 W [68565:70112396925660] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-18 09:01:31.436304 W [68565:70112396925660] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-18 09:01:31.537890 W [68565:70112396925660] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-18 09:01:31.639310 E [68565:70112396925660] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-18 09:01:31.639547 I [68565:70112396925660] (508.1ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 connection attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:574:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:552:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:545:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:304:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:232:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 09:09:54.993229 D [69887:70115975855760] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 09:09:54.994572 T [69887:70115974666960] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-18 09:09:54.994635 D [69887:70115974666960] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-18 09:09:54.994949 D [69887:70115975855760] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 09:09:54.995051 D [69887:70115975855760] SimpleTCPServer -- 
****************** Received request
2012-10-18 09:09:54.995113 T [69887:70115975855760] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-18 09:09:54.995129 D [69887:70115975855760] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-18 09:09:54.995167 D [69887:70115975855760] SimpleTCPServer -- Server closed
2012-10-18 09:09:54.995178 D [69887:70115975855760] SimpleTCPServer -- thread killed
2012-10-18 09:09:55.001688 D [69887:70115975855760] SimpleTCPServer -- Server Restarted
2012-10-18 09:09:55.002433 T [69887:70115974666960] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-18 09:09:55.002507 W [69887:70115974666960] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-18 09:09:55.002583 D [69887:70115974666960] (7.9ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:404:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:373:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:126:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:465:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:121:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 09:09:55.002627 W [69887:70115974666960] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure retry 1 due to ResilientSocket::ConnectionFailure: Connection lost while reading data
2012-10-18 09:09:55.002701 D [69887:70115975801340] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 09:09:55.003662 T [69887:70115974666960] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-18 09:09:55.003738 D [69887:70115974666960] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-18 09:09:55.003805 D [69887:70115975801340] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 09:09:55.003867 D [69887:70115975801340] SimpleTCPServer -- 
****************** Received request
2012-10-18 09:09:55.003877 T [69887:70115975801340] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-18 09:09:55.003892 D [69887:70115975801340] SimpleTCPServer -- Sending Reply
2012-10-18 09:09:55.003901 T [69887:70115975801340] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-18 09:09:55.004495 T [69887:70115974666960] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x16\x00\x00\x00"
2012-10-18 09:09:55.004510 D [69887:70115974666960] (0.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-18 09:09:55.004556 T [69887:70115974666960] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-18 09:09:55.004566 D [69887:70115974666960] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 18 bytes
2012-10-18 09:09:55.006109 D [69887:70115974839660] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 09:09:55.007208 T [69887:70115974666960] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-18 09:09:55.007399 D [69887:70115974666960] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-18 09:09:55.007476 D [69887:70115974839660] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 09:09:55.007541 D [69887:70115974839660] SimpleTCPServer -- 
****************** Received request
2012-10-18 09:09:55.007553 T [69887:70115974839660] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-18 09:09:55.007567 D [69887:70115974839660] SimpleTCPServer -- Sending Reply
2012-10-18 09:09:55.007576 T [69887:70115974839660] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-18 09:09:55.008108 T [69887:70115974666960] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-18 09:09:55.008123 D [69887:70115974666960] (0.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-18 09:09:55.008350 T [69887:70115974666960] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-18 09:09:55.008366 D [69887:70115974666960] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-18 09:09:55.008990 D [69887:70115983014640] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 09:09:55.009569 T [69887:70115974666960] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-18 09:09:55.009746 D [69887:70115974666960] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-18 09:09:55.009825 D [69887:70115983014640] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 09:09:55.009892 D [69887:70115983014640] SimpleTCPServer -- 
****************** Received request
2012-10-18 09:09:55.009903 T [69887:70115983014640] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-18 09:09:58.010911 W [69887:70115974666960] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-18 09:09:58.011050 D [69887:70115974666960] (3001.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:391:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:373:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:111:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:109:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 09:09:58.012386 D [69887:70115983230920] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 09:09:58.012911 T [69887:70115974666960] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-18 09:09:58.012964 D [69887:70115974666960] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-18 09:09:58.013093 D [69887:70115983230920] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 09:09:58.013402 D [69887:70115983230920] SimpleTCPServer -- 
****************** Received request
2012-10-18 09:09:58.013417 T [69887:70115983230920] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-18 09:09:58.013433 D [69887:70115983230920] SimpleTCPServer -- Sending Reply
2012-10-18 09:09:58.013445 T [69887:70115983230920] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-18 09:09:58.013851 T [69887:70115974666960] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-18 09:09:58.013867 D [69887:70115974666960] (0.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-18 09:09:58.014079 T [69887:70115974666960] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-18 09:09:58.014094 D [69887:70115974666960] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-18 09:09:58.015007 D [69887:70115975061140] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 09:09:58.015331 W [69887:70115974666960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-18 09:09:58.516266 W [69887:70115974666960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-18 09:09:59.018309 W [69887:70115974666960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-18 09:09:59.519916 W [69887:70115974666960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-18 09:10:00.021075 W [69887:70115974666960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-18 09:10:00.521864 W [69887:70115974666960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-18 09:10:01.023102 W [69887:70115974666960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-18 09:10:01.523835 W [69887:70115974666960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-18 09:10:02.025226 W [69887:70115974666960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-18 09:10:02.526797 W [69887:70115974666960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-18 09:10:03.028292 E [69887:70115974666960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-18 09:10:03.028541 I [69887:70115974666960] (5013.7ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 connection attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:570:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:548:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:541:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:271:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:269:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:269:in `find'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:269:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:232:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:135:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:135:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 09:10:03.029591 T [69887:70115974666960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-18 09:10:03.029841 D [69887:70115974666960] (0.2ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-18 09:10:03.030053 D [69887:70115975061140] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 09:10:03.030143 D [69887:70115975061140] SimpleTCPServer -- 
****************** Received request
2012-10-18 09:10:03.030159 T [69887:70115975061140] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-18 09:10:03.030179 D [69887:70115975061140] SimpleTCPServer -- Sending Reply
2012-10-18 09:10:03.030191 T [69887:70115975061140] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-18 09:10:03.030944 T [69887:70115974666960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-18 09:10:03.030966 D [69887:70115974666960] (1.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-18 09:10:03.031030 T [69887:70115974666960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-18 09:10:03.031044 D [69887:70115974666960] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-18 09:10:03.033419 T [69887:70115974666960] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-18 09:10:03.033497 D [69887:70115974666960] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-18 09:10:03.033625 D [69887:70115975403380] SimpleTCPServer -- Waiting for a client to connect
2012-10-18 09:10:03.033670 D [69887:70115975403380] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-18 09:10:03.033749 D [69887:70115975403380] SimpleTCPServer -- 
****************** Received request
2012-10-18 09:10:03.033763 T [69887:70115975403380] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-18 09:10:06.034640 W [69887:70115974666960] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-18 09:10:06.034790 D [69887:70115974666960] (3001.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:391:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:373:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:69:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:67:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-18 09:10:06.534849 D [69887:70115975403380] SimpleTCPServer -- Sending Reply
2012-10-18 09:10:06.534951 T [69887:70115975403380] SimpleTCPServer -- Reply -- {"result"=>"sleep"}
2012-10-18 09:10:06.535619 T [69887:70115974666960] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-18 09:10:06.535658 D [69887:70115974666960] (500.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-18 09:10:06.536496 T [69887:70115974666960] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00sleep\x00\x00"
2012-10-18 09:10:06.536533 D [69887:70115974666960] (0.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-18 09:10:06.539085 W [69887:70115974666960] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-18 09:10:06.640539 W [69887:70115974666960] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-18 09:10:06.741671 W [69887:70115974666960] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-18 09:10:06.842696 W [69887:70115974666960] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-18 09:10:06.944408 W [69887:70115974666960] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-18 09:10:07.046270 E [69887:70115974666960] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-18 09:10:07.046501 I [69887:70115974666960] (508.4ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 connection attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:570:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:548:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:541:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:304:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:232:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'