2012-10-12 11:31:12.665756 D [68823:70214360935360] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 11:31:12.669620 T [68823:70214360455900] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> Sending -- #   action    fail attempt     
2012-10-12 11:31:12.669978 D [68823:70214360455900] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> Sent 35 bytes
2012-10-12 11:31:12.670423 D [68823:70214360935360] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 11:31:12.670504 D [68823:70214360935360] SimpleTCPServer -- 
****************** Received request
2012-10-12 11:31:12.670516 T [68823:70214360935360] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 11:31:12.670531 D [68823:70214360935360] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 11:31:12.670608 D [68823:70214360935360] SimpleTCPServer -- Server closed
2012-10-12 11:31:12.670620 D [68823:70214360935360] SimpleTCPServer -- thread killed
2012-10-12 11:31:12.674155 D [68823:70214360935360] SimpleTCPServer -- Server Restarted
2012-10-12 11:31:12.674851 D [68823:70214360910620] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 11:31:12.674881 T [68823:70214360455900] ResilientSocket::TCPClient ["localhost:2000"] -- <== #read Received
2012-10-12 11:31:12.674894 D [68823:70214360455900] (4.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- <== #read Received upto 4 bytes
2012-10-12 11:33:02.840566 D [69275:70343142288320] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 11:33:02.842261 D [69275:70343142288320] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 11:33:02.843041 T [69275:70343138171600] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> Sending -- #   action    fail attempt     
2012-10-12 11:33:02.843240 D [69275:70343138171600] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> Sent 35 bytes
2012-10-12 11:33:02.843364 D [69275:70343142288320] SimpleTCPServer -- 
****************** Received request
2012-10-12 11:33:02.843376 T [69275:70343142288320] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 11:33:02.843390 D [69275:70343142288320] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 11:33:02.843437 D [69275:70343142288320] SimpleTCPServer -- Server closed
2012-10-12 11:33:02.843447 D [69275:70343142288320] SimpleTCPServer -- thread killed
2012-10-12 11:33:02.847096 D [69275:70343142288320] SimpleTCPServer -- Server Restarted
2012-10-12 11:33:02.847243 D [69275:70343142264020] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 11:33:02.848480 D [69275:70343138171600] (5.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- <== #read 4 bytes -- NoMethodError: undefined method `length' for nil:NilClass
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:295:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:277:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:368:in `retry_on_connection_failure'
test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 11:36:02.992025 D [69955:70357499959240] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 11:36:02.993938 D [69955:70357499959240] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 11:36:02.994366 T [69955:70357499468500] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> Sending -- #   action    fail attempt     
2012-10-12 11:36:02.994649 D [69955:70357499959240] SimpleTCPServer -- 
****************** Received request
2012-10-12 11:36:02.994666 T [69955:70357499959240] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 11:36:02.994682 D [69955:70357499959240] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 11:36:02.994777 D [69955:70357499959240] SimpleTCPServer -- Server closed
2012-10-12 11:36:02.994789 D [69955:70357499959240] SimpleTCPServer -- thread killed
2012-10-12 11:36:02.998701 D [69955:70357499959240] SimpleTCPServer -- Server Restarted
2012-10-12 11:36:02.999357 D [69955:70357499937540] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 11:36:02.999425 D [69955:70357499468500] (5.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> Sent 35 bytes
2012-10-12 11:36:02.999509 T [69955:70357499468500] ResilientSocket::TCPClient ["localhost:2000"] -- <== #read -- nil
2012-10-12 11:36:02.999522 W [69955:70357499468500] ResilientSocket::TCPClient ["localhost:2000"] -- #read Connection lost while reading data
2012-10-12 11:36:02.999584 D [69955:70357499468500] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- <== #read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:299:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:277:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:372:in `retry_on_connection_failure'
test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 11:36:02.999652 W [69955:70357499468500] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure Connection failure: Connection lost while reading data. Retry: 1
2012-10-12 11:36:03.000658 D [69955:70357499937540] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 11:36:03.000881 T [69955:70357499468500] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> Sending -- #   action    fail attempt     
2012-10-12 11:36:03.001125 D [69955:70357499937540] SimpleTCPServer -- 
****************** Received request
2012-10-12 11:36:03.001141 T [69955:70357499937540] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-12 11:36:03.001157 D [69955:70357499937540] SimpleTCPServer -- Sending Reply
2012-10-12 11:36:03.001166 T [69955:70357499937540] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-12 11:36:03.001552 D [69955:70357499468500] (0.6ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> Sent 35 bytes
2012-10-12 11:36:03.001673 T [69955:70357499468500] ResilientSocket::TCPClient ["localhost:2000"] -- <== #read -- "\x16\x00\x00\x00"
2012-10-12 11:36:03.001684 D [69955:70357499468500] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- <== #read 4 bytes
2012-10-12 11:36:03.002016 T [69955:70357499468500] ResilientSocket::TCPClient ["localhost:2000"] -- <== #read -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-12 11:36:03.002031 D [69955:70357499468500] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- <== #read 18 bytes
2012-10-12 11:38:58.626505 D [70229:70113408706400] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 11:38:58.628315 T [70229:70113403558620] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 11:38:58.629926 D [70229:70113403558620] (1.6ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 11:38:58.630066 D [70229:70113408706400] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 11:38:58.630144 D [70229:70113408706400] SimpleTCPServer -- 
****************** Received request
2012-10-12 11:38:58.630156 T [70229:70113408706400] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 11:38:58.630171 D [70229:70113408706400] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 11:38:58.630254 D [70229:70113408706400] SimpleTCPServer -- Server closed
2012-10-12 11:38:58.630266 D [70229:70113408706400] SimpleTCPServer -- thread killed
2012-10-12 11:38:58.638421 D [70229:70113408706400] SimpleTCPServer -- Server Restarted
2012-10-12 11:38:58.638600 D [70229:70113408688760] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 11:38:58.638657 T [70229:70113403558620] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 11:38:58.638670 W [70229:70113403558620] ResilientSocket::TCPClient ["localhost:2000"] -- #read Connection lost while reading data
2012-10-12 11:38:58.638737 D [70229:70113403558620] (8.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== Reading 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:299:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:277:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:372:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 11:38:58.638864 W [70229:70113403558620] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure Connection failure: Connection lost while reading data. Retry: 1
2012-10-12 11:38:58.640292 T [70229:70113403558620] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 11:38:58.640343 D [70229:70113408688760] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 11:38:58.641368 D [70229:70113403558620] (1.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 11:38:58.641655 D [70229:70113408688760] SimpleTCPServer -- 
****************** Received request
2012-10-12 11:38:58.641695 T [70229:70113408688760] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-12 11:38:58.641713 D [70229:70113408688760] SimpleTCPServer -- Sending Reply
2012-10-12 11:38:58.641722 T [70229:70113408688760] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-12 11:38:58.641907 T [70229:70113403558620] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x16\x00\x00\x00"
2012-10-12 11:38:58.641922 D [70229:70113403558620] (0.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== Reading 4 bytes
2012-10-12 11:38:58.641989 T [70229:70113403558620] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-12 11:38:58.642000 D [70229:70113403558620] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== Reading 18 bytes
2012-10-12 11:38:58.644216 D [70229:70113407715360] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 11:38:58.644956 T [70229:70113403558620] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 11:38:58.645135 D [70229:70113403558620] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 11:38:58.645233 D [70229:70113407715360] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 11:38:58.645291 D [70229:70113407715360] SimpleTCPServer -- 
****************** Received request
2012-10-12 11:38:58.645302 T [70229:70113407715360] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 11:38:58.645316 D [70229:70113407715360] SimpleTCPServer -- Sending Reply
2012-10-12 11:38:58.645325 T [70229:70113407715360] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 11:38:58.645409 T [70229:70113403558620] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 11:38:58.645420 D [70229:70113403558620] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== Reading 4 bytes
2012-10-12 11:38:58.645468 T [70229:70113403558620] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 11:38:58.645478 D [70229:70113403558620] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== Reading 19 bytes
2012-10-12 11:38:58.646669 D [70229:70113403967460] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 11:38:58.647777 T [70229:70113403558620] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-12 11:38:58.647956 D [70229:70113403558620] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-12 11:38:58.648042 D [70229:70113403967460] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 11:38:58.648111 D [70229:70113403967460] SimpleTCPServer -- 
****************** Received request
2012-10-12 11:38:58.648123 T [70229:70113403967460] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-12 11:39:01.649115 W [70229:70113403558620] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-12 11:39:01.649555 D [70229:70113403558620] (3001.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== Reading 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:284:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:277:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:80:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:78:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 11:39:01.656271 D [70229:70113404090060] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 11:39:01.657693 T [70229:70113403558620] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 11:39:01.657761 D [70229:70113403558620] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 11:39:01.657847 D [70229:70113404090060] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 11:39:01.658871 D [70229:70113404090060] SimpleTCPServer -- 
****************** Received request
2012-10-12 11:39:01.658893 T [70229:70113404090060] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 11:39:01.658913 D [70229:70113404090060] SimpleTCPServer -- Sending Reply
2012-10-12 11:39:01.658924 T [70229:70113404090060] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 11:39:01.659651 T [70229:70113403558620] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 11:39:01.659670 D [70229:70113403558620] (1.9ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== Reading 4 bytes
2012-10-12 11:39:01.659729 T [70229:70113403558620] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 11:39:01.659740 D [70229:70113403558620] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== Reading 19 bytes
2012-10-12 11:39:01.663395 D [70229:70113404238260] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 11:39:01.664062 W [70229:70113403558620] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 11:39:02.165565 W [70229:70113403558620] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 11:39:02.667637 W [70229:70113403558620] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 11:39:03.168872 W [70229:70113403558620] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 11:39:03.670303 W [70229:70113403558620] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 11:39:04.171792 W [70229:70113403558620] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-12 11:39:04.672910 W [70229:70113403558620] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-12 11:39:05.174641 W [70229:70113403558620] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-12 11:39:05.676771 W [70229:70113403558620] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-12 11:39:06.178450 W [70229:70113403558620] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-12 11:39:06.680400 E [70229:70113403558620] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-12 11:39:06.680633 I [70229:70113403558620] (5017.6ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:453:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:431:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:424:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:223:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `each_with_index'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:193:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 11:39:06.682018 T [70229:70113403558620] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 11:39:06.682426 D [70229:70113404238260] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 11:39:06.682531 D [70229:70113403558620] (0.5ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 11:39:06.683374 D [70229:70113404238260] SimpleTCPServer -- 
****************** Received request
2012-10-12 11:39:06.683398 T [70229:70113404238260] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 11:39:06.683420 D [70229:70113404238260] SimpleTCPServer -- Sending Reply
2012-10-12 11:39:06.683435 T [70229:70113404238260] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 11:39:06.684300 T [70229:70113403558620] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 11:39:06.684322 D [70229:70113403558620] (1.8ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== Reading 4 bytes
2012-10-12 11:39:06.684387 T [70229:70113403558620] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 11:39:06.684399 D [70229:70113403558620] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== Reading 19 bytes
2012-10-12 11:39:06.686595 W [70229:70113403558620] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 11:39:06.787131 W [70229:70113403558620] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 11:39:06.887664 W [70229:70113403558620] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 11:39:06.989089 W [70229:70113403558620] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 11:39:07.090564 W [70229:70113403558620] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 11:39:07.192389 E [70229:70113403558620] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-12 11:39:07.192620 I [70229:70113403558620] (506.6ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:453:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:431:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:424:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:230:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:193:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 11:40:11.410947 D [70320:70194497435480] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 11:40:11.412292 T [70320:70194487843540] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 11:40:11.412358 D [70320:70194487843540] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 11:40:11.412468 D [70320:70194497435480] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 11:40:11.412539 D [70320:70194497435480] SimpleTCPServer -- 
****************** Received request
2012-10-12 11:40:11.412551 T [70320:70194497435480] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 11:40:11.412564 D [70320:70194497435480] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 11:40:11.412602 D [70320:70194497435480] SimpleTCPServer -- Server closed
2012-10-12 11:40:11.412612 D [70320:70194497435480] SimpleTCPServer -- thread killed
2012-10-12 11:40:11.420527 D [70320:70194497435480] SimpleTCPServer -- Server Restarted
2012-10-12 11:40:11.420835 T [70320:70194487843540] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 11:40:11.420890 W [70320:70194487843540] ResilientSocket::TCPClient ["localhost:2000"] -- #read Connection lost while reading data
2012-10-12 11:40:11.420983 D [70320:70194487843540] (8.6ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:299:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:277:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:372:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 11:40:11.421076 W [70320:70194487843540] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure Connection failure: Connection lost while reading data. Retry: 1
2012-10-12 11:40:11.421193 D [70320:70194488865660] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 11:40:11.422856 T [70320:70194487843540] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 11:40:11.423028 D [70320:70194488865660] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 11:40:11.423079 D [70320:70194487843540] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 11:40:11.423167 D [70320:70194488865660] SimpleTCPServer -- 
****************** Received request
2012-10-12 11:40:11.423181 T [70320:70194488865660] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-12 11:40:11.423197 D [70320:70194488865660] SimpleTCPServer -- Sending Reply
2012-10-12 11:40:11.423207 T [70320:70194488865660] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-12 11:40:11.423293 T [70320:70194487843540] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x16\x00\x00\x00"
2012-10-12 11:40:11.423304 D [70320:70194487843540] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 11:40:11.423351 T [70320:70194487843540] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-12 11:40:11.423361 D [70320:70194487843540] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 18 bytes
2012-10-12 11:40:11.424862 D [70320:70194488026880] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 11:40:11.426045 T [70320:70194487843540] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 11:40:11.426234 D [70320:70194488026880] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 11:40:11.426297 D [70320:70194488026880] SimpleTCPServer -- 
****************** Received request
2012-10-12 11:40:11.426308 T [70320:70194488026880] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 11:40:11.426322 D [70320:70194488026880] SimpleTCPServer -- Sending Reply
2012-10-12 11:40:11.426331 T [70320:70194488026880] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 11:40:11.426785 D [70320:70194487843540] (0.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 11:40:11.426851 T [70320:70194487843540] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 11:40:11.426862 D [70320:70194487843540] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 11:40:11.426907 T [70320:70194487843540] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 11:40:11.426916 D [70320:70194487843540] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 11:40:11.427926 D [70320:70194496087060] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 11:40:11.428475 T [70320:70194487843540] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-12 11:40:11.428659 D [70320:70194496087060] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 11:40:11.428726 D [70320:70194496087060] SimpleTCPServer -- 
****************** Received request
2012-10-12 11:40:11.428738 T [70320:70194496087060] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-12 11:40:11.428834 D [70320:70194487843540] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-12 11:40:14.429954 W [70320:70194487843540] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-12 11:40:14.430153 D [70320:70194487843540] (3001.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:284:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:277:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:80:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:78:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 11:40:14.431846 D [70320:70194496133060] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 11:40:14.432450 T [70320:70194487843540] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 11:40:14.432561 D [70320:70194487843540] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 11:40:14.432898 D [70320:70194496133060] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 11:40:14.432970 D [70320:70194496133060] SimpleTCPServer -- 
****************** Received request
2012-10-12 11:40:14.432981 T [70320:70194496133060] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 11:40:14.432997 D [70320:70194496133060] SimpleTCPServer -- Sending Reply
2012-10-12 11:40:14.433007 T [70320:70194496133060] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 11:40:14.433471 T [70320:70194487843540] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 11:40:14.433486 D [70320:70194487843540] (0.9ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 11:40:14.433535 T [70320:70194487843540] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 11:40:14.433545 D [70320:70194487843540] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 11:40:14.434552 D [70320:70194496266720] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 11:40:14.435173 W [70320:70194487843540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 11:40:14.936536 W [70320:70194487843540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 11:40:15.438569 W [70320:70194487843540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 11:40:15.941032 W [70320:70194487843540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 11:40:16.442417 W [70320:70194487843540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 11:40:16.943564 W [70320:70194487843540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-12 11:40:17.445145 W [70320:70194487843540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-12 11:40:17.946956 W [70320:70194487843540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-12 11:40:18.448744 W [70320:70194487843540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-12 11:40:18.949651 W [70320:70194487843540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-12 11:40:19.451141 E [70320:70194487843540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-12 11:40:19.451375 I [70320:70194487843540] (5016.9ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:453:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:431:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:424:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:223:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `each_with_index'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:193:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 11:40:19.455051 T [70320:70194487843540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 11:40:19.455131 D [70320:70194487843540] (0.0ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 11:40:19.455210 D [70320:70194496266720] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 11:40:19.455305 D [70320:70194496266720] SimpleTCPServer -- 
****************** Received request
2012-10-12 11:40:19.455321 T [70320:70194496266720] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 11:40:19.455341 D [70320:70194496266720] SimpleTCPServer -- Sending Reply
2012-10-12 11:40:19.455354 T [70320:70194496266720] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 11:40:19.456212 T [70320:70194487843540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 11:40:19.456257 D [70320:70194487843540] (1.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-12 11:40:19.456325 T [70320:70194487843540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 11:40:19.456339 D [70320:70194487843540] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-12 11:40:19.457904 W [70320:70194487843540] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 11:40:19.559735 W [70320:70194487843540] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 11:40:19.660427 W [70320:70194487843540] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 11:40:19.762103 W [70320:70194487843540] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 11:40:19.863972 W [70320:70194487843540] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 11:40:19.964894 E [70320:70194487843540] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-12 11:40:19.964992 I [70320:70194487843540] (507.6ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:453:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:431:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:424:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:230:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:193:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 13:55:32.622372 D [84877:70361036717920] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 13:55:32.624151 D [84877:70361036717920] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 13:55:32.624465 T [84877:70361035266780] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 13:55:32.624521 D [84877:70361035266780] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 13:55:32.624662 D [84877:70361036717920] SimpleTCPServer -- 
****************** Received request
2012-10-12 13:55:32.624675 T [84877:70361036717920] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 13:55:32.624691 D [84877:70361036717920] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 13:55:32.624730 D [84877:70361036717920] SimpleTCPServer -- Server closed
2012-10-12 13:55:32.624741 D [84877:70361036717920] SimpleTCPServer -- thread killed
2012-10-12 13:55:32.631466 D [84877:70361036717920] SimpleTCPServer -- Server Restarted
2012-10-12 13:55:32.631757 T [84877:70361035266780] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 13:55:32.631829 W [84877:70361035266780] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 13:55:32.631911 D [84877:70361035266780] (7.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:375:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 13:55:32.631962 E [84877:70361035266780] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure Connection failure: ResilientSocket::ConnectionFailure: Connection lost while reading data. Giving up after 0 retries
2012-10-12 13:55:32.633378 D [84877:70361039357600] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 13:55:32.634042 D [84877:70361039357600] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 13:55:32.634194 T [84877:70361035266780] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 13:55:32.634314 D [84877:70361039357600] SimpleTCPServer -- 
****************** Received request
2012-10-12 13:55:32.634326 T [84877:70361039357600] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 13:55:32.634341 D [84877:70361039357600] SimpleTCPServer -- Sending Reply
2012-10-12 13:55:32.634351 T [84877:70361039357600] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 13:55:32.635296 D [84877:70361035266780] (1.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 13:55:32.635375 T [84877:70361035266780] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 13:55:32.635480 W [84877:70361035266780] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 13:55:32.635551 D [84877:70361035266780] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:70:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 13:55:32.637369 D [84877:70361035138260] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 13:55:32.643733 D [84877:70361035138260] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 13:55:32.644169 T [84877:70361035266780] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-12 13:55:32.644412 D [84877:70361035138260] SimpleTCPServer -- 
****************** Received request
2012-10-12 13:55:32.644425 T [84877:70361035138260] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-12 13:55:32.644499 D [84877:70361035266780] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-12 13:55:35.645612 W [84877:70361035266780] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-12 13:55:35.645826 D [84877:70361035266780] (3001.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:283:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:80:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:78:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 13:55:35.651443 T [84877:70361035266780] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 13:55:35.651539 D [84877:70361035266780] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 13:55:35.652158 D [84877:70361035633260] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 13:55:35.652202 D [84877:70361035633260] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 13:55:35.652267 D [84877:70361035633260] SimpleTCPServer -- 
****************** Received request
2012-10-12 13:55:35.652278 T [84877:70361035633260] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 13:55:35.652294 D [84877:70361035633260] SimpleTCPServer -- Sending Reply
2012-10-12 13:55:35.652305 T [84877:70361035633260] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 13:55:35.654137 T [84877:70361035266780] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 13:55:35.654233 W [84877:70361035266780] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 13:55:35.654308 D [84877:70361035266780] (2.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:129:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 13:55:35.658219 D [84877:70361039571840] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 13:55:35.662197 W [84877:70361035266780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 13:55:36.163029 W [84877:70361035266780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 13:55:36.664782 W [84877:70361035266780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 13:55:37.167016 W [84877:70361035266780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 13:55:37.670257 W [84877:70361035266780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 13:55:38.171552 W [84877:70361035266780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-12 13:55:38.679117 W [84877:70361035266780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-12 13:55:39.180577 W [84877:70361035266780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-12 13:55:39.682198 W [84877:70361035266780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-12 13:55:40.183773 W [84877:70361035266780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-12 13:55:40.685553 E [84877:70361035266780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-12 13:55:40.685676 I [84877:70361035266780] (5027.6ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ArgumentError: wrong number of arguments (1 for 2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/exceptions.rb:26:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:463:in `new'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:463:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:441:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:434:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:222:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `each_with_index'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:192:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 13:55:40.688555 W [84877:70361035266780] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 13:55:40.790133 W [84877:70361035266780] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 13:55:40.891646 W [84877:70361035266780] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 13:55:40.993564 W [84877:70361035266780] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 13:55:41.094248 W [84877:70361035266780] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 13:55:41.195701 E [84877:70361035266780] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-12 13:55:41.195814 I [84877:70361035266780] (508.1ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ArgumentError: wrong number of arguments (1 for 2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/exceptions.rb:26:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:463:in `new'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:463:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:441:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:434:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:229:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:192:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 13:57:47.497268 D [85122:70239376023820] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 13:57:47.498726 T [85122:70239366896340] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 13:57:47.498817 D [85122:70239366896340] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 13:57:47.498950 D [85122:70239376023820] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 13:57:47.499026 D [85122:70239376023820] SimpleTCPServer -- 
****************** Received request
2012-10-12 13:57:47.499038 T [85122:70239376023820] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 13:57:47.499053 D [85122:70239376023820] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 13:57:47.499132 D [85122:70239376023820] SimpleTCPServer -- Server closed
2012-10-12 13:57:47.499146 D [85122:70239376023820] SimpleTCPServer -- thread killed
2012-10-12 13:57:47.505911 D [85122:70239376023820] SimpleTCPServer -- Server Restarted
2012-10-12 13:57:47.506144 D [85122:70239376015880] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 13:57:47.506363 T [85122:70239366896340] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 13:57:47.506431 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 13:57:47.506506 D [85122:70239366896340] (7.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:375:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 13:57:47.506552 E [85122:70239366896340] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure Connection failure: ResilientSocket::ConnectionFailure: Connection lost while reading data. Giving up after 0 retries
2012-10-12 13:57:47.507196 D [85122:70239367333620] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 13:57:47.507813 T [85122:70239366896340] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 13:57:47.509167 D [85122:70239366896340] (1.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 13:57:47.509449 D [85122:70239367333620] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 13:57:47.509666 D [85122:70239367333620] SimpleTCPServer -- 
****************** Received request
2012-10-12 13:57:47.509679 T [85122:70239367333620] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 13:57:47.509694 D [85122:70239367333620] SimpleTCPServer -- Sending Reply
2012-10-12 13:57:47.509703 T [85122:70239367333620] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 13:57:47.510016 T [85122:70239366896340] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 13:57:47.510217 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 13:57:47.510572 D [85122:70239366896340] (1.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:70:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 13:57:47.512079 D [85122:70239367064740] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 13:57:47.512815 T [85122:70239366896340] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-12 13:57:47.513005 D [85122:70239367064740] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 13:57:47.513076 D [85122:70239367064740] SimpleTCPServer -- 
****************** Received request
2012-10-12 13:57:47.513087 T [85122:70239367064740] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-12 13:57:47.513225 D [85122:70239366896340] (0.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-12 13:57:50.514859 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-12 13:57:50.515094 D [85122:70239366896340] (3001.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:283:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:80:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:78:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 13:57:50.519724 D [85122:70239367320740] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 13:57:50.526359 T [85122:70239366896340] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 13:57:50.526444 D [85122:70239366896340] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 13:57:50.530111 D [85122:70239367320740] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 13:57:50.530203 D [85122:70239367320740] SimpleTCPServer -- 
****************** Received request
2012-10-12 13:57:50.530217 T [85122:70239367320740] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 13:57:50.530232 D [85122:70239367320740] SimpleTCPServer -- Sending Reply
2012-10-12 13:57:50.530242 T [85122:70239367320740] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 13:57:50.530785 T [85122:70239366896340] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 13:57:50.530872 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 13:57:50.530943 D [85122:70239366896340] (4.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:129:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 13:57:50.533368 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 13:57:50.534291 D [85122:70239367590700] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 13:57:51.034581 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 13:57:51.535760 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 13:57:52.037236 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 13:57:52.538659 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 13:57:53.039838 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-12 13:57:53.540923 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-12 13:57:54.041787 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-12 13:57:54.543074 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-12 13:57:55.044695 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-12 13:57:55.546049 E [85122:70239366896340] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-12 13:57:55.546150 I [85122:70239366896340] (5014.0ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:463:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:441:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:434:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:222:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `each_with_index'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:192:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 13:57:55.547159 T [85122:70239366896340] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 13:57:55.547385 D [85122:70239367590700] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 13:57:55.547498 D [85122:70239367590700] SimpleTCPServer -- 
****************** Received request
2012-10-12 13:57:55.547511 T [85122:70239367590700] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 13:57:55.547527 D [85122:70239367590700] SimpleTCPServer -- Sending Reply
2012-10-12 13:57:55.547537 T [85122:70239367590700] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 13:57:55.548290 D [85122:70239366896340] (1.0ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 13:57:55.548436 T [85122:70239366896340] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 13:57:55.548506 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 13:57:55.548571 D [85122:70239366896340] (0.2ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:109:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 13:57:55.549912 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 13:57:55.650565 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 13:57:55.751993 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 13:57:55.853194 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 13:57:55.954789 W [85122:70239366896340] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 13:57:56.055992 E [85122:70239366896340] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-12 13:57:56.056098 I [85122:70239366896340] (506.8ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:463:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:441:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:434:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:229:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:192:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 13:59:57.336644 D [85520:70245608590360] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 13:59:57.338475 D [85520:70245608590360] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 13:59:57.338768 T [85520:70245608020700] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 13:59:57.338908 D [85520:70245608590360] SimpleTCPServer -- 
****************** Received request
2012-10-12 13:59:57.338921 T [85520:70245608590360] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 13:59:57.338936 D [85520:70245608590360] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 13:59:57.339010 D [85520:70245608590360] SimpleTCPServer -- Server closed
2012-10-12 13:59:57.339022 D [85520:70245608590360] SimpleTCPServer -- thread killed
2012-10-12 13:59:57.345742 D [85520:70245608590360] SimpleTCPServer -- Server Restarted
2012-10-12 13:59:57.345875 D [85520:70245608020700] (7.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 13:59:57.345975 T [85520:70245608020700] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 13:59:57.346041 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 13:59:57.346113 D [85520:70245608020700] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:375:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 13:59:57.346151 E [85520:70245608020700] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure Connection failure: ResilientSocket::ConnectionFailure: Connection lost while reading data. Giving up after 0 retries
2012-10-12 13:59:57.348668 D [85520:70245607872100] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 13:59:57.349376 T [85520:70245608020700] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 13:59:57.349543 D [85520:70245608020700] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 13:59:57.349654 D [85520:70245607872100] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 13:59:57.349716 D [85520:70245607872100] SimpleTCPServer -- 
****************** Received request
2012-10-12 13:59:57.349728 T [85520:70245607872100] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 13:59:57.349742 D [85520:70245607872100] SimpleTCPServer -- Sending Reply
2012-10-12 13:59:57.349751 T [85520:70245607872100] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 13:59:57.350656 T [85520:70245608020700] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 13:59:57.351382 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 13:59:57.351455 D [85520:70245608020700] (1.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:70:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 13:59:57.353773 D [85520:70245612170760] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 13:59:57.354358 T [85520:70245608020700] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-12 13:59:57.354544 D [85520:70245608020700] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-12 13:59:57.354643 D [85520:70245612170760] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 13:59:57.354714 D [85520:70245612170760] SimpleTCPServer -- 
****************** Received request
2012-10-12 13:59:57.354726 T [85520:70245612170760] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-12 14:00:00.354848 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-12 14:00:00.355131 D [85520:70245608020700] (3000.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:283:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:80:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:78:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:00:00.356698 D [85520:70245612377960] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:00:00.358435 T [85520:70245608020700] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:00:00.358646 D [85520:70245612377960] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:00:00.358718 D [85520:70245612377960] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:00:00.358730 T [85520:70245612377960] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:00:00.358746 D [85520:70245612377960] SimpleTCPServer -- Sending Reply
2012-10-12 14:00:00.358757 T [85520:70245612377960] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:00:00.358887 D [85520:70245608020700] (0.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:00:00.358998 T [85520:70245608020700] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:00:00.359065 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:00:00.359134 D [85520:70245608020700] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:129:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:00:00.360998 D [85520:70245608302060] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:00:00.361455 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:00:00.862952 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:00:01.364109 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:00:01.865351 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:00:02.366790 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:00:02.868237 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-12 14:00:03.369736 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-12 14:00:03.871367 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-12 14:00:04.372900 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-12 14:00:04.874175 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-12 14:00:05.375156 E [85520:70245608020700] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-12 14:00:05.375266 I [85520:70245608020700] (5015.3ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:463:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:441:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:434:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:222:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `each_with_index'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:192:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:00:05.377611 T [85520:70245608020700] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:00:05.377778 D [85520:70245608020700] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:00:05.378293 D [85520:70245608302060] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:00:05.378438 D [85520:70245608302060] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:00:05.378452 T [85520:70245608302060] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:00:05.378469 D [85520:70245608302060] SimpleTCPServer -- Sending Reply
2012-10-12 14:00:05.378479 T [85520:70245608302060] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:00:05.379133 T [85520:70245608020700] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:00:05.379222 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:00:05.379299 D [85520:70245608020700] (1.5ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:109:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:00:05.381563 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:00:05.483024 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:00:05.583684 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:00:05.684715 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:00:05.785773 W [85520:70245608020700] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:00:05.887253 E [85520:70245608020700] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-12 14:00:05.887361 I [85520:70245608020700] (506.4ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:463:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:441:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:434:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:229:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:192:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:00:58.925677 D [85573:70245948591400] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:00:58.926984 T [85573:70245943565020] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:00:58.927047 D [85573:70245943565020] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:00:58.927156 D [85573:70245948591400] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:00:58.927231 D [85573:70245948591400] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:00:58.927243 T [85573:70245948591400] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:00:58.927256 D [85573:70245948591400] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:00:58.927293 D [85573:70245948591400] SimpleTCPServer -- Server closed
2012-10-12 14:00:58.927304 D [85573:70245948591400] SimpleTCPServer -- thread killed
2012-10-12 14:00:58.933927 D [85573:70245948591400] SimpleTCPServer -- Server Restarted
2012-10-12 14:00:58.936784 T [85573:70245943565020] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:00:58.936838 W [85573:70245943565020] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:00:58.936919 D [85573:70245943565020] (9.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:375:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:00:58.936968 E [85573:70245943565020] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure Connection failure: ResilientSocket::ConnectionFailure: Connection lost while reading data. Giving up after 0 retries
2012-10-12 14:00:58.938703 D [85573:70245943469400] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:00:58.939924 T [85573:70245943565020] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:00:58.940012 D [85573:70245943469400] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:00:58.940066 D [85573:70245943565020] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:00:58.940387 D [85573:70245943469400] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:00:58.940452 T [85573:70245943469400] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:00:58.940469 D [85573:70245943469400] SimpleTCPServer -- Sending Reply
2012-10-12 14:00:58.940479 T [85573:70245943469400] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:00:58.940755 T [85573:70245943565020] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:00:58.940768 D [85573:70245943565020] (0.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:00:58.940926 T [85573:70245943565020] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:00:58.940937 D [85573:70245943565020] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:00:58.944427 D [85573:70245947694240] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:00:58.945108 T [85573:70245943565020] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-12 14:00:58.945204 D [85573:70245947694240] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:00:58.945244 D [85573:70245943565020] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-12 14:00:58.945554 D [85573:70245947694240] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:00:58.945583 T [85573:70245947694240] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-12 14:01:01.946075 W [85573:70245943565020] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-12 14:01:01.946566 D [85573:70245943565020] (3001.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:283:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:80:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:78:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:01:01.952805 D [85573:70245947913760] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:01:01.953566 T [85573:70245943565020] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:01:01.953631 D [85573:70245943565020] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:01:01.954688 D [85573:70245947913760] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:01:01.954780 D [85573:70245947913760] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:01:01.954796 T [85573:70245947913760] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:01:01.954815 D [85573:70245947913760] SimpleTCPServer -- Sending Reply
2012-10-12 14:01:01.954828 T [85573:70245947913760] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:01:01.956135 T [85573:70245943565020] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:01:01.956153 D [85573:70245943565020] (2.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:01:01.956263 T [85573:70245943565020] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:01:01.956275 D [85573:70245943565020] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:01:01.957435 D [85573:70245948039840] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:01:01.957848 W [85573:70245943565020] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:01:02.459119 W [85573:70245943565020] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:01:02.960275 W [85573:70245943565020] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:01:03.461303 W [85573:70245943565020] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:01:03.962874 W [85573:70245943565020] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:01:04.464053 W [85573:70245943565020] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-12 14:01:04.965555 W [85573:70245943565020] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-12 14:01:05.467057 W [85573:70245943565020] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-12 14:01:05.968483 W [85573:70245943565020] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-12 14:01:06.469947 W [85573:70245943565020] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-12 14:01:06.971429 E [85573:70245943565020] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-12 14:01:06.971538 I [85573:70245943565020] (5014.2ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:463:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:441:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:434:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:222:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `each_with_index'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:192:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:01:06.972617 T [85573:70245943565020] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:01:06.972708 D [85573:70245943565020] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:01:06.972997 D [85573:70245948039840] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:01:06.973074 D [85573:70245948039840] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:01:06.973088 T [85573:70245948039840] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:01:06.973106 D [85573:70245948039840] SimpleTCPServer -- Sending Reply
2012-10-12 14:01:06.973117 T [85573:70245948039840] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:01:06.973927 T [85573:70245943565020] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:01:06.973946 D [85573:70245943565020] (1.2ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:01:06.974277 T [85573:70245943565020] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:01:06.974293 D [85573:70245943565020] (0.3ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:01:06.976099 W [85573:70245943565020] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:01:07.077664 W [85573:70245943565020] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:01:07.178556 W [85573:70245943565020] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:01:07.280154 W [85573:70245943565020] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:01:07.382027 W [85573:70245943565020] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:01:07.482950 E [85573:70245943565020] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-12 14:01:07.483067 I [85573:70245943565020] (507.5ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:463:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:441:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:434:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:229:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:192:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:02:51.217714 D [85588:70292869545880] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:02:51.218814 D [85588:70292869545880] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:02:51.219411 T [85588:70292865243860] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:02:51.222055 D [85588:70292869545880] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:02:51.222076 T [85588:70292869545880] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:02:51.222092 D [85588:70292869545880] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:02:51.222168 D [85588:70292869545880] SimpleTCPServer -- Server closed
2012-10-12 14:02:51.222180 D [85588:70292869545880] SimpleTCPServer -- thread killed
2012-10-12 14:02:51.225599 D [85588:70292869545880] SimpleTCPServer -- Server Restarted
2012-10-12 14:02:51.226299 D [85588:70292869353320] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:02:51.226488 D [85588:70292865243860] (7.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:02:51.226732 T [85588:70292865243860] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:02:51.226849 W [85588:70292865243860] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:02:51.226915 D [85588:70292865243860] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:375:in `retry_on_connection_failure'
test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:02:51.226950 E [85588:70292865243860] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure Connection failure: ResilientSocket::ConnectionFailure: Connection lost while reading data. Giving up after 0 retries
2012-10-12 14:05:21.621842 D [85769:70150582291740] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:05:21.623955 D [85769:70150582291740] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:05:21.624319 T [85769:70150577674960] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:05:21.628660 D [85769:70150582291740] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:05:21.628680 T [85769:70150582291740] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:05:21.628696 D [85769:70150582291740] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:05:21.628748 D [85769:70150582291740] SimpleTCPServer -- Server closed
2012-10-12 14:05:21.628774 D [85769:70150582291740] SimpleTCPServer -- thread killed
2012-10-12 14:05:21.629394 D [85769:70150577674960] (5.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:05:21.629475 T [85769:70150577674960] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:05:21.629545 W [85769:70150577674960] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:05:21.629622 D [85769:70150577674960] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:375:in `retry_on_connection_failure'
test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:05:21.629664 E [85769:70150577674960] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure Connection failure: ResilientSocket::ConnectionFailure: Connection lost while reading data. Giving up after 0 retries
2012-10-12 14:07:43.285154 D [86670:70357407772840] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:07:43.287130 T [86670:70357407193820] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:07:43.287221 D [86670:70357407193820] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:07:43.287348 D [86670:70357407772840] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:07:43.287419 D [86670:70357407772840] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:07:43.287430 T [86670:70357407772840] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:07:43.287450 D [86670:70357407772840] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:07:43.287526 D [86670:70357407772840] SimpleTCPServer -- Server closed
2012-10-12 14:07:43.287538 D [86670:70357407772840] SimpleTCPServer -- thread killed
2012-10-12 14:07:43.291364 D [86670:70357407772840] SimpleTCPServer -- Server Restarted
2012-10-12 14:07:43.291673 T [86670:70357407193820] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:07:43.291735 W [86670:70357407193820] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:07:43.291804 D [86670:70357407193820] (4.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:375:in `retry_on_connection_failure'
test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:07:43.291847 E [86670:70357407193820] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure Connection failure: ResilientSocket::ConnectionFailure: Connection lost while reading data. Giving up after 0 retries
2012-10-12 14:08:46.381062 D [86758:70128331265960] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:08:46.382710 D [86758:70128331265960] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:08:46.383593 T [86758:70128326892240] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:08:46.383805 D [86758:70128326892240] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:08:46.383940 D [86758:70128331265960] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:08:46.383952 T [86758:70128331265960] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:08:46.383967 D [86758:70128331265960] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:08:46.384018 D [86758:70128331265960] SimpleTCPServer -- Server closed
2012-10-12 14:08:46.384031 D [86758:70128331265960] SimpleTCPServer -- thread killed
2012-10-12 14:08:46.387766 D [86758:70128331265960] SimpleTCPServer -- Server Restarted
2012-10-12 14:08:46.387879 T [86758:70128326892240] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:08:46.387940 W [86758:70128326892240] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:08:46.388008 D [86758:70128326892240] (4.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:375:in `retry_on_connection_failure'
test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:08:46.388048 E [86758:70128326892240] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure Connection failure: ResilientSocket::ConnectionFailure: Connection lost while reading data. Giving up after 0 retries
2012-10-12 14:09:14.286315 D [86843:70099927650540] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:09:14.287451 D [86843:70099927650540] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:09:14.287808 T [86843:70099927259860] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:09:14.288358 D [86843:70099927259860] (0.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:09:14.288578 D [86843:70099927650540] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:09:14.288590 T [86843:70099927650540] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:09:14.288605 D [86843:70099927650540] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:09:14.288642 D [86843:70099927650540] SimpleTCPServer -- Server closed
2012-10-12 14:09:14.288653 D [86843:70099927650540] SimpleTCPServer -- thread killed
2012-10-12 14:09:14.292351 D [86843:70099927650540] SimpleTCPServer -- Server Restarted
2012-10-12 14:09:14.292622 D [86843:70099927542900] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:09:14.292817 T [86843:70099927259860] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:09:14.292921 W [86843:70099927259860] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:09:14.293044 D [86843:70099927259860] (4.6ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:375:in `retry_on_connection_failure'
test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:09:14.293087 E [86843:70099927259860] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure Connection failure: ResilientSocket::ConnectionFailure: Connection lost while reading data. Giving up after 0 retries
2012-10-12 14:09:24.878192 D [86849:70108244836020] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:09:24.880230 T [86849:70108240370400] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:09:24.880666 D [86849:70108240370400] (0.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:09:24.884095 D [86849:70108244836020] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:09:24.884201 D [86849:70108244836020] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:09:24.884214 T [86849:70108244836020] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:09:24.884229 D [86849:70108244836020] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:09:24.884313 D [86849:70108244836020] SimpleTCPServer -- Server closed
2012-10-12 14:09:24.884324 D [86849:70108244836020] SimpleTCPServer -- thread killed
2012-10-12 14:09:24.890843 D [86849:70108244836020] SimpleTCPServer -- Server Restarted
2012-10-12 14:09:24.892852 D [86849:70108244760320] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:09:24.893589 T [86849:70108240370400] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:09:24.893722 W [86849:70108240370400] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:09:24.893810 D [86849:70108240370400] (13.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:375:in `retry_on_connection_failure'
test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:09:24.893858 E [86849:70108240370400] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure Connection failure: ResilientSocket::ConnectionFailure: Connection lost while reading data. Giving up after 0 retries
2012-10-12 14:11:27.790511 D [86865:70306103159280] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:11:27.793826 D [86865:70306103159280] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:11:27.794126 T [86865:70306094078680] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:11:27.794263 D [86865:70306103159280] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:11:27.794276 T [86865:70306103159280] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:11:27.794291 D [86865:70306103159280] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:11:27.794366 D [86865:70306103159280] SimpleTCPServer -- Server closed
2012-10-12 14:11:27.794377 D [86865:70306103159280] SimpleTCPServer -- thread killed
2012-10-12 14:11:27.802375 D [86865:70306103159280] SimpleTCPServer -- Server Restarted
2012-10-12 14:11:27.802881 D [86865:70306094078680] (8.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:11:27.803347 D [86865:70306095534140] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:11:27.803381 T [86865:70306094078680] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:11:27.803489 W [86865:70306094078680] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:11:27.803573 D [86865:70306094078680] (0.6ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:375:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:11:27.803615 E [86865:70306094078680] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure Connection failure: ResilientSocket::ConnectionFailure: Connection lost while reading data. Giving up after 0 retries
2012-10-12 14:11:27.809372 D [86865:70306103314900] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:11:27.813374 D [86865:70306103314900] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:11:27.813698 T [86865:70306094078680] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:11:27.813762 D [86865:70306094078680] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:11:27.814232 D [86865:70306103314900] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:11:27.814297 T [86865:70306103314900] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:11:27.814316 D [86865:70306103314900] SimpleTCPServer -- Sending Reply
2012-10-12 14:11:27.814326 T [86865:70306103314900] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:11:27.814898 T [86865:70306094078680] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:11:27.814933 D [86865:70306094078680] (1.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:11:27.815186 T [86865:70306094078680] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:11:27.815199 D [86865:70306094078680] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:11:27.816713 D [86865:70306094133000] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:11:27.818735 T [86865:70306094078680] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-12 14:11:27.822977 D [86865:70306094133000] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:11:27.823100 D [86865:70306094133000] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:11:27.823694 T [86865:70306094133000] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-12 14:11:27.833721 D [86865:70306094078680] (15.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-12 14:11:30.835458 W [86865:70306094078680] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-12 14:11:30.835661 D [86865:70306094078680] (3001.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:283:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:80:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:78:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:11:30.838256 D [86865:70306094583300] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:11:30.838996 T [86865:70306094078680] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:11:30.839196 D [86865:70306094583300] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:11:30.839266 D [86865:70306094583300] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:11:30.839277 T [86865:70306094583300] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:11:30.839292 D [86865:70306094583300] SimpleTCPServer -- Sending Reply
2012-10-12 14:11:30.839302 T [86865:70306094583300] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:11:30.839785 D [86865:70306094078680] (0.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:11:30.839852 T [86865:70306094078680] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:11:30.839864 D [86865:70306094078680] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:11:30.839909 T [86865:70306094078680] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:11:30.839918 D [86865:70306094078680] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:11:30.841117 D [86865:70306094786940] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:11:30.842242 W [86865:70306094078680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:11:31.343626 W [86865:70306094078680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:11:31.845427 W [86865:70306094078680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:11:32.346949 W [86865:70306094078680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:11:32.847690 W [86865:70306094078680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:11:33.349265 W [86865:70306094078680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-12 14:11:33.851288 W [86865:70306094078680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-12 14:11:34.352963 W [86865:70306094078680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-12 14:11:34.854255 W [86865:70306094078680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-12 14:11:35.356136 W [86865:70306094078680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-12 14:11:35.857864 E [86865:70306094078680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-12 14:11:35.858101 I [86865:70306094078680] (5017.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 connect attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:464:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:442:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:435:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:222:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `each_with_index'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:192:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:11:35.860561 T [86865:70306094078680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:11:35.860666 D [86865:70306094078680] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:11:35.860772 D [86865:70306094786940] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:11:35.861296 D [86865:70306094786940] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:11:35.861314 T [86865:70306094786940] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:11:35.861333 D [86865:70306094786940] SimpleTCPServer -- Sending Reply
2012-10-12 14:11:35.861347 T [86865:70306094786940] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:11:35.862155 T [86865:70306094078680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:11:35.862173 D [86865:70306094078680] (1.5ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:11:35.862240 T [86865:70306094078680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:11:35.862253 D [86865:70306094078680] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:11:35.867488 W [86865:70306094078680] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:11:35.968975 W [86865:70306094078680] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:11:36.070204 W [86865:70306094078680] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:11:36.171688 W [86865:70306094078680] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:11:36.273459 W [86865:70306094078680] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:11:36.374761 E [86865:70306094078680] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-12 14:11:36.374862 I [86865:70306094078680] (508.8ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 connect attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:464:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:442:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:435:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:229:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:192:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:12:21.434341 D [87041:70156580976300] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:12:21.435912 D [87041:70156580976300] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:12:21.437144 T [87041:70156571335380] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:12:21.437508 D [87041:70156580976300] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:12:21.437526 T [87041:70156580976300] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:12:21.437542 D [87041:70156580976300] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:12:21.437582 D [87041:70156580976300] SimpleTCPServer -- Server closed
2012-10-12 14:12:21.437593 D [87041:70156580976300] SimpleTCPServer -- thread killed
2012-10-12 14:12:21.437619 D [87041:70156571335380] (0.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:12:21.437698 T [87041:70156571335380] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:12:21.437737 W [87041:70156571335380] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:12:21.437811 D [87041:70156571335380] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:375:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:12:21.437851 E [87041:70156571335380] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure Connection failure: ResilientSocket::ConnectionFailure: Connection lost while reading data. Giving up after 0 retries
2012-10-12 14:12:21.440736 D [87041:70156580901360] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:12:21.441795 T [87041:70156571335380] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:12:21.441996 D [87041:70156571335380] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:12:21.442097 D [87041:70156580901360] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:12:21.442156 D [87041:70156580901360] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:12:21.442167 T [87041:70156580901360] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:12:21.442187 D [87041:70156580901360] SimpleTCPServer -- Sending Reply
2012-10-12 14:12:21.442198 T [87041:70156580901360] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:12:21.442282 T [87041:70156571335380] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:12:21.442294 D [87041:70156571335380] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:12:21.442342 T [87041:70156571335380] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:12:21.442352 D [87041:70156571335380] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:12:21.444538 D [87041:70156580806300] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:12:21.445110 T [87041:70156571335380] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-12 14:12:21.445305 D [87041:70156571335380] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-12 14:12:21.445407 D [87041:70156580806300] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:12:21.445472 D [87041:70156580806300] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:12:21.445483 T [87041:70156580806300] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-12 14:12:24.446129 W [87041:70156571335380] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-12 14:12:24.446703 D [87041:70156571335380] (3001.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:283:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:80:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:78:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:12:24.449137 D [87041:70156580768620] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:12:24.451354 T [87041:70156571335380] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:12:24.451461 D [87041:70156571335380] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:12:24.452238 D [87041:70156580768620] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:12:24.452411 D [87041:70156580768620] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:12:24.452472 T [87041:70156580768620] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:12:24.452540 D [87041:70156580768620] SimpleTCPServer -- Sending Reply
2012-10-12 14:12:24.452556 T [87041:70156580768620] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:12:24.453310 T [87041:70156571335380] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:12:24.453332 D [87041:70156571335380] (1.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:12:24.453396 T [87041:70156571335380] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:12:24.453409 D [87041:70156571335380] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:12:24.455185 D [87041:70156580700420] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:12:24.455633 W [87041:70156571335380] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:12:24.956565 W [87041:70156571335380] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:12:25.458081 W [87041:70156571335380] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:12:25.959763 W [87041:70156571335380] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:12:26.460852 W [87041:70156571335380] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:12:26.962428 W [87041:70156571335380] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-12 14:12:27.471306 W [87041:70156571335380] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-12 14:12:27.972661 W [87041:70156571335380] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-12 14:12:28.474509 W [87041:70156571335380] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-12 14:12:28.975726 W [87041:70156571335380] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-12 14:12:29.477227 E [87041:70156571335380] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-12 14:12:29.477356 I [87041:70156571335380] (5022.3ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 connection attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:464:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:442:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:435:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:222:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `each_with_index'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:192:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:12:29.479718 T [87041:70156571335380] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:12:29.480720 D [87041:70156571335380] (0.3ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:12:29.480899 D [87041:70156580700420] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:12:29.480989 D [87041:70156580700420] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:12:29.481004 T [87041:70156580700420] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:12:29.481025 D [87041:70156580700420] SimpleTCPServer -- Sending Reply
2012-10-12 14:12:29.481038 T [87041:70156580700420] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:12:29.481152 T [87041:70156571335380] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:12:29.481169 D [87041:70156571335380] (0.4ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:12:29.481233 T [87041:70156571335380] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:12:29.481247 D [87041:70156571335380] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:12:29.484306 W [87041:70156571335380] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:12:29.586019 W [87041:70156571335380] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:12:29.686868 W [87041:70156571335380] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:12:29.788227 W [87041:70156571335380] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:12:29.889702 W [87041:70156571335380] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:12:29.991171 E [87041:70156571335380] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-12 14:12:29.991270 I [87041:70156571335380] (507.5ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 connection attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:464:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:442:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:435:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:229:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:192:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:12:47.206333 D [87045:70150579411940] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:12:47.208072 D [87045:70150579411940] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:12:47.208415 T [87045:70150577674960] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:12:47.208562 D [87045:70150579411940] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:12:47.208591 T [87045:70150579411940] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:12:47.208608 D [87045:70150579411940] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:12:47.208648 D [87045:70150579411940] SimpleTCPServer -- Server closed
2012-10-12 14:12:47.208659 D [87045:70150579411940] SimpleTCPServer -- thread killed
2012-10-12 14:12:47.216235 D [87045:70150579411940] SimpleTCPServer -- Server Restarted
2012-10-12 14:12:47.216349 D [87045:70150577674960] (7.9ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:12:47.216455 T [87045:70150577674960] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:12:47.216516 W [87045:70150577674960] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:12:47.216592 D [87045:70150577674960] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:375:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:12:47.216631 E [87045:70150577674960] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure Connection failure: ResilientSocket::ConnectionFailure: Connection lost while reading data. Giving up after 0 retries
2012-10-12 14:12:47.219919 D [87045:70150577576460] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:12:47.221431 T [87045:70150577674960] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:12:47.221800 D [87045:70150577576460] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:12:47.221883 D [87045:70150577576460] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:12:47.221895 T [87045:70150577576460] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:12:47.221910 D [87045:70150577576460] SimpleTCPServer -- Sending Reply
2012-10-12 14:12:47.221920 T [87045:70150577576460] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:12:47.222005 D [87045:70150577674960] (0.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:12:47.222067 T [87045:70150577674960] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:12:47.222078 D [87045:70150577674960] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:12:47.222121 T [87045:70150577674960] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:12:47.222131 D [87045:70150577674960] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:12:47.223125 D [87045:70150577633100] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:12:47.223651 D [87045:70150577633100] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:12:47.223794 T [87045:70150577674960] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-12 14:12:47.223911 D [87045:70150577633100] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:12:47.223923 T [87045:70150577633100] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-12 14:12:47.223954 D [87045:70150577674960] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-12 14:12:50.224404 W [87045:70150577674960] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-12 14:12:50.224609 D [87045:70150577674960] (3000.6ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:283:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:80:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:78:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:12:50.226452 D [87045:70150582008980] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:12:50.231400 T [87045:70150577674960] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:12:50.231741 D [87045:70150577674960] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:12:50.231957 D [87045:70150582008980] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:12:50.232720 D [87045:70150582008980] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:12:50.232745 T [87045:70150582008980] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:12:50.232762 D [87045:70150582008980] SimpleTCPServer -- Sending Reply
2012-10-12 14:12:50.232772 T [87045:70150582008980] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:12:50.233556 T [87045:70150577674960] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:12:50.233575 D [87045:70150577674960] (1.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:12:50.233625 T [87045:70150577674960] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:12:50.233634 D [87045:70150577674960] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:12:50.234939 W [87045:70150577674960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:12:50.235026 D [87045:70150582211420] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:12:50.736529 W [87045:70150577674960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:12:51.238157 W [87045:70150577674960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:12:51.739723 W [87045:70150577674960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:12:52.241470 W [87045:70150577674960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:12:52.742897 W [87045:70150577674960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-12 14:12:53.243853 W [87045:70150577674960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-12 14:12:53.745439 W [87045:70150577674960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-12 14:12:54.246344 W [87045:70150577674960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-12 14:12:54.747525 W [87045:70150577674960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-12 14:12:55.248401 E [87045:70150577674960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-12 14:12:55.248547 I [87045:70150577674960] (5014.3ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 connection attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:464:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:442:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:435:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:222:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `each_with_index'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:220:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:192:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:12:55.251859 T [87045:70150577674960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:12:55.252120 D [87045:70150577674960] (0.2ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:12:55.252596 D [87045:70150582211420] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:12:55.252854 D [87045:70150582211420] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:12:55.252958 T [87045:70150582211420] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:12:55.252983 D [87045:70150582211420] SimpleTCPServer -- Sending Reply
2012-10-12 14:12:55.252995 T [87045:70150582211420] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:12:55.253947 T [87045:70150577674960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:12:55.254062 D [87045:70150577674960] (1.9ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:12:55.254260 T [87045:70150577674960] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:12:55.254310 D [87045:70150577674960] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:12:55.256082 W [87045:70150577674960] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:12:55.357732 W [87045:70150577674960] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:12:55.459779 W [87045:70150577674960] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:12:55.561064 W [87045:70150577674960] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:12:55.662512 W [87045:70150577674960] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:12:55.763421 E [87045:70150577674960] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-12 14:12:55.763521 I [87045:70150577674960] (507.9ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 connection attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:464:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:442:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:435:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:229:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:192:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:13:06.764916 D [87048:70168462328720] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:13:06.769874 T [87048:70168457992920] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:13:06.771473 D [87048:70168462328720] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:13:06.771744 D [87048:70168457992920] (1.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:13:06.772295 D [87048:70168462328720] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:13:06.772312 T [87048:70168462328720] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:13:06.772328 D [87048:70168462328720] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:13:06.772371 D [87048:70168462328720] SimpleTCPServer -- Server closed
2012-10-12 14:13:06.772383 D [87048:70168462328720] SimpleTCPServer -- thread killed
2012-10-12 14:13:06.776563 D [87048:70168462328720] SimpleTCPServer -- Server Restarted
2012-10-12 14:13:06.776863 D [87048:70168462257660] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:13:06.777539 T [87048:70168457992920] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:13:06.777781 W [87048:70168457992920] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:13:06.777854 D [87048:70168457992920] (6.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:375:in `retry_on_connection_failure'
test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:13:06.777897 E [87048:70168457992920] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure Connection failure: ResilientSocket::ConnectionFailure: Connection lost while reading data. Giving up after 0 retries
2012-10-12 14:16:58.251849 D [88386:70249315751040] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:16:58.254831 D [88386:70249315751040] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:16:58.255271 T [88386:70249311591120] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:16:58.255588 D [88386:70249315751040] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:16:58.255605 T [88386:70249315751040] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:16:58.255620 D [88386:70249315751040] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:16:58.255787 D [88386:70249315751040] SimpleTCPServer -- Server closed
2012-10-12 14:16:58.255800 D [88386:70249315751040] SimpleTCPServer -- thread killed
2012-10-12 14:16:58.259102 D [88386:70249315751040] SimpleTCPServer -- Server Restarted
2012-10-12 14:16:58.259890 D [88386:70249312140180] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:16:58.260111 D [88386:70249311591120] (4.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:16:58.260367 T [88386:70249311591120] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:16:58.260486 W [88386:70249311591120] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:16:58.260553 D [88386:70249311591120] (0.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:374:in `retry_on_connection_failure'
test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:17:22.961775 D [88543:70100200206280] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:17:22.964023 T [88543:70100195695320] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:17:22.964294 D [88543:70100200206280] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:17:22.964657 D [88543:70100200206280] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:17:22.964692 T [88543:70100200206280] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:17:22.964707 D [88543:70100200206280] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:17:22.964748 D [88543:70100200206280] SimpleTCPServer -- Server closed
2012-10-12 14:17:22.964760 D [88543:70100200206280] SimpleTCPServer -- thread killed
2012-10-12 14:17:22.968853 D [88543:70100200206280] SimpleTCPServer -- Server Restarted
2012-10-12 14:17:22.968934 D [88543:70100195695320] (4.9ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:17:22.969029 T [88543:70100195695320] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:17:22.969086 W [88543:70100195695320] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:17:22.969155 D [88543:70100195695320] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:276:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:374:in `retry_on_connection_failure'
test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:17:46.416583 D [88706:70206697709800] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:17:46.417916 T [88706:70206697462480] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:17:46.418001 D [88706:70206697462480] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:17:46.418214 D [88706:70206697709800] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:17:46.418291 D [88706:70206697709800] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:17:46.418302 T [88706:70206697709800] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:17:46.418316 D [88706:70206697709800] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:17:46.418353 D [88706:70206697709800] SimpleTCPServer -- Server closed
2012-10-12 14:17:46.418364 D [88706:70206697709800] SimpleTCPServer -- thread killed
2012-10-12 14:17:46.421777 D [88706:70206697709800] SimpleTCPServer -- Server Restarted
2012-10-12 14:17:46.422033 D [88706:70206697683840] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:17:46.423022 T [88706:70206697462480] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:17:46.423100 W [88706:70206697462480] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:17:46.423174 D [88706:70206697462480] (5.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:301:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:277:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:375:in `retry_on_connection_failure'
test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:17:46.423216 W [88706:70206697462480] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure retry 1 due to ResilientSocket::ConnectionFailure: Connection lost while reading data
2012-10-12 14:17:46.424675 T [88706:70206697462480] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:17:46.425062 D [88706:70206697683840] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:17:46.425203 D [88706:70206697683840] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:17:46.425264 T [88706:70206697683840] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-12 14:17:46.425285 D [88706:70206697683840] SimpleTCPServer -- Sending Reply
2012-10-12 14:17:46.425295 T [88706:70206697683840] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-12 14:17:46.425800 D [88706:70206697462480] (1.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:17:46.425865 T [88706:70206697462480] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x16\x00\x00\x00"
2012-10-12 14:17:46.425876 D [88706:70206697462480] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:17:46.425921 T [88706:70206697462480] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-12 14:17:46.425930 D [88706:70206697462480] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 18 bytes
2012-10-12 14:17:51.002799 D [88708:70305213956240] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:17:51.004778 T [88708:70305209080540] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:17:51.005027 D [88708:70305213956240] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:17:51.005137 D [88708:70305213956240] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:17:51.005149 T [88708:70305213956240] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:17:51.005164 D [88708:70305213956240] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:17:51.005214 D [88708:70305213956240] SimpleTCPServer -- Server closed
2012-10-12 14:17:51.005228 D [88708:70305213956240] SimpleTCPServer -- thread killed
2012-10-12 14:17:51.011440 D [88708:70305213956240] SimpleTCPServer -- Server Restarted
2012-10-12 14:17:51.011698 D [88708:70305209080540] (6.9ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:17:51.011800 T [88708:70305209080540] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:17:51.011913 W [88708:70305209080540] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:17:51.012208 D [88708:70305209080540] (0.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:301:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:277:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:92:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:375:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:87:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:17:51.012249 W [88708:70305209080540] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure retry 1 due to ResilientSocket::ConnectionFailure: Connection lost while reading data
2012-10-12 14:17:51.013147 D [88708:70305213911080] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:17:51.014149 T [88708:70305209080540] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:17:51.014224 D [88708:70305209080540] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:17:51.014571 D [88708:70305213911080] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:17:51.014875 D [88708:70305213911080] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:17:51.014892 T [88708:70305213911080] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-12 14:17:51.014909 D [88708:70305213911080] SimpleTCPServer -- Sending Reply
2012-10-12 14:17:51.014919 T [88708:70305213911080] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-12 14:17:51.015404 T [88708:70305209080540] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x16\x00\x00\x00"
2012-10-12 14:17:51.015423 D [88708:70305209080540] (1.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:17:51.015470 T [88708:70305209080540] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-12 14:17:51.015480 D [88708:70305209080540] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 18 bytes
2012-10-12 14:17:51.016952 D [88708:70305213127260] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:17:51.017601 D [88708:70305213127260] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:17:51.017849 T [88708:70305209080540] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:17:51.018126 D [88708:70305213127260] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:17:51.018143 T [88708:70305213127260] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:17:51.018157 D [88708:70305213127260] SimpleTCPServer -- Sending Reply
2012-10-12 14:17:51.018166 T [88708:70305213127260] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:17:51.018599 D [88708:70305209080540] (0.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:17:51.018810 T [88708:70305209080540] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:17:51.018851 D [88708:70305209080540] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:17:51.019113 T [88708:70305209080540] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:17:51.019155 D [88708:70305209080540] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:17:51.020184 D [88708:70305209266600] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:17:51.020801 D [88708:70305209266600] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:17:51.021051 T [88708:70305209080540] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-12 14:17:51.021334 D [88708:70305209266600] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:17:51.021351 T [88708:70305209266600] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-12 14:17:51.021574 D [88708:70305209080540] (0.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-12 14:17:54.022063 W [88708:70305209080540] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-12 14:17:54.022483 D [88708:70305209080540] (3000.9ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:284:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:277:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:80:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:78:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:17:54.023995 D [88708:70305209631080] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:17:54.025113 T [88708:70305209080540] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:17:54.025180 D [88708:70305209080540] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:17:54.025446 D [88708:70305209631080] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:17:54.025811 D [88708:70305209631080] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:17:54.025828 T [88708:70305209631080] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:17:54.025847 D [88708:70305209631080] SimpleTCPServer -- Sending Reply
2012-10-12 14:17:54.025860 T [88708:70305209631080] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:17:54.026769 T [88708:70305209080540] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:17:54.026785 D [88708:70305209080540] (1.6ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:17:54.026848 T [88708:70305209080540] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:17:54.026861 D [88708:70305209080540] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:17:54.028992 D [88708:70305209702780] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:17:54.029341 W [88708:70305209080540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:17:54.530781 W [88708:70305209080540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:17:55.032342 W [88708:70305209080540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:17:55.533826 W [88708:70305209080540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:17:56.035345 W [88708:70305209080540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:17:56.537306 W [88708:70305209080540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-12 14:17:57.038959 W [88708:70305209080540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-12 14:17:57.540812 W [88708:70305209080540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-12 14:17:58.041884 W [88708:70305209080540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-12 14:17:58.543089 W [88708:70305209080540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-12 14:17:59.044557 E [88708:70305209080540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-12 14:17:59.044664 I [88708:70305209080540] (5016.0ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 connection attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:465:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:443:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:436:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:223:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `each_with_index'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:193:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:101:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:17:59.046144 T [88708:70305209080540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:17:59.046239 D [88708:70305209080540] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:17:59.046327 D [88708:70305209702780] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:17:59.046427 D [88708:70305209702780] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:17:59.046533 T [88708:70305209702780] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:17:59.046553 D [88708:70305209702780] SimpleTCPServer -- Sending Reply
2012-10-12 14:17:59.046564 T [88708:70305209702780] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:17:59.046967 T [88708:70305209080540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:17:59.046982 D [88708:70305209080540] (0.7ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:17:59.047043 T [88708:70305209080540] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:17:59.047053 D [88708:70305209080540] (0.0ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:17:59.049316 W [88708:70305209080540] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:17:59.150736 W [88708:70305209080540] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:17:59.252711 W [88708:70305209080540] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:17:59.353501 W [88708:70305209080540] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:17:59.454479 W [88708:70305209080540] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:17:59.555710 E [88708:70305209080540] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-12 14:17:59.555817 I [88708:70305209080540] (506.9ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 connection attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:465:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:443:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:436:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:230:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:193:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:43:12.474024 D [94165:70323884299840] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:43:12.476565 T [94165:70323882121940] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:43:12.476667 D [94165:70323882121940] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:43:12.476788 D [94165:70323884299840] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:43:12.476863 D [94165:70323884299840] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:43:12.476926 T [94165:70323884299840] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:43:12.476942 D [94165:70323884299840] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:43:12.477020 D [94165:70323884299840] SimpleTCPServer -- Server closed
2012-10-12 14:43:12.477032 D [94165:70323884299840] SimpleTCPServer -- thread killed
2012-10-12 14:43:12.484377 D [94165:70323884299840] SimpleTCPServer -- Server Restarted
2012-10-12 14:43:12.485291 D [94165:70323890399120] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:43:12.485464 T [94165:70323882121940] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:43:12.485581 W [94165:70323882121940] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:43:12.485673 D [94165:70323882121940] (9.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:277:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:93:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:374:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:88:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:43:12.485723 W [94165:70323882121940] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure retry 1 due to ResilientSocket::ConnectionFailure: Connection lost while reading data
2012-10-12 14:43:12.486880 D [94165:70323890399120] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:43:12.487009 T [94165:70323882121940] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:43:12.487060 D [94165:70323882121940] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:43:12.487564 D [94165:70323890399120] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:43:12.487625 T [94165:70323890399120] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-12 14:43:12.487644 D [94165:70323890399120] SimpleTCPServer -- Sending Reply
2012-10-12 14:43:12.487654 T [94165:70323890399120] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-12 14:43:12.488604 T [94165:70323882121940] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x16\x00\x00\x00"
2012-10-12 14:43:12.488622 D [94165:70323882121940] (1.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:43:12.488674 T [94165:70323882121940] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-12 14:43:12.488684 D [94165:70323882121940] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 18 bytes
2012-10-12 14:43:12.489802 D [94165:70323882302980] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:43:12.490521 T [94165:70323882121940] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:43:12.490705 D [94165:70323882121940] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:43:12.490784 D [94165:70323882302980] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:43:12.490840 D [94165:70323882302980] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:43:12.490851 T [94165:70323882302980] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:43:12.490866 D [94165:70323882302980] SimpleTCPServer -- Sending Reply
2012-10-12 14:43:12.490876 T [94165:70323882302980] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:43:12.491443 T [94165:70323882121940] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:43:12.491511 D [94165:70323882121940] (0.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:43:12.491563 T [94165:70323882121940] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:43:12.491573 D [94165:70323882121940] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:43:12.492561 D [94165:70323890321500] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:43:12.493161 T [94165:70323882121940] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-12 14:43:12.493349 D [94165:70323890321500] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:43:12.493418 D [94165:70323890321500] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:43:12.493429 T [94165:70323890321500] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-12 14:43:12.493525 D [94165:70323882121940] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-12 14:43:15.494680 W [94165:70323882121940] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-12 14:43:15.494880 D [94165:70323882121940] (3001.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:283:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:277:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:81:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:79:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:43:15.497046 D [94165:70323882738660] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:43:15.498752 T [94165:70323882121940] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:43:15.498858 D [94165:70323882121940] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:43:15.498985 D [94165:70323882738660] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:43:15.499066 D [94165:70323882738660] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:43:15.499081 T [94165:70323882738660] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:43:15.499101 D [94165:70323882738660] SimpleTCPServer -- Sending Reply
2012-10-12 14:43:15.499113 T [94165:70323882738660] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:43:15.500541 T [94165:70323882121940] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:43:15.500562 D [94165:70323882121940] (1.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:43:15.500618 T [94165:70323882121940] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:43:15.500629 D [94165:70323882121940] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:43:15.502199 W [94165:70323882121940] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:43:15.502366 D [94165:70323882804880] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:43:16.002937 W [94165:70323882121940] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:43:16.503736 W [94165:70323882121940] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:43:17.005312 W [94165:70323882121940] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:43:17.506295 W [94165:70323882121940] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:43:18.007136 W [94165:70323882121940] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-12 14:43:18.508810 W [94165:70323882121940] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-12 14:43:19.009673 W [94165:70323882121940] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-12 14:43:19.511106 W [94165:70323882121940] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-12 14:43:20.012974 W [94165:70323882121940] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-12 14:43:20.513609 E [94165:70323882121940] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-12 14:43:20.513726 I [94165:70323882121940] (5012.5ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 connection attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:488:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:466:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:459:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:223:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `each_with_index'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:193:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:102:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:102:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:43:20.515084 T [94165:70323882121940] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:43:20.515187 D [94165:70323882121940] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:43:20.515369 D [94165:70323882804880] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:43:20.515454 D [94165:70323882804880] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:43:20.515465 T [94165:70323882804880] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:43:20.515482 D [94165:70323882804880] SimpleTCPServer -- Sending Reply
2012-10-12 14:43:20.515492 T [94165:70323882804880] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:43:20.515611 T [94165:70323882121940] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:43:20.515625 D [94165:70323882121940] (0.4ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:43:20.515687 T [94165:70323882121940] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:43:20.515698 D [94165:70323882121940] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:43:20.517371 W [94165:70323882121940] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:43:20.655235 W [94165:70323882121940] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:43:20.756773 W [94165:70323882121940] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:43:20.858358 W [94165:70323882121940] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:43:20.959889 W [94165:70323882121940] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:43:21.061374 E [94165:70323882121940] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-12 14:43:21.061477 I [94165:70323882121940] (544.8ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 connection attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:488:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:466:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:459:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:230:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:193:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:47:30.639564 D [94994:70108233573700] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:47:30.641858 T [94994:70108231981780] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:47:30.642145 D [94994:70108233573700] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:47:30.642282 D [94994:70108233573700] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:47:30.642295 T [94994:70108233573700] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:47:30.642309 D [94994:70108233573700] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:47:30.642347 D [94994:70108233573700] SimpleTCPServer -- Server closed
2012-10-12 14:47:30.642358 D [94994:70108233573700] SimpleTCPServer -- thread killed
2012-10-12 14:47:30.648926 D [94994:70108233573700] SimpleTCPServer -- Server Restarted
2012-10-12 14:47:30.649261 D [94994:70108233535660] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:47:30.649620 D [94994:70108231981780] (7.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:47:30.650168 T [94994:70108231981780] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:47:30.650293 W [94994:70108231981780] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:47:30.650369 D [94994:70108231981780] (0.6ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:277:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:108:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:374:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:103:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:47:30.650412 W [94994:70108231981780] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure retry 1 due to ResilientSocket::ConnectionFailure: Connection lost while reading data
2012-10-12 14:47:30.651661 T [94994:70108231981780] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:47:30.651815 D [94994:70108231981780] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:47:30.652074 D [94994:70108233535660] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:47:30.652332 D [94994:70108233535660] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:47:30.652348 T [94994:70108233535660] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-12 14:47:30.652365 D [94994:70108233535660] SimpleTCPServer -- Sending Reply
2012-10-12 14:47:30.652374 T [94994:70108233535660] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-12 14:47:30.652716 T [94994:70108231981780] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x16\x00\x00\x00"
2012-10-12 14:47:30.652730 D [94994:70108231981780] (0.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:47:30.653880 T [94994:70108231981780] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-12 14:47:30.653922 D [94994:70108231981780] (1.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 18 bytes
2012-10-12 14:47:30.656512 D [94994:70108232155040] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:47:30.660349 T [94994:70108231981780] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:47:30.660853 D [94994:70108232155040] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:47:30.661285 D [94994:70108231981780] (0.9ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:47:30.661405 D [94994:70108232155040] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:47:30.661418 T [94994:70108232155040] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:47:30.661433 D [94994:70108232155040] SimpleTCPServer -- Sending Reply
2012-10-12 14:47:30.661443 T [94994:70108232155040] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:47:30.662214 T [94994:70108231981780] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:47:30.662237 D [94994:70108231981780] (0.9ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:47:30.662580 T [94994:70108231981780] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:47:30.662655 D [94994:70108231981780] (0.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:47:30.664436 D [94994:70108232388920] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:47:30.665291 T [94994:70108231981780] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-12 14:47:30.665465 D [94994:70108231981780] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-12 14:47:30.665750 D [94994:70108232388920] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:47:30.665933 D [94994:70108232388920] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:47:30.665945 T [94994:70108232388920] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-12 14:47:33.666721 W [94994:70108231981780] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-12 14:47:33.666999 D [94994:70108231981780] (3001.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:283:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:277:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:94:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:92:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:47:34.167029 D [94994:70108232388920] SimpleTCPServer -- Sending Reply
2012-10-12 14:47:34.167075 T [94994:70108232388920] SimpleTCPServer -- Reply -- {"result"=>"sleep"}
2012-10-12 14:47:34.168004 T [94994:70108231981780] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:47:34.168024 D [94994:70108231981780] (500.6ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:47:34.168084 T [94994:70108231981780] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00sleep\x00\x00"
2012-10-12 14:47:34.168094 D [94994:70108231981780] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:47:34.170533 D [94994:70108232676360] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:47:34.175392 T [94994:70108231981780] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-12 14:47:34.175482 D [94994:70108231981780] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-12 14:47:34.176048 D [94994:70108232676360] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:47:34.176128 D [94994:70108232676360] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:47:34.176140 T [94994:70108232676360] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-12 14:47:37.175990 W [94994:70108231981780] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-12 14:47:37.177756 D [94994:70108231981780] (3002.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:283:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:277:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:82:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:80:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:47:37.179521 D [94994:70108232772540] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:47:37.180807 T [94994:70108231981780] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:47:37.180862 D [94994:70108231981780] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:47:37.181190 D [94994:70108232772540] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:47:37.181258 D [94994:70108232772540] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:47:37.181269 T [94994:70108232772540] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:47:37.181333 D [94994:70108232772540] SimpleTCPServer -- Sending Reply
2012-10-12 14:47:37.181346 T [94994:70108232772540] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:47:37.181675 T [94994:70108231981780] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:47:37.181688 D [94994:70108231981780] (0.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:47:37.181734 T [94994:70108231981780] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:47:37.181744 D [94994:70108231981780] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:47:37.184015 W [94994:70108231981780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:47:37.184104 D [94994:70108240285200] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:47:37.685441 W [94994:70108231981780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:47:38.187993 W [94994:70108231981780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:47:38.688803 W [94994:70108231981780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:47:39.189904 W [94994:70108231981780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:47:39.690852 W [94994:70108231981780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-12 14:47:40.195817 W [94994:70108231981780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-12 14:47:40.697215 W [94994:70108231981780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-12 14:47:41.198276 W [94994:70108231981780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-12 14:47:41.700573 W [94994:70108231981780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-12 14:47:42.201527 E [94994:70108231981780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-12 14:47:42.201622 I [94994:70108231981780] (5018.2ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 connection attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:488:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:466:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:459:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:223:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `each_with_index'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:193:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:117:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:117:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:47:42.202396 D [94994:70108240285200] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:47:42.202746 T [94994:70108231981780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:47:42.203014 D [94994:70108231981780] (0.3ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:47:42.203448 D [94994:70108240285200] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:47:42.203460 T [94994:70108240285200] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:47:42.203476 D [94994:70108240285200] SimpleTCPServer -- Sending Reply
2012-10-12 14:47:42.203485 T [94994:70108240285200] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:47:42.203942 T [94994:70108231981780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:47:42.203958 D [94994:70108231981780] (0.6ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:47:42.204005 T [94994:70108231981780] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:47:42.204014 D [94994:70108231981780] (0.0ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:47:42.205687 W [94994:70108231981780] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:47:42.308097 W [94994:70108231981780] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:47:42.409508 W [94994:70108231981780] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:47:42.510895 W [94994:70108231981780] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:47:42.612308 W [94994:70108231981780] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:47:42.715472 E [94994:70108231981780] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-12 14:47:42.715572 I [94994:70108231981780] (510.3ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 connection attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:488:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:466:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:459:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:230:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:193:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:48:31.729954 D [97106:70173941791180] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:48:31.731147 T [97106:70173939948240] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:48:31.731501 D [97106:70173941791180] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:48:31.731808 D [97106:70173941791180] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:48:31.731825 T [97106:70173941791180] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:48:31.731840 D [97106:70173941791180] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:48:31.731878 D [97106:70173941791180] SimpleTCPServer -- Server closed
2012-10-12 14:48:31.731889 D [97106:70173941791180] SimpleTCPServer -- thread killed
2012-10-12 14:48:31.738691 D [97106:70173941791180] SimpleTCPServer -- Server Restarted
2012-10-12 14:48:31.738974 D [97106:70173941766720] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:48:31.739753 D [97106:70173939948240] (8.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:48:31.740163 T [97106:70173939948240] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:48:31.740235 W [97106:70173939948240] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:48:31.740313 D [97106:70173939948240] (0.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:300:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:277:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:111:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:374:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:106:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:48:31.740356 W [97106:70173939948240] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure retry 1 due to ResilientSocket::ConnectionFailure: Connection lost while reading data
2012-10-12 14:48:31.751830 T [97106:70173939948240] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:48:31.751909 D [97106:70173941766720] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:48:31.752237 D [97106:70173939948240] (0.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:48:31.752387 D [97106:70173941766720] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:48:31.752399 T [97106:70173941766720] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-12 14:48:31.752415 D [97106:70173941766720] SimpleTCPServer -- Sending Reply
2012-10-12 14:48:31.752424 T [97106:70173941766720] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-12 14:48:31.753148 T [97106:70173939948240] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x16\x00\x00\x00"
2012-10-12 14:48:31.753183 D [97106:70173939948240] (0.9ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:48:31.753513 T [97106:70173939948240] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-12 14:48:31.753545 D [97106:70173939948240] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 18 bytes
2012-10-12 14:48:31.754365 D [97106:70173940135240] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:48:31.755206 D [97106:70173940135240] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:48:31.755471 T [97106:70173939948240] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:48:31.755718 D [97106:70173939948240] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:48:31.755916 D [97106:70173940135240] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:48:31.755930 T [97106:70173940135240] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:48:31.755943 D [97106:70173940135240] SimpleTCPServer -- Sending Reply
2012-10-12 14:48:31.755952 T [97106:70173940135240] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:48:31.756525 T [97106:70173939948240] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:48:31.756568 D [97106:70173939948240] (0.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:48:31.756787 T [97106:70173939948240] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:48:31.756826 D [97106:70173939948240] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:48:31.757574 D [97106:70173940366260] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:48:31.758393 T [97106:70173939948240] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-12 14:48:31.758580 D [97106:70173940366260] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:48:31.758652 D [97106:70173940366260] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:48:31.758664 T [97106:70173940366260] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-12 14:48:31.759098 D [97106:70173939948240] (0.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-12 14:48:34.761285 W [97106:70173939948240] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-12 14:48:34.761427 D [97106:70173939948240] (3002.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:283:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:277:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:96:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:94:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:48:35.258792 D [97106:70173940366260] SimpleTCPServer -- Sending Reply
2012-10-12 14:48:35.258836 T [97106:70173940366260] SimpleTCPServer -- Reply -- {"result"=>"sleep"}
2012-10-12 14:48:35.259229 T [97106:70173939948240] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:48:35.259247 D [97106:70173939948240] (497.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:48:35.259306 T [97106:70173939948240] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00sleep\x00\x00"
2012-10-12 14:48:35.259316 D [97106:70173939948240] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:48:35.260984 D [97106:70173948383600] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:48:35.261595 T [97106:70173939948240] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-12 14:48:35.261655 D [97106:70173939948240] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-12 14:48:35.262155 D [97106:70173948383600] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:48:35.262323 D [97106:70173948383600] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:48:35.262335 T [97106:70173948383600] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-12 14:48:38.261696 W [97106:70173939948240] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-12 14:48:38.261831 D [97106:70173939948240] (3000.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:283:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:277:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:84:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:82:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:48:38.262831 D [97106:70173948469700] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:48:38.263686 T [97106:70173939948240] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:48:38.263928 D [97106:70173948469700] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:48:38.264121 D [97106:70173948469700] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:48:38.264159 T [97106:70173948469700] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:48:38.264179 D [97106:70173948469700] SimpleTCPServer -- Sending Reply
2012-10-12 14:48:38.264189 T [97106:70173948469700] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:48:38.264282 D [97106:70173939948240] (0.6ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:48:38.264344 T [97106:70173939948240] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:48:38.264356 D [97106:70173939948240] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:48:38.264401 T [97106:70173939948240] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:48:38.264410 D [97106:70173939948240] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:48:38.265736 D [97106:70173948593420] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:48:38.266428 W [97106:70173939948240] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:48:38.767814 W [97106:70173939948240] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:48:39.268839 W [97106:70173939948240] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:48:39.776071 W [97106:70173939948240] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:48:40.277552 W [97106:70173939948240] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:48:40.778748 W [97106:70173939948240] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-12 14:48:41.280273 W [97106:70173939948240] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-12 14:48:41.781468 W [97106:70173939948240] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-12 14:48:42.285373 W [97106:70173939948240] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-12 14:48:42.786964 W [97106:70173939948240] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-12 14:48:43.289119 E [97106:70173939948240] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-12 14:48:43.289217 I [97106:70173939948240] (5023.5ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 connection attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:488:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:466:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:459:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:223:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `each_with_index'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:221:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:193:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:120:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:120:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:48:43.290104 D [97106:70173948593420] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:48:43.290358 T [97106:70173939948240] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:48:43.290436 D [97106:70173939948240] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:48:43.290686 D [97106:70173948593420] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:48:43.290699 T [97106:70173948593420] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:48:43.290715 D [97106:70173948593420] SimpleTCPServer -- Sending Reply
2012-10-12 14:48:43.290725 T [97106:70173948593420] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:48:43.291305 T [97106:70173939948240] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:48:43.291322 D [97106:70173939948240] (0.9ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:48:43.291369 T [97106:70173939948240] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:48:43.291378 D [97106:70173939948240] (0.0ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:48:43.293150 W [97106:70173939948240] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:48:43.394532 W [97106:70173939948240] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:48:43.495883 W [97106:70173939948240] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:48:43.597049 W [97106:70173939948240] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:48:43.697836 W [97106:70173939948240] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:48:43.798624 E [97106:70173939948240] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-12 14:48:43.798733 I [97106:70173939948240] (506.6ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 connection attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:488:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:466:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:459:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:230:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:193:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:52:58.335515 D [239:70142215520920] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:52:58.337509 T [239:70142205844180] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:52:58.337630 D [239:70142215520920] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:52:58.338005 D [239:70142215520920] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:52:58.338022 T [239:70142215520920] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:52:58.338038 D [239:70142215520920] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:52:58.338077 D [239:70142215520920] SimpleTCPServer -- Server closed
2012-10-12 14:52:58.338088 D [239:70142215520920] SimpleTCPServer -- thread killed
2012-10-12 14:52:58.344566 D [239:70142215520920] SimpleTCPServer -- Server Restarted
2012-10-12 14:52:58.344663 D [239:70142205844180] (7.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:52:58.344905 D [239:70142215496480] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:52:58.344984 T [239:70142205844180] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:52:58.345050 W [239:70142205844180] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:52:58.345124 D [239:70142205844180] (0.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:309:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:286:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:111:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:383:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:106:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:52:58.345165 W [239:70142205844180] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure retry 1 due to ResilientSocket::ConnectionFailure: Connection lost while reading data
2012-10-12 14:52:58.346839 T [239:70142205844180] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:52:58.346887 D [239:70142205844180] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:52:58.346946 D [239:70142215496480] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:52:58.347007 D [239:70142215496480] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:52:58.347017 T [239:70142215496480] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-12 14:52:58.347033 D [239:70142215496480] SimpleTCPServer -- Sending Reply
2012-10-12 14:52:58.347043 T [239:70142215496480] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-12 14:52:58.347330 T [239:70142205844180] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x16\x00\x00\x00"
2012-10-12 14:52:58.347343 D [239:70142205844180] (0.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:52:58.347451 T [239:70142205844180] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-12 14:52:58.347462 D [239:70142205844180] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 18 bytes
2012-10-12 14:52:58.348660 D [239:70142205777800] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:52:58.349927 T [239:70142205844180] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:52:58.350116 D [239:70142205777800] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:52:58.350178 D [239:70142205777800] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:52:58.350190 T [239:70142205777800] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:52:58.350203 D [239:70142205777800] SimpleTCPServer -- Sending Reply
2012-10-12 14:52:58.350212 T [239:70142205777800] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:52:58.350660 D [239:70142205844180] (0.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:52:58.350726 T [239:70142205844180] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:52:58.350737 D [239:70142205844180] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:52:58.350781 T [239:70142205844180] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:52:58.350790 D [239:70142205844180] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:52:58.351779 D [239:70142206261960] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:52:58.352283 T [239:70142205844180] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-12 14:52:58.352463 D [239:70142205844180] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-12 14:52:58.352669 D [239:70142206261960] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:52:58.352773 D [239:70142206261960] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:52:58.352785 T [239:70142206261960] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-12 14:53:01.353532 W [239:70142205844180] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-12 14:53:01.353977 D [239:70142205844180] (3001.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:292:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:286:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:96:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:94:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:53:01.853382 D [239:70142206261960] SimpleTCPServer -- Sending Reply
2012-10-12 14:53:01.853502 T [239:70142206261960] SimpleTCPServer -- Reply -- {"result"=>"sleep"}
2012-10-12 14:53:01.854624 T [239:70142205844180] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:53:01.854673 D [239:70142205844180] (499.9ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:53:01.854922 T [239:70142205844180] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00sleep\x00\x00"
2012-10-12 14:53:01.854937 D [239:70142205844180] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:53:01.856442 D [239:70142214186200] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:53:01.858312 T [239:70142205844180] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-12 14:53:01.858449 D [239:70142214186200] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:53:01.858532 D [239:70142214186200] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:53:01.858548 T [239:70142214186200] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-12 14:53:01.859036 D [239:70142205844180] (0.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-12 14:53:04.859687 W [239:70142205844180] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-12 14:53:04.859887 D [239:70142205844180] (3000.8ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:292:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:286:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:84:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:82:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:53:04.860921 D [239:70142206477960] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:53:04.861859 T [239:70142205844180] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:53:04.861939 D [239:70142205844180] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:53:04.862326 D [239:70142206477960] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:53:04.862431 D [239:70142206477960] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:53:04.862443 T [239:70142206477960] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:53:04.862458 D [239:70142206477960] SimpleTCPServer -- Sending Reply
2012-10-12 14:53:04.862469 T [239:70142206477960] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:53:04.862559 T [239:70142205844180] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:53:04.862571 D [239:70142205844180] (0.6ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:53:04.862620 T [239:70142205844180] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:53:04.862630 D [239:70142205844180] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:53:04.863112 D [239:70142206576860] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:53:04.864503 W [239:70142205844180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:53:05.365307 W [239:70142205844180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:53:05.866380 W [239:70142205844180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:53:06.367896 W [239:70142205844180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:53:06.868838 W [239:70142205844180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:53:07.370262 W [239:70142205844180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-12 14:53:07.871536 W [239:70142205844180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-12 14:53:08.372337 W [239:70142205844180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-12 14:53:08.873712 W [239:70142205844180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-12 14:53:09.376788 W [239:70142205844180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-12 14:53:09.878309 E [239:70142205844180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-12 14:53:09.878413 I [239:70142205844180] (5015.4ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 connection attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:497:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:475:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:468:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:224:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:222:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:222:in `each_with_index'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:222:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:193:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:120:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:120:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:53:09.879772 T [239:70142205844180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:53:09.879858 D [239:70142205844180] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:53:09.880218 D [239:70142206576860] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:53:09.880289 D [239:70142206576860] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:53:09.880301 T [239:70142206576860] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:53:09.880317 D [239:70142206576860] SimpleTCPServer -- Sending Reply
2012-10-12 14:53:09.880326 T [239:70142206576860] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:53:09.880416 T [239:70142205844180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:53:09.880428 D [239:70142205844180] (0.5ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:53:09.880476 T [239:70142205844180] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:53:09.880495 D [239:70142205844180] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:53:09.882439 W [239:70142205844180] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:53:09.982960 W [239:70142205844180] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:53:10.083708 W [239:70142205844180] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:53:10.184564 W [239:70142205844180] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:53:10.286062 W [239:70142205844180] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:53:10.387621 E [239:70142205844180] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-12 14:53:10.387730 I [239:70142205844180] (506.8ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 connection attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:497:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:475:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:468:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:231:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:193:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:53:50.651088 D [858:70288244856220] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:53:50.652372 D [858:70288244856220] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:53:50.653161 T [858:70288243120860] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:53:50.653428 D [858:70288244856220] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:53:50.653445 T [858:70288244856220] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-12 14:53:50.653460 D [858:70288244856220] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-12 14:53:50.653508 D [858:70288244856220] SimpleTCPServer -- Server closed
2012-10-12 14:53:50.653519 D [858:70288244856220] SimpleTCPServer -- thread killed
2012-10-12 14:53:50.659966 D [858:70288244856220] SimpleTCPServer -- Server Restarted
2012-10-12 14:53:50.660109 D [858:70288243120860] (6.9ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:53:50.662204 D [858:70288244828780] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:53:50.662469 T [858:70288243120860] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-12 14:53:50.662571 W [858:70288243120860] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-12 14:53:50.662654 D [858:70288243120860] (2.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:311:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:288:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:111:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:385:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:106:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:53:50.662701 W [858:70288243120860] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure retry 1 due to ResilientSocket::ConnectionFailure: Connection lost while reading data
2012-10-12 14:53:50.664698 T [858:70288243120860] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-12 14:53:50.664864 D [858:70288243120860] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-12 14:53:50.665090 D [858:70288244828780] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:53:50.665162 D [858:70288244828780] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:53:50.665174 T [858:70288244828780] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-12 14:53:50.665192 D [858:70288244828780] SimpleTCPServer -- Sending Reply
2012-10-12 14:53:50.665202 T [858:70288244828780] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-12 14:53:50.665798 T [858:70288243120860] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x16\x00\x00\x00"
2012-10-12 14:53:50.665875 D [858:70288243120860] (1.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:53:50.665930 T [858:70288243120860] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-12 14:53:50.665940 D [858:70288243120860] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 18 bytes
2012-10-12 14:53:50.666964 D [858:70288247236200] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:53:50.668036 T [858:70288243120860] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:53:50.668230 D [858:70288243120860] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:53:50.668318 D [858:70288247236200] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:53:50.668453 D [858:70288247236200] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:53:50.668465 T [858:70288247236200] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:53:50.668479 D [858:70288247236200] SimpleTCPServer -- Sending Reply
2012-10-12 14:53:50.668488 T [858:70288247236200] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:53:50.669109 T [858:70288243120860] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:53:50.669124 D [858:70288243120860] (0.9ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:53:50.669171 T [858:70288243120860] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:53:50.669181 D [858:70288243120860] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:53:50.670105 D [858:70288247468880] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:53:50.670739 T [858:70288243120860] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-12 14:53:50.670894 D [858:70288243120860] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-12 14:53:50.671112 D [858:70288247468880] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:53:50.671273 D [858:70288247468880] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:53:50.671305 T [858:70288247468880] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-12 14:53:53.672061 W [858:70288243120860] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-12 14:53:53.672205 D [858:70288243120860] (3001.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:294:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:288:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:96:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:94:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:53:54.172418 D [858:70288247468880] SimpleTCPServer -- Sending Reply
2012-10-12 14:53:54.172462 T [858:70288247468880] SimpleTCPServer -- Reply -- {"result"=>"sleep"}
2012-10-12 14:53:54.172820 T [858:70288243120860] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:53:54.172837 D [858:70288243120860] (500.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:53:54.172938 T [858:70288243120860] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00sleep\x00\x00"
2012-10-12 14:53:54.172949 D [858:70288243120860] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:53:54.174331 D [858:70288247514780] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:53:54.174858 T [858:70288243120860] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-12 14:53:54.174912 D [858:70288243120860] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-12 14:53:54.174986 D [858:70288247514780] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:53:54.175247 D [858:70288247514780] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:53:54.175260 T [858:70288247514780] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-12 14:53:57.175425 W [858:70288243120860] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-12 14:53:57.175561 D [858:70288243120860] (3000.6ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:294:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:288:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:84:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:82:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:53:57.177189 D [858:70288247634140] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:53:57.178115 T [858:70288243120860] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:53:57.178193 D [858:70288247634140] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:53:57.178225 D [858:70288243120860] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:53:57.178624 D [858:70288247634140] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:53:57.178641 T [858:70288247634140] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:53:57.178656 D [858:70288247634140] SimpleTCPServer -- Sending Reply
2012-10-12 14:53:57.178666 T [858:70288247634140] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:53:57.178830 T [858:70288243120860] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:53:57.178843 D [858:70288243120860] (0.6ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:53:57.178992 T [858:70288243120860] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:53:57.179004 D [858:70288243120860] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:53:57.180905 D [858:70288247730020] SimpleTCPServer -- Waiting for a client to connect
2012-10-12 14:53:57.182770 W [858:70288243120860] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:53:57.685100 W [858:70288243120860] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:53:58.186733 W [858:70288243120860] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:53:58.687896 W [858:70288243120860] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:53:59.189396 W [858:70288243120860] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:53:59.690486 W [858:70288243120860] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-12 14:54:00.191841 W [858:70288243120860] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-12 14:54:00.693355 W [858:70288243120860] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-12 14:54:01.194884 W [858:70288243120860] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-12 14:54:01.696336 W [858:70288243120860] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-12 14:54:02.197314 E [858:70288243120860] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-12 14:54:02.197425 I [858:70288243120860] (5017.5ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 connection attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:499:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:477:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:470:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:226:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:224:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:224:in `each_with_index'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:224:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:193:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:120:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:120:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-12 14:54:02.198577 T [858:70288243120860] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-12 14:54:02.198646 D [858:70288243120860] (0.0ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-12 14:54:02.199132 D [858:70288247730020] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-12 14:54:02.199210 D [858:70288247730020] SimpleTCPServer -- 
****************** Received request
2012-10-12 14:54:02.199223 T [858:70288247730020] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-12 14:54:02.199240 D [858:70288247730020] SimpleTCPServer -- Sending Reply
2012-10-12 14:54:02.199251 T [858:70288247730020] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-12 14:54:02.199767 T [858:70288243120860] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-12 14:54:02.199783 D [858:70288243120860] (1.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-12 14:54:02.200240 T [858:70288243120860] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-12 14:54:02.200269 D [858:70288243120860] (0.4ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-12 14:54:02.201489 W [858:70288243120860] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-12 14:54:02.301972 W [858:70288243120860] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-12 14:54:02.402974 W [858:70288243120860] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-12 14:54:02.503963 W [858:70288243120860] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-12 14:54:02.605835 W [858:70288243120860] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-12 14:54:02.707261 E [858:70288243120860] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-12 14:54:02.707366 I [858:70288243120860] (506.7ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 connection attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:499:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:477:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:470:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:233:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:193:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-15 07:37:11.882640 D [6080:70236440923760] SimpleTCPServer -- Waiting for a client to connect
2012-10-15 07:37:11.886870 D [6080:70236440923760] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-15 07:37:11.887341 T [6080:70236439272160] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-15 07:37:11.887517 D [6080:70236440923760] SimpleTCPServer -- 
****************** Received request
2012-10-15 07:37:11.887536 T [6080:70236440923760] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-15 07:37:11.887551 D [6080:70236440923760] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-15 07:37:11.887637 D [6080:70236440923760] SimpleTCPServer -- Server closed
2012-10-15 07:37:11.887648 D [6080:70236440923760] SimpleTCPServer -- thread killed
2012-10-15 07:37:11.894188 D [6080:70236440923760] SimpleTCPServer -- Server Restarted
2012-10-15 07:37:11.894328 D [6080:70236439272160] (7.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-15 07:37:11.894428 T [6080:70236439272160] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-15 07:37:11.894492 W [6080:70236439272160] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-15 07:37:11.894565 D [6080:70236439272160] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:327:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:304:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:111:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:382:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:106:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-15 07:37:11.894606 W [6080:70236439272160] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure retry 1 due to ResilientSocket::ConnectionFailure: Connection lost while reading data
2012-10-15 07:37:11.894683 D [6080:70236440899940] SimpleTCPServer -- Waiting for a client to connect
2012-10-15 07:37:11.895222 T [6080:70236439272160] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-15 07:37:11.895272 D [6080:70236439272160] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-15 07:37:11.895329 D [6080:70236440899940] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-15 07:37:11.895393 D [6080:70236440899940] SimpleTCPServer -- 
****************** Received request
2012-10-15 07:37:11.895404 T [6080:70236440899940] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-15 07:37:11.895419 D [6080:70236440899940] SimpleTCPServer -- Sending Reply
2012-10-15 07:37:11.895435 T [6080:70236440899940] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-15 07:37:11.895535 T [6080:70236439272160] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x16\x00\x00\x00"
2012-10-15 07:37:11.895547 D [6080:70236439272160] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-15 07:37:11.895592 T [6080:70236439272160] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-15 07:37:11.895601 D [6080:70236439272160] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 18 bytes
2012-10-15 07:37:11.899527 D [6080:70236439117340] SimpleTCPServer -- Waiting for a client to connect
2012-10-15 07:37:11.900053 D [6080:70236439117340] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-15 07:37:11.900706 T [6080:70236439272160] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-15 07:37:11.903822 D [6080:70236439117340] SimpleTCPServer -- 
****************** Received request
2012-10-15 07:37:11.903932 T [6080:70236439117340] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-15 07:37:11.903951 D [6080:70236439117340] SimpleTCPServer -- Sending Reply
2012-10-15 07:37:11.903961 T [6080:70236439117340] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-15 07:37:11.904974 D [6080:70236439272160] (4.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-15 07:37:11.905324 T [6080:70236439272160] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-15 07:37:11.905467 D [6080:70236439272160] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-15 07:37:11.905896 T [6080:70236439272160] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-15 07:37:11.905931 D [6080:70236439272160] (0.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-15 07:37:11.907122 D [6080:70236443709680] SimpleTCPServer -- Waiting for a client to connect
2012-10-15 07:37:11.907787 D [6080:70236443709680] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-15 07:37:11.908217 T [6080:70236439272160] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-15 07:37:11.908463 D [6080:70236443709680] SimpleTCPServer -- 
****************** Received request
2012-10-15 07:37:11.908479 T [6080:70236443709680] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-15 07:37:11.908874 D [6080:70236439272160] (0.6ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-15 07:37:14.909606 W [6080:70236439272160] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-15 07:37:14.909862 D [6080:70236439272160] (3000.9ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:310:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:304:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:96:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:94:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-15 07:37:15.409233 D [6080:70236443709680] SimpleTCPServer -- Sending Reply
2012-10-15 07:37:15.409298 T [6080:70236443709680] SimpleTCPServer -- Reply -- {"result"=>"sleep"}
2012-10-15 07:37:15.409637 T [6080:70236439272160] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-15 07:37:15.409655 D [6080:70236439272160] (499.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-15 07:37:15.409722 T [6080:70236439272160] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00sleep\x00\x00"
2012-10-15 07:37:15.409733 D [6080:70236439272160] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-15 07:37:15.412004 D [6080:70236439550780] SimpleTCPServer -- Waiting for a client to connect
2012-10-15 07:37:15.412533 T [6080:70236439272160] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-15 07:37:15.412589 D [6080:70236439272160] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-15 07:37:15.412955 D [6080:70236439550780] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-15 07:37:15.413026 D [6080:70236439550780] SimpleTCPServer -- 
****************** Received request
2012-10-15 07:37:15.413038 T [6080:70236439550780] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-15 07:37:18.413371 W [6080:70236439272160] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-15 07:37:18.413511 D [6080:70236439272160] (3000.9ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:310:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:304:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:84:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:82:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-15 07:37:18.415104 D [6080:70236439634160] SimpleTCPServer -- Waiting for a client to connect
2012-10-15 07:37:18.416085 T [6080:70236439272160] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-15 07:37:18.416264 D [6080:70236439634160] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-15 07:37:18.416332 D [6080:70236439634160] SimpleTCPServer -- 
****************** Received request
2012-10-15 07:37:18.416343 T [6080:70236439634160] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-15 07:37:18.416358 D [6080:70236439634160] SimpleTCPServer -- Sending Reply
2012-10-15 07:37:18.416367 T [6080:70236439634160] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-15 07:37:18.416803 D [6080:70236439272160] (0.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-15 07:37:18.416870 T [6080:70236439272160] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-15 07:37:18.416882 D [6080:70236439272160] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-15 07:37:18.416928 T [6080:70236439272160] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-15 07:37:18.416938 D [6080:70236439272160] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-15 07:37:18.418151 D [6080:70236439762580] SimpleTCPServer -- Waiting for a client to connect
2012-10-15 07:37:18.418602 W [6080:70236439272160] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-15 07:37:18.920069 W [6080:70236439272160] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-15 07:37:19.422183 W [6080:70236439272160] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-15 07:37:19.922988 W [6080:70236439272160] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-15 07:37:20.423619 W [6080:70236439272160] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-15 07:37:20.924851 W [6080:70236439272160] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-15 07:37:21.425925 W [6080:70236439272160] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-15 07:37:21.927070 W [6080:70236439272160] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-15 07:37:22.428558 W [6080:70236439272160] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-15 07:37:22.929588 W [6080:70236439272160] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-15 07:37:23.430719 E [6080:70236439272160] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-15 07:37:23.430818 I [6080:70236439272160] (5012.7ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 connection attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:496:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:474:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:467:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:234:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:232:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:232:in `each_with_index'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:232:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:201:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:120:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:120:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-15 07:37:23.433321 T [6080:70236439272160] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-15 07:37:23.433530 D [6080:70236439272160] (0.2ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-15 07:37:23.433707 D [6080:70236439762580] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-15 07:37:23.433821 D [6080:70236439762580] SimpleTCPServer -- 
****************** Received request
2012-10-15 07:37:23.433834 T [6080:70236439762580] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-15 07:37:23.433850 D [6080:70236439762580] SimpleTCPServer -- Sending Reply
2012-10-15 07:37:23.433860 T [6080:70236439762580] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-15 07:37:23.434631 T [6080:70236439272160] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-15 07:37:23.434648 D [6080:70236439272160] (1.0ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-15 07:37:23.434699 T [6080:70236439272160] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-15 07:37:23.434709 D [6080:70236439272160] (0.0ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-15 07:37:23.436235 W [6080:70236439272160] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-15 07:37:23.536689 W [6080:70236439272160] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-15 07:37:23.637537 W [6080:70236439272160] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-15 07:37:23.738516 W [6080:70236439272160] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-15 07:37:23.839666 W [6080:70236439272160] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-15 07:37:23.941269 E [6080:70236439272160] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-15 07:37:23.941403 I [6080:70236439272160] (506.0ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 connection attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:496:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:474:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:467:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:241:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:201:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-15 07:43:57.049787 D [6347:70199664563820] SimpleTCPServer -- Waiting for a client to connect
2012-10-15 07:43:57.051712 T [6347:70199663614680] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-15 07:43:57.051898 D [6347:70199663614680] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-15 07:43:57.052128 D [6347:70199664563820] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-15 07:43:57.052203 D [6347:70199664563820] SimpleTCPServer -- 
****************** Received request
2012-10-15 07:43:57.052214 T [6347:70199664563820] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>1}
2012-10-15 07:43:57.052229 D [6347:70199664563820] SimpleTCPServer -- Closing client since no reply is being sent back
2012-10-15 07:43:57.052269 D [6347:70199664563820] SimpleTCPServer -- Server closed
2012-10-15 07:43:57.052280 D [6347:70199664563820] SimpleTCPServer -- thread killed
2012-10-15 07:43:57.058958 D [6347:70199664563820] SimpleTCPServer -- Server Restarted
2012-10-15 07:43:57.059144 T [6347:70199663614680] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- nil
2012-10-15 07:43:57.059215 W [6347:70199663614680] ResilientSocket::TCPClient ["localhost:2000"] -- #read server closed the connection before 4 bytes were returned
2012-10-15 07:43:57.059295 D [6347:70199663614680] (7.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ConnectionFailure: Connection lost while reading data
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:327:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:304:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/simple_tcp_server.rb:11:in `read_bson_document'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:111:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:382:in `retry_on_connection_failure'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:106:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-15 07:43:57.059346 W [6347:70199663614680] ResilientSocket::TCPClient ["localhost:2000"] -- #retry_on_connection_failure retry 1 due to ResilientSocket::ConnectionFailure: Connection lost while reading data
2012-10-15 07:43:57.059637 D [6347:70199668961080] SimpleTCPServer -- Waiting for a client to connect
2012-10-15 07:43:57.060770 T [6347:70199663614680] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- #   action    fail attempt     
2012-10-15 07:43:57.060822 D [6347:70199663614680] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 35 bytes
2012-10-15 07:43:57.061008 D [6347:70199668961080] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-15 07:43:57.061325 D [6347:70199668961080] SimpleTCPServer -- 
****************** Received request
2012-10-15 07:43:57.061337 T [6347:70199668961080] SimpleTCPServer -- Request -- {"action"=>"fail", "attempt"=>2}
2012-10-15 07:43:57.061354 D [6347:70199668961080] SimpleTCPServer -- Sending Reply
2012-10-15 07:43:57.061364 T [6347:70199668961080] SimpleTCPServer -- Reply -- {"result"=>"fail"}
2012-10-15 07:43:57.062390 T [6347:70199663614680] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x16\x00\x00\x00"
2012-10-15 07:43:57.062407 D [6347:70199663614680] (1.6ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-15 07:43:57.063858 T [6347:70199663614680] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x05\x00\x00\x00fail\x00\x00"
2012-10-15 07:43:57.063875 D [6347:70199663614680] (1.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 18 bytes
2012-10-15 07:43:57.064968 D [6347:70199667644740] SimpleTCPServer -- Waiting for a client to connect
2012-10-15 07:43:57.065537 T [6347:70199663614680] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-15 07:43:57.065776 D [6347:70199663614680] (0.2ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-15 07:43:57.065988 D [6347:70199667644740] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-15 07:43:57.066050 D [6347:70199667644740] SimpleTCPServer -- 
****************** Received request
2012-10-15 07:43:57.066061 T [6347:70199667644740] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-15 07:43:57.066076 D [6347:70199667644740] SimpleTCPServer -- Sending Reply
2012-10-15 07:43:57.066086 T [6347:70199667644740] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-15 07:43:57.066271 T [6347:70199663614680] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-15 07:43:57.066300 D [6347:70199663614680] (0.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-15 07:43:57.066352 T [6347:70199663614680] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-15 07:43:57.066362 D [6347:70199663614680] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-15 07:43:57.067852 D [6347:70199667925500] SimpleTCPServer -- Waiting for a client to connect
2012-10-15 07:43:57.068355 T [6347:70199663614680] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-15 07:43:57.068540 D [6347:70199667925500] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-15 07:43:57.068606 D [6347:70199667925500] SimpleTCPServer -- 
****************** Received request
2012-10-15 07:43:57.068617 T [6347:70199667925500] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-15 07:43:57.068713 D [6347:70199663614680] (0.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-15 07:44:00.069863 W [6347:70199663614680] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-15 07:44:00.070061 D [6347:70199663614680] (3001.3ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:310:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:304:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:96:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:94:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-15 07:44:00.569653 D [6347:70199667925500] SimpleTCPServer -- Sending Reply
2012-10-15 07:44:00.569707 T [6347:70199667925500] SimpleTCPServer -- Reply -- {"result"=>"sleep"}
2012-10-15 07:44:00.570147 T [6347:70199663614680] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-15 07:44:00.570172 D [6347:70199663614680] (499.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-15 07:44:00.570345 T [6347:70199663614680] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00sleep\x00\x00"
2012-10-15 07:44:00.570359 D [6347:70199663614680] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-15 07:44:00.572329 D [6347:70199664083580] SimpleTCPServer -- Waiting for a client to connect
2012-10-15 07:44:00.573595 T [6347:70199663614680] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending -- )   action    sleep duration       @ 
2012-10-15 07:44:00.573659 D [6347:70199663614680] (0.0ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 41 bytes
2012-10-15 07:44:00.573743 D [6347:70199664083580] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-15 07:44:00.573852 D [6347:70199664083580] SimpleTCPServer -- 
****************** Received request
2012-10-15 07:44:00.573865 T [6347:70199664083580] SimpleTCPServer -- Request -- {"action"=>"sleep", "duration"=>3.5}
2012-10-15 07:44:03.574774 W [6347:70199663614680] ResilientSocket::TCPClient ["localhost:2000"] -- #read Timeout waiting for server to reply
2012-10-15 07:44:03.575109 D [6347:70199663614680] (3001.4ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes -- ResilientSocket::ReadTimeout: Timedout after 3.0 seconds trying to read from localhost:2000
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:310:in `block in read'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_debug'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:304:in `read'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:84:in `block (5 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:82:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-15 07:44:03.578264 D [6347:70199664181960] SimpleTCPServer -- Waiting for a client to connect
2012-10-15 07:44:03.579630 T [6347:70199663614680] ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-15 07:44:03.579731 D [6347:70199664181960] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-15 07:44:03.580061 D [6347:70199664181960] SimpleTCPServer -- 
****************** Received request
2012-10-15 07:44:03.580078 T [6347:70199664181960] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-15 07:44:03.580104 D [6347:70199664181960] SimpleTCPServer -- Sending Reply
2012-10-15 07:44:03.580118 T [6347:70199664181960] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-15 07:44:03.580175 D [6347:70199663614680] (0.5ms) ResilientSocket::TCPClient ["localhost:2000"] -- #write ==> sent 23 bytes
2012-10-15 07:44:03.580900 T [6347:70199663614680] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-15 07:44:03.580947 D [6347:70199663614680] (0.7ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 4 bytes
2012-10-15 07:44:03.581022 T [6347:70199663614680] ResilientSocket::TCPClient ["localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-15 07:44:03.581035 D [6347:70199663614680] (0.1ms) ResilientSocket::TCPClient ["localhost:2000"] -- #read <== read 19 bytes
2012-10-15 07:44:03.582281 D [6347:70199664283240] SimpleTCPServer -- Waiting for a client to connect
2012-10-15 07:44:03.582747 W [6347:70199663614680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-15 07:44:04.083719 W [6347:70199663614680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-15 07:44:04.585135 W [6347:70199663614680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-15 07:44:05.086385 W [6347:70199663614680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-15 07:44:05.587839 W [6347:70199663614680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-15 07:44:06.089707 W [6347:70199663614680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 6
2012-10-15 07:44:06.591624 W [6347:70199663614680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 7
2012-10-15 07:44:07.093142 W [6347:70199663614680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 8
2012-10-15 07:44:07.594050 W [6347:70199663614680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 9
2012-10-15 07:44:08.096043 W [6347:70199663614680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 10
2012-10-15 07:44:08.597585 E [6347:70199663614680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 10 retries
2012-10-15 07:44:08.597719 I [6347:70199663614680] (5015.6ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- Connecting to server localhost:1999 -- ResilientSocket::ConnectionFailure: After 10 connection attempts to host 'localhost:1999': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:496:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:474:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:467:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:234:in `block in connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:232:in `each'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:232:in `each_with_index'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:232:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:201:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:120:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:120:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
2012-10-15 07:44:08.598803 D [6347:70199664283240] SimpleTCPServer -- Client connected, waiting for data from client
2012-10-15 07:44:08.599193 T [6347:70199663614680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sending --    action    test1  
2012-10-15 07:44:08.599285 D [6347:70199663614680] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #write ==> sent 23 bytes
2012-10-15 07:44:08.599688 D [6347:70199664283240] SimpleTCPServer -- 
****************** Received request
2012-10-15 07:44:08.599702 T [6347:70199664283240] SimpleTCPServer -- Request -- {"action"=>"test1"}
2012-10-15 07:44:08.599720 D [6347:70199664283240] SimpleTCPServer -- Sending Reply
2012-10-15 07:44:08.599731 T [6347:70199664283240] SimpleTCPServer -- Reply -- {"result"=>"test1"}
2012-10-15 07:44:08.600173 T [6347:70199663614680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x17\x00\x00\x00"
2012-10-15 07:44:08.600192 D [6347:70199663614680] (0.9ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 4 bytes
2012-10-15 07:44:08.600275 T [6347:70199663614680] ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== received -- "\x02result\x00\x06\x00\x00\x00test1\x00\x00"
2012-10-15 07:44:08.600326 D [6347:70199663614680] (0.1ms) ResilientSocket::TCPClient ["localhost:1999", "localhost:2000"] -- #read <== read 19 bytes
2012-10-15 07:44:08.601916 W [6347:70199663614680] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 1
2012-10-15 07:44:08.703389 W [6347:70199663614680] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 2
2012-10-15 07:44:08.805279 W [6347:70199663614680] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 3
2012-10-15 07:44:08.906189 W [6347:70199663614680] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 4
2012-10-15 07:44:09.007756 W [6347:70199663614680] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Retry: 5
2012-10-15 07:44:09.108625 E [6347:70199663614680] ResilientSocket::TCPClient ["localhost:3300"] -- Connection failure: Errno::ECONNREFUSED: Connection refused - connect(2). Giving up after 5 retries
2012-10-15 07:44:09.108746 I [6347:70199663614680] (507.6ms) ResilientSocket::TCPClient ["localhost:3300"] -- Connecting to server localhost:3300 -- ResilientSocket::ConnectionFailure: After 5 connection attempts to host 'localhost:3300': Errno::ECONNREFUSED: Connection refused - connect(2)
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:496:in `rescue in block in connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:474:in `block in connect_to_server'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/semantic_logger-0.7.0/lib/semantic_logger/base.rb:181:in `benchmark_info'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:467:in `connect_to_server'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:241:in `connect'
/Users/rmorrison/Sandbox/resilient_socket/lib/resilient_socket/tcp_client.rb:201:in `initialize'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `new'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:21:in `block (4 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:330:in `assert_raises'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/assertions.rb:69:in `assert_raise'
/Users/rmorrison/Sandbox/resilient_socket/test/tcp_client_test.rb:20:in `block (3 levels) in <class:TCPClientTest>'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `call'
/Users/rmorrison/.rvm/gems/ruby-1.9.3-p194/gems/shoulda-context-1.0.0/lib/shoulda/context/context.rb:398:in `block in create_test_from_should_hash'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:949:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit/testcase.rb:17:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:787:in `block in _run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `map'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:780:in `_run_suite'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:565:in `block in _run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:563:in `_run_suites'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:746:in `_run_anything'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:909:in `run_tests'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:896:in `block in _run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `each'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:895:in `_run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:21:in `run'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
/Users/rmorrison/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'