tag:blogger.com,1999:blog-8508935345157628191.post791715187036164377..comments2024-03-27T22:41:50.041-07:00Comments on Phill Barber's Blog: Lessons learned from a connection leak in productionPhill Barberhttp://www.blogger.com/profile/18399785532886246472noreply@blogger.comBlogger12125tag:blogger.com,1999:blog-8508935345157628191.post-90993731695013962082016-11-17T11:43:07.356-08:002016-11-17T11:43:07.356-08:00Hey Phil... thanks for your help... much appreciat...Hey Phil... thanks for your help... much appreciated! I did eventually find what was causing our issue, and it was that we were using ApacheHttpClient with DropWizard 0.6.2, which behind the scenes was creating an MultiThreadedHttpConnectionManager with a default configuration - and that default configuration only allowed 2 concurrent http connections at a time (more info here: http://tinyurl.com/mtcm2connections<br />)! So the problem was that we thought we were passing in config to bump this # up, but in reality it wasn't being used, and our server was only processing 2 requests at a time ;-)euzekahttps://www.blogger.com/profile/14551724099746803725noreply@blogger.comtag:blogger.com,1999:blog-8508935345157628191.post-15849274129059422442016-11-01T18:23:33.804-07:002016-11-01T18:23:33.804-07:00Tip: Assuming you're on the app server (localh...Tip: Assuming you're on the app server (localhost) hit the admin port (5001 in my case) for metrics and filter for the key stats to avoid the plethora of info normally thrown back at you:<br /><br />> curl localhost:5001/metrics | grep -Po "org.apache.http.conn.HttpClientConnectionManager..+?":{.+?}<br />Anonymoushttps://www.blogger.com/profile/00299479009542616519noreply@blogger.comtag:blogger.com,1999:blog-8508935345157628191.post-56673004671252994442016-10-24T13:32:55.036-07:002016-10-24T13:32:55.036-07:00Hi euzeka.
With Jersey 1.x (haven't had a c...Hi euzeka. <br /><br />With Jersey 1.x (haven't had a chance to play with Jersey 2.x yet) if you get a response (i.e. there wasn't some kind of IOException) then you'll always want to close it. When you consume the response (i.e. in your case read the json body) you will probably find in most circumstances that once the entity has been read, the .close() method will have been invoked on the response. For example, if you went clientResponse.getEntity(String.class) this would have the effect of calling close() in most circumstances. However, I think it's still worth having a finally block to make 100% sure that it is called. <br /><br />I'd also recommend writing a test that proves there's a problem if you remove your call to .close() in your finally block... see my post here regarding that one... http://phillbarber.blogspot.co.uk/2015/02/how-to-test-for-connection-leaks.html<br /><br />ThanksPhill Barberhttps://www.blogger.com/profile/18399785532886246472noreply@blogger.comtag:blogger.com,1999:blog-8508935345157628191.post-54821573481801325172016-10-20T07:53:32.679-07:002016-10-20T07:53:32.679-07:00hey! in your example, you can always close the cli...hey! in your example, you can always close the clientResponse, as you were only looking for status, and didn't need the response body. But what if your code returned a large json response body? Would you only close it in fail scenarios, ie when you had an exception?euzekahttps://www.blogger.com/profile/14551724099746803725noreply@blogger.comtag:blogger.com,1999:blog-8508935345157628191.post-29750696176296497172016-10-20T07:52:24.850-07:002016-10-20T07:52:24.850-07:00This comment has been removed by the author.euzekahttps://www.blogger.com/profile/14551724099746803725noreply@blogger.comtag:blogger.com,1999:blog-8508935345157628191.post-61871977281550044062016-06-10T03:09:36.694-07:002016-06-10T03:09:36.694-07:00Awesome summary. Just stumbled over the same sh..t...Awesome summary. Just stumbled over the same sh..t (-:Anonymoushttps://www.blogger.com/profile/02639462672022581747noreply@blogger.comtag:blogger.com,1999:blog-8508935345157628191.post-9791363420330367242016-06-06T10:02:09.732-07:002016-06-06T10:02:09.732-07:00Thanks for the post Phill! My company was running ...Thanks for the post Phill! My company was running performance test before prod deploy and ran into the same issue. Anonymoushttps://www.blogger.com/profile/13359607125771224695noreply@blogger.comtag:blogger.com,1999:blog-8508935345157628191.post-13031958301893059362015-07-04T14:02:50.010-07:002015-07-04T14:02:50.010-07:00Hi Victor, sorry for taking so long to respond.
F...Hi Victor, sorry for taking so long to respond.<br /><br />From memory, I think we just made a bash script that looped forever and piped the number of connections in the CLOSE_WAIT state into mailx which spammed the team until we had proved we had fixed it.<br /><br />Hope this helps,<br /><br />ThanksPhill Barberhttps://www.blogger.com/profile/18399785532886246472noreply@blogger.comtag:blogger.com,1999:blog-8508935345157628191.post-9062136893796970402015-06-01T13:05:34.360-07:002015-06-01T13:05:34.360-07:00Great post, Phill. May I ask how you set up your a...Great post, Phill. May I ask how you set up your alerts on the connection pool? Background thread?Anonymoushttps://www.blogger.com/profile/02729441442158176813noreply@blogger.comtag:blogger.com,1999:blog-8508935345157628191.post-47951936695361214012015-02-02T13:10:19.222-08:002015-02-02T13:10:19.222-08:00Adam, Sorry for taking almost 3 months to reply! ...Adam, Sorry for taking almost 3 months to reply! <br /><br />The code you're after is here: https://github.com/phillbarber/connection-leak-test/blob/master/src/test/java/com/github/phillbarber/connectionleak/AcceptanceTestThatExaminesConnectionPoolBeforeAndAfterRun.java<br /><br />You'll notice that it's part of a project with a few other techniques to test for connection leaks. I have written more about this on another article here: http://phillbarber.blogspot.co.uk/2015/02/how-to-test-for-connection-leaks.html <br /><br />ThanksPhill Barberhttps://www.blogger.com/profile/18399785532886246472noreply@blogger.comtag:blogger.com,1999:blog-8508935345157628191.post-90702269054363290952014-11-26T00:56:07.276-08:002014-11-26T00:56:07.276-08:00Great article! I've ran into exactly the same ...Great article! I've ran into exactly the same issue.<br />Is there any chance you could update this to include your implementation of the getLeasedConnections() method?Anonymoushttps://www.blogger.com/profile/01941177233549171737noreply@blogger.comtag:blogger.com,1999:blog-8508935345157628191.post-72755837598858352182014-03-06T07:39:16.043-08:002014-03-06T07:39:16.043-08:00Nice post.
"Make sure your alerts alert"...Nice post.<br /><br />"Make sure your alerts alert" testers should test this... ie break something and check the dashboards/monitoring emails...Anonymousnoreply@blogger.com