H2O is a nice fast tools for data science work. I have discussed this topic in the following blogs:
H2O vs Sparkling Water
Sparking Water Shell: Cloud size under 12 Exception
Access Sparkling Water via R Studio
Running H2O Cluster in Background and at Specific Port Number
Recently we run into some weird issue when using H2O when using R Studio. For unknown reasons, it throws the following error messages:
score <- as.data.frame(main[,c('id', 'my_test_score')])
ERROR: Unexpected HTTP Status code: 500 Server Error (url = http://enkbda1node05.enkitec.com:26000/99/Rapids)
java.lang.IllegalStateException
[1] "java.lang.IllegalStateException: Ref-count mismatch for vec $04ffc52a0000ffffffff$hdfs://ENKBDA1-ns/user/wzhou/test/data/test1/part-00000-8516f9f8-3d5f-164b-cf7e-18ca7e8d467f-d000.snappy.parquet: REFCNT = 2, should be 1"
[2] " water.rapids.Session.sanity_check_refs(Session.java:341)"
[3] " water.rapids.Session.exec(Session.java:83)"
[4] " water.rapids.Rapids.exec(Rapids.java:93)"
[5] " water.api.RapidsHandler.exec(RapidsHandler.java:38)"
[6] " sun.reflect.GeneratedMethodAccessor44.invoke(Unknown Source)"
[7] " sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)"
[8] " java.lang.reflect.Method.invoke(Method.java:498)"
[9] " water.api.Handler.handle(Handler.java:63)"
[10] " water.api.RequestServer.serve(RequestServer.java:448)"
[11] " water.api.RequestServer.doGeneric(RequestServer.java:297)"
[12] " water.api.RequestServer.doPost(RequestServer.java:223)"
[13] " javax.servlet.http.HttpServlet.service(HttpServlet.java:707)"
[14] " javax.servlet.http.HttpServlet.service(HttpServlet.java:790)"
[15] " ai.h2o.org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)"
[16] " ai.h2o.org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:503)"
[17] " ai.h2o.org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)"
[18] " ai.h2o.org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429)"
[19] " ai.h2o.org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)"
[20] " ai.h2o.org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)"
[21] " ai.h2o.org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)"
[22] " ai.h2o.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)"
[23] " water.JettyHTTPD$LoginHandler.handle(JettyHTTPD.java:189)"
[24] " ai.h2o.org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)"
[25] " ai.h2o.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)"
[26] " ai.h2o.org.eclipse.jetty.server.Server.handle(Server.java:370)"
[27] " ai.h2o.org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)"
[28] " ai.h2o.org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)"
[29] " ai.h2o.org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982)"
[30] " ai.h2o.org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043)"
[31] " ai.h2o.org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)"
[32] " ai.h2o.org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)"
[33] " ai.h2o.org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)"
[34] " ai.h2o.org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)"
[35] " ai.h2o.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)"
[36] " ai.h2o.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)"
[37] " java.lang.Thread.run(Thread.java:745)"
Error in .h2o.doSafeREST(h2oRestApiVersion = h2oRestApiVersion, urlSuffix = page, :
ERROR MESSAGE:
Ref-count mismatch for vec $04ffc52a0000ffffffff$hdfs://ENKBDA1-ns/user/wzhou/test/data/test1/part-00000-8516f9f8-3d5f-164b-cf7e-18ca7e8d467f-d000.snappy.parquet: REFCNT = 2, should be 1
Tried different ways and still got the same error. Had to bounce the H2O cluster and get rid of this error. Then further investigation found out the steps we can reproduce this issue. If we delete some or all H2O frames from H2O UI, we could run into this issue.
Basically, just run getFrames command, then select the frames want to be deleted, click Delete selected frames at the bottom. Run getFrames again. It should show those frames deleted. But if I go to R Studio, run h2o.ls(), it will show the exact Ref-count mismatch error. It seems like the frames were deleted from H2O UI perspective, but not from R Studio’s perspective.
Ok, we found out the cause. How to resolve it? Check out H2O source code at https://github.com/h2oai/h2o-3/blob/master/h2o-core/src/main/java/water/rapids/Session.java.
we could see that this error is thrown during sanity_check_refs function. The function also has the following interesting description
Check that ref counts are in a consistent state.
* This should only be called between calls to Rapids expressions (otherwise may blow false-positives).
It seems there is something not handled well in other part of the code. If inconsistent in the object references, some other part of the code might fail. It looks bugs to me.
After some investigation, found out h2o.removeAll() running from R Studio should fix this issue.
My Big Data World 
Pingback: Use Python for H2O | My Big Data World
Pingback: Parquet File Can not Be Read in Sparkling Water H2O | My Big Data World