Tuesday, January 11, 2011

How to avoid excessive logging when using Hive JDBC

By default, the Hive JDBC driver outputs a HUGE amount of log messages at the INFO log level.  That makes it very hard to track the log statements added by you.  I spent quite some time trying to change the log level to ERROR by tinkering with the hive-log4j.properties and hive-exec-log4j.properties files.   This approach was not successful. The Hive JDBC driver does not seem to read these log4j config files.  Hence, in order to change the log level, you must add the following lines near the beginning of your program:

from org.apache.log4j import Logger as Logger
from org.apache.log4j import Level as Level
rootLogger = Logger.getRootLogger()
rootLogger.setLevel(Level.ERROR)

Note: The above code is in Jython.  Something very similar will work in Java.

1 comment:

mc said...

This is nice blog danjo, keep the good posts coming frequently :)