James Moore's blog

If you’re including a jar, and you see errors like this when you try to use something defined in that jar file:

/home/james/dev/radoop/./test/../lib/radoop.rb:15:in `method_missing': cannot link Java class org.apache.hadoop.mapred.TextInputFormat (NameError)

Try running with jruby -d and you might get a more informative message:

Script started on Tue 29 Jul 2008 10:54:05 AM PDT
java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory
        at org.apache.hadoop.mapred.FileInputFormat.<clinit>(FileInputFormat.java:49)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:247)

In this case, I needed to include commons-logging-1.0.4.jar and commons-logging-api-1.0.4.jar. I added them, and reran, and got this:

java.lang.ExceptionInInitializerError
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:247)
...
Caused by: org.apache.commons.logging.LogConfigurationException: org.apache.commons.logging.LogConfigurationException: No suitable Log constructor [Ljava.lang.Class;@1edf84f for org.apache.commons.logging.impl.Log4JLogger (Caused by java.lang.NoClassDefFoundError: org/apache/log4j/Category) (Caused by org.apache.commons.logging.LogConfigurationException: No suitable Log constructor [Ljava.lang.Class;@1edf84f for org.apache.commons.logging.impl.Log4JLogger (Caused by java.lang.NoClassDefFoundError: org/apache/log4j/Category))

So I added log4j-1.2.13.jar, and now I’m getting:

/home/james/jruby/lib/ruby/site_ruby/1.8/builtin/javasupport/core_ext/module.rb:16 warning: instance variable @java_aliases not initialized

Here’s the relevant part of module.rb (line 16 is the last line):

def include_package(package_name)
  if defined? @included_packages
    @included_packages << package_name      
    return
  end
  @included_packages = [package_name]
  @java_aliases = {} unless @java_aliases

So at this point I’m done - I don’t care about that warning message (this sort of assignment is a normal Ruby idiom).

If you see this kind of error:

2008-07-28 10:54:08,747 INFO org.apache.hadoop.mapred.JobTracker: problem cleaning system directory: /home/james/dfsTmp/mapred/system
java.io.IOException: No FileSystem for scheme: http
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1277)
        at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1291)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:203)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:108)
        at org.apache.hadoop.mapred.JobTracker.(JobTracker.java:717)
        at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:141)
        at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:2319)

You’ve probably set this in hadoop-site.xml:

<property>
        <name>fs.default.name</name>
        <value>http://localhost:54310</value>
    </property>

Don’t use http:// in front of localhost - in 0.17.1 it should look like:

<property>
        <name>fs.default.name</name>
        <value>hdfs://localhost:54310</value>
    </property>

Make sure you aren’t trying to track too many changes with the NetBeans mercurial plugin:

NetBeans bug about userdirs and mercurial