Here’s a one-liner to get the current gems, in the form of a gem-install command:

gem list | cut -f 1 -d ' ' | grep -v -e '\*\*\*' | xargs echo sudo gem install

If you’re including a jar, and you see errors like this when you try to use something defined in that jar file:

/home/james/dev/radoop/./test/../lib/radoop.rb:15:in `method_missing': cannot link Java class org.apache.hadoop.mapred.TextInputFormat (NameError)

Try running with jruby -d and you might get a more informative message:

Script started on Tue 29 Jul 2008 10:54:05 AM PDT
java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory
        at org.apache.hadoop.mapred.FileInputFormat.<clinit>(FileInputFormat.java:49)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:247)

In this case, I needed to include commons-logging-1.0.4.jar and commons-logging-api-1.0.4.jar. I added them, and reran, and got this:

java.lang.ExceptionInInitializerError
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:247)
...
Caused by: org.apache.commons.logging.LogConfigurationException: org.apache.commons.logging.LogConfigurationException: No suitable Log constructor [Ljava.lang.Class;@1edf84f for org.apache.commons.logging.impl.Log4JLogger (Caused by java.lang.NoClassDefFoundError: org/apache/log4j/Category) (Caused by org.apache.commons.logging.LogConfigurationException: No suitable Log constructor [Ljava.lang.Class;@1edf84f for org.apache.commons.logging.impl.Log4JLogger (Caused by java.lang.NoClassDefFoundError: org/apache/log4j/Category))

So I added log4j-1.2.13.jar, and now I’m getting:

/home/james/jruby/lib/ruby/site_ruby/1.8/builtin/javasupport/core_ext/module.rb:16 warning: instance variable @java_aliases not initialized

Here’s the relevant part of module.rb (line 16 is the last line):

def include_package(package_name)
  if defined? @included_packages
    @included_packages << package_name      
    return
  end
  @included_packages = [package_name]
  @java_aliases = {} unless @java_aliases

So at this point I’m done - I don’t care about that warning message (this sort of assignment is a normal Ruby idiom).

If you see this kind of error:

2008-07-28 10:54:08,747 INFO org.apache.hadoop.mapred.JobTracker: problem cleaning system directory: /home/james/dfsTmp/mapred/system
java.io.IOException: No FileSystem for scheme: http
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1277)
        at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1291)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:203)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:108)
        at org.apache.hadoop.mapred.JobTracker.(JobTracker.java:717)
        at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:141)
        at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:2319)

You’ve probably set this in hadoop-site.xml:

<property>
        <name>fs.default.name</name>
        <value>http://localhost:54310</value>
    </property>

Don’t use http:// in front of localhost - in 0.17.1 it should look like:

<property>
        <name>fs.default.name</name>
        <value>hdfs://localhost:54310</value>
    </property>

Make sure you aren’t trying to track too many changes with the NetBeans mercurial plugin:

NetBeans bug about userdirs and mercurial

I was at the Google Seattle Conference on Scalability here in Seattle on Saturday. Jerry Morrison from Google talked about Google Maps on mobile devices, and the thing that stuck out for me was that right now map use on mobile is heavier on the weekends, and the web site is used more during the week.

I’ve switched over to building my own netbeans instead of grabbing it from hudson. Once a day, I’ve got a cron job that does:

First, pull netbeans using mercurial:

hg clone http://hg.netbeans.org/main ~/dev/netbeans

Then set up this script to build it every day:

#!/bin/sh
export JAVA_HOME=/usr/lib/jvm/java-1.5.0-sun-1.5.0.13
export PATH=$JAVA_HOME/bin:$PATH
cd ~/dev/netbeans
hg pull
hg up
ant

In theory, you should be able to build netbeans with version 6, but it dies on me even with the -Dpermit.jdk6.builds=true option set.

I don’t bother to get a reduced version of netbeans with just the Ruby support.

FoxyProxy makes it very easy to talk to your Hadoop cluster running on EC2.

Run ssh with the -D command:

ssh -D 2324 ec2-75-101-XXX-XX.compute-1.amazonaws.com

Tell FoxyProxy to “use SOCKS proxy for DNS lookups” (tools > foxyproxy > more > global settings > use SOCKS proxy for DNS lookups)

Configure foxyproxy with rules for when to use local port 2324. Use wildcards like httpec2internal*.

All the features I cared about worked when set up this way.

(And of course the choice of 2324 isn’t special - use any port you like.)

Here’s a screenshot of the the proxy settings I use. It’s been a while since I configured it - I suspect that at least one of these lines is obsolete:

If you’re trying to build netbeans from the git repository, and you’re on Ubuntu 7.10, you’ll probably get:

    james@madra:~/dev/netbeans$ ant
    Buildfile: build.xml

    BUILD FAILED
    /home/james/dev/netbeans/build.xml:45: The following error occurred while executing this line:
    /home/james/dev/netbeans/nbbuild/build.xml:72: No supported regular expression matcher found: java.lang.ClassNotFoundException: org.apache.tools.ant.util.regexp.Jdk14RegexpMatcher

    Total time: 0 seconds

To fix, just install ant-optional:

sudo apt-get install ant-optional

You’ll find building instructions here.

The most interesting thing I saw yesterday was John Lam and Jimmy Schementi’s presentation on IronRuby. They’ve got Ruby running on the client in Silverlight - they’re calling the combination Silverline. Assuming that it’s going to work just fine in Moonlight, this is hot - no more switching back and forth between Ruby and Javascript.

Avi Bryant’s work on Ruby with the GemStone VM was also interesting. If the performance numbers GemStone talked about turn out to be true in the final version, this could be a very fast Ruby. They’ve got some interesting technology around distributed objects (think of a cluster of machines as being a single, shared object space, with transactions on changes to your objects), but it’s a whole different way of looking at the world. Worth watching. Talking to some of their people at their party, some of them get that moving to an open source model is going to be a requirement for them to be taken seriously - hopefully they’ll solve their issues of how to change their business model.

The (paper) message boards downstairs are covered with companies looking for Ruby people. It’s a great time to be looking for work. My favorite was the one from square circle triangle - it’s a faux ticket to Australia.

If you've got an array that contains objects with nested class names like this:

module Foo
  class Bar
    def to_xml options = {} # :nodoc:
      builder = options[:builder] || Builder::XmlMarkup.new(options)
      builder.some_tag "some value"
    end
  end
end

def test_array_with_to_xml
  f = Foo::Bar.new
  puts [f].to_xml
end

test_array_with_to_xml

You'll end up with invalid xml:

<?xml version="1.0" encoding="UTF-8"?>
<foo/bars type="array">
  <some_tag>some value</some_tag>
</foo/bars>

That's because the array #to_xml (it's in /usr/local/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/activesupport/coreext/array/conversions.rb on my machine) looks at the class of every object it contains, and then just calls #underscore and #pluralize on it like so:

options[:root]     ||= all? { |e| e.is_a?(first.class) && first.class.to_s != "Hash" } ? first.class.to_s.underscore.pluralize : "records"

That doesn't work if the class name is nested.

The workaround is to specify the root:

puts [f].to_xml(:root => "bars")

Status codes

Curious about how Rails deals with HTTP status codes? Take a look at statuscodes.rb (on my system, it's in /usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.1/lib/actioncontroller/status_codes.rb). When you do a

render :status => :reset_content

It's just looking up the string "reset content" in it's hash of codes-to-error messages. The error message strings are converted into stringswithunderscores, so "Reset Content" ends up as match for :reset_content.

Exceptions

In rescue.rb (on my system, /usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.1/lib/action_controller/rescue.rb), you'll see where some of the Rails exceptions are turned into status codes for you.

This is where you'll see things like ActiveRecord::RecordNotFound turning into a :not_found, which finally turns into a 404.

Controller status codes

In addition to the status codes for exceptions, Rails uses these status codes when it generates a controller for you:

• :created (201) when #create is successful
• :unprocessable_entity (422) when #create or #update fails

Sucessfull methods return :ok (200) otherwise.

The Rails code

status_codes.rb

module ActionController
  module StatusCodes #:nodoc:
    # Defines the standard HTTP status codes, by integer, with their
    # corresponding default message texts.
    # Source: http://www.iana.org/assignments/http-status-codes
    STATUS_CODES = {
      100 => "Continue",
      101 => "Switching Protocols",
      102 => "Processing",

      200 => "OK",
      201 => "Created",
      202 => "Accepted",
      203 => "Non-Authoritative Information",
      204 => "No Content",
      205 => "Reset Content",
      206 => "Partial Content",
      207 => "Multi-Status",
      226 => "IM Used",

      300 => "Multiple Choices",
      301 => "Moved Permanently",
      302 => "Found",
      303 => "See Other",
      304 => "Not Modified",
      305 => "Use Proxy",
      307 => "Temporary Redirect",

      400 => "Bad Request",
      401 => "Unauthorized",
      402 => "Payment Required",
      403 => "Forbidden",
      404 => "Not Found",
      405 => "Method Not Allowed",
      406 => "Not Acceptable",
      407 => "Proxy Authentication Required",
      408 => "Request Timeout",
      409 => "Conflict",
      410 => "Gone",
      411 => "Length Required",
      412 => "Precondition Failed",
      413 => "Request Entity Too Large",
      414 => "Request-URI Too Long",
      415 => "Unsupported Media Type",
      416 => "Requested Range Not Satisfiable",
      417 => "Expectation Failed",
      422 => "Unprocessable Entity",
      423 => "Locked",
      424 => "Failed Dependency",
      426 => "Upgrade Required",

      500 => "Internal Server Error",
      501 => "Not Implemented",
      502 => "Bad Gateway",
      503 => "Service Unavailable",
      504 => "Gateway Timeout",
      505 => "HTTP Version Not Supported",
      507 => "Insufficient Storage",
      510 => "Not Extended"
    }

    # Provides a symbol-to-fixnum lookup for converting a symbol (like
    # :created or :not_implemented) into its corresponding HTTP status
    # code (like 200 or 501).
    SYMBOL_TO_STATUS_CODE = STATUS_CODES.inject({}) do |hash, (code, message)|
      hash[message.gsub(/ /, "").underscore.to_sym] = code
      hash
    end

    # Given a status parameter, determine whether it needs to be converted
    # to a string. If it is a fixnum, use the STATUS_CODES hash to lookup
    # the default message. If it is a symbol, use the SYMBOL_TO_STATUS_CODE
    # hash to convert it.
    def interpret_status(status)
      case status
      when Fixnum then
        "#{status} #{STATUS_CODES[status]}".strip
      when Symbol then
        interpret_status(SYMBOL_TO_STATUS_CODE[status] ||
          "500 Unknown Status #{status.inspect}")
      else
        status.to_s
      end
    end
    private :interpret_status

  end
end

rescue.rb (excerpt)

DEFAULT_RESCUE_RESPONSE = :internal_server_error
DEFAULT_RESCUE_RESPONSES = {
  'ActionController::RoutingError'             => :not_found,
  'ActionController::UnknownAction'            => :not_found,
  'ActiveRecord::RecordNotFound'               => :not_found,
  'ActiveRecord::StaleObjectError'             => :conflict,
  'ActiveRecord::RecordInvalid'                => :unprocessable_entity,
  'ActiveRecord::RecordNotSaved'               => :unprocessable_entity,
  'ActionController::MethodNotAllowed'         => :method_not_allowed,
  'ActionController::NotImplemented'           => :not_implemented,
  'ActionController::InvalidAuthenticityToken' => :unprocessable_entity
}

DEFAULT_RESCUE_TEMPLATE = 'diagnostics'
DEFAULT_RESCUE_TEMPLATES = {
  'ActionController::MissingTemplate' => 'missing_template',
  'ActionController::RoutingError'    => 'routing_error',
  'ActionController::UnknownAction'   => 'unknown_action',
  'ActionView::TemplateError'         => 'template_error'
}

vlad will create mongrel configuration files for you, but you're probably going to need to set the right mongrel variables.

For example, I have a demo that looks like:

set :mongrel_port, 4001
set :mongrel_servers, 3
set :mongrel_user, :restphone

I'm not a big fan of line-oriented interfaces, so when I found the new (well, new-ish) Asterisk AJAX Manager Interface, AKA "AJAM," I figured it was a better fit for Ruby than the original.

My needs right now are very simple - set variables and redirect to a new extension. The variables that are set send Asterisk off to a new RESTPhone menu, where all the heavy lifting is done.

At some point this should become a gem, but here's the current, very simple, incarnation of how to talk to Asterisk via AJAM.

require 'rubygems'
require 'net/http'
require 'uri'
require 'xmlsimple'
require 'cgi'
require File.dirname(__FILE__) + '/rest_phone_utilities/uri_mixin'
require 'pp'
require 'pathname'

class AsteriskAjaxManager
  # The authorization cookie from the manager.
  attr_accessor :auth_cookie

  # The URI to send commands to.
  attr_accessor :uri

  # The Asterisk username
  attr_accessor :username
  # The Asterisk secret
  attr_accessor :secret

  # Takes:
  # 
  #   :uri => The URI for the Asterisk http server, not including /asterisk
  #   :path => The AJAM path (default: '/asterisk')
  #   :username => The Asterisk manager username (default: mark)
  #   :secret => The Asterisk manager password (default: mysecret)
  #   
  # (mark/mysecret is the default user in the Asterisk configuration file
  # samples.)
  def initialize(args = {})
    final_uri = args[:uri]
    final_uri ||= 'http://localhost:8088'
    final_uri = URI.parse(final_uri) unless URI === final_uri
    final_uri.path = '/asterisk' if final_uri.path.empty?
    final_uri.path = '/' if final_uri.path.empty?
    self.uri = final_uri
    self.username = args[:username] || 'mark'
    self.secret = args[:secret] || 'mysecret'
  end  

  def login
    login_uri = uri
    login_uri.path = (Pathname.new(uri.path) + 'manager').to_s
    login_uri.query = "action=login&username=#{username}&secret=#{secret}"

    response = Net::HTTP.get_response(login_uri)

    # The response should include an authorization cookie that we use 
    # on future requests.
    self.auth_cookie = response.response['set-cookie']
  end

  def execute
    login
    yield self
    logout
  end

  def logout

  end

  # ActionQuery is the http query sent off to AJAM.
  class ActionQuery
    attr_accessor :query
    # Takes:
    # 
    # [<tt>:uri</tt>]
    #   The uri (usually 'http://localhost:8088')
    # [<tt>:action</tt>]
    #   The action.  (Case-insensitive string or symbol)
    # [<tt>:params</tt>]
    #   Parameters for the action
    #   
    # Actions are:
    # 
    # status::
    # sippeers::
    def initialize(args = {})
      u = args[:uri]
      u = UriWithQueryCreation.parse(u.to_s)
      u.path = '/asterisk/mxml'
      u.append_hash_to_query 'action' => args[:action]
      u.append_hash_to_query args[:params]
      self.query = u
    end

    def to_s
      query.to_s
    end

    def net_http_obj
      Net::HTTP::Get.new("#{query.path}?#{query.query}")
    end
  end

  def sip_peers
    args = append_standard_arguments(:action => :sippeers)
    result = do_ajam_cmd_return_xml(args)
    result['response'][1..-2].map {|m| m['generic'].first}
  end

  def extension_state args = {}
    args = append_standard_arguments(:action => :extensionstate, :params => args)
    result = do_ajam_cmd_return_xml(args)
    result['response'].first['generic'].first
  end

  def redirect args = {}
    args = append_standard_arguments(
      :action => :redirect,
      :params => args
    )
    result = do_ajam_cmd_return_xml(args)
    result = result['response'].first['generic'].first
    if result['response'] =~ /Error/
      raise AsteriskAjaxManagerError, result['message']
    end
  end

  def setvar args = {}
    args = append_standard_arguments(
      :action => :setvar,
      :params => args
    )
    result = do_ajam_cmd_return_xml(args)
    result = result['response'].first['generic'].first
    if result['response'] =~ /Error/
      raise AsteriskAjaxManagerError, result['message']
    end
  end

  def append_standard_arguments(args = {})
    args.merge(:uri => uri, :cookie => auth_cookie)
  end

  # Takes:
  #   :uri => the uri (usually http://localhost) as a String or URI object
  #   :action => the action (status, SIPpeers, etc) as a symbol
  #   :cookie => the authorization cookie
  def do_ajam_cmd(args = {})
    u = args[:uri]
    u = UriWithQueryCreation.parse(u.to_s)
    result = Net::HTTP.start(u.host, u.port) do |h|
      q = ActionQuery.new :uri => 'http://localhost:8088/asterisk/mxml', :action => args[:action].to_s,
        :params => args[:params]
      req = q.net_http_obj
      req.add_field('Cookie', args[:cookie])
      h.request(req)
    end
    result.body
  end

  def do_ajam_cmd_return_xml(args = {})
    result = do_ajam_cmd(args)
    XmlSimple.xml_in(result)
  end

  class AsteriskAjaxManagerError < RuntimeError
    class LoginError < AsteriskAjaxManagerError
    end
  end
end

If you're using the Asterisk manager REDIRECT command, and your calls spend a lot of their time in macros, you'll probably need to change MACRO_RECURSION. I was seeing messages like this:

[Dec  9 14:23:29] ERROR[19509]: app_macro.c:193 _macro_exec: Macro():  possible infinite loop detected.  Returning early.

The macro code is deciding that's it too deep, since the REDIRECT just pushes another level onto the stack.

I'm working around this by just making the MACRO_RECURSION large in my extensions.ael file:

s = {
  MACRO_RECURSION=10000;
  ...

Which, according to "dialplan show" turns into the old style:

Set(MACRO_RECURSION=$[10000])

Note about adding the variable

bld=`wget -q -O - 'http://deadlock.netbeans.org/hudson/job/ruby/lastSuccessfulBuild/buildNumber'`
wget -P /home/james/Desktop "http://deadlock.netbeans.org/hudson/job/ruby/lastSuccessfulBuild/artifact/ruby/rubyide/dist/netbeans-rubyide-hudson-$bld.zip"
cd /home/james/Desktop
rm -rf nbrubyide
unzip "netbeans-rubyide-hudson-$bld.zip"
newdir=nb$bld
mkdir $newdir
mv nbrubyide/* $newdir