sqlup - how to back up your EC2 MySQL server to S3

Posted on June 23, 2007

I wanted a backup system for MySQL that could do a few things:

  • A single step backup of all of the data files, including all of the files for both MyISAM and InnoDB databases.
  • Backups of the completed transaction logs.
  • Continuous backups of the current transaction log.

For the first, you can’t just tar up the files of a running MySQL database. The problem is coherency; while you’re tarring up the files, MySQL is continuing to write to them. You’re not going to end up with something as useful as a backup. So, you need to do you tar in the contex t of a “FLUSH TABLES WITH READ LOCK” statement.

The second is pretty simple. Once MySQL has finished reading to a log file, it’s safe to just copy that file.

The third is a little trickier. It’s OK to read the file, but you’re not guaranteed that anything past the latest transaction is going to be valid. It’s possible that your read is going to start when the next transaction is only partially written, so you need to be able to record the point in the file that marks where your valid transactions stop.

So, here’s “sqlup” –a utility that does all of these things for you, and then writes the files up to S3. It’s available as a ruby gem:

gem install sqlup

You can get general help with:

sqlup help

Or -h gives you the command-line options

sqlup -h

How to use class methods to simplify your models

Posted on June 14, 2007

The short answer

Use class methods, and call them on your collection objects.

Given Customer has_many Orders, and Orders has_one Payments, add class methods to your Order model:

class Order < ActiveRecord::Base
  has_one :payment
  belongs_to :customer

  def self.paid
    find :all, :include => :payment, :conditions => ['payments.order_id']
  end

  # Returns the unpaid orders.  See <tt>paid</tt> for more information.
  def self.unpaid
    find :all, :include => :payment, :conditions => ['payments.order_id is null']
  end
end

You can call these class methods through your collection methods:

c = Customer.find(12)
c.orders # This is the collection method added by has_many :orders
c.orders.paid 
c.orders.unpaid

The long answer

# Our simple example is for a Customer
# that has_many Orders, and each Order
# has_one Payment.
class Customer < ActiveRecord::Base
  # See <tt>Customer::paid</tt> for how to get
  # all of the paid and unpayed orders.
  has_many :orders
end

class Order < ActiveRecord::Base
  has_one :payment
  belongs_to :customer

  # Returns all the paid orders.  A paid order
  # is any order that has a corresponding Payment
  # object.
  #
  # You can use this after a collection method.  In
  # this example, Customer has_many Orders, and Orders
  # has_one Payment, so you can do:
  #   
  #   c = Customer.find(12)
  #   c.orders # This is the collection method added by has_many :orders
  #   c.orders.paid 
  #   c.orders.unpaid
  #
  # The key thing to notice here is that this isn't a 
  # regular method.  It's a method of the class itself,
  # and Rails lets you call these class methods on the 
  # collection object.
  #
  # Here's another way to think about it: collection objects
  # don't return a set of objects.  This line:
  #
  #   c.orders
  # 
  # doesn't return an array of orders.  What it returns is
  # an "association proxy."  The proxy knows how you've called
  # it, and if you call it by itself, it'll just return
  # an array of the objects you're looking for.
  #
  # If you call it with something else attached to it, like:
  # 
  #   c.order.paid
  # 
  # what happens is a little different.  
  # 
  # <tt>c.order</tt> returns
  # the association proxy, and then the association proxy is
  # sent the :paid message.  The association proxy doesn't have
  # a method called :paid defined, so #method_missing is called.
  # The association proxy does know what kind of objects it holds
  # (Orders) so it asks the Order class if the Order class can
  # respond to the :paid message.  Order::paid does exist, since
  # we defined it as a method on the Order class.  The association
  # proxy starts to build a query using #with_scope, so the 
  # class method that looks like it would just return every unpaid
  # order only returns the unpaid orders that are in the right scope.
  # In this case, the scope is Orders that have the right
  # customer_id.
  def self.paid
    find :all, :include => :payment, :conditions => ['payments.order_id']
  end

  # Returns the unpaid orders.  See <tt>paid</tt> for more information.
  def self.unpaid
    find :all, :include => :payment, :conditions => ['payments.order_id is null']
  end
end

# In a real payment, you'd want to include more information.
# For this example, the fact that a payment exists means
# that the Order is paid.
class Payment < ActiveRecord::Base
  belongs_to :order
end

The trick is that methods on the class itself can be used in finders.

Useful links

Posted on June 03, 2007

What I'm looking at

Mac admin

Glassfish and JRuby

Amazon's S3 storage service

Deployment (Capistrano, Rake, Mongrel, Monit)

Rubyforge

Markdown

MySQL

Testing

Asterisk

Apache