Perspectives : Writings from The Killswitch Collective

DataMapper::AbstractAdapter 101

Fri, 27 Feb 2009 12:20:20 -0800

DataMapper is an object relational mapper for ruby with an interface somewhat similar to ActiveRecord's. More than sprinkles on top of a generic SQL adapter, DataMapper is a design pattern for defining repositories and the models that love them. DataMapper differs from ActiveRecord and ActiveResource in that models are encapsulated from repositories while queries and collections communicate between them. One of the most significant advantages to this approach lies in the ability to develop models separately from their repository.

Over the next several articles we will use the Twitter API to explore how DataMapper expects custom adapters to work, executes CRUD requests, handles associations and works with multiple repositories. For now we will concentrate on the basics of DataMapper and querying users from the Twitter API service. Those comfortable with ruby or ActiveRecord should be able to follow along, however I strongly recommend spending time with DataMapper's fantastic documentation if you have not already.

Modeling Our Models

First things first, we should install DataMapper and define a model to represent a user account from the Twitter API. To do that we need to define a module using the DataMapper::Resource module and describe the properties Twitter provides.

gem install datamapper

require 'dm-core'

class User
  include DataMapper::Resource

  property :id, Integer, :field => 'user_id'
  property :name, String
  property :screen_name, String
  property :email, String
  property :location, String
  property :description, Text, :lazy => false
  property :profile_image_url, String
  property :url, String
  property :protected, Boolean
  property :followers_count, Integer
end

DataMapper differs from the ActiveRecord family in that fields are defined in your model rather than being created from the repository's schema. This allows models to be built without an adapter making a connection and avoids the headaches of ActiveRecord-style migrations. Additionally you may specify the :field name to be used for each property, allowing antiquated or confusing field names to be user friendly. Defining a model's properties also allows DataMapper to intelligently calculate by type which fields should be lazily loaded, which may also be customized by passing the :lazy option a true or false value. In our User model we have set the lazy option to false as Twitter provides a user's description by default, and there is no use in wasting an API hit if we do not need to (Twitter limits API calls by the hour). A more in-depth description of DataMapper's lazy fields can be found in the documentation.

The Base Adapter

For the foundation of our Twitter adapter we need to catch any authentication options passed to the initialization as well as write a method for communicating with Twitter.

require 'cgi'
require 'open-uri'
require 'rubygems'
require 'dm-core'
require 'xmlsimple'

module DataMapper
  module Adapters
    class TwitterAdapter

      # Clients can provide DataMapper with a URI string or hash of options when
      # initializing an adapter. We can store these values and use them for each
      # request to the Twitter service if the client provides them. Depending on
      # your repository you may wish to verify authentication here rather than 
      # waiting for the initial request.
      #
      # name:: Name of the adapter
      # uri_or_options:: A uri string, or hash of options used to initialize the adapter
      #
      def initialize(name, uri_or_options)
        # don't forget to phone home!
        super(name, uri_or_options)

        case uri_or_options
        when Hash
          user = uri_or_options[:user] || ''
          pass = uri_or_options[:pass] || ''
          @auth = user.blank? || pass.blank? ? nil : [user, path]
        end
      end

      private

      # Requests a resource from the Twitter API. If the adapter was initialized
      # with the :user and :pass options, they will be used to authenticate the request.
      #
      # method:: Path to follow the base URI 'http://twitter.com'
      # params:: Hash of key/value pairs to be used as the query string
      # returns:: XmlSimple representation of the response from Twitter
      #
      def request(method, params = {})
        uri = "http://twitter.com/#{method}"
        options = {:http_basic_authentication => @auth}

        unless params.blank?
          query = params.map { |k,v| "%s=%s" % [CGI.escape(k), CGI.escape(v)] }
          uri << "?#{query.join('&')}"
        end

        result = open(uri, options)
        return XmlSimple.xml_in(result.read, {'ForceArray' => false})
      end

    end
  end
end

Now we can go ahead and see if we are on the right path, we should get a NotImplementedError when we try to perform any action with our repository.

DataMapper.setup(:default => {
  :adapter => 'twitter',
  :user => 'kscollective',
  :pass => 'snark snark'
})

User.first(:screen_name => 'kscollective') # => NotImplementedError

Fetching Heffalumps And Woozles

In order to continue building our adapter we have to be able to understand the queries and collections DataMapper uses to mediate between the models and adapters. DataMapper::AbstractAdapter defines #read_one and #read_many, both of which accept the query as the single parameter. The query object allows us to determine which model and fields to query along with any possible conditions to limit our results by. Queries also tell our adapter about any possible offsets, limits or ordering, but we will come back to that another day.

Query#model

Each query belongs to a model which we can use to load and return instances fetched from our repository, and may also be used to customize repository requests by type. We can even use the model to DRY up our adapter and work with a single #read method.

class << TwitterAdapter

  def read_one(query)
    read(query, query.model, false)
  end

  def read_many(query)
    Collection.new(query) do |set|
                  read(query, set, true)
    end
  end

  private

  # Each read has a query and returns a set, #read_one and #read_many should provide
  # the set to load the results into. When called by #read_many nothing needs to be returned
  # as the collection is filling itself, however we must return what ever object #read_one 
  # should return back to the client code.
  #
  def read(query, set, many = true)
    raise NotImplementedError # to be filled in later
  end

end

Query#fields

Each query contains a subset of the model's properties specifying which fields to query as well as the order of our values when instantiating each result. Twitter does not provide a means to filter out fields so we will use #fields only to order the values we pass to the model builders.

class << TwitterAdapter
  
  # Map the values from item into an array of the same size and order as Query#fields
  # [id, name, something_else, title] => [1, 'Hello World', nil, 'Cats!']
  #
  def parse_user_values(query, item)
    return query.fields.map { |f| item[f.field.to_s] }
  end
end

Query#conditions

The meat of most queries, conditions is an array of tuples containing the operator, property and value to be considered when executing the request. Each operator may be any of the standard 'SQL' operators (:eql, :in, :gt[e], :lt[e]) and should be used to match the property with one or more values. Because Twitter only provides a method to query individual users by id, email or screen name we can write a method to create an array of key/value pair queries.

class << TwitterAdapter

  def generate_users_query(query)
    result = Array.new
    fields = ['user_id', 'email', 'screen_name']
    
    conditions = query.conditions.select do |condition|
      condition[0] == :eql and fields.include?(condition[1].field)
    end
    
    # each item in conditions is a [operator, property, value] tuple
    for operator, property, value in conditions
      # if an array, each value must be queried individually
      [value].flatten.each { |v| result << [property.field, v] }
    end
    
    return result
  end

end

Mind The Gap!

What used to be the hardest part in using third party resources has now become a matter of building the request, parsing the request and loading the model.

def read(query, set, many = true)
  queries = generate_user_queries(query)
  
  for key, value in queries
    twitter_user = request("users/show.xml", {key => value})
    next if twitter_user.blank? or twitter_user['screen_name'].blank?
    user_values = parse_user_values(query, item)
    many ? set.load(user_values) : (break set.load(user_values, query))
  end
  
  return
end

Recess!

At this point you should be able to query users from Twitter without parsing a single query string or XML response.

DataMapper.setup(:default => {:adapter => 'Twitter'})

user = User.first(:screen_name => 'KSCollective')
puts user.url # => http://www.killswitchcollective.com

As you can see DataMapper can make working with third party repositories as automagical as ActiveRecord. DataMapper itself comes with adapters for the major databases with additional adapters available for CouchDB, Google Video and many others. In a few weeks we will continue building our Twitter adapter, leveraging the power of associations. Until then I highly recommend reviewing DataMappers documentation, the adapters included with the dm-core gem and the base adapter we have just built.

Introducing FlexibleCSV

Thu, 19 Feb 2009 12:46:53 -0800

Get FlexibleCSV from GitHub

A Challenge in Flexibility

As part of a contact management system we are building for a client, I encountered a unique challenge with allowing users to upload and import their contacts from CSV files. Usually this would not be a problem, except that in this case there was no standardization to what the header names would be or what order the columns were in. Because the FasterCSV gem relies on using the header names as access keys, this process was suddenly quite complicated.

One solution would be to create a user interface that would display our database fields, their CSV columns and allow them to pair them up. For example, my database column is 'email' but their CSV column is 'Email Address', so they could mark those as equivalent. What would I do, however, for the users who have a "Full Name" column when I use 'first_name' and 'last_name' database columns? Suddenly the user interface could get very complicated and confusing.

Introducing FlexibleCSV

Instead, I developed FlexibleCSV, a gem that allows you to parse through a CSV file without knowing exactly what the headers are named. By providing a list of possible header names, you can access all the CSV columns with a uniform interface.

require 'flexible_csv'

# Arbitrary CSV data
csv_data1 = %Q{Full Name, Email Address\nJohn Doe, john@doe.com}
csv_data2 = %Q{Email, Name\njohn@doe.com, John Doe}

parser = FlexibleCsv.new do |csv|
  csv.column :full_name, "Name", "Full Name", "Client Name"
  csv.column :email, "Email", "Email Address"
end

parser.parse(csv_data1).each do |row|
  puts row.full_name #=> 'John Doe'
  puts row.email     #=> 'john@doe.com'
end

parser.parse(csv_data2).each do |row|
  puts row.full_name #=> 'John Doe'
  puts row.email     #=> 'john@doe.com'
end

Both data sets can now be accessed using the uniform #full_name and #email accessors.

Handling Complexity with Adapters

Going back to my original example, how would we handle CSV files that separated first and last names when my database uses the full name? Or vis versa? Though I considered adding this kind of functionality to the FlexibleCSV gem, ultimately I thought it best to keep that kind of logic in a separate adapter class. For example:

require 'flexible_csv'

# Arbitrary CSV data
csv_data1 = %Q{Full Name\nJohn Doe}
csv_data2 = %Q{First Name, Last Name\nJohn,Doe}

parser = FlexibleCsv.new do |csv|
  csv.column :full_name, "Name", "Full Name", "Client Name"
  csv.column :first_name, "First Name", "First"
  csv.column :last_name, "Last Name", "Last", "Surname"
end

class CsvAdapter
  def initialize(row)
    @row = row
  end

  def full_name
    row.full_name || "#{row.first_name} #{row.last_name}"
  end

  def last_name
    row.last_name || row.full_name.split(' ').last
  end

  def first_name
    row.first_name || row.full_name.split(' ').first
  end

  def method_missing(method_name, *args)
    row.send(method_name, *args)
  end
end

parser.parse(csv_data1).each do |row|
  ad_row = CsvAdapter.new(row)
  puts ad_row.full_name  #=> 'John Doe'
  puts ad_row.first_name #=> 'John'
  puts ad_row.last_name  #=> 'Doe'
end

parser.parse(csv_data2).each do |row|
  ad_row = CsvAdapter.new(row)
  puts ad_row.full_name  #=> 'John Doe'
  puts ad_row.first_name #=> 'John'
  puts ad_row.last_name  #=> 'Doe'
end

Using the adapter class, we can once again access each row of data from any CSV file with a uniform interface.

Go Get It!

To use the FlexibleCSV gem, you can follow or fork the project on GitHub or just install the gem:

sudo gem install chrisjpowers-flexible_csv

Transparency in the Social Realm

Tue, 10 Feb 2009 06:52:12 -0800

There is a universal rule that people, companies and organizations must adhere to when participating in the Social Media Realm: Remain Transparent!

All too often, those trying to maximize the all-too-appealing benefits of conducting social campaigns forget the transparency tenet. Truthfully, I understand the temptation, especially for marketers, advertisers and PR professionals. People tend to respond more favorably to communications that they believe are altruistic or sincere, communications that do not possess ulterior motives.

But transparency goes both ways and applies, if not even more so, to those who maintain and run these sites. Again, I understand the temptation. These sites are cash cows (Facebook's total implied value is around $15 billion). While just about every industry and company is struggling in these financial times, the social media companies that run sites like Facebook and Hulu continue to add more users each month, which in turn means larger advertising premiums. However, the decision makers of these sites must be wary that the monetization of the communities they've created doesn't mitigate the user's trust.

Case in point, let's look at what Hulu recently did to assuage user rage over the removal of a popular TV show on the site. Hulu, at the request of the content provider, FX Network, removed "It's Always Sunny In Philadelphia" from the site's downloads, and it did so without any form of notification to its users. "Sunny" was one of the most popular shows on Hulu and its many fans began emailing, posting and tweeting away in protest. Below is the letter posted by Jason Kilar, CEO of Hulu, to the site's users.

On January 9, we removed nearly 3 seasons of full episodes of "It's Always Sunny in Philadelphia." We did this at the request of the content owner. Despite Hulu's opinion and position on such content removals (which we share liberally with all of our content partners), these things do happen and will continue to happen on the Hulu service with regards to some television series. As power users of Hulu have seen, we've added a large amount of content to the library each month, and every once in a while we are required to remove some content as well.

This note, however, is not about the fact that episodes of "It's Always Sunny in Philadelphia" were taken down. Rather, this note is to communicate to our users that we screwed up royally with regards to how we handled this specific content removal and to apologize for our lack of strong execution. We gave effectively no notice to our users that these "Sunny" episodes would be coming off the service. We handled this in precisely the opposite way that we should have. We believe that our users deserve the decency of a reasonable warning before content is taken down from the Hulu service. Please accept our apologies.

Given the very reasonable user feedback that we have received on this topic (we read every twitter, email and post), we have just re-posted all of the episodes that we had previously removed. I'd like to point out to our users that the content owner in this case - FX Networks - was very quick to say yes to our request to give users reasonable advance notice here, despite the fact that it was the Hulu team that dropped the ball. We have re-posted all of the episodes in the interest of giving people advance notice before the episodes will be taken down two weeks from today. The episodes will be taken down on January 25, 2009. Unfortunately we do not have the permission to keep the specific episodes up on Hulu beyond that. We hope that the additional two weeks of availability will help to address some of the frustration that was felt over the past few days.

The team at Hulu is doing our best to make lemonade out of lemons on this one, but it's not easy given how poorly we executed here. Please know that we will do our best to learn from this mistake such that the Hulu user experience benefits in other ways down the road.

Sincerely,

Jason Kilar, CEO, Hulu

This is a textbook example of how to diffuse a social uprising. Kilar directly addressed Hulu's users, accepted blame for the way the situation was handled, offered a remedy and then promised to learn from this mistake (I may just disseminate this letter to politicians).

Think about this example when your company or organization decides to go Social. Remember that the user is wary and easily slighted. Remain transparent and reap the rewards!

Thanks to Muhammad Saleem (@msaleem on Twitter) for writing a great article on this subject.