Subscribe:

Pages

Using LibSVM from Ruby with data in files

In previous posts, I looked at RubySVM and how to set up models with data held within the memory of a running Ruby process. I have found this does not scale up to larger datasets of images, taking up to a gigabyte in storage on disk, and so I have adopted a different approach, constructing training/test files on disk and then calling the command line svm-train and svm-predict functions. Ruby then drives the whole process, automating the construction of the datasets, selection of the best models, and final evaluation. I use the UCI Letter Recognition dataset discussed earlier as an example, but the approach is fully general.

Calling System Commands

The first step in controlling external programs is calling them, and for this Ruby's backtick syntax is ideal. `ls` will call the ls command, returning the result as a string, which we can store with result = `ls`: result.split("\n") then produces an array of filenames in the current directory.

This backtick notation supports the interpolation of arbitrary Ruby code, just like in
strings. Thus, writing `ls *.#{extension}` will use the value of extension to list file of the given extension only.

Try it from irb:

peter@peter-desktop:~/Code$ irb
irb(main):001:0> `ls`
=> "instances.html\njruby.jar\ntest-instances.rb\ntrial-kmeans.rb\nweather.arff\nweka.jar\n"
irb(main):002:0> result = `ls`
=> "instances.html\njruby.jar\ntest-instances.rb\ntrial-kmeans.rb\nweather.arff\nweka.jar\n"
irb(main):003:0> result.split("\n")
=> ["instances.html", "jruby.jar", "test-instances.rb", "trial-kmeans.rb", "weather.arff", "weka.jar"]
irb(main):004:0> extension = "rb"
=> "rb"
irb(main):005:0> `ls *.#{extension}`.split("\n")
=> ["test-instances.rb", "trial-kmeans.rb"]

Constructing the Datasets

The data is read in, just like I did before, creating an array @@data containing instances of the LetterInstance datatype. I then use input parameters to partition the dataset into training, cross-validation and test datasets:

# Read in letter-recognition UCI data

LetterInstance = Struct.new(:letter, :x_box, :y_box, :width, :height, 
                            :on_pixels, :x_bar, :y_bar, :x2bar, :y2bar, 
                            :xybar, :x2ybar, :xy2bar, :x_ege, :xegvy, 
                            :y_ege, :yegvx)

@@data = []

File.open("letter-recognition.data", "r") do |f|
  f.each_line do |line|
    @@data << LetterInstance.new(*line.strip.split(","))
  end
end

# -- using given proportion for training/cross-validation sets, create three files

if ARGV.length != 2
  puts "To use: ruby train-models.rb TRAINING_PROPORTION CROSS_PROPORTION"
  exit
end

@@training_proportion = ARGV[0].to_f
@@cross_validation_proportion = ARGV[1].to_f

@@num_training = (@@data.length * @@training_proportion).floor
@@num_cross = (@@num_training * @@cross_validation_proportion).floor

# Select Train/Cross/Test sets
# -- note, cross-validation size is taken out of the training-set

@@data.shuffle!

TrainSet = @@data[@@num_cross...@@num_training]
CrossSet = @@data[0...@@num_cross]
TestSet = @@data[@@num_training..-1]

The three datasets must now be saved onto disk, to form the datasets to pass to the SVM. The file format for an SVM places each instance onto a single line in the file. Each instance has its class first, and then its attribute values. Every item is separated by a space. The class must be a number, and the attribute values should be in the range [0,1] or [-1,1]. The attribute-values are written by writing the index of the attribute, followed by a colon, and then the value. An example from the training file is:

17 0:0.466666666666667 1:0.6 2:0.666666666666667 3:0.533333333333333 ....

The advantage of this notation is that it supports sparse data - any attribute that has value 0 can simply be ignored.

The following code constructs the required data:

# Create SVM format of features in instance, rescaling to [0,1]
def make_data item
  str = ""
  (item.length-1).times do |i|
    str << "#{i}:#{item[i+1].to_f/15.0} "
  end
  str
end

def save_file(dataset, filename)
  File.open(filename, "w") do |f|
    dataset.each_with_index do |item, index|
      f.puts "#{item.letter.ord - 65} #{make_data(item)}"
    end
  end
end

save_file(TrainSet, "training-data.dat")
save_file(CrossSet, "cross-data.dat")
save_file(TestSet, "test-data.dat")

Training the Models

I create each model using svm-train, supplying it a training data set and set of parameters. As I am creating SVMs with an RBF kernel, there are two parameters that particularly interest me: cost and gamma. I will save a model file for each pair of parameters, embedding the parameter value in the name. This is all done by string interpolation in the backtick call to svm-train. 

# -- train models using Cost/Gamma values and external system
Costs = [2**-5,2**-3,2**-1,1,2,8,2**5,2**8,2**10,2**13,2**15]
Gammas = [2**-15,2**-12,2**-8,2**-5,2**-3,2**-1,2,2**3,2**5,2**7,2**9]

Costs.each do |cost|
  Gammas.each do |gamma|
    `svm-train -s 1 -t 2 -d 3 -r 0 -e 0.001 -c #{cost} -g #{gamma} training-data.dat l-#{cost.to_f}-#{gamma.to_f}.model`
  end
end

Evaluating a Model

svm-predict is the command-line tool for evaluating a model on a given dataset. It produces two sets of outputs. A file containing the actual predictions of the model for each instance in the dataset, and a printed output of the model's accuracy. This accuracy is a percentage correct. We can retrieve it using a regular expression match on the result of running svm-predict.

Our models are defined by their cost and gamma values, using the format above. Given the filename of a dataset to test on, the following function will run svm-predict and parse its output to produce the model's accuracy. The actual predictions are sent to the same file - I don't want to keep them.

# Test the model on given dataset, and parse output to get accuracy
def get_accuracy(cost, gamma, dataset)
  result = `svm-predict #{dataset} l-#{cost.to_f}-#{gamma.to_f}.model test.out`
  result.match(/\d+.\d+/)[0].to_f
end

Selecting a Model and Final Evaluation

Having trained models for every combination of cost and gamma, we need to find the one that does best on the cross-validation dataset, and then evaluate it on the test set. This is straightforward:

# Search for best cost/gamma pairing, based on result on cross-validation dataset
@@best_cost = Costs[0]
@@best_gamma = Gammas[0]
@@best_accuracy = 0.0
Costs.each do |cost|
  Gammas.each do |gamma|
    accuracy = get_accuracy(cost, gamma, "cross-data.dat")
    if accuracy > @@best_accuracy
      @@best_accuracy, @@best_cost, @@best_gamma = accuracy, cost, gamma
    end
  end
end

puts "Best cost #{@@best_cost}, Best gamma #{@@best_gamma}"
puts "Test score is: #{get_accuracy(@@best_cost, @@best_gamma, "test-data.dat")}"


Summary

Running the letter-recognition task with 2000 training examples using files, instead of in memory, took 11 minutes in total, instead of the 35 minutes my in-memory solution required. So the saving in time is considerable.

Accuracy with 2000 training examples was 82%.

I also ran the system using 72% of the data (14,400 instances) for training, 8% (1600 instances) as cross-validation, and the remaining 20% (4000 instances) as test data. I searched across 121 combinations of Cost/Gamma pairs using an RBF kernel. The best fit had a cost of 1/32, gamma of 32, and produced an accuracy of 97.5%. This took 6 hours to run.
Labels: ,