Working with these datasets is a useful exercise to learn about different machine-learning applications, and particularly for exploring the capabilities of learning algorithms with which you are unfamiliar. You can compare your own experimental results with those of previous researchers, and make sure you are using the algorithms correctly. Also, if and when you create your own learning algorithm, you can evaluate its performance on these standard datasets, and hence compare it with previous algorithms.
However, the format of the datasets is not consistent, although most of the data can be viewed in standard text editors. Before we can begin exploring these datasets within a toolkit such as WEKA, we must convert the data into a format that WEKA can understand. One way would be to write a tool to convert the original data format into WEKA's ARFF file format, and then work with the graphical tool. A different way, which follows on from my previous post, is to use jRuby to talk directly to WEKA; jRuby can then be used to read in and analyse the custom data format, before developing and evaluating machine-learning algorithms through the WEKA API.
Reading UCI Datasets
Using WEKA with files like the UCI datasets means converting an external data format into the internal format used within WEKA. As I covered in my previous post, WEKA instances must be constructed using a set of Attribute instances, where each Attribute describes its name, type (numeric or nominal) and the range of values it can hold, if nominal. Setting up these attributes is the tedious step. You have to look in the description of the data, and extract the attributes names, type and values, then code it into Ruby. The following was created using information from the 'names' file of the Mushroom dataset, to create an array of Attributes and a Label attribute:
# Convert provided arguments into a FastVector
# -- use mainly in making nominal attributes
def make_vector *args
atts = FastVector.new(args.length)
args.each {|arg| atts.addElement arg}
return atts
end
# == Create the attributes
Attributes = [
Attribute.new("cap-shape", make_vector("b", "c", "x", "f", "k", "s")),
Attribute.new("cap_surface", make_vector("f", "g", "y", "s")),
Attribute.new("cap_colour", make_vector("n", "b", "c", "g", "r", "p", "u", "e", "w", "y")),
Attribute.new("bruises?", make_vector("t", "f")),
Attribute.new("odour", make_vector("a", "l", "c", "y", "f", "m", "n", "p", "s")),
Attribute.new("gill-attachment", make_vector("a", "d", "f", "n")),
Attribute.new("gill-spacing", make_vector("c", "w", "d")),
Attribute.new("gill-size", make_vector("b", "n")),
Attribute.new("gill-colour", make_vector("k", "n", "b", "h", "g", "r", "o", "p", "u", "e", "w", "y")),
Attribute.new("stalk-shape", make_vector("e", "t")),
Attribute.new("stalk-root", make_vector("b", "c", "u", "e", "z", "r", "?")),
Attribute.new("stalk-surface-above-ring", make_vector("f", "y", "k", "s")),
Attribute.new("stalk-surface-below-ring", make_vector("f", "y", "k", "s")),
Attribute.new("stalk-colour-above-ring", make_vector("n", "b", "c", "g", "o", "p", "e", "w", "y")),
Attribute.new("stalk-colour-below-ring", make_vector("n", "b", "c", "g", "o", "p", "e", "w", "y")),
Attribute.new("veil-type", make_vector("p", "u")),
Attribute.new("veil-colour", make_vector("n", "o", "w", "y")),
Attribute.new("ring-number", make_vector("n", "o", "t")),
Attribute.new("ring-type", make_vector("c", "e", "f", "l", "n", "p", "s", "z")),
Attribute.new("spore-print-colour", make_vector("k", "n", "b", "h", "r", "o", "u", "w", "y")),
Attribute.new("population", make_vector("a", "c", "n", "s", "v", "y")),
Attribute.new("habitat", make_vector("g", "l", "m", "p", "u", "w", "d"))
]
Label = Attribute.new("class", make_vector("p", "e"))
The 'data' file of the Mushroom dataset contains lines of characters, in the following format:
p,x,s,n,t,p,f,c,n,k,e,e,s,s,w,w,p,w,o,p,k,s,u
e,x,s,y,t,a,f,c,b,k,e,c,s,s,w,w,p,w,o,p,n,n,g
e,b,s,w,t,l,f,c,b,n,e,c,s,s,w,w,p,w,o,p,n,n,m
p,x,y,w,t,p,f,c,n,n,e,e,s,s,w,w,p,w,o,p,k,s,u
e,x,s,g,f,n,f,w,b,k,t,e,s,s,w,w,p,w,o,e,n,a,g
Each line represents one instance, which later we will need to convert into a WEKA instance. The trick with the WEKA instances is that each must be linked to the dataset in which it lies. When I load the data, I don't know which dataset I will want each line to be added to. So, I construct instances of the following class as I read in the data. The class has a method which will build an instance and put it in the appropriate dataset, when called.
class UciInstance
attr_reader :features, :label
def initialize(features, class_label)
@features = features
@label = class_label
end
# Convert raw data into WEKA Instance and add to dataset
# -- assumes features.length == Attributes.length
def make_instance dataset
instance = Instance.new(Attributes.length + 1)
Attributes.length.times do |i|
instance.setValue(Attributes[i], features[i])
end
instance.setValue(Label, label)
instance.setDataset dataset
dataset.add instance
end
end
# -- routine to read in a UCI dataset, and return a list of UciInstances
# -- modify the parts to extract the features and class label, based on
# input data format
def read_csv filename
data = []
File.open(filename, "r") do |f|
f.each_line do |line|
items = line.rstrip.split(",")
features = items[1, items.length-1]
class_label = items[0]
data << UciInstance.new(features, class_label)
end
end
return data
end
DataSet = read_csv("agaricus-lepiota.data")
The above code will produce an array, DataSet, containing the items from the original dataset.
Comparing Algorithms
WEKA contains a huge number of algorithms. Just to name two, there is J48 for building decision-tree classifiers, and IB1 for constructing an instance-based classifier. Which one should we use? The answer to that question can be complex, and depends a lot on how intelligible we want the machine-learning model to be (decision trees and rules are much better for explaining outcomes than neural networks, for example). But a straightforward empirical question is to ask which of the two produces the best result on a given dataset. There is a fairly standard set of comparison techniques in machine learning, all based around the idea of a held-out test set, which is a dataset not used in training the model.
A held-out test set is required to prevent the following algorithm being the best learning algorithm:
- When learning, remember every training instance and its classification.
- When tested, see if the test instance is one of the ones we know.
- If it is, then return the classification we already know.
- If it is not, then return a random result.
The simplest approach is to split the complete dataset into a Training and a Test partition. We build the classifiers on the Training set and then evaluate both on the same Test set. In this way, the only variable is the type of learning algorithm - the training data and test data are the same. (A more precise way of evaluating algorithms is based on cross validation.)
Constructing the datasets
In the earlier section, we used a function to construct an array of UciInstance objects from the data file. To build the training and test sets, we first partition that array into two sections, based on a target proportion. Then we need to construct a WekaTrainingSet and WekaTestSet, which are instances of the WEKA Instances class.
# Randomly partition given array of items into two arrays, using given proportion
def split_array(array, proportion)
array1, array2 = [], []
array.each do |item|
if rand < proportion
array1 << item
else
array2 << item
end
end
return [array1, array2]
end
TrainingSet, TestSet = split_array(DataSet, 0.1)
WekaTrainingSet = Instances.new("Mushrooms training dataset",
make_vector(*(Attributes.clone << Label)),
TrainingSet.length)
WekaTrainingSet.setClassIndex(WekaTrainingSet.numAttributes() - 1)
WekaTestSet = Instances.new("Mushrooms test dataset",
make_vector(*(Attributes.clone << Label)),
TestSet.length)
WekaTestSet.setClassIndex(WekaTestSet.numAttributes() - 1)
TrainingSet.each { |instance| instance.make_instance(WekaTrainingSet) }
TestSet.each { |instance| instance.make_instance(WekaTestSet) }
puts "Training set size: #{WekaTrainingSet.numInstances}"
puts "Test set size: #{WekaTestSet.numInstances}"
Constructing the models
Building the models is simple using WEKA's API. We simply create instances of the classifiers and train them on the training set:
# == kNN classifier
include_class "weka.classifiers.lazy.IB1"
ib1 = IB1.new
ib1.buildClassifier WekaTrainingSet
# == J48 classifier
include_class "weka.classifiers.trees.J48"
j48 = J48.new
j48.buildClassifier WekaTrainingSet
Evaluating
The evaluation procedure can be wrapped into a function which takes a classifier and a dataset, and returns the percentage of the dataset which the classifier has got correct:
# Check result of classifying each instance against the target result
# and return percentage correct
def evaluate_classifier(classifier, testset)
num_correct = 0
testset.numInstances.times do |i|
begin
if testset.instance(i).classValue == classifier.classifyInstance(testset.instance(i))
num_correct += 1
end
rescue java.lang.Exception
end
end
return 100*num_correct.quo(testset.numInstances)
end
puts "Performance of IB1 is: #{evaluate_classifier(ib1, WekaTestSet)}%"
puts "Performance of J48 is: #{evaluate_classifier(j48, WekaTestSet)}%"
Result
When the above is run, we get an output like:
Training set size: 787
Test set size: 7337
Performance of IB1 is: 100.0%
Performance of J48 is: 99.90459315796647%
which tells us that both algorithms do very well on the test set when trained on only 10% of the original dataset.
Summary
That is a lot of work, initially, to set up the functions to interact with WEKA. Looking back, we see that UciInstance, evaluate_classifier, and the code to convert an array of UciInstance objects into an example of WEKA Instances are all reusable pieces of code. They can be packaged up into a library file to be used in future work with WEKA and similar datasets.
