The Weka data mining library makes available an extensive range of functionality, some of it hidden unless you delve into the API documentation. So, over the next few weeks, I intend to work through some of this documentation, and get a clearer picture of what is available and how to use it from jRuby. I am currently using the developer version 3.7, which you can find on the Weka website.
I start by opening up the Weka documentation in a browser, and this reveals a number of packages, some with many subpackages. In alphabetical order, the main packages are:
weka.associations:
The items in this package are for learning association rules: such rules are relations between variables in the data. There are two interfaces and several classes. The classes include standard algorithms such as APriori.
weka.classifiers:
The items in this package are for constructing classification models: these are models which predict the value of a specified attribute from the remaining attributes. This package is quite complex, and has several layers to it. There are four interfaces and a number of classes. The more familiar classifiers, such as J48 or LibSVM, are subclasses of the abstract class Classifier, and are grouped within subpackages. Some of the other classes, such as EnsembleLibrary, are for handling groups of classifiers.
The subpackages bring together the main groups of classificaion algorithms, such as bayes, functions, trees etc. These subpackages contain classes with concrete implementations of classification models, such as NaiveBayes and J48.
weka.clusterers:
This package contains concrete classes for a number of standard clustering algorithms, including Cobweb and SimpleKMeans: a clustering algorithm groups instances of data based on their similarity. Also, some interfaces and abstract classes are present for organising the clustering algorithms into groups, such as those which can estimate density for an instance.
weka.core:
This package contains many 'day-to-day' classes such as Instance and Instances. There are also classes such as EuclideanDistance and EditDistance which compute distances between instances or strings, respectively. MathematicalExpression is a class for evaluating strings containing expressions adhering to a simple grammar. There are over 70 classes in total, so plenty to explore.
weka.datagenerators and weka.estimators:
These packages provide ways to generate and estimate data based on distributions of values.
weka.experiment:
This package contains some interesting 'organisational' interfaces and classes. For example, the 'ResultsProducer' is an interface for objects which work with different randomisations of a dataset. The CrossValidationResultProducer is an implementation of this interface which organises a run of n-fold cross validation.
A collection of classes starting 'Result' provide ways of outputting data in standard formats, such as for GnuPlot, HTML etc.
This is the kind of material I tend to start coding myself, and perhaps I could save some time by using this package instead!
weka.filters:
Filters are techniques for modifying the dataset, either by modifying the number or balance of instances, or by altering the values of attributes themselves. Some useful examples include PrincipalComponents, which performs a transformation of the data, and RandomProjection, which reduces the dimensionality of data.
weka.gui:
Many of these classes are for the Weka graphical environment, but some components can be usefully reused in our own GUIs. These include visualisations of trees and other datatypes, which are under subpackages such as treeviewer and visualize.
Using libsvm from jruby
The latest libsvm distribution contains a Java version of the libsvm system. I want to use this from jruby, but there is a problem. The java version of libsvm has been written based directly on the C code, and this means class names do not have capital letters, which upsets the import process.
Correcting this is fairly simple. All the important classes need to have capital letter names. Having done this and recompiled the code, I am making it available here. You can unpack the code (using 'jar xf libsvm.jar') to find the source files. Or, you can just 'require "libsvm"' to use the contents within a jruby program.
Using this from jruby is not as smooth as it might be: some convenience functions to convert ruby data into the SVM format would be helpful, but for now here is a simple example of learning and then categorising some vectors. (The example is adapted from that in a great introduction to SVMs.)
In terms of performance, you don't appear to lose anything by switching to jruby and the java version of libsvm: I reran my experiment from an earlier post (with suitable changes to the code) and it ran in the same amount of time, and, of course, got pretty much the same result.
Correcting this is fairly simple. All the important classes need to have capital letter names. Having done this and recompiled the code, I am making it available here. You can unpack the code (using 'jar xf libsvm.jar') to find the source files. Or, you can just 'require "libsvm"' to use the contents within a jruby program.
Using this from jruby is not as smooth as it might be: some convenience functions to convert ruby data into the SVM format would be helpful, but for now here is a simple example of learning and then categorising some vectors. (The example is adapted from that in a great introduction to SVMs.)
require "java"
require "libsvm"
import "libsvm.Parameter"
import "libsvm.Model"
import "libsvm.Problem"
import "libsvm.Node"
import "libsvm.Svm"
puts "Classification with LIBSVM"
puts "--------------------------"
# Sample dataset: the 'Play Tennis' dataset
# from T. Mitchell, Machine Learning (1997)
# --------------------------------------------
# Labels for each instance in the training set
# 1 = Play, 0 = Not
@@labels = [0, 0, 1, 1, 1, 0, 1, 0, 1, 1, 1, 1, 1, 0]
# Recoding the attribute values into range [0, 1]
@@instances = [
[0.0,1.0,1.0,0.0],
[0.0,1.0,1.0,1.0],
[0.5,1.0,1.0,0.0],
[1.0,0.5,1.0,0.0],
[1.0,0.0,0.0,0.0],
[1.0,0.0,0.0,1.0],
[0.5,0.0,0.0,1.0],
[0.0,0.5,1.0,0.0],
[0.0,0.0,0.0,0.0],
[1.0,0.5,0.0,0.0],
[0.0,0.5,0.0,1.0],
[0.5,0.5,1.0,1.0],
[0.5,1.0,0.0,0.0],
[1.0,0.5,1.0,1.0]
]
# create some arbitrary train/test split
@@training_labels = @@labels.slice(0,10)
@@training_instances = @@instances.slice(0,10)
@@test_labels = @@labels.slice(10,14)
@@test_instances = @@instances.slice(10,14)
# convert vector of values into a vector of Nodes
def convert values
ns = Node[values.size].new
values.each_with_index do |v, i|
n = Node.new
n.index = i
n.value = v
ns[i] = n
end
return ns
end
# Define kernel parameters
# -- changing these makes the difference between something working or not
@@pa = Parameter.new
@@pa.C = 10
@@pa.svm_type = Parameter::NU_SVC
@@pa.degree = 1
@@pa.coef0 = 0
@@pa.eps= 0.001
@@pa.probability = 0
@@pa.nu = 0.5
@@pa.gamma = 100
@@sp = Problem.new
# Add documents to the training set
@@sp.l = @@training_labels.size
@@sp.x = Node[@@training_instances.size][@@training_instances[0].size].new
@@training_instances.each_with_index do |instance, i|
instance.each_with_index do |v, j|
n = Node.new
n.index = j
n.value = v
@@sp.x[i][j] = n
end
end
@@sp.y = Java::double[@@training_labels.size].new
@@training_labels.each_with_index {|v, i| @@sp.y[i] = v}
# Try four different Kernels
@@kernels = [ Parameter::LINEAR, Parameter::POLY, Parameter::RBF, Parameter::SIGMOID ]
@@kernel_names = [ 'Linear', 'Polynomial', 'Radial basis function', 'Sigmoid' ]
@@kernels.each_index do |j|
# Iterate and over each kernel type
@@pa.kernel_type = @@kernels[j]
m = Svm.svm_train(@@sp, @@pa)
errors = 0
# Test kernel performance on the training set
@@training_labels.each_index do |i|
pred = Svm.svm_predict(m, @@sp.x[i])
puts "Prediction: #{pred}, True label: #{@@training_labels[i]}, Kernel: #{@@kernel_names[j]}"
errors += 1 if @@labels[i] != pred
end
puts "Kernel #{@@kernel_names[j]} made #{errors} errors on the training set"
# Test kernel performance on the test set
errors = 0
@@test_labels.each_index do |i|
pred = Svm.svm_predict(m, convert(@@test_instances[i]))
puts "\t Prediction: #{pred}, True label: #{@@test_labels[i]}"
errors += 1 if @@test_labels[i] != pred
end
puts "Kernel #{@@kernel_names[j]} made #{errors} errors on the test set \n\n"
end
In terms of performance, you don't appear to lose anything by switching to jruby and the java version of libsvm: I reran my experiment from an earlier post (with suitable changes to the code) and it ran in the same amount of time, and, of course, got pretty much the same result.
