Subscribe:

Pages

Division Pitfall in Ruby

Today, I want to highlight a pitfall in Ruby which keeps catching me out, and that is with division. Simple functions are the staple of computing with Ruby, many working out quantities, such as proportions. In this example, I wanted to compute the proportion of numbers exceeding a particular value in a given list of numbers:
def proportion_exceeding(numbers, threshold)
large_numbers = numbers.find_all {|num| num > threshold}
return large_numbers.size / numbers.size
end

The function is very simple. It takes an array of numbers and a threshold as input, builds a new array by finding all the numbers exceeding the threshold, and then returns the proportion of the large numbers in relation to the original array. But there is a problem:

> proportion_exceeding([1,2,3,4,5,6], 3.5)
=> 0

The function returns 0. What goes wrong is that the division symbol / is being applied to two integers, tries to return the nearest integer to the solution, and so returns 0 for anything less than 1.

> 12/3
=> 4
> 5/3
=> 1
> 1/3
=> 0

Numeric Classes in Ruby

To understand what is going on, we should first remind ourselves of the numeric classes available in Ruby. Numbers come in various forms. We can ask Ruby to tell us the classes by using the class method on different numbers:

> 5.class
Fixnum
> 2.0.class
Float
> (2 ** 1000).class
Bignum

These are the basic three. The Fixnum is your regular integer class, holding numbers up to your machine's word size. Float is similarly your a floating-point class. You can find out the range for the latter using Float::MAX and Float::MIN

> Float::MAX
=> 1.79769313486232e+308
> Float::MIN
=> 2.2250738585072e-308

Finally, the Bignum is used to hold extremely large integers: try printing 2**n for varying sizes of n to see how large a number Ruby can handle!

Division of Numbers

The problem Ruby has with basic arithmetic is that it can be asked to add or subtract or multiply or divide any combination of these numbers:

> 2.class
=> Fixnum
> 3.5.class
=> Float
> 2 + 3.5
=> 5.5
> (2 + 3.5).class
=> Float

When given numbers from two different numbers to add, subtract or multiply together, Ruby will use type conversion to turn all the numbers into their most general form. In the example above, 2 can be represented either as a Fixnum or a Float, so it is turned into a Float to suit the second number. The problem comes with division. Alone of the basic arithmetic operators, division can yield a result of a different type to its arguments: 1 divided by 2 is a half, which is not a Fixnum.

Solution

My solution is to get into the habit of not using '/'. In virtually all cases I prefer an accurate result rather than an integer. Instead, I use the method 'quo', for quotient.

> 1.quo(2)
=> (1/2)
> (1.quo(2)).class
=> Rational
> 1.quo(2) * 2
=> (1/1)
> 1/2 * 2
=> 0
> 2.5.quo(7)
=> 0.357142857142857

If you look at the second call above, you will see there is yet another class of number, called Rational. The rationals are those numbers which can be represented as exact fractions, such as 1/2 or 3/7. The 'quo' method is quite smart. Where possible, it will create a Rational class, which is exact. Where not possible, it will return a Float, as in the last line above. (Apart from 'quo', the other method to remember is 'fdiv' which always returns a Float.)

The revised function looks like:
def proportion_exceeding(numbers, threshold)
large_numbers = numbers.find_all {|num| num > threshold}
return large_numbers.size.quo(numbers.size)
end


> proportion_exceeding([1,2,3,4,5,6], 3.5)
=> (1/2)
> proportion_exceeding([1,2,3,4,5,6], 4.5)
=> (1/3)

Dialogs in wxRuby

A dialog is an independent window that your application can use to provide or require some information. Typical examples are the file dialog, to locate a file, and the printer dialog, to configure a print job. wxRuby provides a number of standard dialogs to provide platform standard tools to get information from the user or provide information to the user. This post introduces the simplest dialogs, message dialogs, moves on to ways to retrieve information from the user, and finally considers custom-built dialogs. To explore how these dialogs are used, it is convenient to run them within the simplest possible wxRuby program:
require 'wx'

Wx::App.run do
#
# Put dialog code here
#
exit
end


Message Dialog

The Wx::MessageDialog class is used to display information to the user. We can set the title of the dialog box, the text that is displayed, and the icon. Finally, we can change the buttons on the dialog, to get some feedback from the user. Let us begin with the simplest application, displaying some information.



md = Wx::MessageDialog.new(
nil,
"Text within dialog\ncan be on multiple lines",
"Title of Dialog",
Wx::ICON_INFORMATION)
md.show_modal

The variable 'md' is set up to hold an instance of the MessageDialog class. The instance takes four parameters:
  1. the parent of the dialog: usually the parent frame, so the dialog is centred on it;
  2. the text to display in the body of the dialog;
  3. the title of the dialog; and
  4. the style.
The use of styles is important within wxRuby, as almost every widget within the library can be customised using one or more flags provided within its style parameter. In the example above, the style is simply Wx::ICON_INFORMATION, which tells the dialog to use an information icon on display. The choices for the icon include Wx::ICON_INFORMATION, Wx::ICON_QUESTION, and Wx::ICON_ERROR. The dialog is run by calling its 'show_modal' method. 'Modal' means that the dialog will take over interactions with the application whilst it is on the screen. This is to prevent the user doing something else in the meantime, such as trying to open another file before finishing with the dialog that is currently displayed.

Our second example shows how to get information back from a user. The MessageDialog provides control over which buttons will be displayed in the dialog, and what their labels might be. I have combined the style for an error icon (Wx::ICON_ERROR) with the style for yes/no buttons (Wx::YES_NO), to get the dialog below.

How to see which button the user clicked? The answer is to check the return value from the call to md.show_modal to see if it equals Wx::ID_YES, which is what you get back if you click on the 'yes' button. The other button styles include Wx::ID_CANCEL and Wx::ID_OK to use 'ok' and 'cancel' buttons. Note that styles are joined using bitwise or, '|'.




md = Wx::MessageDialog.new(
nil,
"Error in program. Shall we continue?",
"Error Dialog",
Wx::YES_NO | Wx::ICON_ERROR)
if md.show_modal == Wx::ID_YES
puts "Clicked YES"
else
puts "Clicked NO"
end


File Dialogs


After the MessageDialog, there is a welcome sense of deja vu with the remaining dialogs. Let us consider the file dialog, which has a place in most applications as a means to identify the file to work on.


The file dialog can be created just by providing it a parent window and a dialog. By default, you will get an 'open file' dialog. If you would like a 'save file' dialog, add the following as the third argument: ':style => Wx::FD_SAVE | Wx::FD_OVERWRITE_PROMPT'. Having created the dialog, we see if the 'Open' or 'Save' button was clicked by checking if the return value is Wx::ID_OK. The name of the filename is retrieved by accessing the 'filename' attribute of the dialog itself.

fd = Wx::FileDialog.new(
nil,
"Title of Dialog"
)
if fd.show_modal == Wx::ID_OK
puts "User selected file: #{fd.filename}"
end


Tip:
If you would like your file dialog to open up at the directory you left it in, then store your file dialog in an instance variable of your topmost frame and create it in your frame's initialize method.

Selection Dialogs
Apart from the file dialog, more general ways to retrieve information from the user include the TextEntryDialog, PasswordDialog, and the selection dialogs. The selection dialogs let the user choose one or more items from a list. Below is shown the MultiSelectionDialog. All of these dialogs work in very similar ways, with slightly different methods to retrieve the final information.



mcd = Wx::MultiChoiceDialog.new(
nil,
"Choose one or more languages:",
"Example dialog",
["C", "C++", "Java", "Lisp", "Ruby", "Scheme"]
)
if mcd.show_modal == Wx::ID_OK
puts "You selected: #{mcd.selections}"
end
exit
end


Custom Dialogs


Missing from the standard wxRuby toolbox is a way to retrieve numeric information from the user. Also, what happens if you want more than one piece of information at a time? Indeed, can we create our own dialogs within wxRuby? The answer is yes, we can construct our own dialogs, by subclassing the Dialog class. We need to fill out the dialog with the widgets of our choosing, catch events when they change to save off the current values, and read the values when the dialog exits. The code below contains various techniques I have yet to cover in this weblog, but hopefully will still be understandable.

There is a magic touch: wxRuby provides a very clever method to create and layout buttons for us too. The method: create_separated_button_sizer will generate a Sizer with the horizontal separator line and buttons based on the flags you provide it. Possible buttons include: OK, CANCEL, YES, NO and HELP. What is particularly clever about this method is that it will put the buttons in the right order and with the right spacing depending on the operating system you are using!


require 'wx'

class NumberDialog < Wx::Dialog
attr_reader :value
def initialize(parent, title, label, min_value=0, max_value=100)
super(parent, Wx::ID_ANY, title)


spin_control = Wx::SpinCtrl.new(self, Wx::ID_ANY,
:min => min_value, :max => max_value)
evt_spinctrl(spin_control) {|e| @value = spin_control.value}

items_sizer = Wx::BoxSizer.new(Wx::HORIZONTAL)
items_sizer.add(Wx::StaticText.new(self, Wx::ID_ANY, label))
items_sizer.add(spin_control, 1, Wx::GROW)

main_sizer = Wx::BoxSizer.new(Wx::VERTICAL)
main_sizer.add(items_sizer, 0, Wx::ALL | Wx::GROW, 10)
main_sizer.add(create_separated_button_sizer(Wx::OK | Wx::CANCEL),
0, Wx::ALL | Wx::GROW, 5)

set_sizer main_sizer

end
end

Wx::App.run do
nd = NumberDialog.new(nil, "Sample numbers", "Choose:", 10, 50)
if nd.show_modal == Wx::ID_OK
puts "Value is: #{nd.value}"
end
exit
end

Finally ...

This post has not touched on all the standard dialogs or methods for using a custom dialog. There are dialogs to select colours, fonts, and to manage a sequence of tasks through a Wizard. Nor have I covered all the options that most of these dialogs provide. But I have tried to cover the more important ones for information input and display. Once you understand these, you should find the principles for using the remaining dialogs are similar.

Displaying Images in wxRuby

An important area of scientific exploration is the analysis of images. My favourites are images of planetary surfaces, such as those the Galileo spacecraft took of Ganymede (http://www2.jpl.nasa.gov/galileo/images/ganymede/ganyimages.html). I want to take collections of such images, process them a little, and pass them to a classification system for analysis. But first, it would be nice to view the images. I will want to do more than a standard image viewer, because there is usually a lot of information that can be taken from an image. In this post, I look at the elements of wxRuby which let us select and display an image within a frame.

Starting Point

I begin with a simple application, which opens up a frame and displays a menu. I have added an 'open' menu item, which will be responsible for opening an image and displaying it in the frame.

require 'wx'

class ImageFrame < Wx::Frame
def initialize
super(nil, :title => "Image Displayer")
set_menu_bar make_menu_bar
create_status_bar 2
end

def make_menu_bar
# construct menu
menu_bar = Wx::MenuBar.new
# -- menu
image_menu = Wx::Menu.new
image_menu.append(Wx::ID_ABOUT)
image_menu.append(Wx::ID_OPEN, "&Open", "Open an image file")
image_menu.append(Wx::ID_EXIT)
evt_menu(Wx::ID_ABOUT) {about_program}
evt_menu(Wx::ID_OPEN) {puts "open_image"}
evt_menu(Wx::ID_EXIT) {exit_program}

menu_bar.append(image_menu, "&Images")

menu_bar
end

def about_program
Wx::MessageDialog.new(self,
"This program lets you choose an image file\nto display",
"About Program",
Wx::ICON_INFORMATION | Wx::OK).show_modal

end

def exit_program
close
end
end

Wx::App.run do
ImageFrame.new.show
end

File Dialog

The standard dialogs within a graphical library make it easy to add functionality to your application in a format your users will expect. The file dialog is the way to get a filename from the user. Here, the user must identify an image file to open and display.

The general pattern is first to create an instance of the FileDialog object. Second, show the dialog, which returns the ID number of the button the user clicked: we will only act if the dialog returns 'Wx::ID_OK', to indicate that the user clicked the 'OK' button. Finally, we ask the dialog instance for the name of the filename. Here is the code, which should be placed inside the open_image method; for now, it will only display the name of the selected file:

def open_image
fd = Wx::FileDialog.new(self, "Open an image file",
:style => Wx::FD_FILE_MUST_EXIST)
if fd.show_modal == Wx::ID_OK
puts "Selected #{fd.filename}"
end
end

The 'style' keyword tells the file dialog to only accept existing files, so the user cannot type in an unknown filename. For this method to be called, we must make a change to the make_menu_bar method: evt_menu(Wx::ID_OPEN) {open_image}

Displaying an Image

Once we have the filename of an image, we can open and display the image in the frame. The image object is known as a Bitmap, and is created from the path of the image to load. A call to is_ok checks that nothing went wrong, and then the Bitmap can be passed to our display widget. We replace the puts call above with:

begin
image = Wx::Bitmap.new(fd.get_path)
if image.is_ok
@display.set_image image
set_status_text(fd.get_filename, 1)
end
rescue Exception # catch any problems in setting up the bitmap
return
end

The display widget is responsible for showing the image inside our frame. As the image may be larger than the frame size, we will need the widget to have scrollbars: we construct a subclass of ScrolledWindow. Whenever we open an image, we tell the display widget about the new image by calling set_image. Then, the widget updates its scrollbars to the image's width and height, and refreshes its display.

The last bit of code is to handle the actual drawing of the bitmap. This is managed through an on_draw method, which is called everytime the widget must update its display. The on_draw method is called with a pointer to the 'device context' on which it must draw. Our method is very simple: we clear the device, and then draw the bitmap, if we have one.

class ImageDisplay < Wx::ScrolledWindow
def initialize parent
super parent
@image = nil
end

def set_image image
@image = image
set_scrollbars(1, 1, @image.width, @image.height)
refresh
end

def on_draw dc
dc.set_background Wx::WHITE_BRUSH
dc.clear
return if @image == nil
dc.draw_bitmap(@image, 0, 0, true)
end
end


Telling the Frame to Display the Widget

Apart from learning about individual widgets, the user of a GUI library like wxRuby must learn how to display widgets within a frame in the intended places. The way to do this is with 'Sizers'. Each widget has a sizer to manage the widgets within it. The simplest sizers are the BoxSizers, which lay out their children in either a horizontal or a vertical direction. Widgets are usually constructed and added to sizers in the initialize method of their parent.

For the program here, we made a vertical sizer. The widget to display is then added to the sizer. The call sizer.add(@display, 1, Wx::GROW) tells the sizer which widget to add, that it should expand the widget to fill the space, and that the widget should grow as the frame grows around it. Finally, the sizer is set as the frame's main sizer. The final initialize method for ImageFrame is:

def initialize
super(nil, :title => "Image Displayer")
set_menu_bar make_menu_bar
create_status_bar 2
sizer = Wx::BoxSizer.new Wx::VERTICAL
@display = ImageDisplay.new self
sizer.add(@display, 1, Wx::GROW)
set_sizer sizer
end

Graphical Interfaces for Your Ruby Programs

Sometimes, the plain text interface to Ruby is not enough. A graphical user interface can be convenient, allowing you to, for instance, select the files you want to analyse and see the results directly. When we create such an interface, our program moves along the dimension from being a tool, which you just use and forget about, to an application, which you use and spend a lot of time with. My favourite library for graphical interfaces is wxWidgets (http://www.wxwidgets.org/), which is cross platform and easy to use. Ruby has its own binding to wxWidgets, called wxRuby (http://wxruby.rubyforge.org/wiki/wiki.pl).

wxRuby recently made it to version 2.0, and has become a robust platform for developing software with a graphical user interface, or GUI. Let us get started with wxRuby, and create a simple program.

Installation

Installation is covered in http://wxruby.rubyforge.org/wiki/wiki.pl?Installation. You will need wxWidgets installed on your Linux platform: get it from your repository. On Ubuntu, I used synaptic to search for wxWidgets, and installed libwxgtk2.8-0, which also pulled in libwxbase2.8-0. Once you have these, rubygems does the rest: sudo gem install wxruby.

Simplest Example

The simplest example just creates a basic frame. The code is in three pieces. The first piece is the loader, which tries to install wxruby; if rubygems is not installed by default (should not happen in newer Ruby distributions) then it will require rubygems before trying to find wxruby. The second piece defines the SimpleFrame class. This class is a subclass of the wxWidgets class Frame. The namespace for all the wxruby material is 'Wx'. (I like to keep namespaces within my code, but some like to 'include wx' at the start, and then the 'Wx::' parts are optional.)

The SimpleFrame class has an initialize method which calls its parent class. The nil tells the parent there is no parent frame for the SimpleFrame to relate to. The second argument uses a keyword-style format to set the title of the frame.

The third part of the code starts the wx application, and, within the block, creates and shows a SimpleFrame.
# This block ensures that rubygems is present before requiring wx.
begin
require 'wx'
rescue LoadError => no_wx_err
begin
require 'rubygems'
require 'wx'
rescue LoadError
raise no_wx_err
end
end

class SimpleFrame < Wx::Frame
def initialize
super(nil, :title => "Simple Frame Title")
end
end

Wx::App.run do
SimpleFrame.new.show
end

Adding a Menu and Status Bar

The hardest part in learning about any GUI library is to figure out how the different elements can be placed onto the screen. Two basic elements are the menu and status bar. The menu places the commands for your application in an easy to access place, and the status bar provides a method to feed information back to the user.

Adding a menu bar is a three step process. First, a set of menus must be created, containing any submenus or menuitems. Second, these menus must be added to a menubar. And third, the menubar must be added to the frame.

Step 1: creating the menus.

A menu is an instance of Wx::Menu. Before creating a menu, we must create an instance of Wx::Menu and name it. Next, we must add the items we want to display to the menu. Here we meet a useful feature of wxWidgets, which is the standard identifier. Every widget (instance of a wx class) within wxWidgets has a unique ID number. This ID number can be assigned by a user, but there are also several standard ones provided in the library. Useful ones include: ID_ABOUT, ID_EXIT, ID_OPEN, ID_SAVE. If you use a standard identifier for a menu item or button, wxWidgets will automatically give it a name, icon and appearance standard for the platform you are using.

To create a program menu with an About and Quit button, we use:

program_menu = Wx::Menu.new
program_menu.append(Wx::ID_ABOUT)
program_menu.append(Wx::ID_EXIT)

How do we make the menu item do something when you click on it? We associate the menu item widget with a ruby method, so that the method is called every time the menu item is clicked. When we know the ID number of the widget, we can use that to make the association. The evt_menu method in the construction of the menu associates the block with the given ID number. The complete listing below provides some examples of things to do to suit the menu items.

evt_menu(Wx::ID_ABOUT) {about_program}


Step 2: Add the menus to a menubar

The menubar is an instance of Wx::MenuBar. You add menus to the menubar using the append method of menubar. The append method takes the menu itself and the title of the menu. Use an ampersand, &, in the title of the menu to provide the little line for a keyboard shortcut to that menu.

Step 3: Add the menubar to the frame

The easiest step of the three: just tell the frame to set_menu_bar to your menubar instance. Below is the full code to initialise the frame with the menubar:

def initialize
super(nil, :title => "Simple Frame with Menus")
set_menu_bar make_menu_bar
end

def make_menu_bar
# construct menu
menu_bar = Wx::MenuBar.new
# -- menu 1
program_menu = Wx::Menu.new
program_menu.append(Wx::ID_ABOUT)
program_menu.append(Wx::ID_EXIT)
evt_menu(Wx::ID_ABOUT) {about_program}
evt_menu(Wx::ID_EXIT) {exit_program}

menu_bar.append(program_menu, "&Program")

menu_bar
end

Adding a status bar

The status bar is the little strip along the bottom of the screen which contains instructions or information for the user. The bar can be divided into a number of pieces, so different information can be displayed. It is very simple to create a status bar: call the create_status_bar method on the frame, provide it with the number of pieces (fields) you want, and it will be set. Putting text into a field of the status bar is equally easy: call the set_status_text method on the frame, providing it with the string to display and the index (from 0) of the field to display in.

Getting free help

Something I find very useful about status bars and wxWidgets is that each menu item is given a help string. This help string gets displayed in field 0 of the status bar, if you have one! So, include a status bar, and you get free help for your users.

Non-standard menu items

You may want to create your own menu item, not one of the standard ones. To do this, you need to create an instance of MenuItem yourself. This needs at least three items. The first is the Menu in which the item will sit. The second is an ID number - you can write Wx::ID_ANY if you don't care what it is. The third item is the name of the menu. The fourth is an optional help string, which will display in any status bar. The method append_item attaches the item to a menu. evt_menu is again used to associate a block with the menu, and wxRuby is smart enough to recognise an instance of a menu item as well as a widget ID as its parameter.

new_item = Wx::MenuItem.new(program_menu, Wx::ID_ANY, "Special", "Special menu")
program_menu.append_item(new_item)
evt_menu(new_item) {puts "Special clicked"}

Put it all together, and you should get something like the following screen shot:

Below is the complete program:
# This block ensures that rubygems is present before requiring wx.
begin
require 'wx'
rescue LoadError => no_wx_err
begin
require 'rubygems'
require 'wx'
rescue LoadError
raise no_wx_err
end
end

class SimpleFrame < Wx::Frame
def initialize
super(nil, :title => "Simple Frame with Menus")
set_menu_bar make_menu_bar
create_status_bar 2
set_status_text("Hello", 1)
end

def make_menu_bar
# construct menu
menu_bar = Wx::MenuBar.new
# -- menu 1
program_menu = Wx::Menu.new
program_menu.append(Wx::ID_ABOUT)
program_menu.append(Wx::ID_EXIT)
evt_menu(Wx::ID_ABOUT) {about_program}
evt_menu(Wx::ID_EXIT) {exit_program}

new_item = Wx::MenuItem.new(program_menu,
Wx::ID_ANY, "Special", "Special menu")
program_menu.append_item(new_item)
evt_menu(new_item) {puts "Special clicked"}

menu_bar.append(program_menu, "&Planet")

menu_bar
end

def about_program
Wx::MessageDialog.new(self,
"Our first wxRuby program",
"About Program",
Wx::ICON_INFORMATION | Wx::OK).show_modal
end

def exit_program
close
end
end

Wx::App.run do
SimpleFrame.new.show
end

Tip: In Ubuntu, associate ruby with your .rb files, and you can run your wxRuby applications by double clicking on the icon!

Constructing a Wrapper around RSRuby

In my previous post, I looked at creating some graphs and doing other analysis within RSRuby. The problem with using RSRuby directly is that the syntax can be clunky; there are many things that interfere with simply getting the result. For example, to save a graph of data, you must start up an instance of R, tell it to create a file of the right type, plot the graph, then tell the device you are finished, etc. What would be convenient, at least for simple cases, is to have a single function which does all the above in one go. In this post, I explore how easy it is to create a wrapper around some of the RSRuby calls, and so devise a personal library of functions with just the right level of exposed complexity.

Creating a Wrapper

I shall give my wrapper library a simple name, 'L', as I don't want long names to type in. All methods are module methods. 'L' includes an instance of R and the methods in 'L' will do the hard work of interacting with R. Here is the start of the library:

require 'rsruby'

module L
R = RSRuby.instance # keep a constant R interpreter
end

This is very simple, of course. The instance of R is accessible as 'L::R', and can be called in the usual way. I think this is important — the wrapper library does not prevent all the usual ways of interacting with R, but will make some functions simpler.

Accessing Statistical Functions

Let us add some simple statistical functions. I would like to compute the mean of an array of numbers by calling: 'L::mean [1,2,3,4,5,6,7]', and I can do this by providing functions like:

# --- stats
def L.mean items
R.mean items
end

def L.stddev items
R.sd items
end

This library is for my own use, so I can use whatever names I like. I sometimes prefer more verbose forms than R uses, as you can see in the stddev function.

Graphical Functions

The process of creating and saving a graph in RSRuby required a few steps. We can simplify those steps using our wrapper, creating a function to accept the name of the final graph along with the parameters for constructing the graph. We use here the usual Ruby trick of combining the hash map arguments into a single Hashmap, so save_histogram("sample.png", data, :main => "Title", :xlab => "x label") will treat the parameters :main => "Title, :xlab => "x label" as a single argument, labelled params, which can then be passed to R's own hist function.

def L.save_histogram(filename, data, params)
R.png filename
R.hist(data, params)
R.eval_R("dev.off()")
end

Reading Data

The machine-learning and data-mining communities have a fairly standard format for representing data instances: comma-separated values, or CSV. Each line of a text file is taken to represent a single data instance. The features of the data are separated by commas. For example, a table of data representing information about cars might hold features about the colour, engine size and whether there was a roofrack:

white, 1800, no
blue, 1200, yes
...

Many such files contain header information, describing what the column headings are, for instance. We would like a function that will read such data into a Ruby data structure. The aims of the function are:
  1. to accept as input the filename of the data to read in;
  2. to optionally accept a number of header lines to ignore;
  3. to optionally accept an alternative separator symbol to the comma;
  4. to return an array in which each element is an array of the features for one instance; and
  5. to convert the feature values into Integer or Float types, where appropriate.
The functions I created are:

# --- reading in data
# try to convert string item into an Integer or a Float, else return item
def L.convert_item item
begin
Integer item
rescue
begin
Float item
rescue
item
end
end
end

# convert every item in given list of items
def L.convert_items items
items.collect {|i| L.convert_item i}
end

# return a list of the lines from given file
# -- ignore the top ignore_n lines
# -- convert every item in each line using the above
def L.read_data_file(filename, ignore_n=0, split_char=",")
data = []

file = File.open(filename, "r")
ignore_n.times { file.gets }
while line = file.gets
unless line.strip == "" # ignore blank lines
data << L.convert_items(line.split(split_char))
end
end
file.close

data
end


Example

Finally, we can put all these pieces together by simplifying our example from last time. There, we took an example dataset which had a number of fields in comma-separated format. That is, each line represents a single data instance, and the information for each instance's features is separated by a comma. The first five lines of our example dataset are header information, which we do not need. Reading in these data, extracting the 11th feature from the data, printing the feature's mean and standard deviation, and finally creating a histogram of its values, can now be done quite directly:

# -- process the image.txt data
image_data = L::read_data_file("image.txt", 5, ",")
column_data = image_data.collect{|i| i[10]}
puts "Mean: #{L::mean(column_data)} SD: #{L::stddev(column_data)}"
L::save_histogram("image.png", image_data.collect{|i| i[10]},
:xlab => "X from image", :main=>"From image")

Just a Beginning ...

Of course, what I've shown above is just the beginning. It does demonstrate how easy Ruby makes it to provide tailored libraries to simplify daily tasks. Collect a few dozen of such functions together, and suddenly we have a budding wrapper library. How far you go to a complete wrapper library really depends on your stamina - the R project comes with a manual of over 1600 pages!

Plotting Simple Graphs with RSRuby

R has a powerful set of plotting and graphics tools for visualising data. I shall look at some of the options for generating graphs, fitting a line to a graph, and plotting data that is stored in a text file. If you are familiar with R, the method used here may seem a bit odd, as I call R from Ruby code as a standalone program, generating results with minimal user input. Working with R is often more of an interactive process, where graphs are built up piece by piece, and shown on the screen. Instead, I shall immediately save graphs out to file, and then use an image viewer program.

Plotting Points on a Graph

The simplest graph is obtained by plotting data points and drawing lines between those points. For instance, the graph below was obtained from the list of x,y coordinates: (1,10) (2,5) (3,2) (4,5) (5,10) using a simple call to RSRuby.
The calling code goes through the following steps:
  1. Creates an instance @@r of the R interpreter.
  2. Instructs the interpreter to use the named PNG file for its graphical output.
  3. Plots the graph.
  4. Tells the interpreter that plotting is over, and the PNG file can be written.
The plot command can take a wide range of parameters. The basic ones are the list of x coordinates, and the list of y coordinates. The RSRuby bridge is kind to us here, as the lists are Ruby arrays. The points (1,10) (2,5) (3,2) (4,5) (5,10) are passed to the interpreter as the two arrays, first of the x-coordinates [1,2,3,4,5] and second of the y-coordinates [10,5,2,5,10]. The rest of the arguments use Ruby's special hashmap notation to give the effect of keyword arguments. The :xlab and :ylab keywords specify the labels for the x and y axes respectively. :type specifies how the points and line are plotted: "n" stands for show nothing, "p" for show the points only, "l" for draw lines between the points, "b" for plot both, "c" for draw lines and leave gaps for the points. :main sets the title of the graph, and :col lets you colour the points and lines plotted.
require 'rsruby'

@@r = RSRuby.instance

@@r.png("sample.png")
@@r.plot([1,2,3,4,5], [10,5,2,5,10],
:xlab => "x label", :ylab => "y label",
:type => "b", :main => "Simple graph",
:col => "blue")
@@r.eval_R("dev.off()")

Fitting a Line to Data

R provides the 'lm' function for computing linear models. This function appears very powerful, and relies on the caller to provide it with a formula, or linear model, to fit against. Consider the following data, which fall around the line y=x:

(1, 1.2) (1.9, 2.1) (3.1, 2.8) (3.8, 4.1) (4.7, 5.2)

In order to fit a linear model to these data, we place the x and y coordinates in variables called, perhaps, xs and ys. Then, in R, we call lm with the argument "xs ~ ys", to define the formula for the linear model. Below, I give a rather clunky way to do this, which involves assigning variables called xs and ys in the R interpreter - I hope to improve this in later posts. What is returned is a Ruby object defining the line of fit, and we can print out some summary information about the computed linear model. By supplying the intercept and y values of our fitted line to the 'abline' method, a line will be added to our graph giving the best fit.

@@r.png("sample2.png")
x = [1, 1.9, 3.1, 3.8, 4.7]
y = [1.2, 2.1, 2.8, 4.1, 5.2]
@@r.assign('x', x)
@@r.assign('y', y)
fit = @@r.lm('x ~ y')
@@r.plot(x, y)
@@r.abline(fit["coefficients"]["(Intercept)"], fit["coefficients"]["y"])
puts fit["coefficients"]
@@r.eval_R("dev.off()")

$ ruby sample-1.rb
{"(Intercept)"=>0.0875346260387804, "y"=>0.913138108428967}
Histograms from a Text File

To finish this post, I want to show how to plot a different kind of graph, using data from a text file. The text file I shall use is an example from UCI Irvine's Machine Learning Repository (http://archive.ics.uci.edu/ml/), and I grabbed the 'Image Segmentation' dataset for illustration. The UCI data is usually stored as comma separated data, one instance per row, with some header information at the top. Let's plot a histogram of the values in the 11th column of data. (Don't ask why - I just want to see how to do it!)

First, we store the data in a file, call it "image.txt". Second, we open the file into Ruby, ignore the first five lines, which contain header information, and then begin working for real. We read each line, split it up using the comma separator character, and store the 11th item in that array as a floating point number. The code is:

data = []
file = File.new("image.txt", "r") # open the file for reading
5.times {file.gets} # ignore the header
while line = file.gets # read all remaining lines
data << line.split(",")[10].to_f # add 11th item to 'data' as a float
end
file.close # close the file

Finally, generating a histogram from this is fairly simple:

@@r.png("sample3.png")
@@r.hist(data, :xlab => "Some data", :main => "Histogram of some data")
@@r.eval_R("dev.off()")

Analysing Numeric Data in Text Files with Ruby

A fairly typical scenario I find myself in is having a program run an experiment, outputting its results into a text file. I want to automate the data analysis so that large amounts of data are converted into tables, graphs and summarising statistics with as little effort as possible. This is the kind of work computers should be doing for us.

To illustrate a simple case, I have an optimisation program running various experiments, and producing text output, which consists of many lines like:

results-14-51-3-0.txt, 26, 7, 128, 18, 0.1, 4.62769, 6.45275e+07, 27.6596, 8, 256, 19, 0.1, 9.25538, 4.79442e+08, 27.6596, 9, 512, 21, 0.1, 18.5108, 3.75805e+09, 27.6596
results-14-51-3-1.txt, 21, 7, 128, 16, 0.1, 5.72952, 1.14546e+07, 22.3404, 8, 256, 17, 0.1, 11.459, 6.22772e+07, 22.3404, 9, 512, 18, 0.1, 22.9181, 3.37488e+08, 22.3404
...
results-14-59-3-0.txt, 3, 7, 128, 3, 0.35, 35.84, 53775.3, 3.57143, 8, 256, 3, 0.45, 80.2133, 257532, 3.19149, 9, 512, 3, 0.45, 160.427, 1.03329e+06, 3.19149
results-14-59-3-1.txt, 24, 7, 128, 16, 0.1, 5.01333, 2.77391e+07, 25.5319, 8, 256, 20, 0.1, 10.0267, 2.46826e+08, 25.5319, 9, 512, 20, 0.1, 20.0533, 1.39406e+09, 25.5319
...


These provide the following information:
  1. within the name is encoded values for 'n' 'm' 'k', defining the experiment, and the 'instance' number; each experiment has been run 20 times.
  2. next is the actual number of minima located in the file.
  3. finally we have three sets of seven values, encoding: l and "2 to the power of l", the number of samples made, the number of minima located, values for gamma, r_appr and T_gamma_r, and finally N_appr, which is an estimate of the actual number of minima.
I want to collect some aggregate results out of these data and generate the results in a format suitable to use in LaTeX. (One of the great benefits of using LaTeX for writing articles is that its text format can be created by your programs!) The first time I tried this, I imported the file into a spreadsheet program as CSV, wrote some cell formulae, and then manually copied all the results over to my text editor - not a process I want to repeat.

Classes for the Data

As we are using Ruby, we first need some classes to hold the data. There will be a class for the individual results:

class Results_l
attr_reader :l, :num_found, :gamma, :r_appr, :t_gamma_r, :n_appr

def initialize(l, pow_l, num_found, gamma, r_appr, t_gammar_r, n_appr)
@l = l.to_i
@num_found = num_found.to_i
@gamma = gamma.to_f
@r_appr = r_appr.to_f
@t_gammar_r = t_gammar_r.to_f
@n_appr = n_appr.to_f
end
end

And a class for the complete experiment:

class Results_nmk
attr_reader :n, :m, :k, :instance, :true_N, :results_1, :results_2, :results_3

def initialize(n, m, k, instance, true_N, results_1, results_2, results_3)
@n = n
@m = m
@k = k
@instance = instance
@true_N = true_N
@results_1 = results_1
@results_2 = results_2
@results_3 = results_3
end
end

Reading in the Data

The data is stored in consecutive lines of a text file. Our main program will read in each text line, extract the fields that it needs, and construct instances of the above classes with the relevant data. Each line will be stored as an instance of Result_nmk, and these instances will be stored in an array of results. Ruby provides a number of useful techniques to achieve this.

First, we need to split a string up into pieces, based on a separator token. The String.split method is key to making our process here work. Assume the variable line holds one of the lines read in from the text file, then line.split(",") will divide the line up into an array of strings, separating it at each of the commas.

"a,b,c,dd,ee".split -> ["a", "b", "c", "dd", "ee"]

Second, we want some of the strings to be treated as numbers, not as strings. We can do this with the String.to_i and String.to_f methods, which convert strings into Fixnums or Floats, respectively.

"25.2".to_f -> 25.2
"19x1".to_i -> 19 # notice that the initial number is read, and the rest ignored


Third, we need to extract items from the input array and construct a class instance using these items as its input parameters. Ruby provides a nice way to 'explode' an array when passing it to a method, so that each item in the array gets seen by the method as a separate input argument.

For analysing my experiment data, I split the input text line and stored it in a local variable called line_arr. Then I could create an instance of Results_l by exploding the relevant part of line_arr, e.g. Results_l.new(*line_arr[2..8])

The complete method to read in and construct class instances for my data file follows. I have made the classes themselves responsible for converting their arguments from strings to Fixnums or Floats as required.

def analyse_file filename
results = []
File.open(filename).each do |line|
line_arr = line.split(",")
results << Results_nmk.new(
get_n(line_arr[0]),
get_m(line_arr[0]),
get_k(line_arr[0]),
get_instance(line_arr[0]),
line_arr[1].to_i,
Results_l.new(*line_arr[2..8]),
Results_l.new(*line_arr[9..15]),
Results_l.new(*line_arr[16..22]))
end

The get_n, get_m, get_k and get_instance methods all work by splitting the "results-14-55-3-0.txt" name string on the "-" separator, and converting the relevant part of the array to an integer. For example:

def get_k name
name.split("-")[3].to_i
end

Calculating Statistics

In my experiment, I have 20 instances for each value of the three key variables, n, m and k. For my tables, I want to aggregate these results to compute mean values and some extra relations. The output should be in a LaTeX friendly form, so I can just copy the output directly into my article. I didn't need anything too complex here, but of course, the opportunity is there to automate more complex statistical analysis.

Again, there are two steps, three if you include the output as something separate.

The first step is to collect together values for each of the items I am interested in. For each value of n, m, k and l there are 20 experimental results. I want to collect all of these 20 results together for each set of n, m, k and l. This step is complicated by the fact that each instance of Result_nmk contains three instances of Result_l! I would like to use an iterator to step through an array of possible nmkl values, so what I do first is create an array of 4-tuples, where each 4-tuple is an array of n,m,k,l values. The code to create the array of possible nmkl values is below. Notice the 'unless' clauses, which make sure each nmkl value is only stored once.

nmkl_values = []
results.each do |result|
nmkl_value_1 = [result.n, result.m ,result.k, result.results_1.l]
nmkl_values << nmkl_value_1 unless nmkl_values.include? nmkl_value_1
nmkl_value_2 = [result.n, result.m ,result.k, result.results_2.l]
nmkl_values << nmkl_value_2 unless nmkl_values.include? nmkl_value_2
nmkl_value_3 = [result.n, result.m ,result.k, result.results_3.l]
nmkl_values << nmkl_value_3 unless nmkl_values.include? nmkl_value_3
end

We can now iterate through the list of nmkl values, and retrieve matching Result_nmk instances and, though it, Result_l. We will add the following methods to Result_nmk:

class Results_nmk
def matches_nmkl?(n, m, k, l)
@n == n && @m == m && @k == k &&
(@results_1.l == l || @results_2.l == l || @results_3.l == l)
end

def get_result_l l
if @results_1.l == l
@results_1
elsif @results_2.l == l
@results_2
elsif @results_3.l == l
@results_3
else
raise Exception # there must be a legal value for l
end
end

The second step is to compute the results and generate output in the correct format. Computing the results is simple enough, as I only care about the mean for each set of instances; I compute this value using RSRuby. To get the results into the correct format for LaTeX, I print out each line of results as text, with the numbers separated by an ampersand sign. These lines can then be copied into my article, where the header and other formatting information is waiting for them. An instance of R is initialised through RSRuby at the start of the program with:

require 'rsruby'
@@r_instance = RSRuby.instance

The code is straightforward: run through each one of the nmkl values, retrieve the values for each of the items of interest, and compute the average. Within the 'puts' I use the format: #{"%5.2f" % true_N} which displays every number with 2 decimal places. First is the general loop, and second is one of the methods which gets called to collect the values. Because of the way I have constructed my data, sometimes you want a list of the overall results, of type Result_mnk, and sometimes you want a list of the individual results, of type Result_l. My code computes both these lists, and passes them to the individual methods for computing the desired averages. The method 'show_mean' takes an array as an argument, uses R to compute the mean, and returns a string formatted neatly.

nmkl_values.each do |nmkl_value|
instances = results.find_all {|result| result.matches_nmkl?(*nmkl_value)}
l_instances = instances.collect{|result| result.get_result_l(nmkl_value[3])}

avg_true_N = show_mean(get_true_N(instances))
avg_beta_m = show_mean(get_beta_m(l_instances))
avg_gamma = show_mean(get_gamma(l_instances))
avg_r_appr = show_mean(get_r_appr(l_instances))
avg_n_appr = show_mean(get_n_appr(l_instances))
avg_n_appr_by_n = show_mean(get_n_appr_by_n(instances, nmkl_value[3]))

puts "#{"%5.2f" % nmkl_value[1]} & #{avg_true_N} & $2^#{nmkl_value[3]}$ & #{avg_beta_m} &
#{avg_gamma} & #{avg_r_appr} & #{avg_n_appr} & #{avg_n_appr_by_n} \\"

end

def show_mean values
"%5.2f" % @@r_instance.mean(values)
end

def get_n_appr(l_instances)
l_instances.collect {|instance| instance.n_appr}
end

The final output is:

51.00 & 22.40 & $2^7$ & 16.60 & 0.12 & 8.23 & 23.76 & 1.07 \\
51.00 & 22.40 & $2^8$ & 18.45 & 0.13 & 17.17 & 23.93 & 1.06 \\
51.00 & 22.40 & $2^9$ & 19.20 & 0.12 & 33.23 & 24.21 & 1.07 \\
...


Summary

In this post, I have looked at how to process a text file line by line, and turn it into a collection of Ruby objects. The information in these objects has then been aggregated and processed, using a statistical library to compute some data, before producing the results ready to include in an article. The reward for taking the trouble on such a program comes the second or third time you have to use it, when your data are analysed immediately. The potential to automate routine data analysis and generate tables and graphs with Ruby is enormous. This post, on a small scale, encapsulates what this weblog is about. Using Ruby in conjunction with other programs to generate or process data can save a lot of time and effort, leaving more time to think about the conclusions from your experiments. In later posts, I will explore ways in which Ruby can help us analyse data smarter.