| Previous: What Is? |
Getting Started
The following short tutorial walks the user through the application of the malibu workbench using a number of companion bash scripts (intended to run on GNU/Linux). While these scripts are not necessary to use malibu, they provide a simplified means to run an experiment and evaluate its results.
1. Install malibu program (See Instructions)
2. Download datasets
Use the getdata script located in the script directory e.g.
[malibu]$ ./script/getdata uci
This will take some time and does not give any output. Alternatively, the datasets can be downloaded individually at the UCI Dataset Repository. A list of the dataset you will need can be found in the getdata script. Download them to the directory "data/uci" in the malibu directory.
3. Examine example experiment configurations
The malibu workbench has four groups of parameters: dataset format, experiment, algorithm and evaluation. Here, we combine the experiment parameters with the dataset format parameters. This is a natural grouping because the type of experiment is heavily dependent on the dataset.
In the script directory, you will find the experiments script. This script contains eight variable definitions describing 10-fold cross-validation experiments for 8 of the 13 datasets downloaded by the getdata script.
[malibu]$ more script/experiments
4. Examine example algorithm configurations
In the script directory, you will also file the algorithms script. This script contains a set of functions that act as a simple interface to a number of example algorithm configurations available in malibu. Each function has a number of optional arguments.
[malibu]$ more script/algorithms
5. Running the example experiments and algorithms
One method to use a malibu algorithm or run an experiment is through the use of the source command. This command temporarily loads the functions and variables defined in the scripts into the current bash shell:
[malibu]$ source ./script/experiments
[malibu]$ source ./script/algorithms
Once the scripts have been loaded, an experiment can be run as follows:
[malibu]$ c45 $hepatitis
# or
[malibu]$ glibsvm 0.0001-33*2 $chess
Each function defines a set of optional parameters. A description can be found in the algorithms script. The results of each experiment are written to the standard output (stdout), so it is best to redirect the command to a file (error can be redirected to a separate file):
[malibu]$ c45 $hepatitis > output.txt
# or
[malibu]$ c45 $hepatitis > output.txt 2> errors.txt
Some of the available machine learning algorithms may take considerable time to complete. Thus, it is often desireable to run such processes in the background; this is accomplished by appending a "&" to the end of the line:
[malibu]$ c45 $hepatitis > output.txt &
Finally, if the algorithms are run on a remote machine, you might not be able to stay connected durning the duration of the process. Normally, this is remedied by prefixing the line with the nohup command (this is wrong in this case):
When you run c45, you are running a bash function, not a command; nohup only works with commands. Therefore, you must do the following to use nohup:
[malibu]$ prefix=nohup
[malibu]$ c45 $hepatitis > output.txt &
[malibu]$ prefix=
Another method for running algorithms and experiments is the benchmark script. This script takes two quoted parameters <algorithms> and <experiments>, e.g.
[malibu]$ ./script/benchmark "c45 tree" "hepatitis chess"
The above example runs both the c45 and the tree algorithms on both the hepatitis and chess experiments. When using the benchmark script, you cannot set the optional algorithm parameters on the command line. Also, the "$" must be excluded for the experiments.
The benchmark script writes a results file for each algorithm and experiment combination to the output directory. The files are named first by the algorithm then the experiment name with a separating underscore. In addition, a set of average metrics (average in the case of multiple run experiments) is written to the standard output stream for each algorithm/experiment. It is best to also redirect the output of this script to a file summarizing the performance of each algorithm/experiment combination. Likewise, benchmarking may take considerable time; thus it is recommended you run benchmarking using nohup and in the background (see 6).
5. Evaluating the results
The malibu workbench contains an application for evaluating your classifiers called "evaluate". This program takes a single command line argument describing the format for your results and a results file redirected to stdin (see below).
[malibu]$ ./bin/evaluate roc < c45_ionosphere.txt
In the example above (9), the evaluate program writes (to the console) to stdout a ROC plot (in text form) formatted for GNUPlot, which describes the success of the C45 tree over the ionosphere dataset. To obtain, an actual ROC plot image use one of the two examples below.
[malibu]$ ./bin/evaluate roc < c45_ionosphere.txt | gnuplot
or
[malibu]$ ./bin/evaluate roc < c45_ionosphere.txt > out
[malibu]$ gnuplot out
In both cases, your ROC plot will be found in a file called roc.png (the Plot you see below). In the second part of the example, you can edit the out file to change the properites of the ROC plot, e.g. title.
The evaluate program is also capable of printing a number of ROC curves on the same plot. For example, lets say you want to compare several learning algorithms over the same dataset. In this case, you used the benchmark script to run c45, c45boost and adtree over the ionosphere dataset. To use a lower ROC plot to compare these algorithm over the same dataset, use the "roc.d" format.
[malibu]$ cat *ionosphere.txt | ./bin/evaluate roc.d | gnuplot
or
[malibu]$ cat *ionosphere.txt | ./bin/evaluate roc.d > out
[malibu]$ gnuplot out
The resulting plot can be seen below.
The evaluate program also has formats for writing metrics. Below is an example of writing out metrics for the previously mentioned experiment.
[malibu]$ cat *ionosphere.txt | ./bin/evaluate
The resulting metrics can be seen below (edited for clarity).
Algorithm Accuracy Sensitivity Specificity AUC_ROC |
| Previous: What Is? |