View on GitHub

ParkBench: Advanced Features

Download this project as a .zip file Download this project as a tar.gz file

First, did you read the 10-minute introduction to ParkBench?

Quick Links

Calling external commands

So far, we assumed that the execution of an experiment occurs in the runExperiment() method of your experiment class. Of course, you can write Java experiment code outside of that method (using any classes and packages you like), and call it from within runExperiment() --this is Java after all. But what if some of your experiment code is not in Java? ParkBench provides some helper classes and objects that allow you to run commands at the command line, as well as read and parse their output. Therefore, you can also use external programs in a ParkBench experiment suite.

A simple way of doing so is to create an experiment that inherits from the CommandExperiment class. An empty experiment looks as follows:

import ca.uqac.lif.parkbench.*;

class ProcedureB extends CommandExperiment {

  public ProcedureB() {
    super("Procedure B");
  }
  
  public Experiment newExperiment() {
    return new ProcedureB();
  }
  
  public void createCommand(Parameters input, List<String> command) {
  }
  
  public void readOutput(String output, Parameters results) {
  }
}

The first part of the file is the same boilerplate code as in a regular experiment. In this case, the work happens in the createCommand() and readOutput() methods.

Generating the command to run

The method createCommand() is responsible for creating the command-line string that will be called. Since that external call will probably depend on the experiment's concrete parameters, these parameters are passed as the argument input; you can read from them as usual. The resulting command line is written into the command argument; it is a list of Strings, each of which is a part of the command to run.

Suppose for example that Procedure B takes two numerical parameters, x and y, and requires to call the external program myprogram. Suppose also that the value of x must be passed as is to myprogram, while the value of y must be passed as the -a command-line switch. Method createCommand() could look like this:

  public void createCommand(Parameters input, List<String> command) {
    int x = input.getNumber("x").intValue();
    int y = input.getNumber("y").intValue();
    command.add("myprogram").add(x).add("-a " + y);
  }

The first two lines extract the values of x and y as before. The last line creates the command line string. It first adds myprogram, followed by the first argument (the value of x), followed by the second (the value of y passed as the command-line argument -a). If x=2 and y=3, this would result in the following command-line string:

myprogram 2 -a 3

Processing the output

The second part is to do something with the output of the program. To this end, method readOutput() is called once the command has run. It contains in argument output the String of what the command sent to the standard output (if that output contained multiple lines, these lines are present in the string). After processing that string content, you should, as usual, put whatever results of your experiments into the results parameter map.

Suppose that mycommand prints to the standrd output a single number, which we will use as the output z. We need to write the following code:

  public void readOutput(String output, Parameters results) {
    int result = Integer.parseInt(output);
    results.put("z", result);
  }

That's it. If you have already read the 10-minute tutorial, you can add instances of ProcedureB to the same experiment suite we created earlier.

Of course, it seldom happens that the program's output contains directly the value we are looking for; more often than not, we need to extract that value from a more complicated output. For example, mycommand could output something like this:

The output value of this program is 3. Good bye!

In such a case, you can use Java's standard regular expression matching to parse such a string. Method readOutput() could look like:

  public void readOutput(String output, Parameters results) {
    Pattern p = Pattern.compile("The output value of this program is (\\d+)");
    Matcher m = p.matcher(output);
    if (m.find()) {
      results.put("z", Integer.parseInt(m.group(1)));
    }
  }

This document is by no means a tutorial on regular expressions, but the point is to show how you can use whatever code you wish to break the program's output and make experiment results out of it.

Advantages of extending the CommandExperiment class over running the commands by yourself are multiple:

Experiment prerequisites

Sometimes your experiment will require the presence of other artifacts (external files, connection to a database, etc.) before it can run; these are called prerequisites. You can write code taking care of that directly in the experiments's runExperiment() method, but this method is not available to you in the case of CommandExperiments and some other experiment classes. A cleaner way of doing so is to use two methods defined for every Experiment: prerequisitesFulfilled() and fulfillPrerequisites().

Let us create a new experiment class, ProcedureC, with the same input parameters x and y as for the other experiments. We don't care about what it does; however suppose that procedure C, in order to run, requires that the values of x and y be in a text file, separated by a comma.

Check for prerequisites

Method prerequisitesFulfilled(), as its name implies, is there to check whether the experiment's prerequisites are fulfilled. In the case of ProcedureC, this means checking that the input file corresponding to that experiment exists in the file system. Suppose that every experiment instance has its own input file, which we name experimentC-x-y.csv (you can use whatever naming convention you wish). Then checking that the experiment's prerequisite is fulfilled can be written as:

  public boolean prerequisitesFulfilled(Parameters input) {
    int x = input.getNumber("x").intValue();
    int y = input.getNumber("y").intValue();
    String filename = "experimentC-" + x + "-" + y + ".csv";
    return CommandRunner.fileExists(filename);
  }

Lines 1-3 build the filename from the experiment's input parameters. Line 4 calls fileExists(), a helper method of class CommandRunner which checks the existence of the file. (You can use whatever other means you wish to check for that, but ParkBench provides a bunch of helper methods for the most common tasks.) If the file exists, fileExists() returns true, and the experiment's prerequisites are fulfilled. Otherwise the method returns false.

The effects of such a method have an impact in ParkBench's web interface. When an experiment has prerequisites that are not fulfilled, its status square is different (Not ready) than when it does not have prerequisites or these prerequisites are already fulfilled (Ready).

Generate prerequisites

The second step is to take care of generating the experiment's prerequisites. This is the task of method fulfillPrerequisites(). As we have already said, for ProcedureC this amounts to generating a text file with values of x and y separated by a comma. Implementing this is straightforward:

  public boolean prerequisitesFulfilled(Parameters input) {
    int x = input.getNumber("x").intValue();
    int y = input.getNumber("y").intValue();
    String filename = "experimentC-" + x + "-" + y + ".csv";
    String file_contents = x + "," + y;
    FileReadWrite.writeToFile(filename, file_contents);
  }

The last two lines generate the file's contents, and write them into the proper filename. Here, FileReadWrite is another convenience class that allows you to easily write character strings into a file.

Note here that the part of code generating the filename from input parameters is repeated in both prerequisitesFulfilled() and fulfillPrerequisites(). A cleaner implementation would refactor that functionality into a separate method (say getFilename()), which would be called by both prerequisite methods. From here on, you are free to implement the inner workings of the experiment as you would do for any other piece of code --good coding practices still apply.

This, again, can be seen in the web interface. When an experiment is generating its prerequisites (which still count into the experiment's global running time), its status square (Prerequisites) is different than when the experiment itself is running (Running). ParkBench is wise, however: if an experiment declares that its prerequisites are already fulfilled, it will not call its fulfillPrerequisites() method another time before running it.

Calling other commands

Generating prerequisites can be done directly as Java code (as for running the experiment itself), or can use external commands. ParkBench provides the CommandRunner class to help you launch external programs, receive their output, as well as manipulate filenames (change their extension, get the base name, etc.). As usual, if these functions do not fit your needs, you are free to write whatever Java code you like.

Resuming an interrupted experiment suite

We said in the first part of this tutorial that ParkBench was resistant to crashes and interruptions by regularly saving the state of every experiment in a JSON file. This makes sure that whatever experiment results that were already computed are not lost, but what happens of the experiments that were not executed?

It is possible to use the JSON file to reload an experiment suite and put it back into the same state as it was when that file was saved. This means that ParkBench will recreate the same experiment instances with the same input parameters, and that any experiments that were already completed will be marked as such (complete with all their output values). From that point on, is it possible to start any experiment in the suite, whose results will be added to the existing results when it completes. Hence it is possible to load an "empty" experiment suite, run a few experiments, save the suite to a file, reload that file at a later time to run a few more experiments, save the file again, etc.

To load an experiment suite with a JSON file, one simply needs to pass the filename as a command-line parameter to the experiment suite; for example:

java -jar MyExperimentSuite.jar --interactive Untitled.json

Note that it does not matter if the ExperimentSuite object and the JSON file define experiments with different parameters, as long as they contain experiments of the same classes. ParkBench will overwrite whatever experiments instances the ExperimentSuite contains with those defined in the file.

Merging and running tests on multiple machines

If you use the --merge option, ParkBench will merge, rather than overwrite, the tests from the JSON file. Only tests that have the status DONE in the file will be imported in the test suite.

This option makes it easy to split the execution of your test suite across multiple machines. Simply…

  1. Start the same test suite on every computer.
  2. Select different test instances to run on each (using e.g. filtering through the web interface)
  3. Save the JSON file resulting from the execution of each test suite.
  4. In one instance of the test suite, merge the JSON files produced by the others. Only tests that are finished will be imported.

It is possible to keep track of which test was executed on which machine, as every test carries in its parameters the IP address of the host where it was run. That value remains empty until the test is started.

Creating plots

ParkBench can also auto-create (and auto-update) plots created from experiment results. For example, you may want to create a two-dimensional, x-y plot where each dot represents a value of the x input parameter and a value of the z input parameter. The simplest way to do so is to add the following lines at the end of the setup() method of your experiment suite:

  Plot plot = new ScatterPlot("The plot's title")
    .withLines()
    .setParameterX("x").setCaptionX("Value of x")
    .setParameterY("z").setCaptionY("Value of z");
    b.addPlotToAll(plot);
  }

Methods setParameterX() and setParameterY() tell the plot what values in each experiment to pick for the x and y coordinates (in this case, we use x and z). Methods setCaptionX() and setCaptionY() provide the caption for the x and y axis in the plot (they are optional).

Restart your experiment suite in interactive mode and re-run the experiments. This time, in the graphical interface, you can select the "Plots" sub-menu at the top of the page, which will lead you to a display of all plots associated with your experiment suite. You will also notice that the plots auto-update periodically (every few seconds), so that you can see the trends in the results as the data from the experiments progressively comes in. Buttons for each plot also allow you to manually refresh a graph, download the graph locally in PDF format, and download a stand-alone GnuPlot file that generates that graph. (As a matter of fact, ParkBench uses GnuPlot in the background to produce the images.)

As the plot above is two-dimensional, ParkBench will create one data series with every combination of experiment parameters that remains. For example, in our case, ProcedureA has two input parameters: x and y. Parameter x is used for the x-axis, but parameter y is not used. Therefore, ParkBench will treat all experiments with y=1 as one data series, using the values of x and z to plot a line of a given colour. Similarly, all experiments with y=2 will be another data series, which ParkBench will plot as another line of a different colour, etc. Moreover, the name of the experiment also counts as a parameter, so experiments of ProcedureA with y=0 will be in a different data series as the experiments of ProcedureB with y=0.