Regression Testing the ThreadFix CLl with JUnit Parameterized Tests

Many applications have some form of external API that allows users to call actions or return information from outside of the UI. As the functionality of an application grows, the number of available API calls will likely (and will hopefully) grow alongside it. With something like a REST API, unwanted changes could cause calls to return different outputs, but in wrapper APIs like a command line interface or language-specific API, it can also cause outright errors and exceptions.

Now regression suites can be powerful tools that simulate a wide range of possible interactions that a user can have with an application. When you introduce several versions of a specific API clients (in this case a command line interface), then you multiply the possible number of ways that a user could interact with your application. Luckily, there are a couple of useful JUnit features to handle these situations.

Parameterized Tests
Parameterized tests are useful for simulating situations where you want to execute the same action, but under slightly different conditions. In this example, we want to create a team with the command line interface, but using one of three different CLI jars on each run.

The first step is to annotate your test class to use JUnit’s Parameterized class:

public class CommandLineIT extends BaseDataTest {

The next step is to declare an instance variable that will represent our parameter for each test run. Along with this, you define a public constructor that takes your parameter and sets it to the corresponding instance variable. Our use case only requires one parameter to be set each test run (versionNumber):

private String versionNumber;
public CommandLineIT(String versionNumber) {
    this.versionNumber = versionNumber;

Lastly, you create a method that returns a two-dimensional array. Each of the values in each of the inner arrays will be assigned to the corresponding parameter for that test run. In our case, the one value for each inner array will be set as the API version number for that run (The API versions are represented as Integer-compatible Strings in order to simplify comparisons between versions. More on this in the next section.) Also make sure to properly annotate the method to indicate that it’s what defines your parameter values.

public static List<String[]> getVersions() {
    return Arrays.asList(new String[][] {

Now we have our test class set up to run as parameterized tests. For each test that we’ve defined in this class, it will run once for each set of parameters. Now we can test the functionality of creating teams using the 2.1, 2.2, and 2.3 API jars.

If you have your IntelliJ configured to run and show JUnit results, it even cleanly displays the results of each parameter run on a test method.


Now say we’re focusing on a test that simulates a feature added in version 2.2. It would be misleading to attempt to execute that functionality from an earlier API that couldn’t have possibly incorporated that feature, and it would only lead to false positive test failures. JUnit provides another tool to handle these cases.

If you’ve used JUnit, you’re likely familiar with assertTrue() or assertFalse() and how they set expected conditions for a successful test run. However, there is also a lesser-used assumeTrue() method which defines conditions for a relevant test run. From the JUnit documentation:

A set of methods useful for stating assumptions about the conditions in which a test is meaningful. A failed assumption does not mean the code is broken, but that the test provides no useful information.

This description fits the bill for our scenario: a CLI attempting to make a call that didn’t exist when it was released. To implement this into our suite, we first create a checkVersion method in our CommandLineUtils class.

public static void checkVersion(int introductionVersion) {
    int currentVersion = Integer.valueOf(cliVersion);
    assumeTrue(introductionVersion <= currentVersion);

For our case, where we’re testing major versions of an API, we can cast our version numbers to Integer values and compare them to determine whether this API should have this action. It is worth nothing that an ordered Enum would accomplish the same result and even provide better support for minor and patch versions if required.

You can call this from the beginning of your newer test methods, passing in the introductionVersion number where that API call was first created.

public void testSearchTagByName() {
    String tagName = getName();
    dbUtils.createTag(tagName, "Application");
    JSONObject response = cliUtils.searchTagByName(tagName);
    assertTrue("Tag was not found.", cliUtils.isCommandResponseSuccessful(response));

Tests that do not meet assumptions are not reported as failed, so they don’t skew test results.


Hopefully this can help out in situations where a feature may not have one cut-and-dry entry point or context for testing. Feel free to leave any comments or notes for improvement below!

About Dan Cornell

A globally recognized application security expert and the creator of ThreadFix, Dan Cornell holds 20 years of experience architecting, developing and securing web-based software systems. As the Chief Technology Officer and a Principal at Denim Group, Ltd, the parent company of ThreadFix, he leads the technology team to help Fortune 500 companies and government organizations integrate security throughout the development process.