Recitation 9: Property-Based Testing

Overview

In this recitation, you will use fast-check to write property-based tests for PierogIO, the pierogi delivery service you tested in lecture last week. The codebase is riddled with bugs! Working in your project teams, you'll write property-based tests to find and fix as many bugs as possible. A small prize will be awarded to the team that finds the most bugs at the start of Thursday's lecture.

Setup

1. One person in your team should go to the PierogIO repository and click the "Use this template" button in the top right corner before selecting the "Create repository" option, as shown below.

   

   That person should then choose to create a private repository, select "CMU-313" as the owner, and name it "pierogio-your-team-name" before hitting the "Create repository" button.

   

2. After creating the repository, that person should invite the rest of the team as collaborators. To do so, go to the Settings page for your repository, then select "Collaborators and teams" from the menu on the left. You can then either click the Add team button and type in / select your team, or you can click Add people and manually add the rest of your team to the repository.

   

3. Finally, everyone should be able to access the repository. You can then use Codespaces to open up the repository or you can clone it to your machine and install it as a DevContainer (CMD/CTRL+SHIFT+P: "Rebuild and Reopen in Dev Container"). You should now be able to interact with the codebase.

   

   Codespaces is just an IDE in the Cloud

   Be aware that any changes you make within the Codespace won't be saved to GitHub unless and until you push your changes. Similarly, you won't see any changes that your team members make until you pull those changes.

Background: fast-check

To write property-based tests in fast-check, you need two things: (1) a way to generate inputs at random (known as an arbitrary), and (2) a predicate that checks whether the property holds on a given input. To help get you started, we have provided starter test code in ./tests/properties/. Below is an example of a property-based test from ./tests/properties/order.properties.spec.js:

// we describe our property in natural language
it('subtotal should always be non-negative integer', () => {

  // we call fc.assert to assert that a given property should be true for all generated inputs
  fc.assert(

    // we use fc.property to construct the property we want to test
    // fc.property takes n + 1 arguments
    fc.property(
        // the first n arguments specify the arbitraries that
        // should be used to construct inputs (n=1 in this case)
        orderArb,

        // the last argument to fc.property is a predicate that takes n arguments
        // and returns true if the property is satisfied on that input
        // each argument corresponds to a concrete value that is generated by the i-th arbitrary
        (order) => {

            // we supply the generated inputs to the function that we want to test
            const result = subtotal(order);

            // we check our property over the result
            // in this case, we don't find any bugs!
            // but we can verify that our property works by intentionally breaking the code
            // (temporarily) and having it negate the price before returning the value
            return result >= 0 && Number.isInteger(result);

        }
    ),

    // below, we provide extra optional arguments to `fc.assert`
    // numRuns specifies how many random test cases to generate and check
    { numRuns: 50 }

  );

});

Generating Inputs via Arbitraries

Arbitraries are responsible for generating random inputs to the function that you would like to test. To help get you started, we have provided orderArb which will create random orders.

To test other functions, you'll need to write your own arbitraries. For example, to test functions that require a context object (like total(), tax(), or discount()), you'll need to create a contextArb. Look at how orderArb is defined and use the fast-check documentation to create arbitraries for other inputs.

Verifying Properties with Predicates

The predicate is a function that returns true if the property holds for the given inputs, and false otherwise. In the example above, our predicate checks two things:

(order) => {
    const result = subtotal(order);
    return result >= 0 && Number.isInteger(result);
}

This predicate verifies that:

1. The subtotal is non-negative (result >= 0)
2. The subtotal is an integer (Number.isInteger(result))

When writing predicates, think about what should always be true about your function's output, regardless of the input. Common properties include:

• Range properties: The output is within expected bounds (e.g., non-negative, less than a maximum)
• Type properties: The output has the expected type (e.g., integer, string, array)
• Relationship properties: The output relates to the input in a predictable way (e.g., output length equals input length)
• Invariant properties: Certain conditions never change (e.g., sorted arrays stay sorted after filtering)

If fast-check finds an input where your predicate returns false, it will report a test failure and show you the counterexample.

Running Tests

To run the property-based tests:

npm test

Alternatively, click the Play icon to the left of the line number to run tests directly in the IDE.

Activity: Bug Hunt Challenge

Your team's goal is to find as many bugs as possible using property-based testing. For each bug you find:

1. Write a property-based test that fails due to the bug
2. Document the bug by adding a comment above the test explaining what property is violated
3. Fix the bug in the source code and push your changes to main
4. Verify that your test now passes

Getting Started

Since you already wrote example-based tests for PierogIO during lecture, you're familiar with the codebase. Follow these steps to make the most of your recitation time:

1. Set up your environment (5 minutes): Clone your team's repository and run npm test to ensure everything works
2. Pick a module (2 minutes): Have each pair in your team claim a different module to focus on (e.g., delivery, tax, discounts, total)
3. Iterate: Write properties, find bugs, fix bugs (45 minutes): Repeat this loop as many times as possible:
   • Come up with a property that should hold for your module
   • Turn it into a property-based test
   • Run the test: Does it fail?
     • Yes → You found a bug! Fix it in the source code, verify the test passes, then repeat
     • No → Great! Your code satisfies that property. Now repeat with a new property
4. Share findings (5 minutes): Gather as a team and count how many bugs you found

Working as a Team

• Pair up: Work in pairs of 2 on different modules to cover more ground
• Rotate: Consider switching pairs halfway through to share knowledge
• Share findings: Use the last 5 minutes to discuss what each pair discovered
• Push often: Commit and push your changes regularly so teammates can see your progress

Submission

Before leaving recitation, each student should submit individually to Gradescope:

1. A brief reflection (3-5 sentences) answering:

   • What bugs did your team find?
   • What was the most interesting property you wrote?
   • What was challenging about property-based testing?

2. Your team's repository URL

The team with the most bugs found wins a prize! Winners will be announced at the start of Thursday's lecture.