Incrementality Example

In this example, we will run our build system and show off simple incrementality with a task that reads a string from a file.

ReadStringFromFile task

Create the pie/examples directory, and create the pie/examples/incrementality.rs file with the following contents:

#![allow(unused_imports, unused_variables)]

use std::io::{self, Read};
use std::path::{Path, PathBuf};

use dev_shared::{create_temp_dir, write_until_modified};
use pie::{Context, Task};
use pie::context::top_down::TopDownContext;
use pie::stamp::FileStamper;

/// Task that reads a string from a file.
#[derive(Clone, Ord, PartialOrd, Eq, PartialEq, Hash, Debug)]
struct ReadStringFromFile(PathBuf, FileStamper);

impl ReadStringFromFile {
  fn new(path: impl AsRef<Path>, stamper: FileStamper) -> Self {
    Self(path.as_ref().to_path_buf(), stamper)
  }
}

impl Task for ReadStringFromFile {
  type Output = Result<String, io::ErrorKind>;
  fn execute<C: Context<Self>>(&self, context: &mut C) -> Self::Output {
    println!("Reading from {} with {:?} stamper", self.0.file_name().unwrap().to_string_lossy(), self.1);
    let file = context.require_file_with_stamper(&self.0, self.1).map_err(|e| e.kind())?;
    if let Some(mut file) = file {
      let mut string = String::new();
      file.read_to_string(&mut string).map_err(|e| e.kind())?;
      Ok(string)
    } else {
      Err(io::ErrorKind::NotFound)
    }
  }
}

The ReadStringFromFile task is similar to the one we defined earlier in a test, but this one accepts a FileStamper as input, and propagates errors by returning a Result. We cannot use std::io::Error as the error in the Result, because it does not implement Clone nor Eq, which need to be implemented for task outputs. Therefore, we use std::io::ErrorKind which does implement these traits.

Exploring incrementality

We’ve implemented the task, now add a main function to pie/examples/incrementality.rs:

fn main() -> Result<(), io::Error> {
  let temp_dir = create_temp_dir()?;
  let input_file = temp_dir.path().join("input.txt");
  write_until_modified(&input_file, "Hi")?;

  let mut context = TopDownContext::new();
  let read_task = ReadStringFromFile::new(&input_file, FileStamper::Modified);

  println!("A) New task: expect `read_task` to execute");
  // `read_task` is new, meaning that we have no cached output for it, thus it must be executed.
  let output = context.require_task(&read_task)?;
  assert_eq!(&output, "Hi");

  Ok(())
}

We create a temporary file, create a task, create a context, and require our first task. Run this example with cargo run --example incremental. You should see the println! in ReadStringFromFile appear in your console as the incremental context correctly determines that this task is new (i.e., has no output) and must be executed. It should look something like:

   Compiling pie v0.1.0 (/pie)
    Finished dev [unoptimized + debuginfo] target(s) in 0.60s
     Running `target/debug/examples/incremental`
A) New task: expect `read_task` to execute
Reading from input.txt with Modified stamper

Reuse

If we require the task again, what should happen?

Insert the following code into the main method:

  println!("\nB) Reuse: expect no execution");
  // `read_task` is not new and its file dependency is still consistent. It is consistent because the modified time of
  // `input_file` has not changed, thus the modified stamp is equal.
  let output = context.require_task(&read_task)?;
  assert_eq!(&output, "Hi");

Running with cargo run --example incremental should produce output like:

   Compiling pie v0.1.0 (/pie)
    Finished dev [unoptimized + debuginfo] target(s) in 0.34s
     Running `target/debug/examples/incremental`
A) New task: expect `read_task` to execute
Reading from input.txt with Modified stamper

B) Reuse: expect no execution

We don’t see the println! from ReadStringFromFile, so it was not executed, so our incremental build system has correctly reused its output!

Normally we would write a test to confirm that the task was executed the first time, and that it was not executed the second time. However, this is not trivial. How do we know if the task was executed? We could track it with a global mutable boolean that ReadStringFromFile keeps track of, but this quickly becomes a mess. Therefore, we will look into creating a proper testing infrastructure in the next chapter.

For now, we will continue this example with a couple more interesting cases. The comments in the code explain in more detail why the build system behaves in this way.

Inconsistent file dependency

Insert into the main method:

  write_until_modified(&input_file, "Hello")?;
  println!("\nC) Inconsistent file dependency: expect `read_task` to execute");
  // The file dependency of `read_task` is inconsistent due to the changed modified time of `input_file`.
  let output = context.require_task(&read_task)?;
  assert_eq!(&output, "Hello");

If we change the file (using write_until_modified to ensure that the modified time changes to trigger the Modified file stamper) and require the task, it should execute, because the file dependency of the task is no longer consistent.

Different tasks

Insert into the main method:

  let input_file_b = temp_dir.path().join("input_b.txt");
  write_until_modified(&input_file_b, "Test")?;
  let read_task_b_modified = ReadStringFromFile::new(&input_file_b, FileStamper::Modified);
  let read_task_b_exists = ReadStringFromFile::new(&input_file_b, FileStamper::Exists);
  println!("\nD) Different tasks: expect `read_task_b_modified` and `read_task_b_exists` to execute");
  // Task `read_task`, `read_task_b_modified` and `read_task_b_exists` are different, due to their `Eq` implementation
  // determining that their paths and stampers are different. Therefore, `read_task_b_modified` and `read_task_b_exists`
  // are new tasks, and must be executed.
  let output = context.require_task(&read_task_b_modified)?;
  assert_eq!(&output, "Test");
  let output = context.require_task(&read_task_b_exists)?;
  assert_eq!(&output, "Test");

The identity of tasks is determined by their Eq and Hash implementations, which are typically derived to compare and hash all their fields. Therefore, if we create read tasks for different input file input_file_b and different stamper FileStamper::Exists, these read tasks are not equal to the existing read task, and thus are new tasks with a different identity. We require read_task_b_modified and read_task_b_exists, they are new, and are therefore executed.

Same file different stampers

Insert into the main method:

  write_until_modified(&input_file_b, "Test Test")?;
  println!("\nE) Different stampers: expect only `read_task_b_modified` to execute");
  // Both `read_task_b_modified` and `read_task_b_exists` read from the same file, but they use different stampers.
  // Therefore, `read_task_b_modified` must be executed because the modified time has changed, but `read_task_b_exists`
  // will not be executed because its file dependency stamper only checks for existence of the file, and the existence
  // of the file has not changed.
  //
  // Note that using an `Exists` stamper for this task does not make a lot of sense, since it will only read the file
  // on first execute and when it is recreated. But this is just to demonstrate different stampers.
  let output = context.require_task(&read_task_b_modified)?;
  assert_eq!(&output, "Test Test");
  let output = context.require_task(&read_task_b_exists)?;
  assert_eq!(&output, "Test");

Here we write to input_file_b and then require read_task_b_modified and read_task_b_exists. We expect read_task_b_modified to be executed, but read_task_b_exists to be skipped, because its file dependency only checks for the existence of the input file, which has not changed. This shows that tasks can depend on the same file with different stampers, which influences whether the tasks are affected by a file change individually.

Of course, using an Exists stamper for ReadStringFromFile does not make a lot of sense, but this is for demonstration purposes only.

Running cargo run --example incremental now should produce output like:

   Compiling pie v0.1.0 (/pie)
    Finished dev [unoptimized + debuginfo] target(s) in 0.35s
     Running `target/debug/examples/incremental`
A) New task: expect `read_task` to execute
Reading from input.txt with Modified stamper

B) Reuse: expect no execution

C) Inconsistent file dependency: expect `read_task` to execute
Reading from input.txt with Modified stamper

D) Different tasks: expect `read_task_b_modified` and `read_task_b_exists` to execute
Reading from input_b.txt with Modified stamper
Reading from input_b.txt with Exists stamper

E) Different stampers: expect only `read_task_b_modified` to execute
Reading from input_b.txt with Modified stamper

Feel free to experiment more with this example (or new example files) before continuing. In the next chapter, we will define minimality and soundness, set up an infrastructure for testing those properties, and fix issues uncovered by testing.