Stamps

To check whether we need to execute a task, we need to check the dependencies of that task to see if any of them are inconsistent. To make this consistency checking configurable, we will use stamps. A dependency is inconsistent if after stamping, the new stamp is different from the old stamp. Therefore, we will implement a FileStamper that stamps files and produces a FileStamp, and an OutputStamper that stamps task outputs and produces an OutputStamp.

Add the stamp module to pie/src/lib.rs:

This module is public as users of the library will construct stampers.

File stamps

Create the pie/src/stamp.rs file and add:

use std::fmt::Debug;
use std::io;
use std::path::Path;
use std::time::SystemTime;

use crate::fs::metadata;

#[derive(Copy, Clone, Ord, PartialOrd, Eq, PartialEq, Hash, Debug)]
pub enum FileStamper {
  Exists,
  Modified,
}

#[derive(Copy, Clone, Ord, PartialOrd, Eq, PartialEq, Hash, Debug)]
pub enum FileStamp {
  Exists(bool),
  Modified(Option<SystemTime>),
}

impl FileStamper {
  pub fn stamp(&self, path: impl AsRef<Path>) -> Result<FileStamp, io::Error> {
    match self {
      FileStamper::Exists => {
        Ok(FileStamp::Exists(path.as_ref().try_exists()?))
      }
      FileStamper::Modified => {
        let Some(metadata) = metadata(path)? else {
          return Ok(FileStamp::Modified(None));
        };
        Ok(FileStamp::Modified(Some(metadata.modified()?)))
      }
    }
  }
}

We’re implementing FileStamper as an enum for simplicity.

A FileStamper has a single method stamp which takes something that can be dereferenced to a path, and produces a FileStamp or an error if creating the stamp failed. For now, we implement only two kinds of file stampers: Exists and Modified. The Exists stamper just returns a boolean indicating whether a file exists. It can be used to create a file dependency where a task behaves differently based on whether a file exists or not. The Modified stamper returns the last modification date if the file exists, or None if the file does not exist.

We derive Eq for stamps so that we can compare them. Equal (same) stamps indicate a consistent dependency, unequal (different) indicates inconsistent. We also derive Eq for stampers, because the stamper of a dependency could change, making the dependency inconsistent.

Task output stamps

We implement task output stampers in a similar way. Add to pie/src/stamp.rs:

#[derive(Copy, Clone, Ord, PartialOrd, Eq, PartialEq, Hash, Debug)]
pub enum OutputStamper {
  Inconsequential,
  Equals,
}

#[derive(Clone, Eq, PartialEq, Debug)]
pub enum OutputStamp<O> {
  Inconsequential,
  Equals(O),
}

impl OutputStamper {
  pub fn stamp<O>(&self, output: O) -> OutputStamp<O> {
    match self {
      OutputStamper::Inconsequential => OutputStamp::Inconsequential,
      OutputStamper::Equals => OutputStamp::Equals(output),
    }
  }
}

The Inconsequential stamper simply ignores the output and always returns the same stamp (thus is always equal). It can be used to create a task dependency where we are interested in some side effect of a task, but don’t care about its output. The Equals stamper simply wraps the output of a task, so the stamp is equal when the output is equal.

Output stamps are generic over the task output type O.

Tests

Finally, we write some tests. Add to pie/src/stamp.rs:

#[cfg(test)]
mod test {
  use std::fs::{remove_file, write};
  use std::io;

  use dev_shared::create_temp_file;

  use super::*;

  #[test]
  fn test_exists_file_stamper() -> Result<(), io::Error> {
    let stamper = FileStamper::Exists;
    let temp_file = create_temp_file()?;
    let stamp = stamper.stamp(&temp_file)?;
    assert_eq!(stamp, stamper.stamp(&temp_file)?);

    remove_file(&temp_file)?;
    assert_ne!(stamp, stamper.stamp(&temp_file)?);

    Ok(())
  }

  #[test]
  fn test_modified_file_stamper() -> Result<(), io::Error> {
    let stamper = FileStamper::Modified;
    let temp_file = create_temp_file()?;
    let stamp = stamper.stamp(&temp_file)?;
    assert_eq!(stamp, stamper.stamp(&temp_file)?);

    write(&temp_file, format!("{:?}", stamp))?;
    let new_stamp = stamper.stamp(&temp_file)?;
    assert_ne!(stamp, new_stamp);
    let stamp = new_stamp;

    remove_file(&temp_file)?;
    assert_ne!(stamp, stamper.stamp(&temp_file)?);

    Ok(())
  }

  #[test]
  fn test_inconsequential_output_stamper() {
    let stamper = OutputStamper::Inconsequential;
    let stamp = stamper.stamp(&1);
    assert_eq!(stamp, stamper.stamp(&1));
    assert_eq!(stamp, stamper.stamp(&2));
  }

  #[test]
  fn test_equals_output_stamper() {
    let stamper = OutputStamper::Equals;
    let stamp = stamper.stamp(&1);
    assert_eq!(stamp, stamper.stamp(&1));
    assert_ne!(stamp, stamper.stamp(&2));
  }
}

We test file stamps by creating a stamp, changing the file, creating a new stamp, and then compare the stamps. We test task output stamps by just passing a different output value to the stamp function, and then compare the stamps.

Run cargo test to test the stamp implementation. However, a test could fail on some operating systems. Do continue to the next subsection if this happens.

Testing with file modified time, correctly

Unfortunately, these tests may fail on some operating systems (Linux and Windows in my testing), due to an imprecise file last modified timer. What can happen is that we write to a file, making the OS update its modified time to 1000 (as an example, not a real timestamp), then very quickly write to the file again, making the OS update its modified time to 1000 again. Then, our test will fail because the stamp didn’t change even though we expect it to change.

This can happen with an imprecise timer that only increases once every millisecond (again, an example, not a real number) when we perform writes in between that millisecond. Even worse, our test can be flaky, sometimes succeeding if we write in between those milliseconds, sometimes failing if we write within a millisecond.

To solve this, add a function to the filesystem testing utility crate. Change dev_shared/src/lib.rs:

The write_until_modified function writes to the file, but ensures its modified time will change. Now change the tests in pie/src/stamp.rs to use this function:

Now we use write_until_modified to write to the file, ensuring its modified time will change, ensuring the stamp will change when it should. Run cargo test to confirm the stamp implementation, which should succeed now.

Stamps in Context

We now have a module dedicated to stamps. However, stampers are constructed by users of the library that author tasks, and they need to pass in these stampers when creating dependencies. Therefore, we need to update the Context trait to allow passing in these stampers.

Change Context in pie/src/lib.rs:

We add the require_file_with_stamper method which allow passing in a stamper. We add a default implementation for require_file that passes in a default stamper. The default is provided by default_require_file_stamper which can be overridden by context implementations.

Now apply the same to tasks, changing Context again in pie/src/lib.rs:

Update NonIncrementalContext in src/context/non_incremental.rs to implement the new methods:

We just ignore the stampers in NonIncrementalContext, as they are only needed for incrementality.

Run cargo test to confirm everything still works.