Requiring Files

Since build systems frequently interact with files, and changes to files can affect tasks, we need to keep track of file dependencies. Therefore, we will extend the Context API with methods to require files, enabling tasks to specify dynamic dependencies to files.

Add a method to the Context trait in pie/src/lib.rs:

require_file is similar to requiring a task, but instead takes a path to a file or directory on the filesystem as input. We use AsRef<Path> as the type for the path, so that we can pass anything in that can dereference to a path. For example, str has an AsRef<Path> implementation, so we can just use "test.txt" as a path.

As an output, we return Result<Option<File>, io::Error>, with File being a handle to an open file. The reason for this complicated type is:

• An incremental context will want to read the metadata (such as the last modified date) of the file, or create a hash over the file, to be able to detect changes. Because getting metadata or reading the file can fail, and we want to propagate this error, we return a Result with io::Error as the error type.
• Tasks can create a dependency to a file that does not exist, and the existence of that file affects the task. For example, a task that prints true or false based on if a file exists. If the file does not exist (or it is a directory), we cannot open it, so we cannot return a File, hence we use Option<File> to return None.
• Otherwise, we return Ok(Some(file)) so that the task can read the opened file.

Now we need to implement this method for NonIncrementalContext. However, because we will be performing similar file system operations in the incremental context as well, we will create some utility functions for this first.

Filesystem utilities

Add the fs module to pie/src/lib.rs:

Create file pie/src/fs.rs with:

use std::{fs, io};
use std::fs::{File, Metadata};
use std::path::Path;

/// Gets the metadata for given `path`, returning:
/// - `Ok(Some(metadata))` if a file or directory exists at given path,
/// - `Ok(None)` if no file or directory exists at given path,
/// - `Err(e)` if there was an error getting the metadata for given path.
pub fn metadata(path: impl AsRef<Path>) -> Result<Option<Metadata>, io::Error> {
  match fs::metadata(path) {
    Err(e) if e.kind() == io::ErrorKind::NotFound => Ok(None),
    Err(e) => Err(e),
    Ok(m) => Ok(Some(m))
  }
}

/// Attempt to open file at given `path`, returning:
/// - `Ok(Some(file))` if the file exists at given path,
/// - `Ok(None)` if no file exists at given path (but a directory could exist at given path),
/// - `Err(e)` if there was an error getting the metadata for given path, or if there was an error opening the file.
///
/// This function is necessary due to Windows returning an error when attempting to open a directory.
pub fn open_if_file(path: impl AsRef<Path>) -> Result<Option<File>, io::Error> {
  let file = match metadata(&path)? {
    Some(metadata) if metadata.is_file() => Some(File::open(&path)?),
    _ => None,
  };
  Ok(file)
}

The metadata function gets the filesystem metadata given a path, and open_if_file opens the file for given path. The reason for these functions is that the standard library function std::fs::metadata treats non-existent files as an error, whereas we don’t want to treat it as an error and just return None. Furthermore, open_if_file works around an issue where opening a directory on Windows (and possibly other operating systems) is an error, where we want to treat it as None again. The documentation comments explain the exact behaviour.

We will write some tests to confirm the behaviour, but for that we need utilities to create temporary files and directories. Furthermore, we will be writing more unit tests – but also integration tests – in this tutorial, so we will set up these utilities in such a way that they are reachable by both unit and integration tests. The only way to do that right now in Rust, is to create a separate package and have the pie package depend on it.

And yes, we went from adding file dependencies, to creating file system utilities, to testing those file system utilities, to creating testing utilities, and now to making a crate for those testing utilities. Sorry about that 😅, we will start unwinding this stack soon!

Create the dev_shared package

Next to the pie directory, create a directory named dev_shared (i.e., the pibs/dev_shared directory where pibs is your workspace directory). Create the dev_shared/Cargo.toml file with the following contents:

[package]
name = "dev_shared"
version = "0.1.0"
edition = "2021"

[dependencies]
tempfile = "3"

We’ve added the tempfile dependency here already, which is a library that creates and automatically cleans up temporary files and directories.

Create the main library file dev_shared/src/lib.rs, with functions for creating temporary files and directories:

use std::io;

use tempfile::{NamedTempFile, TempDir};

/// Creates a new temporary file that gets cleaned up when dropped.
pub fn create_temp_file() -> Result<NamedTempFile, io::Error> { NamedTempFile::new() }

/// Creates a new temporary directory that gets cleaned up when dropped.
pub fn create_temp_dir() -> Result<TempDir, io::Error> { TempDir::new() }

Now add the dev_shared package to the members of the workspace in Cargo.toml (i.e., the pibs/Cargo.toml file where pibs is your workspace directory):

Your directory structure should now look like this:

pibs
├── pie
│   ├── Cargo.toml
│   └── src
│       ├── context
│       │   ├── non_incremental.rs
│       │   └── mod.rs
│       ├── lib.rs
│       └── fs.rs
├── Cargo.toml
└── dev_shared
    ├── Cargo.toml
    └── src
        └── lib.rs

To access these utility functions in the pie crate, add a dependency to dev_shared in pie/Cargo.toml along with another create that will help testing:

We’re using a path dependency to have pie depend on the dev_shared package at "../dev_shared" in the workspace.

We’ve also added a dependency to assert_matches, which is a handy library for asserting that a value matches a pattern. Note that these dependencies are added under dev-dependencies, indicating that these dependencies are only available when running tests, benchmarks, and examples. Therefore, users of our library will not depend on these libraries, which is good, because temporary file management and assertions are not necessary to users of the library.

Testing filesystem utilities

Back to testing our filesystem utilities. Add the following tests to pie/src/fs.rs:

#[cfg(test)]
mod test {
  use std::fs::remove_file;
  use std::io;

  use assert_matches::assert_matches;

  use dev_shared::{create_temp_dir, create_temp_file};

  use super::*;

  #[test]
  fn test_metadata_ok() -> Result<(), io::Error> {
    let temp_file = create_temp_file()?;
    let metadata = metadata(temp_file)?;
    assert_matches!(metadata, Some(metadata) => {
      assert!(metadata.is_file());
    });
    Ok(())
  }

  #[test]
  fn test_metadata_none() -> Result<(), io::Error> {
    let temp_file = create_temp_file()?;
    remove_file(&temp_file)?;
    let metadata = metadata(&temp_file)?;
    assert!(metadata.is_none());
    Ok(())
  }

  #[test]
  fn test_open_if_file() -> Result<(), io::Error> {
    let temp_file = create_temp_file()?;
    let file = open_if_file(&temp_file)?;
    assert!(file.is_some());
    Ok(())
  }

  #[test]
  fn test_open_if_file_non_existent() -> Result<(), io::Error> {
    let temp_file = create_temp_file()?;
    remove_file(&temp_file)?;
    let file = open_if_file(&temp_file)?;
    assert!(file.is_none());
    Ok(())
  }

  #[test]
  fn test_open_if_file_on_directory() -> Result<(), io::Error> {
    let temp_dir = create_temp_dir()?;
    let file = open_if_file(temp_dir)?;
    assert!(file.is_none());
    Ok(())
  }
}

We test whether the functions conform to the specified behaviour. Unfortunately, we can’t easily test when metadata and open_if_file should return an error, because we cannot disable read permissions on files via the Rust standard library.

We use our create_temp_file and create_temp_dir utility functions to create temporary files and directories. The tempfile library takes care of deleting temporary files when they go out of scope (at the end of the test).

We use assert_matches! to assert that metadata is Some(metadata), binding metadata in the => { ... } block in which we assert that the metadata describes a file. We will use this macro more in future integration tests.

Implement require_file

Now we are done unwinding our stack and have filesystem and testing utilities. Make the non-incremental context compatible by changing pie/src/context/non_incremental.rs:

Since the non-incremental context does not track anything, we only try to open the file and return it, matching the contract in the documentation comment of the Context::require_file trait method.

Confirm everything works with cargo test.