Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions NEWS.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# fireSense_dataPrepFit 1.3.0.9011 (development version)

* New parameter `skipUnusedFitPrep` (default FALSE: nothing changes). TRUE, in a job that cannot fit, validate, profile or hold out a fold (`fireSenseUtils::fitDataNeeded()`: `doFit` FALSE, no `heldOutFold`, `fireSense_spreadFit`'s `mode` without "validate" or "profile"), leaves out the spread covariates, the slowest step: about 2.5 minutes of every hindcast year-job. The spread fire data (new event `prepSpreadFitFireData`, the fire half of `prepSpreadFitData`) and `prepIgnitionFitData` and `prepEscapeFitData` still run, so `spreadFirePolys`, `escapeFireSet`, `ignitionFirePoints` and the escape covariates are what a full run makes. The same parameter of `fireSense_spreadFit` (`.globals = list(skipUnusedFitPrep = TRUE)`); init stops if they differ on it or on `doFit`. Not in the cache keys. Needs fireSenseUtils >= 0.2.3.9167 (the version assumed for the PR that adds `fitDataNeeded()`).

* `fireSense_ignitionCovariates` has a new column `coreFraction`: the share of each coarse pixel's flammable cells that are in the core of `rasterToMatchLargeELF` (1 in the core, 0 in the buffer, fractional at the edge; computed per data year from `flammableRTMs`, with `fireSenseUtils::inELFCore()` and the aggregation of the coarse covariates). The ignitions are counted in the core only, so fireSense_ignitionFit weights each pixel by it; without it the fit spread the core's ignitions over core and buffer and hindcasts simulated about 0.8 of the observed ignitions. It is an exposure, not a covariate (`fireSenseUtils::coreFractionTxt`, fireSenseUtils >= 0.2.3.9161), and the escape formula leaves it out. The `prepIgnitionFitData` event re-runs once (its code changed). Used with PredictiveEcology/fireSense_ignitionFit (`Depends-on` there).
* New parameter `spreadClimateCandidates` (default: the default ignition variables plus `Tmax_sm` and `PPT_sm`): `climateVariablesForFire$spread = "auto"` now chooses among it, not among `climateVariablesForFire$ignition`, which IgnitionFit still uses whole and unchanged. The candidates' layers are prepared with `climateVariables`, but only the chosen variable enters the spread tables. Precipitation is scored as decreasing with dryness. The default `auto` choice can now be `Tmax_sm` or `PPT_sm`, so spread fits and results can change; they need `Tmaxsm` and `PPTsm` in `fireSenseUtils::climateCovRanges` (PredictiveEcology/fireSenseUtils, branch `climate-ranges-summer`). It is a module parameter, so changing it re-runs the cached events.

Expand Down
83 changes: 66 additions & 17 deletions fireSense_dataPrepFit.R
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ defineModule(sim, list(
reqdPkgs = list("data.table", "digest", "fastDummies", "googledrive", "httr2", "qs2", "Require",
"PredictiveEcology/reproducible@development (>= 3.2.1.9042)", # CacheGeo re-reads a changed local file
"PredictiveEcology/climateData@development (>= 2.2.3.9006)",
"PredictiveEcology/fireSenseUtils@development (>= 0.2.3.9161)", # 9161: coreFractionTxt; 9159: climateCovRanges Tmaxsm, PPTsm (#224); 9141: spreadFitLedgerInputs(), spreadFitLedgerInputLists(); 9135: fireSenseFireCauses; 9131: fireCauseCodes(); 9130: climateRasterToDataTable(Index) reads only those cells; 9128: burnableHaTxt
"PredictiveEcology/fireSenseUtils@development (>= 0.2.3.9169)", # 9169: fitDataNeeded() (#234); 9161: coreFractionTxt; 9159: climateCovRanges Tmaxsm, PPTsm (#224); 9141: spreadFitLedgerInputs(), spreadFitLedgerInputLists(); 9135: fireSenseFireCauses; 9131: fireCauseCodes(); 9130: climateRasterToDataTable(Index) reads only those cells; 9128: burnableHaTxt
"FOR-CAST/fireregimetools@main (>= 0.1.0.9008)",
"ggplot2", "methods", "parallel", "purrr", "raster", "sf", "sp", "stats", "utils",
"PredictiveEcology/LandR@development (>= 1.2.0.9015)",
Expand Down Expand Up @@ -138,6 +138,16 @@ defineModule(sim, list(
"takes that fit's climate variable(s); when it is fitted, `climateVariablesForFire$spread`",
"as given (`\"auto\"`: chosen afresh). Set it for every fireSense module at once with",
"`.globals = list(doFit = ...)`.")),
defineParameter("skipUnusedFitPrep", "logical", default = FALSE,
desc = paste("FALSE (default): unchanged. TRUE: a job that cannot fit, validate, profile or hold out a fold",
"(`fireSenseUtils::fitDataNeeded()`: `doFit` FALSE, no `heldOutFold`, `fireSense_spreadFit`'s `mode`",
"without \"validate\" or \"profile\") makes only the spread fire data (`prepSpreadFitFireData`: the",
"fire polygons, `escapeFireSet`, hence the same `ignitionFirePoints` and escape covariates) and",
"leaves out the spread covariates, the slowest step and the one only a fit reads.",
"The same parameter of `fireSense_spreadFit`: set it for both with",
"`.globals = list(skipUnusedFitPrep = TRUE)`; init stops if they differ on it or on `doFit`.",
"Not part of the cache keys."),
cacheKey = FALSE),
defineParameter("fitFileSuffix", "character", "", NA, NA,
paste("Text put after the model tag in the ledger file names, e.g. `\"_boundfix_rep81\"`, so a test run",
"writes and reads (`\"latest\"`) only its own files, never the production ones. `\"\"` (the",
Expand Down Expand Up @@ -206,6 +216,11 @@ defineModule(sim, list(
list(spreadFitPreRun = sim$spreadFitPreRun), fireSenseUtils::escapeFireSetDeps()))),
prepEscapeFitData = list(.cacheExtra = quote(c(
list(spreadFitPreRun = sim$spreadFitPreRun), fireSenseUtils::escapeFireSetDeps()))),
prepSpreadFitFireData = list(.cacheExtra = quote(c(list(
fireSenseUtils::bufferToArea, fireSenseUtils::escapedFires, fireSenseUtils::escapeDefinitionOf,
fireSenseUtils::fireCauseCodes, fireSenseUtils::rasterFireBufferDT,
fireSenseUtils::rasterFireSpreadPoints, spreadFitPreRun = sim$spreadFitPreRun),
fireSenseUtils::harmonizeFireDataDeps(), fireSenseUtils::escapeFireSetDeps()))),
prepSpreadFitData = list(.cacheExtra = quote(c(list(
fireSenseUtils::bufferToArea, fireSenseUtils::chooseDomSecFuelClasses,
fireSenseUtils::climateRasterToDataTable, fireSenseUtils::dataYearBefore,
Expand Down Expand Up @@ -347,13 +362,13 @@ defineModule(sim, list(
createsOutput("fireBufferedListDT", "list",
paste("list of data.tables with fire id, `pixelID`, and buffer status, of the fires the spread",
"model is fitted to: those that start in the core with at least `minBurnableFireSizeHa` on burnable pixels."),
events = "prepSpreadFitData"),
events = c("prepSpreadFitFireData", "prepSpreadFitData")),
createsOutput("fuelClassTable", "data.table",
"Fuel class assigned to each tree species; only created if fuel classes are estimated.",
events = "dataPrepBuild"),
createsOutput("spreadFirePolys", "list",
"List of annual fire polygons of the escaped fires, core and buffer start, matched to their harmonized start points (`escapeFireSet`); `spreadFirePoints` and `fireBufferedListDT` have the core-start fires only.",
events = c("dataPrepBuild", "prepSpreadFitData")),
events = c("dataPrepBuild", "prepSpreadFitFireData", "prepSpreadFitData")),
createsOutput("fireSense_annualSpreadFitCovariates", "list",
"List of data.tables, one per fire year, of `pixelID` and the spread climate covariates in the fire buffers (every flammable pixel with `wholeLandscapeSpreadCovariates`).",
events = "prepSpreadFitData"),
Expand Down Expand Up @@ -423,15 +438,15 @@ defineModule(sim, list(
paste("List of ignition points, one per fire year, for the fires larger than one pixel that start in the core,",
"harmonized with `spreadFirePolys`. Column `fireSenseUtils::burnableHaTxt` is each fire's",
"burned area (ha) on burnable pixels, the spread fit's target size."),
events = c(".inputObjects", "prepSpreadFitData")),
events = c(".inputObjects", "prepSpreadFitFireData", "prepSpreadFitData")),

createsOutput("escapeFireSet", "SpatVector",
paste("The start of every escaped fire (`escapeSizeHa`, `fireCauses`), core and buffer, from",
"`fireSenseUtils::escapeFireSet()` with `spreadFirePoints` as starts: `YEAR`, the fire ID,",
"`polyHa`, `burnableHa`, `pixelID` and `core`. Only `core` fires are in `spreadFirePoints` and",
"`fireBufferedListDT`; `spreadFirePolys` keeps all, so the ignition and escape data can leave out",
"the NFDB points of buffer-start fires."),
events = "prepSpreadFitData"),
events = c("prepSpreadFitFireData", "prepSpreadFitData")),
createsOutput("fireSense_spreadFitCorePixels", "integer",
paste("`pixelID`s (cells of `rasterToMatch`) in the ELF's core, where `rasterToMatchLargeELF` is",
"`fireSenseUtils::fireSenseELFCoreValue`; every pixel without `rasterToMatchLargeELF`. The spread",
Expand Down Expand Up @@ -473,7 +488,7 @@ doEvent.fireSense_dataPrepFit = function(sim, eventTime, eventType) {
eventType,
init = {
## schedule future event(s)
for (ev in prepEventsToSchedule(P(sim)$whichModulesToPrepare))
for (ev in prepEventsToSchedule(P(sim)$whichModulesToPrepare, fitPrepNeeded(sim)))
sim <- scheduleEvent(sim, start(sim), "fireSense_dataPrepFit", ev, eventPriority = 1)

sim <- Init(sim)
Expand All @@ -496,6 +511,9 @@ doEvent.fireSense_dataPrepFit = function(sim, eventTime, eventType) {
prepSpreadFitData = {
sim <- prepare_SpreadFit(sim)
},
prepSpreadFitFireData = {
sim <- prepare_SpreadFitFires(sim)
},
plotAndMessage = {
sim <- plotAndMessage(sim)
},
Expand Down Expand Up @@ -837,6 +855,26 @@ dataPrepBuild <- function(sim) {
}


#' Prepare the fire data for fireSense_spreadFit
#'
#' The fire half of `prepare_SpreadFit()`: the fire polygons and start points, `fireBufferedListDT` and
#' `escapeFireSet`, which the ignition and escape data also need. A job that only predicts
#' (`skipUnusedFitPrep`) runs only this, so those objects are what a full run makes. Sets `mod$allYears`.
#'
#' @param sim a `simList`.
#' @return the `simList`, invisibly, with `fireBufferedListDT`, `spreadFirePoints`, `spreadFirePolys` and
#' `escapeFireSet`.
prepare_SpreadFitFires <- function(sim) {
mod$allYears <- fitYearNames(Par) # it runs before prepare_IgnitionFit, which set them
## sim$fireBufferedListDT is made in these functions
if (P(sim)$useRasterizedFireForSpread) {
sim <- prepare_SpreadFitFire_Raster(sim)
} else {
sim <- prepare_SpreadFitFire_Vector(sim)
}
invisible(sim)
}

#' Prepare the covariates and formula for fireSense_spreadFit
#'
#' Buffers each fire, joins the buffers to the fuel covariates of their data year and to annual
Expand All @@ -848,7 +886,7 @@ dataPrepBuild <- function(sim) {
#' `spreadFirePoints` and `spreadFirePolys`.
prepare_SpreadFit <- function(sim) {

mod$allYears <- fitYearNames(Par) # it runs before prepare_IgnitionFit, which set them
sim <- prepare_SpreadFitFires(sim)

## prep veg data ---------------------------------------------------------------------------------
doAssertion <- getOption("fireSenseUtils.assertions", TRUE)
Expand Down Expand Up @@ -927,14 +965,6 @@ prepare_SpreadFit <- function(sim) {
})
vegData <- rbindlist(vegData)

# prep the fire data ####
# sim$fireBufferedListDT is made in these functions
if (P(sim)$useRasterizedFireForSpread) {
sim <- prepare_SpreadFitFire_Raster(sim)
} else {
sim <- prepare_SpreadFitFire_Vector(sim)
}

## the pixels of the ELF's core, where the spread fit scores area burned
sim$fireSense_spreadFitCorePixels <- which(fireSenseUtils::inELFCore(seq_len(terra::ncell(sim$rasterToMatch)), sim$rasterToMatchLargeELF,
grid = sim$rasterToMatch))
Expand Down Expand Up @@ -1506,18 +1536,37 @@ fitYearNames <- function(Par) {
#' exists, so a project still naming it stops.
#'
#' @param which character, `P(sim)$whichModulesToPrepare`.
#' @param fitPrep logical; `FALSE` makes only the spread fire data (`prepSpreadFitFireData`), which the ignition and
#' escape data need, not the spread covariates (`fitPrepNeeded()`).
#' @return character, event names in the order to schedule them.
prepEventsToSchedule <- function(which) {
prepEventsToSchedule <- function(which, fitPrep = TRUE) {
if ("fireSense_EscapeFit" %in% which)
stop("fireSense_EscapeFit no longer exists as a module; escape data are prepared with ",
"fireSense_ignitionFit. Remove it from parameter whichModulesToPrepare.")
if (!all(which %in% c("fireSense_spreadFit", "fireSense_ignitionFit")))
stop("unrecognized module to prepare - review parameter whichModulesToPrepare")
## spread first: its fire start points place the ignitions of fire polygons without an ignition point
c(if ("fireSense_spreadFit" %in% which) "prepSpreadFitData",
c(if ("fireSense_spreadFit" %in% which && fitPrep) "prepSpreadFitData",
if ("fireSense_spreadFit" %in% which && !fitPrep && "fireSense_ignitionFit" %in% which) "prepSpreadFitFireData",
if ("fireSense_ignitionFit" %in% which) c("prepIgnitionFitData", "prepEscapeFitData"))
}

#' Will a fit, profile or validation read the spread covariates?
#'
#' Always, unless `skipUnusedFitPrep` and `fireSenseUtils::fitDataNeeded()` is `FALSE` for this job's `doFit`,
#' `heldOutFold` and `fireSense_spreadFit`'s `mode`. The ledger is not known here (`Init()` reads it after the
#' events are scheduled), so `doFit` NA counts as a job that may fit.
#'
#' @param sim a `simList`.
#' @return logical, length one.
fitPrepNeeded <- function(sim) {
if (!isTRUE(P(sim)$skipUnusedFitPrep)) return(TRUE)
for (nm in c("skipUnusedFitPrep", "doFit"))
SpaDES.core::paramCheckOtherMods(sim, nm, ifSetButDifferent = "error")
fireSenseUtils::fitDataNeeded(P(sim)$doFit, SpaDES.core::paramCheckOtherMods(sim, "heldOutFold"),
params(sim)[["fireSense_spreadFit"]][["mode"]])
}

#' Prepare the covariates and formula for the escape part of fireSense_ignitionFit
#'
#' Adds to the ignition covariates the number of escapes: ignitions that became a fire polygon
Expand Down
2 changes: 1 addition & 1 deletion tests/testthat/test-metadata.R
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ test_that("parameters are the expected names", {
"forestedLCC", "fuelClassCol", "fuelCovariates", "heldOutFold", "igAggFactor",
"minBufferSize", "minBurnableFireSizeHa", "minCovariateProp", "nonflammableLCC",
"nonForestCanBeYoungAge", "scanfiVersion", "sppEquivCol", "spreadClimateCandidates", "spreadFitFilename",
"spreadFitGoogleDriveFolder", "fitFileSuffix", "spreadIntercept", "targetFuelClasses",
"spreadFitGoogleDriveFolder", "fitFileSuffix", "skipUnusedFitPrep", "spreadIntercept", "targetFuelClasses",
"useRasterizedFireForSpread", "wholeLandscapeSpreadCovariates", "whichModulesToPrepare"))
)
})
Expand Down
81 changes: 81 additions & 0 deletions tests/testthat/test-skipUnusedFitPrep.R
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
## `skipUnusedFitPrep`: a job that only predicts from the ledger's fits makes only the spread FIRE data
## (`prepSpreadFitFireData`), not the spread covariates. The ignition and escape data are made as in a full run,
## from the same fire data, so `ignitionFirePoints` and `fireSense_escapeCovariates` are the same.

bothFits <- c("fireSense_ignitionFit", "fireSense_spreadFit")
all4 <- c("prepSpreadFitData", "prepIgnitionFitData", "prepEscapeFitData")
fireOnly <- c("prepSpreadFitFireData", "prepIgnitionFitData", "prepEscapeFitData")

## a simList with this module current, `doFit` as given, and optionally a spreadFit with its own parameters
prepSim <- function(doFit = FALSE, heldOutFold = NA, spreadFit = NULL, skip = TRUE) {
sim <- suppressMessages(SpaDES.core::simInit())
sim@params <- list(fireSense_dataPrepFit = list(doFit = doFit, heldOutFold = heldOutFold,
skipUnusedFitPrep = skip))
if (!is.null(spreadFit)) sim@params$fireSense_spreadFit <- c(list(skipUnusedFitPrep = skip, doFit = doFit), spreadFit)
sim@current <- data.table::data.table(moduleName = "fireSense_dataPrepFit")
sim
}
scheduled <- function(sim) prepEventsToSchedule(bothFits, fitPrepNeeded(sim))

test_that("by default every prep event is scheduled, as before", {
expect_identical(scheduled(prepSim(FALSE, skip = FALSE)), all4)
})

test_that("skipUnusedFitPrep: a predict-only job makes the fire data, not the spread covariates", {
expect_identical(scheduled(prepSim(FALSE)), fireOnly)
expect_identical(scheduled(prepSim(FALSE, spreadFit = list(mode = "fit", heldOutFold = NA))), fireOnly)
})

test_that("skipUnusedFitPrep: a job that may fit, validate, profile or hold out a fold prepares everything", {
expect_identical(scheduled(prepSim(TRUE)), all4)
expect_identical(scheduled(prepSim(NA)), all4) # the ledgers decide: not known here
expect_identical(scheduled(prepSim(FALSE, heldOutFold = 2L)), all4)
expect_identical(scheduled(prepSim(FALSE, spreadFit = list(mode = c("fit", "validate")))), all4)
expect_identical(scheduled(prepSim(FALSE, spreadFit = list(mode = "profile"))), all4)
})

test_that("a project that prepares only the spread fit schedules nothing when it only predicts", {
expect_length(prepEventsToSchedule("fireSense_spreadFit", FALSE), 0L)
})

test_that("the modules must agree on skipUnusedFitPrep and doFit", {
sim <- prepSim(FALSE, spreadFit = list())
sim@params$fireSense_spreadFit$skipUnusedFitPrep <- FALSE
expect_error(fitPrepNeeded(sim), "skipUnusedFitPrep")
sim <- prepSim(FALSE, spreadFit = list())
sim@params$fireSense_spreadFit$doFit <- NA
expect_error(fitPrepNeeded(sim), "doFit")
})

## ignitionFirePoints and the escape covariates read the fire data (spreadFirePolys, escapeFireSet) and nothing else
## that only the spread covariates make; the fire half is one function that a full run calls too
test_that("the fire data are made by one function that the full spread prep calls", {
defs <- moduleDefs()
expect_true("prepare_SpreadFitFires" %in% all.names(defs[["prepare_SpreadFit"]]))
expect_false(any(c("prepare_SpreadFitFire_Raster", "prepare_SpreadFitFire_Vector") %in%
all.names(defs[["prepare_SpreadFit"]])))
md <- SpaDES.core::moduleMetadata(module = moduleName, path = modulePath)
made <- md$outputObjects$objectName[vapply(md$outputObjects$events, function(e) "prepSpreadFitFireData" %in% e,
logical(1))]
expect_true(all(c("spreadFirePolys", "escapeFireSet") %in% made))
expect_false(any(c("fireSense_annualSpreadFitCovariates", "fireSense_spreadFormula") %in% made))
})

test_that("prepare_SpreadFitFires runs the raster or the vector builder, as the full run did", {
for (raster in c(TRUE, FALSE)) {
sim <- suppressMessages(SpaDES.core::simInit())
sim@params <- list(fireSense_dataPrepFit = list(useRasterizedFireForSpread = raster))
sim@current <- data.table::data.table(moduleName = "fireSense_dataPrepFit")
ran <- character()
e <- new.env(parent = environment(prepare_SpreadFitFires))
e$mod <- new.env()
e$Par <- list(dataYears = 2000, fireYears = 2001:2002)
e$prepare_SpreadFitFire_Raster <- function(sim) { ran <<- c(ran, "raster"); sim }
e$prepare_SpreadFitFire_Vector <- function(sim) { ran <<- c(ran, "vector"); sim }
f <- prepare_SpreadFitFires
environment(f) <- e
f(sim)
expect_identical(ran, if (raster) "raster" else "vector")
expect_false(is.null(e$mod$allYears))
}
})
Loading