Data submission instructions
-
All forecasts should be submitted directly to the model-output/ folder. Data in this directory should be
-added to the repository through a pull request so that automatic data validation checks are run.
+
All forecasts should be submitted directly to the model-output folder. Data in this directory
+should be added to the repository through a pull request so that
+automatic validation checks
+are run (although teams are highly encouraged to do local validations before doing the data
+submission).
These instructions provide detail about the data format as well as validation that you can do prior to this
pull request. In addition, we describe metadata
that each model should provide in the model-metadata folder.
@@ -402,7 +405,7 @@
What is a forecast
Target Data
County-level forecasts for total WNV diseases cases in each remaining
-calendar month of 2024 through October should be produced and submitted
+calendar month of 2024 through December should be produced and submitted
by their respective monthly due dates, beginning on April 30, 2024.
Please note that these monthly county-level forecasts differ from
previous national WNV forecasting challenge targets. This page includes
@@ -412,7 +415,8 @@
Target Data
All forecasts should be submitted directly to the model-output folder
in this repository (more information on the submission folders is
available here). Forecast data should be added to the repository through
-a pull request so that automatic data validation checks are run.
+a pull request so that automatic
data validation
+checks are run.
These instructions provide detail about the data format as well as
validation that you can do prior to this pull request. In addition, we
describe the metadata that each model should
@@ -646,6 +650,22 @@
Evaluation criteria
general performance of submitted forecasts. Participants may publish
their own forecasts and results at any time.
+
+
Local Validation
+
Teams are highly encouraged to validate their data locally before
+doing their pull request. To do this, install the
+hubValidations package:
+
remotes::install_github("Infectious-Disease-Modeling-Hubs/hubValidations")
+
then set your working directory to the base repository path, and
+run:
+
require(hubValidations)
+hubValidations::validate_submission(
+ file_path='CDPH-AZO/2024-05-31-CDPH-AZO.csv',
+ hub_path='.'
+)
+
where the file_path should be replaced with your
+submission file.
+
diff --git a/model-output/README.md b/model-output/README.md
index ccba4f3..6a18ff7 100644
--- a/model-output/README.md
+++ b/model-output/README.md
@@ -5,7 +5,7 @@ This folder contains a set of subdirectories, one for each model, that contains
# Data submission instructions
-All forecasts should be submitted directly to the [model-output](./) folder. Data in this directory should be added to the repository through a pull request so that automatic [data validation checks](../.github/workflows) are run.
+All forecasts should be submitted directly to the [model-output](../model-output) folder. Data in this directory should be added to the repository through a pull request so that automatic [validation checks](../.github/workflows/model-output-pull.yml) are run (although teams are highly encouraged to do [local validations](#Local-Validation) before submitting).
These instructions provide detail about the [data format](#Data-formatting) as well as [validation](#Forecast-validation) that you can do prior to this pull request. In addition, we describe [metadata](https://github.com/Infectious-Disease-Modeling-Hubs/hubTemplate/blob/master/model-metadata/README.md) that each model should provide in the model-metadata folder.
@@ -225,3 +225,23 @@ To ensure that forecasting is done in real-time, all forecasts are required to b
## Evaluation criteria
When reported data for 2024 are available, an analysis will be conducted using multiple scoring metrics, including the logarithmic score and the multibin logarithmic score, to assess and compare forecasts across all counties at each time point. A joint manuscript will be prepared to disseminate findings on this comparison and the general performance of submitted forecasts. Participants may publish their own forecasts and results at any time.
+
+## Local Validation
+
+Teams are highly encouraged to validate their data locally before doing their pull request. To do this, install the `hubValidations` package:
+
+```R
+remotes::install_github("Infectious-Disease-Modeling-Hubs/hubValidations")
+```
+
+then set your working directory to the base repository path, and run:
+
+```R
+require(hubValidations)
+hubValidations::validate_submission(
+ file_path='CDPH-AZO/2024-05-31-CDPH-AZO.csv',
+ hub_path='.'
+)
+```
+
+where the `file_path` should be replaced with your submission file.