Adding local validation instructions

model-output/README.html

@@ -367,8 +367,11 @@ <h1>Model outputs folder</h1>
 </div>
 <div id="data-submission-instructions" class="section level1">
 <h1>Data submission instructions</h1>
-<p>All forecasts should be submitted directly to the <a href="./">model-output/</a> folder. Data in this directory should be
-added to the repository through a pull request so that automatic <a href="../.github/workflows">data validation checks</a> are run.</p>
+<p>All forecasts should be submitted directly to the <a href="../model-output">model-output</a> folder. Data in this directory
+should be added to the repository through a pull request so that
+automatic <a href="../.github/workflows/model-output-pull.yml">validation checks</a>
+are run (although teams are highly encouraged to do <a href="#Local-Validation">local validations</a> before doing the data
+submission).</p>
 <p>These instructions provide detail about the <a href="#Data-formatting">data format</a> as well as <a href="#Forecast-validation">validation</a> that you can do prior to this
 pull request. In addition, we describe <a href="https://github.com/Infectious-Disease-Modeling-Hubs/hubTemplate/blob/master/model-metadata/README.md">metadata</a>
 that each model should provide in the model-metadata folder.</p>
@@ -402,7 +405,7 @@ <h2>What is a forecast</h2>
 <div id="target-data" class="section level2">
 <h2>Target Data</h2>
 <p>County-level forecasts for total WNV diseases cases in each remaining
-calendar month of 2024 through October should be produced and submitted
+calendar month of 2024 through December should be produced and submitted
 by their respective monthly due dates, beginning on April 30, 2024.
 Please note that these monthly county-level forecasts differ from
 previous national WNV forecasting challenge targets. This page includes
@@ -412,7 +415,8 @@ <h2>Target Data</h2>
 <p>All forecasts should be submitted directly to the model-output folder
 in this repository (more information on the submission folders is
 available here). Forecast data should be added to the repository through
-a pull request so that automatic data validation checks are run.</p>
+a pull request so that automatic <a href="../.github/workflows/model-output-pull.yml">data validation
+checks</a> are run.</p>
 <p>These instructions provide detail about the data format as well as
 validation that you can do prior to this pull request. In addition, we
 describe the <a href="#Metadata">metadata</a> that each model should
@@ -646,6 +650,22 @@ <h2>Evaluation criteria</h2>
 general performance of submitted forecasts. Participants may publish
 their own forecasts and results at any time.</p>
 </div>
+<div id="local-validation" class="section level2">
+<h2>Local Validation</h2>
+<p>Teams are highly encouraged to validate their data locally before
+doing their pull request. To do this, install the
+<code>hubValidations</code> package:</p>
+<pre class="r"><code>remotes::install_github(&quot;Infectious-Disease-Modeling-Hubs/hubValidations&quot;)</code></pre>
+<p>then set your working directory to the base repository path, and
+run:</p>
+<pre class="r"><code>require(hubValidations)
+hubValidations::validate_submission(
+  file_path=&#39;CDPH-AZO/2024-05-31-CDPH-AZO.csv&#39;,
+  hub_path=&#39;.&#39;
+)</code></pre>
+<p>where the <code>file_path</code> should be replaced with your
+submission file.</p>
+</div>
 </div>
 
 

model-output/README.md

@@ -5,7 +5,7 @@ This folder contains a set of subdirectories, one for each model, that contains
 
 # Data submission instructions
 
-All forecasts should be submitted directly to the [model-output](./) folder. Data in this directory should be added to the repository through a pull request so that automatic [data validation checks](../.github/workflows) are run.
+All forecasts should be submitted directly to the [model-output](../model-output) folder. Data in this directory should be added to the repository through a pull request so that automatic [validation checks](../.github/workflows/model-output-pull.yml) are run (although teams are highly encouraged to do [local validations](#Local-Validation) before submitting).
 
 These instructions provide detail about the [data format](#Data-formatting) as well as [validation](#Forecast-validation) that you can do prior to this pull request. In addition, we describe [metadata](https://github.com/Infectious-Disease-Modeling-Hubs/hubTemplate/blob/master/model-metadata/README.md) that each model should provide in the model-metadata folder.
 
@@ -225,3 +225,23 @@ To ensure that forecasting is done in real-time, all forecasts are required to b
 ## Evaluation criteria
 
 When reported data for 2024 are available, an analysis will be conducted using multiple scoring metrics, including the logarithmic score and the multibin logarithmic score, to assess and compare forecasts across all counties at each time point. A joint manuscript will be prepared to disseminate findings on this comparison and the general performance of submitted forecasts. Participants may publish their own forecasts and results at any time. 
+
+## Local Validation
+
+Teams are highly encouraged to validate their data locally before doing their pull request. To do this, install the `hubValidations` package:
+
+```R
+remotes::install_github("Infectious-Disease-Modeling-Hubs/hubValidations")
+```
+
+then set your working directory to the base repository path, and run:
+
+```R
+require(hubValidations)
+hubValidations::validate_submission(
+  file_path='CDPH-AZO/2024-05-31-CDPH-AZO.csv',
+  hub_path='.'
+)
+```
+
+where the `file_path` should be replaced with your submission file.