Updating model-metadata instructions

model-metadata/CDPH-AZO.yml

@@ -0,0 +1,27 @@
+team_name: "California Department of Public Health"
+team_abbr: "CDPH"
+model_name: "Always Zero cases baseline model."
+model_abbr: "AZO"
+model_version: "1.0"
+model_contributors: [
+  {
+    "name": "Hector M. Sanchez C.",
+    "affiliation": "California Department of Public Health",
+    "email": "hector.sanchez-castellanos@cdph.ca.gov"
+  },
+  {
+    "name": "Tomas M. Leon",
+    "affiliation": "California Department of Public Health",
+    "email": "tomas.leon@cdph.ca.gov"
+  }
+]
+website_url: "https://github.com/cdphmodeling/wnvca-2024"
+license: "CC-BY-4.0"
+citation: ""
+team_funding: ""
+designated_model: false
+methods: "Predicts zero cases for every situation."
+data_inputs: "None"
+methods_long: "Baseline model to use as a comparison for a non-informative prediction."
+ensemble_of_models: false
+ensemble_of_hub_models: false

model-metadata/CDPH-EQV.yml

@@ -0,0 +1,27 @@
+team_name: "California Department of Public Health"
+team_abbr: "CDPH"
+model_name: "Equiprobable cases bins baseline model."
+model_abbr: "EQV"
+model_version: "1.0"
+model_contributors: [
+  {
+    "name": "Hector M. Sanchez C.",
+    "affiliation": "California Department of Public Health",
+    "email": "hector.sanchez-castellanos@cdph.ca.gov"
+  },
+  {
+    "name": "Tomas M. Leon",
+    "affiliation": "California Department of Public Health",
+    "email": "tomas.leon@cdph.ca.gov"
+  }
+]
+website_url: "https://github.com/cdphmodeling/wnvca-2024"
+license: "CC-BY-4.0"
+citation: ""
+team_funding: ""
+designated_model: false
+methods: "Assign the same probability to each bin."
+data_inputs: ""
+methods_long: "Baseline model that assigns equal probabilities to all the cases bins."
+ensemble_of_models: false
+ensemble_of_hub_models: false

model-metadata/README.html

@@ -496,40 +496,42 @@ <h3>team_funding</h3>
 </div>
 <div id="data-validation" class="section level1">
 <h1>Data validation</h1>
-<p>We strongly suggest to validate the output data before submitting it,
-as the pull request might take some time to perform the required tests
-(which might delay the submission process). To run the validation follow
-these steps:</p>
+<p>We strongly suggest to validate the model-metadata before submitting
+it, as the pull request might take some time to perform the required
+tests (which might delay the submission process). To run the validation
+follow these steps:</p>
 <ol style="list-style-type: decimal">
 <li>Create a fork of the <code>WNVCA-2024</code> repository and then
 clone the fork to your computer.</li>
 <li>Create a draft of the model metadata file for your model and place
-it in the <code>model-metadata</code> folder of this clone.</li>
+it in the <code>model-metadata</code> folder.</li>
 <li>Install the <code>hubValidations</code> package for R by running the
 following command from within an R session:</li>
 </ol>
 <pre class="r"><code>remotes::install_github(&quot;Infectious-Disease-Modeling-Hubs/hubValidations&quot;)</code></pre>
 <ol start="4" style="list-style-type: decimal">
 <li>Validate your draft metadata file by running the following command
-in an R session:</li>
+in an R session (with the working directory being in the root of the
+repository):</li>
 </ol>
 <pre class="r"><code>hubValidations::validate_model_metadata(
-    hub_path=&quot;&lt;path to your clone of the hub repository&gt;&quot;,
-    file_path=&quot;&lt;name of your metadata file&gt;&quot;
+  hub_path=&quot;.&quot;, 
+  # file_path=&quot;CDPH-AZO.yml&quot;
 )</code></pre>
 <p>For example, if your working directory is the root of the hub
 repository, you can use a command similar to the following:</p>
-<pre class="r"><code>hubValidations::validate_model_metadata(
-  hub_path=&quot;.&quot;, 
-  file_path=&quot;UMass-trends_ensemble.yml&quot;
-)</code></pre>
 <p>If all is well, you should see output similar to the following:</p>
 <pre><code>✔ model-metadata-schema.json: File exists at path hub-config/model-metadata-schema.json.
-✔ UMass-trends_ensemble.yml: File exists at path model-metadata/UMass-trends_ensemble.yml.
-✔ UMass-trends_ensemble.yml: Metadata file extension is &quot;yml&quot; or &quot;yaml&quot;.
-✔ UMass-trends_ensemble.yml: Metadata file directory name matches &quot;model-metadata&quot;.
-✔ UMass-trends_ensemble.yml: Metadata file contents are consistent with schema specifications.
-✔ UMass-trends_ensemble.yml: Metadata file name matches the `model_id` specified within the metadata file.</code></pre>
+✔ CDPH-AZO.yml: File exists at path model-metadata/CDPH-AZO.yml.
+✔ CDPH-AZO.yml: Metadata file extension is &quot;yml&quot; or &quot;yaml&quot;.
+✔ CDPH-AZO.yml: Metadata file directory name matches &quot;model-metadata&quot;.
+✔ CDPH-AZO.yml: Metadata file contents are consistent with schema specifications.
+✔ CDPH-AZO.yml: Metadata file name matches the `model_id` specified within the metadata file.</code></pre>
+</div>
+<div id="examplestemplates" class="section level1">
+<h1>Examples/Templates</h1>
+<p>Please see our <a href="./CDPH-AZO.yml">CDPH-AZO.yml</a> and <a href="./CDPH-EQV.yml">CDPH-EQV.yml</a> models examples for a template to
+follow for your models.</p>
 </div>
 
 

model-metadata/README.md

@@ -1,9 +1,3 @@
----
-output:
-  html_document: default
-  pdf_document: default
----
-
 # Model metadata
 
 
@@ -15,7 +9,7 @@ These instructions provide detail about the [data format](#Data-format) as well
 
 ## Required variables
 
-This section describes each of the variables (keys) in the yaml document. Please order the variables in this order.
+This section describes each of the variables (keys) in the yaml document. Please order the variables as follows.
 
 ### team_name
 The name of your team that is less than 50 characters.
@@ -102,43 +96,40 @@ One or more citations to manuscripts or preprints with additional model details.
 
 Any information about funding source(s) for the team or members of the team that would be natural to include on any resulting  publications. For example, "National Institutes of General Medical Sciences (R01GM123456). The content is solely the responsibility of the authors and does not necessarily represent the official views of NIGMS."
 
-# Data validation
+# YML validation
 
-We strongly suggest to validate the output data before submitting it, as the pull request might take some time to perform the required tests (which might delay the submission process). To run the validation follow these steps:
+We strongly recommend to validate the `model-metadata` before submitting it, as the pull request might take some time to perform the required tests (which might delay the submission process). To run the validation follow these steps:
 
 1. Create a fork of the `WNVCA-2024` repository and then clone the fork to your computer.
-2. Create a draft of the model metadata file for your model and place it in the `model-metadata` folder of this clone.
+2. Create a draft of the model metadata file for your model and place it in the `model-metadata` folder.
 3. Install the `hubValidations` package for R by running the following command from within an R session:
 
 ``` r
 remotes::install_github("Infectious-Disease-Modeling-Hubs/hubValidations")
 ```
-4. Validate your draft metadata file by running the following command in an R session:
-``` r
-hubValidations::validate_model_metadata(
-    hub_path="<path to your clone of the hub repository>",
-    file_path="<name of your metadata file>"
-)
-```
+4. Validate your draft metadata file by running the following command in an R session (with the working directory being in the root of the repository):
 
-For example, if your working directory is the root of the hub repository, you can use a command similar to the following:
 ``` r
 hubValidations::validate_model_metadata(
   hub_path=".", 
-  file_path="UMass-trends_ensemble.yml"
+  file_path="CDPH-AZO.yml"
 )
 ```
 
+For example, if your working directory is the root of the hub repository, you can use a command similar to the following:
+
+
 If all is well, you should see output similar to the following:
+
 ```
 ✔ model-metadata-schema.json: File exists at path hub-config/model-metadata-schema.json.
-✔ UMass-trends_ensemble.yml: File exists at path model-metadata/UMass-trends_ensemble.yml.
-✔ UMass-trends_ensemble.yml: Metadata file extension is "yml" or "yaml".
-✔ UMass-trends_ensemble.yml: Metadata file directory name matches "model-metadata".
-✔ UMass-trends_ensemble.yml: Metadata file contents are consistent with schema specifications.
-✔ UMass-trends_ensemble.yml: Metadata file name matches the `model_id` specified within the metadata file.
+✔ CDPH-AZO.yml: File exists at path model-metadata/CDPH-AZO.yml.
+✔ CDPH-AZO.yml: Metadata file extension is "yml" or "yaml".
+✔ CDPH-AZO.yml: Metadata file directory name matches "model-metadata".
+✔ CDPH-AZO.yml: Metadata file contents are consistent with schema specifications.
+✔ CDPH-AZO.yml: Metadata file name matches the `model_id` specified within the metadata file.
 ```
 
 # Examples/Templates
 
-Please see our CDPH-AZO.yml and CDPH-EQV.yml models examples for a template to follow for your models.
+Please have a look at our [CDPH-AZO.yml](./CDPH-AZO.yml) and [CDPH-EQV.yml](./CDPH-EQV.yml) specification files as examples on how to setup your own!

model-output/README.html

@@ -370,8 +370,7 @@ <h1>Data submission instructions</h1>
 <p>All forecasts should be submitted directly to the <a href="../model-output">model-output</a> folder. Data in this directory
 should be added to the repository through a pull request so that
 automatic <a href="../.github/workflows/model-output-pull.yml">validation checks</a>
-are run (although teams are highly encouraged to do <a href="#Local-Validation">local validations</a> before doing the data
-submission).</p>
+are run (although teams are highly encouraged to do <a href="#Local-Validation">local validations</a> before submitting).</p>
 <p>These instructions provide detail about the <a href="#Data-formatting">data format</a> as well as <a href="#Forecast-validation">validation</a> that you can do prior to this
 pull request. In addition, we describe <a href="https://github.com/Infectious-Disease-Modeling-Hubs/hubTemplate/blob/master/model-metadata/README.md">metadata</a>
 that each model should provide in the model-metadata folder.</p>
@@ -554,7 +553,7 @@ <h3><code>output_type_id</code></h3>
 <li><code>8</code>: (6, 8]</li>
 <li><code>10</code>: (8, 10]</li>
 <li><code>15</code>: (10, 15]</li>
-<li><code>200</code>: (15, 200]</li>
+<li><code>200</code>: (15, infinity]</li>
 </ul>
 </div>
 <div id="type" class="section level3">
@@ -610,7 +609,7 @@ <h3>Example tables</h3>
 <h2>Forecast validation</h2>
 <p>To ensure proper data formatting, pull requests for new data in
 <code>model-output/</code> will be automatically run. Optionally, you
-may also run these validations locally.</p>
+may also run these validations <a href="#Local-Validation">locally</a>.</p>
 <div id="pull-request-forecast-validation" class="section level3">
 <h3>Pull request forecast validation</h3>
 <p>When a pull request is submitted, the data are validated through <a href="https://docs.github.com/en/actions">Github Actions</a> which runs
@@ -619,10 +618,44 @@ <h3>Pull request forecast validation</h3>
 the requirements above. Please <a href="https://github.com/Chipdelmal/WNVCA-2024">let us know</a> if you
 are facing issues while running the tests.</p>
 </div>
-<div id="local-forecast-validation" class="section level3">
-<h3>Local forecast validation</h3>
-<p>Optionally, you may validate a forecast file locally before
-submitting it to the hub in a pull request.</p>
+<div id="local-validation" class="section level3">
+<h3>Local Validation</h3>
+<p>Teams are highly encouraged to validate their data locally before
+doing their pull request. To do this, install the
+<code>hubValidations</code> package:</p>
+<pre class="r"><code>remotes::install_github(&quot;Infectious-Disease-Modeling-Hubs/hubValidations&quot;)</code></pre>
+<p>then set your working directory to the base repository path, and
+run:</p>
+<pre class="r"><code>require(hubValidations)
+hubValidations::validate_submission(
+  file_path=&#39;CDPH-AZO/2024-05-31-CDPH-AZO.csv&#39;,
+  hub_path=&#39;.&#39;
+)</code></pre>
+<p>where the <code>file_path</code> should be replaced with your
+submission file. If the validation ran correctly, you should see
+something like:</p>
+<pre><code>✔ WNVCA-2024: All hub config files are valid.
+✔ 2024-05-31-CDPH-AZO.csv: File exists at path model-output/CDPH-AZO/2024-05-31-CDPH-AZO.csv.
+✔ 2024-05-31-CDPH-AZO.csv: File name &quot;2024-05-31-CDPH-AZO.csv&quot; is valid.
+✔ 2024-05-31-CDPH-AZO.csv: File directory name matches `model_id` metadata in file name.
+✔ 2024-05-31-CDPH-AZO.csv: `round_id` is valid.
+✔ 2024-05-31-CDPH-AZO.csv: File is accepted hub format.
+✔ 2024-05-31-CDPH-AZO.csv: Metadata file exists at path model-metadata/CDPH-AZO.yml.
+✔ 2024-05-31-CDPH-AZO.csv: File could be read successfully.
+✔ 2024-05-31-CDPH-AZO.csv: `round_id_col` name is valid.
+✔ 2024-05-31-CDPH-AZO.csv: `round_id` column &quot;origin_date&quot; contains a single, unique round ID value.
+✔ 2024-05-31-CDPH-AZO.csv: All `round_id_col` &quot;origin_date&quot; values match submission `round_id` from file name.
+✔ 2024-05-31-CDPH-AZO.csv: Column names are consistent with expected round task IDs and std column names.
+✔ 2024-05-31-CDPH-AZO.csv: Column data types match hub schema.
+✔ 2024-05-31-CDPH-AZO.csv: `tbl` contains valid values/value combinations.
+✔ 2024-05-31-CDPH-AZO.csv: All combinations of task ID column/`output_type`/`output_type_id` values are unique.
+✔ 2024-05-31-CDPH-AZO.csv: Required task ID/output type/output type ID combinations all present.
+✔ 2024-05-31-CDPH-AZO.csv: Values in column `value` all valid with respect to modeling task config.
+✔ 2024-05-31-CDPH-AZO.csv: Values in `value` column are non-decreasing as output_type_ids increase for all unique task ID value/output type combinations of
+  quantile or cdf output types.
+ℹ 2024-05-31-CDPH-AZO.csv: No pmf output types to check for sum of 1. Check skipped.
+! 2024-05-31-CDPH-AZO.csv: Submission time must be within accepted submission window for round.  Current time 2024-03-26 09:09:58.40993 is outside window
+  2024-05-16 PDT--2024-06-08 23:59:59 PDT.</code></pre>
 </div>
 </div>
 <div id="monthly-ensemble-build" class="section level2">
@@ -650,22 +683,6 @@ <h2>Evaluation criteria</h2>
 general performance of submitted forecasts. Participants may publish
 their own forecasts and results at any time.</p>
 </div>
-<div id="local-validation" class="section level2">
-<h2>Local Validation</h2>
-<p>Teams are highly encouraged to validate their data locally before
-doing their pull request. To do this, install the
-<code>hubValidations</code> package:</p>
-<pre class="r"><code>remotes::install_github(&quot;Infectious-Disease-Modeling-Hubs/hubValidations&quot;)</code></pre>
-<p>then set your working directory to the base repository path, and
-run:</p>
-<pre class="r"><code>require(hubValidations)
-hubValidations::validate_submission(
-  file_path=&#39;CDPH-AZO/2024-05-31-CDPH-AZO.csv&#39;,
-  hub_path=&#39;.&#39;
-)</code></pre>
-<p>where the <code>file_path</code> should be replaced with your
-submission file.</p>
-</div>
 </div>
 
 

model-output/README.md

@@ -221,12 +221,37 @@ then set your working directory to the base repository path, and run:
 ```R
 require(hubValidations)
 hubValidations::validate_submission(
-  file_path='CDPH-AZO/2024-05-31-CDPH-AZO.csv',
-  hub_path='.'
+  hub_path='.',
+  file_path='CDPH-AZO/2024-05-31-CDPH-AZO.csv'
 )
 ```
 
-where the `file_path` should be replaced with your submission file.
+where the `file_path` should be replaced with your submission file. If the validation ran correctly, you should see something like:
+
+```
+✔ WNVCA-2024: All hub config files are valid.
+✔ 2024-05-31-CDPH-AZO.csv: File exists at path model-output/CDPH-AZO/2024-05-31-CDPH-AZO.csv.
+✔ 2024-05-31-CDPH-AZO.csv: File name "2024-05-31-CDPH-AZO.csv" is valid.
+✔ 2024-05-31-CDPH-AZO.csv: File directory name matches `model_id` metadata in file name.
+✔ 2024-05-31-CDPH-AZO.csv: `round_id` is valid.
+✔ 2024-05-31-CDPH-AZO.csv: File is accepted hub format.
+✔ 2024-05-31-CDPH-AZO.csv: Metadata file exists at path model-metadata/CDPH-AZO.yml.
+✔ 2024-05-31-CDPH-AZO.csv: File could be read successfully.
+✔ 2024-05-31-CDPH-AZO.csv: `round_id_col` name is valid.
+✔ 2024-05-31-CDPH-AZO.csv: `round_id` column "origin_date" contains a single, unique round ID value.
+✔ 2024-05-31-CDPH-AZO.csv: All `round_id_col` "origin_date" values match submission `round_id` from file name.
+✔ 2024-05-31-CDPH-AZO.csv: Column names are consistent with expected round task IDs and std column names.
+✔ 2024-05-31-CDPH-AZO.csv: Column data types match hub schema.
+✔ 2024-05-31-CDPH-AZO.csv: `tbl` contains valid values/value combinations.
+✔ 2024-05-31-CDPH-AZO.csv: All combinations of task ID column/`output_type`/`output_type_id` values are unique.
+✔ 2024-05-31-CDPH-AZO.csv: Required task ID/output type/output type ID combinations all present.
+✔ 2024-05-31-CDPH-AZO.csv: Values in column `value` all valid with respect to modeling task config.
+✔ 2024-05-31-CDPH-AZO.csv: Values in `value` column are non-decreasing as output_type_ids increase for all unique task ID value/output type combinations of
+  quantile or cdf output types.
+ℹ 2024-05-31-CDPH-AZO.csv: No pmf output types to check for sum of 1. Check skipped.
+! 2024-05-31-CDPH-AZO.csv: Submission time must be within accepted submission window for round.  Current time 2024-03-26 09:09:58.40993 is outside window
+  2024-05-16 PDT--2024-06-08 23:59:59 PDT.
+  ```
 
 
 ## Monthly ensemble build