ECMWF Forecast Refactor#11

Open

sdc50 wants to merge 295 commits intoerdc:masterfrom sdc50:refactor_run_ecmwf_forecast_process

+4,722-2,410

This pull request is big! We're only showing the most recent 250 commits

Commits on Sep 2, 2015

Updated AutoRoutePy
snowman2
committed

Commits on Sep 11, 2015

removed passwords
Alan Snow
committed
removed passwords
Alan Snow
committed

Commits on Sep 18, 2015

Added RHEL 7 netcdf4 installation instructions.
Alan Snow
committed
Added RHEL 7 netcdf4 installation instructions.
Alan Snow
committed
Updated wording.
Alan Snow
committed
Updated wording.
Alan Snow
committed
Updated RHEL 7 netCDF4 installation instructions.
Alan Snow
committed
Updated RHEL 7 netCDF4 installation instructions.
Alan Snow
committed

Commits on Sep 25, 2015

Updated dataset manager
snowman2
committed
Updated dataset manager
snowman2
committed
Merge branch 'master' of https://github.com/erdc-cm/spt_ecmwf_autorapid_process
snowman2
committed
Merge branch 'master' of https://github.com/erdc-cm/spt_ecmwf_autorapid_process
snowman2
committed

Commits on Sep 29, 2015

Updated scripts for CKAN 2.5
snowman2
committed
Updated scripts for CKAN 2.5
snowman2
committed

Commits on Sep 30, 2015

Updated dataset maanger
snowman2
committed
Updated dataset maanger
snowman2
committed

Commits on Oct 6, 2015

Modified code to upload geoserver floodmap files as layer groups
snowman2
committed
Modified code to upload geoserver floodmap files as layer groups
snowman2
committed
Updated AutoRoute
snowman2
committed
Updated AutoRoute
snowman2
committed
Modified directory delete method
snowman2
committed
Updated dataset manager
snowman2
committed
Updated dataset manager
snowman2
committed

Commits on Oct 9, 2015

remove old file
snowman2
committed
remove old file
snowman2
committed
merge
snowman2
committed
merge
snowman2
committed
Merge branch 'master' of https://github.com/erdc-cm/spt_ecmwf_autorapid_process
snowman2
committed
Merge branch 'master' of https://github.com/erdc-cm/spt_ecmwf_autorapid_process
snowman2
committed
added ftp function
snowman2
committed
added ftp function
snowman2
committed
Merge branch 'master' of https://github.com/erdc-cm/spt_ecmwf_autorapid_process
snowman2
committed
Fixed bug
snowman2
committed
Updated dataset manager
snowman2
committed
Updated indents for loops
snowman2
committed

Commits on Oct 13, 2015

Made it so that your passwords and usernames in file not uploaded to repo
snowman2
committed

Commits on Oct 14, 2015

Syntax bug fix
snowman2
committed
Updated prereq installation instructions
Alan Snow
committed
Updated redhat dependencies
Alan Snow
committed

Commits on Oct 19, 2015

Added waiting for request to complete due to connection timeouts
snowman2
committed
Added error catching for AutoRAPID process GeoServer Upload
snowman2
committed
Updated dataset manager
snowman2
committed
Merge branch 'master' of https://github.com/erdc-cm/spt_ecmwf_autorapid_process
snowman2
committed
Improved download method to include multiple attempts until wait time exceeded
snowman2
committed

Commits on Oct 27, 2015

Updated RAPID installation steps
Alan Snow
committed

Commits on Oct 30, 2015

Added RedHat instructions for HTCondor
Alan Snow
committed
updated error exception for geoserver
snowman2
committed
updated cron to work with RedHat
snowman2
committed
Merge branch 'master' of https://github.com/erdc-cm/spt_ecmwf_autorapid_process
snowman2
committed

Commits on Nov 10, 2015

FIxed geoserver import method
snowman2
committed
Added method to run ECMWF low res at full resolution
snowman2
committed
Modified code to run ecmwf low res at full temporal resolution
snowman2
committed
Added old code for reference and fixed max wait time
snowman2
committed
Improved datetime search in forecast name to remove characters
snowman2
committed
Modified query to search for full resolution forecasts
snowman2
committed
Merge branch 'master' of https://github.com/erdc-cm/spt_ecmwf_autorapid_process
snowman2
committed
Updated ZS_dtM
snowman2
committed
Updated to work with full resolution of the data
snowman2
committed
Remove reduntant operation for extracting high res data
snowman2
committed
Bug fix for name of variable
snowman2
committed
Updated AutoRoutePy version
snowman2
committed
Updated function name to new verison
snowman2
committed
Updated version of AutoRoutePy
snowman2
committed
Added sorting and filtering
snowman2
committed

Commits on Nov 12, 2015

Updated code to work with AutoRoute-py multiprocess mode
snowman2
committed
Remove submodule AutoRoute-py
snowman2
committed
Updated README to reflect installation of AutoRoute-py
snowman2
committed
Fixed name
snowman2
committed

Commits on Nov 16, 2015

Modified code to look for return period file
snowman2
committed

Commits on Dec 17, 2015

Updated dataset manager to new version
snowman2
committed
Updated script for backwards compatibility, bug fixes
snowman2
committed

Commits on Dec 31, 2015

Made comid_lat_lon_z file optional.
snowman2
committed

Commits on Jan 4, 2016

Updated clean log script to skip unrecognized files.
snowman2
committed

Commits on Jan 19, 2016

Updated AutoRoutePy name
Alan Snow
committed

Commits on Jan 26, 2016

Merge branch 'master' of https://github.com/erdc-cm/spt_ecmwf_autorapid_process
snowman2
committed

Commits on Feb 9, 2016

updated river id to always be rivid. Set fill_value to zero
snowman2
committed

Commits on Feb 11, 2016

Remove negative values and noise from simulations
snowman2
committed
Modified code to use RAPIDDataset
snowman2
committed
Added initial attempt at seasonal initialization method.
snowman2
committed
Removed debug comments
snowman2
committed

Commits on Feb 16, 2016

Removed unnecessary weight table info loading.
snowman2
committed

Commits on Feb 19, 2016

Move seasonal init at beginning to use multiprocessing
snowman2
committed
Modified to unzip gz files
snowman2
committed
Modified to use extractnested code
snowman2
committed

Commits on Feb 22, 2016

Fixed folder name deriving method. Moved attempt for ftpclient in try except loop.
snowman2
committed
Added identification for new ECMWF grids and weight table name function
snowman2
committed
Modified method to search for weight table name.
snowman2
committed
Added search for new forecast files
snowman2
committed
dataIdentify/getWeightTableName bug fixes
snowman2
committed

Commits on Feb 26, 2016

Added date_string option
snowman2
committed
modified code to identify grid for unit conversions
snowman2
committed
Bug fix with new grid name function use
snowman2
committed
Bug fix with weight table name search.
snowman2
committed
Bug fix with weight table file search
snowman2
committed

Commits on Mar 7, 2016

Bug fix in intit script
snowman2
committed
fixed import statement helper_functions
snowman2
committed
Reorganized streamfow assimilation functions
snowman2
committed
new folder name bug fix
snowman2
committed
Bug fix getting ensemble number from qout
snowman2
committed
Bug fix
snowman2
committed

Commits on Mar 9, 2016

updated dataset manager
snowman2
committed
Updated licensing.
snowman2
committed
Changed to BSD-3 Clause
Alan D. Snow
committed
Added license badge
Alan D. Snow
committed
Removed unecessary print statements
snowman2
committed
Merge branch 'master' of https://github.com/erdc-cm/spt_ecmwf_autorapid_process
snowman2
committed

Commits on Mar 10, 2016

Added zenodo badge
Alan D. Snow
committed

Commits on Mar 15, 2016

Added option for using multiprocessing instead of htcondor
snowman2
committed
Fixed file name in comments
snowman2
committed
Merge branch 'master' of https://github.com/erdc-cm/spt_ecmwf_autorapid_process
snowman2
committed
Updated readme for new multiprocess version. Modified htcondor install instructions.
snowman2
committed
header fix
snowman2
committed
reorder
snowman2
committed
condor restart instructions
snowman2
committed

Commits on Mar 28, 2016

Updated RAPIDpy imports
snowman2
committed

Commits on Apr 5, 2016

Update ftp_ecmwf_download.py
Alan D. Snow
committed
Update ftp_ecmwf_download.py
Alan D. Snow
committed

Commits on Apr 6, 2016

Added RAPIDpy watershed generation info
Alan D. Snow
committed

Commits on Apr 7, 2016

updated dataset manager
snowman2
committed
Modified warning point generation to give the time of peak.
snowman2
committed
Trim date for peak
snowman2
committed

Commits on May 3, 2016

Delete forecast file daily to save space
cecsr
committed
Merge branch 'master' of https://github.com/CI-WATER/spt_ecmwf_autorapid_process
cecsr
committed
Undo personal stuff for merge with alan's code
crae2244
committed
Merge pull request erdc#2 from CI-WATER/curtisdev
Alan D. Snow
committed

Commits on May 13, 2016

modified for graduated thresholds
snowman2
committed

Commits on May 16, 2016

bug fix naming return period variables
snowman2
committed

Commits on Jun 2, 2016

modified to reflect most recent autoroute version
snowman2
committed
Merge branch 'master' of https://github.com/erdc-cm/spt_ecmwf_autorapid_process
snowman2
committed

Commits on Jun 22, 2016

bug fix multiprocess get job output
snowman2
committed
added info for how it works
snowman2
committed
bug fix multiprocessing set job info outside of if statement
snowman2
committed
modified path for qinit file
snowman2
committed
changed COMID to rivid
snowman2
committed

Commits on Jun 29, 2016

Added JAWRA publication
Alan D. Snow
authored

Commits on Jul 26, 2016

bug fix as first Qinit file is in RAPID input directory
snowman2
committed

Commits on Aug 29, 2016

updated dataset manager
snowman2
committed
reorganize repo, make python write logfile
snowman2
committed
modified print functions for python 2/3
snowman2
committed
modified print to be python 2/3 compatible
snowman2
committed
fixed imports to be compatible with python 2/3
snowman2
committed
updated README with table for function and examples of use
snowman2
committed
fixed table
snowman2
committed
updated spacing
snowman2
committed
minor update
snowman2
committed
minor fixes
snowman2
committed
header changes
snowman2
committed
added tethys_dataset_services to dependencies
snowman2
committed
added seasonal initialization from seasonal average file
snowman2
committed
fixed import netCDF4
snowman2
committed
modified run mode to be hourly with lock file
snowman2
committed

Commits on Aug 30, 2016

modified method for lock based on last forecast date
snowman2
committed
updated cron instructions
snowman2
committed
made AutoRoute install optional
snowman2
committed
minor modification for directory name example
snowman2
committed
added full path to folder when downloading
snowman2
committed
added comment about commented testing code
snowman2
committed
updated path for forecasts to download folder
snowman2
committed
added lock release if no forecasts found
snowman2
committed

Commits on Aug 31, 2016

modified job name
snowman2
committed
added traceback to print exception
snowman2
committed
moved new version to default for forecast file search
snowman2
committed

Commits on Sep 7, 2016

Update README.md
Alan D. Snow
authored

Commits on Sep 16, 2016

added rivid attribute to warning point output
snowman2
committed

Commits on Sep 20, 2016

converted rivid to int before json dumps
snowman2
committed

Commits on Nov 3, 2016

modified code to reset lock info file & instructions
snowman2
committed
updated docs
snowman2
committed
added lock reset function to init import
snowman2
committed
updated instructions for file name
snowman2
committed

Commits on Dec 7, 2016

modified code to work with spt_dataset_manager as package
snowman2
committed

Commits on Dec 19, 2016

Added option to not delete past forecasts
snowman2
committed
bug fix if statement
snowman2
committed

Commits on Mar 30, 2017

added conda env yml file for creating environment in conda
Alan D. Snow
committed
updated headers
Alan D. Snow
committed

Commits on Apr 27, 2017

Commits on Jun 14, 2017

added ignore for pycharm .idea
snowman2
committed
moved rapid_process.py to ecmwf_rapid_process.py
snowman2
committed
renamed spt_ecmwf_autorapid_process to spt_process
snowman2
committed
rename to spt_compute; move process lock to separate file
snowman2
committed
fixed comment to be spt_compute
snowman2
committed
initial design for SPT WRF process
snowman2
committed

Commits on Jun 16, 2017

renamed to spt_compute; renamed to ecmwf_forecast_process; added lsm_forecast_process
snowman2
committed
fixed print statements for py3
snowman2
committed
removed unused functions; fix for clean logs
snowman2
committed
added start of function to generate streamflow init
snowman2
committed
fixed import for determine_start_end_timestep; added new init function
snowman2
committed
remve m3 file generated
snowman2
committed
added tests for wrf forecast
snowman2
committed
removed commented out cleanup method
snowman2
committed
added comparison of Qout file using xarray
snowman2
committed
fix getting date by sorting input list of wrf grids; update Qout file for linux as it is more stable
Alan D. Snow
committed

Commits on Jun 20, 2017

added historical tests generating warning points; fix tests to cleanup after each one
Alan D. Snow
committed
added historical setup
Alan D. Snow
committed
removed duplicate test
Alan D. Snow
committed

Commits on Jun 21, 2017

added seasonal input to tests
Alan D. Snow
committed
added time selection for qinit
Alan D. Snow
committed
modified tests to compare the warning json files produced
Alan D. Snow
committed

Commits on Jun 24, 2017

modified to allow to run by region in collaboration with Mohit 3d274eb
Alan D. Snow
committed
fix lock file name
Alan D. Snow
committed
python 3 fixes; improved debugging
Alan D. Snow
committed
added traceback printing for exceptions
Alan D. Snow
committed

Commits on Jul 7, 2017

added Travis CI yml for testing
snowman2
committed
added badge for testing
snowman2
committed
added travis yml for testing
snowman2
committed
removed python pin
snowman2
committed
added test requirments
snowman2
committed
added installation for gfortran; removed osx miniconda install
snowman2
committed
added unicode literals for python2
snowman2
committed
modified warnings to use unicode literals
snowman2
committed
Merge pull request erdc#7 from erdc-cm/wrf
Alan D. Snow
authored

Commits on Jul 11, 2017

added basic ECMWF forecast test
Alan D. Snow
committed
relaxed decimal comparison from 7 to 5
Alan D. Snow
committed
updated tests to 20161221 tag of RAPID
Alan D. Snow
committed
Merge pull request erdc#8 from erdc-cm/tests
Alan D. Snow
authored

Commits on Jul 19, 2017

Update README.md
Alan D. Snow
authored
Update README.md
Alan D. Snow
authored
Update README.md
Alan D. Snow
authored
Update README.md
Alan D. Snow
authored

Commits on Aug 11, 2017

modified to only generate geojson
Alan D. Snow
committed
Added test for generating ecmwf warnings
Alan D. Snow
committed
updated paths to build links
Alan D. Snow
committed
Merge branch 'geojson'
Alan D. Snow
committed
formatting
Alan D. Snow
committed
updated tests for seasonal initialization for ecmwf
Alan D. Snow
committed

Commits on Aug 14, 2017

compressed warning points geojson files
Alan D. Snow
committed
added pandas dependency
Alan D. Snow
committed
update warning points generation file formatting
Alan D. Snow
committed
updated to format required packages array
Alan D. Snow
committed
use xarray to analyze data for generating warning points
Alan D. Snow
committed
use xarray instead of RAPIDDataset
Alan D. Snow
committed
added ignore for pylint
Alan D. Snow
committed
added option for return period threshold to generate warting; updated tests
Alan D. Snow
committed
modified return period to have warnings in all bins
Alan D. Snow
committed
added parameter for warning flow threshold for LSM forecast
Alan D. Snow
committed
typo fix
Alan D. Snow
committed

Commits on Aug 29, 2017

updated to import from spt_compute
Alan Snow
committed
added qforcing file search
Alan Snow
committed
added forcing option for forecasts. However only supports constant flow
Alan D. Snow
committed
merged in HPC testing stuff
Alan Snow
committed
update helper functions
Alan Snow
committed
update folder date
Alan Snow
committed
Merge pull request erdc#9 from erdc-cm/hpc
Alan D. Snow
authored

Commits on Aug 30, 2017

to 2.0.1
Alan Snow
committed

Commits on Sep 14, 2017

Fix travis badge path.
Alan D. Snow
authored
Update README.md
Alan D. Snow
authored

Commits on May 18, 2018