Merge pull request #21 from FRBs/typing_bugs

Typing and bugs
FRBs · Jan 31, 2022 · d612e8e · d612e8e
2 parents a07d00c + b8fc515
commit d612e8e
Show file tree

Hide file tree

Showing 36 changed files with 1,861 additions and 1,446 deletions.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -26,9 +26,6 @@ jobs:
       run: |
         python -m pip install -U pip setuptools wheel
         python -m pip install -e .[tests]
-    #- name: Simple test
-    #  run: |
-    #    python examples/dedisperse.py examples/tutorial.fil 300
     - name: Lint with flake8
       run: |
         pip install flake8 wemake-python-styleguide

diff --git a/.gitignore b/.gitignore
@@ -56,7 +56,7 @@ pytestdebug.log
 docs/_build/
 docs/_static/
 docs/_templates/
-doc/_build/
+docs/jupyter_execute/
 
 # Jupyter Notebook
 .ipynb_checkpoints

diff --git a/HISTORY.rst b/HISTORY.rst
@@ -0,0 +1,39 @@
+1.0.0 (2022-02-01)
+++++++++++++++++++
+
+- First stable release of the new API.
+- Adds Multifile support for ``SIGPROC Filterbank`` files. (`#13 <https://github.com/FRBs/sigpyproc3/pull/13>`_)
+- Adds Numba backend for kernel functions in place of the earlier `pybind11 <https://github.com/pybind/pybind11>`_. (`#17 <https://github.com/FRBs/sigpyproc3/pull/17>`_)
+- Removes dependence on `pybind11 <https://github.com/pybind/pybind11>`_ and `fftw3 <http://www.fftw.org/>`_.
+- Adds Packaging structure as for the new API. (`#16 <https://github.com/FRBs/sigpyproc3/pull/16>`_)
+- Adds ``io`` module for reading and writing of ``SIGPROC`` files.
+- Adds fast median and mean filter (`bottleneck <https://github.com/pydata/bottleneck>`_) and Higher-Order running stats.
+- Support for command-line utilities
+- Adds CHIME telescope code to ``params``.
+
+
+0.5.5 (2021-03-31)
+++++++++++++++++++
+
+- Fixes major installation and Compilation errors on Mac.
+- Adds SRT telescope code to ``params``.
+- Adds more tests to increase code covergae.
+
+
+0.5.0 (2020-11-18)
+++++++++++++++++++
+
+- Major update: replacing ctypes with `pybind11 <https://github.com/pybind/pybind11>`_.
+- Adds coverage tests and online docs on the readthedocs platform.
+- Fixes the cyclic import in modules.
+
+
+0.1.1 (2020-08-14)
+++++++++++++++++++
+
+- First python3 stable release.
+- Adds an arg ``only_valid_samples`` to ``FilterbankBlock.dedisperse()`` (`#2 <https://github.com/FRBs/sigpyproc3/pull/2>`_)
+- Adds a new reader ``FilReader.readDedispersedBlock()``  (`#1 <https://github.com/FRBs/sigpyproc3/pull/1>`_)
+- Adds coverage tests and progress bar using `tqdm <https://github.com/tqdm/tqdm>`_.
+- Adds new methods ``removeBandpass``, ``removeZeroDM``, ``splitToBands``.
+- Adds more stable and accurate ``getStats`` (`ref <https://doi.org/10.2172/1028931>`_).
diff --git a/README.md b/README.md
@@ -3,42 +3,38 @@
 [![GitHub CI](https://github.com/FRBs/sigpyproc3/workflows/GitHub%20CI/badge.svg)](https://github.com/FRBs/sigpyproc3/actions)
 [![Docs](https://readthedocs.org/projects/sigpyproc3/badge/?version=latest)](https://sigpyproc3.readthedocs.io/en/latest/?badge=latest)
 [![codecov](https://codecov.io/gh/FRBs/sigpyproc3/branch/master/graph/badge.svg)](https://codecov.io/gh/FRBs/sigpyproc3)
+[![License](https://img.shields.io/github/license/FRBs/sigpyproc3)](https://github.com/FRBs/sigpyproc3/blob/main/LICENSE)
 [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
 
-`sigpyproc` is a pulsar and FRB data analysis library for python. It provides an OOP approach to pulsar data handling through the use of objects representing different data types (e.g. [SIGPROC filterbank](http://sigproc.sourceforge.net), time-series, fourier-series, etc.).
+`sigpyproc` is a pulsar and FRB data analysis library for python. It provides an OOP approach to pulsar data handling through the use of objects representing different data types (e.g. [SIGPROC filterbank](http://sigproc.sourceforge.net), time-series, fourier-series, etc.). As pulsar data processing is often time critical, speed is maintained using
+the excellent [numba](https://numba.pydata.org) library.
 
-As pulsar data processing is often time critical, speed is maintained through the use of compiled C++ code that are accessed via the excellent `pybind11 <https://pybind11.readthedocs.io/>`_ library. Additional performance increases are obtained via the use of multi-threading with OpenMP, a threading library standard to most linux and mac systems.
-
-`sigpyproc` was initially intended to be an Python wrapper for the [SIGPROC](http://sigproc.sourceforge.net) pulsar signal processing toolbox, but over time it has developed and become an independent project in its own right. Unlike [SIGPROC](http://sigproc.sourceforge.net) and [PRESTO](https://github.com/scottransom/presto), `sigpyproc` does not currently have full capabilities as a piece of FRB/pulsar searching software. Instead, `sigpyproc` provides data manipulation routines which are well suited to preprocessing and micro-management of pulsar data. The structure of the package also makes it an ideal development environment, with a simple plug-and-play system with new modules and extensions.
+## Installation
 
-## Basic Usage
-
-```python
-from sigpyproc.readers import FilReader
-myFil = FilReader("tutorial.fil")
+The quickest way to install the package is to use [pip](https://pip.pypa.io):
 
+```bash
+pip install -U git+https://github.com/FRBs/sigpyproc3
 ```
 
-## Installation
+Note that you will need Python (>=3.8) installed to use `sigpyproc`.
+Also check out the [installation documentation page](https://sigpyproc3.readthedocs.io/en/latest/install.html) for more options.
 
-You need Python 3.8 or later to run sigpyproc.
-Install via pip:
+## Legacy Version
 
-```bash
-pip install git+https://github.com/FRBs/sigpyproc3
-```
+`sigpyproc` is currently undergoing major developements which will modify the existing API in order to be a modern python
+replacemet for [SIGPROC](http://sigproc.sourceforge.net). To use the older API, you can install the ``legacy``
+branch of this repo, or install the last released version 0.5.5.
 
-Or, download / clone this repository, and then run
+## Usage
 
-```bash
-python -m pip install .
+```python
+from sigpyproc.readers import FilReader
+fil = FilReader("tutorial.fil")
 ```
 
-### Test the installation
+Check out the tutorials and API docs on [the docs page](https://sigpyproc3.readthedocs.io) for example usage and more info.
 
-You can execute some unit and benchmark tests using [pytest](https://docs.pytest.org) to make sure that the installation went alright. In the root directory of the source code
-execute the following command:
+## Contributing
 
-```bash
-python -m pytest -v tests
-```
+Check out [the developer documentation](https://sigpyproc3.readthedocs.io/en/latest/dev.html) for more info about getting started.
diff --git a/docs/changes.rst b/docs/changes.rst
@@ -0,0 +1,4 @@
+Changelog
+=========
+
+.. include:: ../HISTORY.rst
diff --git a/docs/conf.py b/docs/conf.py
@@ -20,8 +20,8 @@
 # -- Project information -----------------------------------------------------
 
 project = "sigpyproc3"
-copyright = "2022, Ewan Barr & Fast Radio Burst Software"
-author = "Ewan Barr"
+copyright = "2020, Fast Radio Burst Software"
+author = "Fast Radio Burst Software"
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the
@@ -37,14 +37,10 @@
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
 extensions = [
-    "myst_parser",
     "sphinx.ext.autodoc",
     "sphinx.ext.napoleon",
     "sphinx.ext.intersphinx",
-    "sphinx.ext.viewcode",
-    "nbsphinx",
-    "IPython.sphinxext.ipython_console_highlighting",
-    "IPython.sphinxext.ipython_directive",
+    "myst_nb",
 ]
 
 # Add any paths that contain templates here, relative to this directory.
@@ -62,29 +58,27 @@
 # a list of builtin themes.
 
 html_theme = "sphinx_book_theme"
+html_title = "sigpyproc3"
 html_theme_options = {
-    "logo_only": False,
-    "display_version": True,
-    "prev_next_buttons_location": "bottom",
-    "style_external_links": False,
-    # Toc options
-    "collapse_navigation": True,
-    "sticky_navigation": True,
-    "navigation_depth": 4,
-    "includehidden": True,
-    "titles_only": False,
+    "repository_url": "https://github.com/FRBs/sigpyproc3",
+    "use_repository_button": True,
+    "use_issues_button": True,
+    "use_download_button": True,
 }
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
 # \html_static_path = ["_static"]
 
+jupyter_execute_notebooks = "auto"
+execution_timeout = -1
 
 # -- Extension configuration -------------------------------------------------
 
 autoclass_content = "class"  # include both class docstring and __init__
 autodoc_member_order = "bysource"
 autodoc_typehints = "none"
+autodoc_inherit_docstrings = True
 
 napoleon_google_docstring = False
 napoleon_numpy_docstring = True

diff --git a/docs/dev.rst b/docs/dev.rst
@@ -1,72 +1,61 @@
 .. _dev:
 
-Developer guide
-===============
+Developer documentation
+=======================
 
 Here we will cover all the steps required to add new functionality to the
 ``sigpyproc`` package. To do this, we will first consider adding a new function
-to the :class:`~sigpyproc.Filterbank.Filterbank` class, before going on to extend
-the function to deal with an arbitrary data format.
+to the :class:`~sigpyproc.base.Filterbank` class.
 
 
 Adding a new function: ``bandpass()``
 -------------------------------------
 **Aim:** Add a new function called bandpass, which will return the total power
 as a function of frequency for our observation.
 
-**Files to be modified:** ``sigpyproc/Filterbank.py``, ``c_src/kernels.hpp``
-, ``c_src/bindings.cpp``.
+**Files to be modified:** ``sigpyproc/base.py``, ``sigpyproc/core/kernels.py``.
 
-Step 1: Write the Python part
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-The first step is to write the Python side of the function. As this function
+Step 1: Write the API part
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+The first step is to write the user API of the function. As this function
 will run on data with both time and frequency resolution,
-it belongs in the :class:`~sigpyproc.Filterbank.Filterbank` class.
+it belongs in the :class:`~sigpyproc.base.Filterbank` class.
 
 .. code-block:: python
 
     def bandpass(self, gulp=512, **kwargs):
-        bpass_ar = np.zeros(self.header.nchans, dtype="float64")
+        bpass_ar = np.zeros(self.header.nchans, dtype=np.float32)
         num_samples = 0
-        for nsamps, ii, data in self.readPlan(gulp, **kwargs):
-            lib.getBpass(data, bpass_ar, self.header.nchans, nsamps)
+        for nsamps, ii, data in self.read_plan(**plan_kwargs):
+            kernel.extract_bpass(data, bpass_ar, self.header.nchans, nsamps)
             num_samples += nsamps
         bpass_ar = bpass_ar / num_samples
-        return TimeSeries(bpass_ar, self.header.newHeader({"nchans": 1}))
+        return TimeSeries(bpass_ar, self.header.new_header({"nchans": 1}))
 
 Looking at the important lines, we have:
-Return an instance of the :class:`~sigpyproc.TimeSeries.TimeSeries` class.
-The :class:`~sigpyproc.TimeSeries.TimeSeries` class takes two arguments,
+Return an instance of the :class:`~sigpyproc.timeseries.TimeSeries` class.
+The :class:`~sigpyproc.timeseries.TimeSeries` class takes two arguments,
 an instance of :py:obj:`numpy.ndarray` and an instance of
-:class:`~sigpyproc.Header.Header`.
+:class:`~sigpyproc.header.Header`.
 
 Now we have something similar to a normal :py:obj:`numpy.ndarray`,
 which exports several other methods for convenience.
 
-Step 2: Write the C++ part
-^^^^^^^^^^^^^^^^^^^^^^^^^^
-We called a C++ library function named getBpass. This function belongs in the
-``c_src/kernels.hpp`` file (or anywhere you please, as long as it is properly
-binded with the `pybind11 <https://pybind11.readthedocs.io/>`_ library).
-In ``kernels.hpp``, our function looks like:
-
-.. code-block:: C++
-
-    template <class T>
-    void getBpass(T* inbuffer, double* outbuffer, int nchans,
-                int nsamps) {
-    #pragma omp parallel for default(shared)
-        for (int jj = 0; jj < nchans; jj++) {
-            for (int ii = 0; ii < nsamps; ii++) {
-                outbuffer[jj] += inbuffer[(nchans * ii) + jj];
-            }
-        }
-    }
+Step 2: Write the core Numba kernel
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+We called a kernel function named ``extract_bpass``. This function belongs in the
+``sigpyproc/core/kernels.py`` file. In ``kernels.py``, our function looks like:
+
+.. code-block:: python
+
+    @njit(["void(u1[:], f4[:], i4, i4)", "void(f4[:], f4[:], i4, i4)"], cache=True, parallel=True)
+    def extract_bpass(inarray, outarray, nchans, nsamps):
+        for ichan in prange(nchans):
+            for isamp in range(nsamps):
+                outarray[ichan] += inarray[nchans * isamp + ichan]
 
 This function receives a block of data and sums that block along the time axis.
-We use an optional C pre-processor directive to enable OpenMP threading.
-Realistically this call is not required here, but it is left here of a clear
-example of how to quicly multi-thread a system of loops.
+We use the jit compiler directive ``parallel`` to enable OpenMP threading.
 
 
 Reporting an issue
@@ -80,3 +69,5 @@ please provide the details to reproduce the issue.
 Contributing code or documentation
 ----------------------------------
 
+An excellent place to start is the `AstroPy developer docs
+<https://docs.astropy.org/en/stable/development/workflow/development_workflow.html>`_.
diff --git a/docs/index.rst b/docs/index.rst
@@ -1,16 +1,52 @@
-.. include:: ../README.md
-   :parser: myst_parser.sphinx_
+sigpyproc
+=========
 
+**sigpyproc** is a FRB/pulsar data analysis library for Python. It provides an
+OOP approach to pulsar data handling through the use of objects representing
+different data types (e.g.
+`SIGPROC filterbank <http://sigproc.sourceforge.net>`_,
+`PSRFITS <https://www.atnf.csiro.au/research/pulsar/psrfits_definition/Psrfits.html>`_,
+time series, fourier series, etc.).
+
+As pulsar data processing is often time critical, speed is maintained using
+the excellent `numba <https://numba.pydata.org/>`_ library.
+
+`sigpyproc` is intended to be an Python alternative for the
+`SIGPROC filterbank <http://sigproc.sourceforge.net>`_ pulsar signal processing toolbox.
+Over time it has also developed and become an independent project in its own right.
+Unlike `SIGPROC <http://sigproc.sourceforge.net>`_ and `PRESTO <https://github.com/scottransom/presto>`_,
+`sigpyproc` does not currently have full capabilities as a piece of FRB/pulsar searching software.
+Instead, `sigpyproc` provides data manipulation routines which are well suited to preprocessing
+and micro-management of pulsar data. The structure of the package also makes it an ideal development
+environment, with a simple plug-and-play system with new modules and extensions.
+
+.. image:: https://github.com/FRBs/sigpyproc3/workflows/GitHub%20CI/badge.svg
+  :target: https://github.com/FRBs/sigpyproc3/actions
+.. image:: https://readthedocs.org/projects/sigpyproc3/badge/?version=latest
+  :target: https://sigpyproc3.readthedocs.io/en/latest/?badge=latest
+  :alt: Documentation Status
+.. image:: https://codecov.io/gh/FRBs/sigpyproc3/branch/master/graph/badge.svg
+  :target: https://codecov.io/gh/FRBs/sigpyproc3
+.. image:: https://img.shields.io/github/license/FRBs/sigpyproc3
+  :target: https://github.com/FRBs/sigpyproc3/blob/main/LICENSE
+.. image:: https://img.shields.io/badge/code%20style-black-000000.svg
+  :target: https://github.com/psf/black
+
+
+Contents
+--------
 
 .. toctree::
-  :maxdepth: 2
-  :caption: Documentation
+  :maxdepth: 1
+  :caption: User Guide
 
+  install
   modules
   dev
+  changes
 
 .. toctree::
-  :maxdepth: 2
+  :maxdepth: 1
   :caption: Tutorials
 
   tutorials/quickstart.ipynb