Developer Knowledge Base

This page is a collection of assorted tasks and topics developers from the NIST OSCAL Team will need or want as part of development activities. Documentation items, when specifically relevant to OSCAL, could and should make their way into official documentation with commits to the git repository's Markdown files. For official documentation, bug reports for error corrections and feature request issues identify and track work items. Changes to information here are not tracked that way. See the note on the bottom of this page. Information this page and children pages are for supporting activities underlying OSCAL development: code snippets for testing data format snippets (JSON, XML, YAML), software configuration, and common queries in different querying languages (JMESPath for JSON, XPath for XML, etc.).

NIST Dev Lunch and Learn Archive

20230421 - Basics of GitHub Actions

• recording
• example code repo

20230602 - VS Code, DevContainers, and interactive presentation tools

• recording

OSCAL Tips and Tricks

Submodule Management

Updating the Metaschema submodule in a PR

Switching to a fork sub-module for a draft PR

1. Open the OSCAL repository on your workstation.
2. Checkout your new branch from the target branch (develop or main):

$ git remote -v # confirm origin is usnistgov/OSCAL
me      git@github.com:youruser/OSCAL.git (fetch)
me      git@github.com:youruser/OSCAL.git (push)
origin  git@github.com:usnistgov/OSCAL.git (fetch)
origin  git@github.com:usnistgov/OSCAL.git (push)
$ git fetch --all
$ git checkout --track origin/develop
$ git checkout -b new-pr-to-develop # create PR branch
$ git push me HEAD # push to your fork
$ git submodule update --init --recursive
$ cd build/metaschema/
$ git fetch --all
$ git remote -v # confirm origin is usnistgov/metaschema
me      git@github.com:youruser/metaschema.git (fetch)
me      git@github.com:youruser/metaschema.git (push)
origin  git@github.com:usnistgov/metaschema.git (fetch)
origin  git@github.com:usnistgov/metaschema.git (push)
$ git checkout --track me/metaschema-fork-updated
$ cd ..
$ git submodule set-url build/metaschema git@github.com:youruser/metaschema.git
$ git add .gitmodules build/metaschema
$ git commit -m "[WIP] Test metaschema sub-module change to test tooling to fix model issue."

Model Documentation Pipeline Hacking

How to setup GitHub Repository

1. Make sure that submodules are initialized

   git submodule update --init --recursive
   

2. (Optional) Update submodules recursively

   git pull --recurse-submodules
   

How to run the pipeline

1. cd into the build/ directory

   cd ./build
   

2. Start the development stack:

   docker-compose up -d
   

3. Exec into the cli container

   docker-compose exec cli /bin/bash
   

4. Generate the model documentation

   In the cli container:

   cd ./build/ci-cd
   ./generate-model-documentation.sh -v
   

   or combine the full path

   ./build/ci-cd/generate-model-documentation.sh -v
   

5. Preview the changes

   In another window, with Hugo installed:

   cd docs
   hugo serve
   

6. View the updated documentation

   Visit http://localhost:1313/OSCAL/reference/. The "release" will depend on the branch you are in.

Software

Docker

Developers of the OSCAL project are encouraged to rely on a shared Docker image to ensure their environment is consistent with the rest of the team and CI/CD. Currently developers can choose to build the image themselves, or use the private Docker registry.

Note: currently the Docker registry is only accessible to members of the csd773 Docker Hub organization for administrative reasons, though we hope to make the registry public at some point.

Obtaining the image from the registry

If you are a OSCAL team member that does not have an account with Docker Hub that has access to the csd773 organization, ask on the team Gitter for instructions.

Otherwise, logging in is simple.

Using Docker desktop

If you are using Docker desktop, simply sign in via the button the top right corner.

Using the Docker CLI

Run the following command and follow the instructions:

docker login

Pulling the image

Once you have logged in, simply pull the image as follows:

docker pull csd773/oscal-common-env:develop

Building the image manually

Building the image locally is simple:

# From the root of the OSCAL project
# Force uninitialize submodules (optional)
# git submodule deinit -f .
# Ensure your submodules are up to date with the current commit
git submodule update --init --recursive
# Build the image
./build/build-oscal-env-dockerfile.sh

The script build-oscal-env-dockerfile.sh will tell you the built image's tag. E.g.

# Script output from build-oscal-env-dockerfile.sh
Built and tagged csd773/oscal-common-env:fixed-a-thing, to push run:
    docker push csd773/oscal-common-env:fixed-a-thing

Note that the tag of the built image is derived from your current branch name. For more details see "Controlling the Docker image in CI".

Using the image

To run a single command within the container run the following, replacing csd773/oscal-common-env:fixed-a-thing with the desired tag:

docker run \
    -v $(pwd):/oscal \
    csd773/oscal-common-env:fixed-a-thing \
    your-command-here

To run an interactive session, prepend the -it flags and do not put a command:

docker run -it \
    -v $(pwd):/oscal \
    csd773/oscal-common-env:fixed-a-thing

In both of these scenarios, the Docker image is passed the current working directory (e.g. the root of the repository) and maps it to /oscal in the container.

For more options like port mapping (required to access the Hugo serve web server), consult the Docker documentation

Controlling the Docker image in CI

In the future we plan to rely on Docker images for CI/CD more. GitHub actions workflows that depend on the Docker images use the csd773 registry (building locally each run would take forever). Because of this, GitHub workflows relying on CI will attempt to:

1. Pull the image with a tag corresponding to the current branch of the PR
2. Pull the image with the develop tag

This is done to provide users with an "escape hatch" for modifying the environment within a PR.

Users that have logged in to the Docker registry can build and push new images as follows:

1. Build the image as instructed in the "Building the image manually" section

2. Push the resulting tagged image to the registry as instructed by the script output:

   docker push csd773/oscal-common-env:$TAG # where $TAG is the tag of your built image

OxygenXML

Useful Plugins

• Java Classes Generator
• JSON Schema Documentation Generator
• OpenAPI Tester
• Oxygen Emmet Plugin
• Oxygen Emmet Plugin
• XSD to JSON Schema converter
• XSpec Framework
• XSpec Helper Viewer

Note: Plugins in Oxygen can be installed automatically or manually

Scenario Transformation Setup

This is an opinionated view, but Oxygen has a variety of configuration options for setting up a scenario for applying one or more XSLT transformations against a source document, a scenario for executing a XProc pipeline, and many more.

Below is how some of us use Editor Variables and other configuration elements. As a rule, some of us will use a variable for the source document and hard-code the path to the transform or XProc pipeline when running the scenarios interactively.

1. Open the XSLT stylesheet you will use for the transform, for example nist-metaschema-COMPOSE.xsl.
2. Open an input file for transforms in a XSLT stylesheet, like the oscal_ssp_metaschema.xml Metaschema module.
3. Open the scenario menu or click the Ctrl+Shift+C keyboard shortcut.
4. In the scenario menu, provide an appropriate name like nist-metaschema-COMPOSE.
5. For the XML URL for the input file, use the variable ${currentFileURL}.
6. For the XSL URL for the stylesheet, do not use a variable, but hard-code the path toe transform on the workstation like /path/to/code/repos/oscal/branches/origin/develop/build/metaschema/toolchains/xslt-M4/nist-metaschema-COMPOSE.xsl

This configuration will allow you to switch different inputs for testing while using the same stylesheet.

Code

Java-based Tooling

How do I invoke Java dependencies directly from a pom.xml file?

In several places across our stack we make use of Java-based tooling such as Saxon and Calabash. The version of these tools can be pinned using POM files, such as the one present in the build/ directory. Below is a simple template script that can be used to invoke a Java dependency directly from a POM file.

#!/usr/bin/env bash

# Fail early if an error occurs
set -Eeuo pipefail

if ! [ -x "$(command -v mvn)" ]; then
  echo 'Error: Maven (mvn) is not in the PATH, is it installed?' >&2
  exit 1
fi

SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd)"
OSCAL_DIR="${SCRIPT_DIR}/../../../.." # Edit this to resolve to the OSCAL root from the script's directory
POM_FILE="${OSCAL_DIR}/build/pom.xml"
# Grab any additional files relative to the root OSCAL directory (like stylesheets)

MAIN_CLASS="net.sf.saxon.Transform" #edit this to target your application's main class. below are a few examples:
# XML Calabash: "com.xmlcalabash.drivers.Main"
# Saxon: "net.sf.saxon.Transform"

# Perform any argument processing here, such as preselecting stylesheets in Saxon, etc.
# Note here "${*// /\\ }" is a shell expansion that escapes spaces within arguments.
# For more information, see https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html
ARGS=${*// /\\ }

mvn \
    -f "$POM_FILE" \
    exec:java \
    -Dexec.mainClass="$MAIN_CLASS" \
    -Dexec.args="$ARGS"

Note that this template is often extended to parse arguments directly. See an example of this in the profile resolution Saxon wrapper.

This approach has several key advantages:

1. The script can refer to resources invariably to the user's current working directory, making it more portable.
2. The user does not have to worry about downloading, copying, or otherwise managing any of the Java dependencies.
3. A lot of the complexity of consuming stylesheets and other operations can be hidden from the user, making our stack more approachable.

XPath

How do I check for multiple kinds of elements at the same time with XPath 3.0?

Let's say we want to look for the following OSCAL syntax elements in XML at the same time:

• any definition of a field with the name "with-id" in one or more target documents
• any definition of a flag with the name "pattern" in one or more target documents
• any definition of an assembly with the name "matching" in one or more target documents

Apply the following query against the target document(s) (for this example any Metaschema definition in src/metaschema/*.xml), observing use of the | ("union") operator:

//(
    define-flag[@name='pattern']
   | define-assembly[@name='matching']
   | define-field[@name='with-id']
 )

Here is a sneakier way to do something similar:

//@name[.=('pattern','matching','with-id')]/parent::*

parent::* can also be shortened to .. for those who prefer.

Mathematically, this path returns a superset of the nodes returned by the first path given, although in a particular metaschema they are likely to be the same.

This takes advantage of a feature of general comparison operators in XPath (=, <, >, <=, >=) namely that they support comparing sequences, i.e. many values on either side of the operand, hence .=('a','b','c').

Troubleshooting

GitHub Actions

Why do workflows on some runners return HTTP status code 0, failing builds or opening issues?

lychee or markdown-link-check will run an automated scan of the destination of almost every link (there is an exclude list) for every Markdown file in the repo and in the generated HTML that runs the website. Infrequently, GitHub's websites and external websites as well will rate-limit the runner's attempt open the link target or there is an internal failure on the runner. The resulting HTTP status is 0 (and that is not a normal one in the 200, 400, or 500 range). The build failure will indicate the link or an issue will be automatically opened that looks like the report below.

FILE: build/ci-cd/README.md

[✖] https://nodejs.org/en/

[✓] https://github.com/jessedc/ajv-cli

2 links checked.

If you can open the full URL reliably in your web browser on a developer workstation, you can safely presume this is a the infrequent hiccup on a GitHub Actions runner and close the issue as a false positive.