Clashing constraints in the assessment layer - Interpreting the XPath

[wendellpiez]

The assessment-common metaschema module contains some rules that can benefit from elucidation:

OSCAL/src/metaschema/oscal_assessment-common_metaschema.xml

Lines 71 to 74 in 4f02dac

	<has-cardinality target="part[has-oscal-namespace('http://csrc.nist.gov/ns/oscal') and @name=('objective','assessment-objective')]" max-occurs="1" />
	<has-cardinality target="part[has-oscal-namespace('http://csrc.nist.gov/ns/oscal') and @name=('assessment','assessment-method')]/prop[has-oscal-namespace(('http://csrc.nist.gov/ns/oscal','http://csrc.nist.gov/ns/rmf')) and @name='method']" min-occurs="1" max-occurs="1" />
	<has-cardinality target="part[has-oscal-namespace('http://csrc.nist.gov/ns/oscal') and @name=('assessment','assessment-method')]/part[has-oscal-namespace('http://csrc.nist.gov/ns/oscal') and @name=('objects','assessment-objects')]" min-occurs="1" max-occurs="1" />
	<has-cardinality target="part[has-oscal-namespace('http://csrc.nist.gov/ns/oscal') and @name=('objective','assessment-objective')]/prop[has-oscal-namespace('http://csrc.nist.gov/ns/oscal') and @name='method-id']" min-occurs="1" />

All of these rules pertain to elements defined as "local-objective", including <objective-and-methods>.

RULE 1:

<has-cardinality target="part[has-oscal-namespace('http://csrc.nist.gov/ns/oscal') and @name=('objective','assessment-objective')]" max-occurs="1" />

There may not be more than one part named objective or assessment-objective

RULE 2:

<has-cardinality target="part[has-oscal-namespace('http://csrc.nist.gov/ns/oscal') and @name=('assessment','assessment-method')]/prop[has-oscal-namespace(('http://csrc.nist.gov/ns/oscal','http://csrc.nist.gov/ns/rmf')) and @name='method']" min-occurs="1" max-occurs="1" />

There must be a single prop named method in a part named assessment or assessment-method. Yes, this entails the requirement that the part also exists.

RULE 3:

<has-cardinality target="part[has-oscal-namespace('http://csrc.nist.gov/ns/oscal') and @name=('assessment','assessment-method')]/part[has-oscal-namespace('http://csrc.nist.gov/ns/oscal') and @name=('objects','assessment-objects')]" min-occurs="1" max-occurs="1" />

Likewise there must be a single prop named objects or assessment-objects in a part named assessment or assessment-method.

RULE 4:

<has-cardinality target="part[has-oscal-namespace('http://csrc.nist.gov/ns/oscal') and @name=('objective','assessment-objective')]/prop[has-oscal-namespace('http://csrc.nist.gov/ns/oscal') and @name='method-id']" min-occurs="1" />

And there must be at least one prop named method-id in a part named objective or assessment-objective as well.

EXAMPLE:
A valid construct would look like this (untested), with @ns values being implicitly assigned to the OSCAL namespace:

<objectives-and-methods control-id="token">
   <description>
      <p>Paragraphs and (block-level) markup contents.</p>
   </description>
   <part name="assessment" id="A1">
      <prop name="method" value="EXAMINE"/>
      <prop name="objects" value="(objects)"/>
      <p>some prose</p>
   </part>
   <part name="objective" id="O1">
      <prop name="method-id" value="A1.EXAMINE"/>
      <p>more prose</p>
   </part>      
</objectives-and-methods>

If anything seems amiss here, it could be either (a) errors in the above, (b) expressions not correctly registering designers' intent, or (c) design problems.

In view of the possibility of (a) could we verify my interpretation against current tooling?

[iMichaela]

@wendellpiez - Thank you for the detailed explanation of the XPath describing the constraints. Please note that objectives and assessment are deprecated and assessment-objective and assessment-method are to be used. This is not important but wanted to note it for other users of the example above.

Regarding RULE: 3 you translate it as referring to a prop, but I see in the Path part being listed. Can you please confirm?

Here is my sample (exerpt):

        <objectives-and-methods control-id="ac-6.1">
            <description>
                <p>docuemnting AC-06(01) assessment objectives and methods</p>
            </description>
            <prop name="marking" value="ac-6.1_om"/>
            <part id="ac-6.1_mtd" name="assessment-method">
                <prop name="method" value="TEST">
                    <remarks>
                        <p>some text</p>
                    </remarks>
                </prop>
                <prop name="assessment-objects" value="this-object"/> 
                <p>some more info</p>               
            </part>
            <part id="ac-6.1_obj" name="assessment-objective">
                <prop name="method-id" value="some-objective-value">
                    <remarks>
                        <p>some obj text</p>
                    </remarks>
                </prop>
                <p>some more obj info</p>
            </part>
        </objectives-and-methods>

It fails the oscal-cli validation with the following errors. PLEASE NOTE, first error indicates a part with name="assessment-objects" is missing. A prop exists:

Validation identified the following in file '/Users/miorga/Desktop/ALL_PROJECTS/OSCAL/_0SCAL Examples/2023 Workshop Examples/ap.xml'.
[ERROR] [/assessment-plan/local-definitions[1]/objectives-and-methods[1]] The cardinality '0' is below the required minimum '1' for items matching the expression 'part[has-oscal-namespace('http://csrc.nist.gov/ns/oscal') and @name=('assessment','assessment-method')]/part[has-oscal-namespace('http://csrc.nist.gov/ns/oscal') and @name=('objects','assessment-objects')]'.
[ERROR] [/assessment-plan/local-definitions[1]/objectives-and-methods[1]/part[1]/prop[1]/@name] Value 'method' doesn't match one of 'alt-identifier, label, marking, or sort-id' at path '/assessment-plan/local-definitions[1]/objectives-and-methods[1]/part[1]/prop[1]/@name'
[ERROR] [/assessment-plan/local-definitions[1]/objectives-and-methods[1]/part[1]/prop[2]/@name] Value 'assessment-objects' doesn't match one of 'alt-identifier, label, marking, or sort-id' at path '/assessment-plan/local-definitions[1]/objectives-and-methods[1]/part[1]/prop[2]/@name'
[ERROR] [/assessment-plan/local-definitions[1]/objectives-and-methods[1]/part[2]/prop[1]/@name] Value 'method-id' doesn't match one of 'alt-identifier, label, marking, or sort-id' at path '/assessment-plan/local-definitions[1]/objectives-and-methods[1]/part[2]/prop[1]/@name'
```

Also important to note, values `method` and `method-id` do not appear to be valid due to rigid constraints we have today in place. 

[iMichaela]

Changing prop to part:

       <objectives-and-methods control-id="ac-6.1">
            <description>
                <p>docuemnting AC-06(01) assessment objectives and methods</p>
            </description>
            <prop name="marking" value="ac-6.1_om"/>
            <part id="ac-6.1_mtd" name="assessment-method">
                <prop name="method" value="TEST">
                    <remarks>
                        <p>some text</p>
                    </remarks>
                </prop>
                <p>some more info</p> 
                <part name="assessment-objects" id="ac-6.1_objects"/>
                <link href="https://www.ifa.gov/assessment_objects"/>              
            </part>
            <part id="ac-6.1_obj" name="assessment-objective">
                <prop name="method-id" value="some-objective-value">
                    <remarks>
                        <p>some obj text</p>
                    </remarks>
                </prop>
                <p>some more obj info</p>
            </part>
        </objectives-and-methods>

reduces the number of errors to only 2, but they are still conflicting and confusing.

Validation identified the following in file '/Users/miorga/Desktop/ALL_PROJECTS/OSCAL/_0SCAL Examples/2023 Workshop Examples/ap.xml'.
[ERROR] [/assessment-plan/local-definitions[1]/objectives-and-methods[1]/part[1]/prop[1]/@name] Value 'method' doesn't match one of 'alt-identifier, label, marking, or sort-id' at path '/assessment-plan/local-definitions[1]/objectives-and-methods[1]/part[1]/prop[1]/@name'
[ERROR] [/assessment-plan/local-definitions[1]/objectives-and-methods[1]/part[2]/prop[1]/@name] Value 'method-id' doesn't match one of 'alt-identifier, label, marking, or sort-id' at path '/assessment-plan/local-definitions[1]/objectives-and-methods[1]/part[2]/prop[1]/@name'

**Should this analysis conclude that OSCAL constraints have a bug revealed by the oscal-cli or that the oscal-cli has a bug which should be reported? Can you please confirm the listed values for the part/@name are documented as valid values. I am not able to find them in the OSCAL-Reference, and I would appreciate your expert support on debugging if they are allowed but oscal-cli is reporting them as not being allowed, If OSCAL allows them - where do we have it in the OSCAL-Reference.

[wendellpiez]

This is interesting. It suggests a different set of errors, probably 'allowed-values' errors on the part/prop.

It could be either of (b) or (c), assuming my reading of the first XPaths is correct. Someone closer to oscal-cli can say which.

I can also look more closely to identify the rules that are being fired here, to help determine if there are logical conflicts between the rules.

(In a perfect world, "reference examples" would already have flushed out those problems, but here we are.)

[wendellpiez]

So here are two allowed-values constraints in oscal_assessment-common_metaschema that appear to bear on the case (plus also another cardinality requirement):

OSCAL/src/metaschema/oscal_assessment-common_metaschema.xml

Lines 1769 to 1777 in 4f02dac

	<allowed-values target=".[@name='objective']/prop[has-oscal-namespace('http://csrc.nist.gov/ns/oscal')]/@name">
	<enum value="method">The assessment method to use. This typically appears on parts with the name "objective".</enum>
	</allowed-values>
	<has-cardinality target=".[@name='objective']/prop[has-oscal-namespace('http://csrc.nist.gov/ns/oscal') and @name='method']" min-occurs="1"/>
	<allowed-values target=".[@name='objective']/prop[has-oscal-namespace('http://csrc.nist.gov/ns/oscal') and @name='method']/@value">
	<enum value="INTERVIEW">The process of holding discussions with individuals or groups of individuals within an organization to once again, facilitate assessor understanding, achieve clarification, or obtain evidence.</enum>
	<enum value="EXAMINE">The process of reviewing, inspecting, observing, studying, or analyzing one or more assessment objects (i.e., specifications, mechanisms, or activities).</enum>
	<enum value="TEST">The process of exercising one or more assessment objects (i.e., activities or mechanisms) under specified conditions to compare actual with expected behavior.</enum>
	</allowed-values>

Note that they cover only part[@name='objective'], not assessment-objective as you have it (and as the rule already cited allows).

I suggest trying objective and meanwhile this is an issue for the OSCAL designers. (I can help support and I am sure oscal-cli team will as well.)

[iMichaela]

Do you think that by deprecating 'objective' and 'assessment' and not updating here, we lost the constraints ?I do not see 'method' and 'method-id' for part/prop. The documentation pipeline is not picking them up.... I'll also try 'objectives' and let you know.

[brian-comply0]

The core OSCAL syntax (under the NIST OSCAL namespace @ns="http://csrc.nist.gov/ns/oscal") only allows a part[@name="assessment-method"] (or part[@name="method"] - depreciated) to have one of the following three properties:

• label
• sort-id
• alt-identifier

However, you'll notice the 800-53r5 catalog, has part[@name="assessment-method"] with prop[@name="method"]. This is only valid because it is specifically marked as an RMF extension to the core OSCAL syntax. (@ns="http://csrc.nist.gov/ns/rmf")

This was apparently changed to be an RMF extension at some point after AJ's example was created. Simply add the RFM namespace to the example above and that error will no longer be flagged.

The core OSCAL metaschema needs to do a better job of only including core OSCAL, and not RMF extensions. Those RMF extensions need to be defined and documented separately, just like the FedRAMP extensions.

As for the four HAS CARDINALITY rules flagged above, here is my analysis:

• RULE 1: either move this to the oscal_control-common_metaschema.xml (inserted between lines 81 and 82) or drop it. If kept, documentation needs to be improved.

• RULE 2: remove this from the AP metaschema. It is incorrect for two reasons:

  • "method" is only valid in the RMF namespace making it an OSCAL extension. Enforcing it in the core OSCAL syntax is inappropriate. This should be treated similar to FedRAMP extensions.

  • As "method" is not valid in the core OSCAL syntax, any reference to it with that namespace is incorrect.

• RULE 3: either move this to the oscal_control-common_metaschema.xml (inserted between lines 81 and 82) or drop it. If kept, documentation needs to be improved. This is the one I believe is implemented poorly as described in OSCAL issue #1896.

• RULE 4: This is invalid and should be removed. "method-id" is not a valid property name in the core OSCAL syntax and does not exist in any OSCAL documentation aside from this constraint.

[iMichaela]

This was apparently changed to be an RMF extension at some point after AJ's example was created. Simply add the RFM namespace to the example above and that error will no longer be flagged.

@brian-comply0 - Thank you for your suggestion. Unfortunately, the RMF namespace is not applicable to an example like the one I created since the example is not RMF specific. It is part of the core OSCAL model, and if those constraints are needed in the OSCAL AP, then we need to properly define them and fix the bug revealed while working on an OSCAL Assessment Plan (AP) example. The bug might appear in other models as well, and further investigation is needed.

prop/@name and part/@name constraint issues are also identified in #1972.

[brian-comply0]

I see what you are saying. In that case, the core OSCAL syntax should allow the "method" property in that context; however, the core OSCAL syntax should enumerate allowed values of "EXAMINE", "INTERVIEW" and "TEST" while setting allow-others to "yes". NIST RMF Team can further define that in the context of RMF no other values are acceptable. Best Regards, Brian J. Ruf, CISSP, CCSP, PMP | Director of Cybersecurity | Easy Dynamics Corp<https://www.easydynamics.com/> T: 703-214-3410 ***@***.***

…

---- Interested in developing OSCAL-enabled tools? Check out https://OSCAL.io<https://oscal.io/>

________________________________ From: Michaela Iorga ***@***.***> Sent: Tuesday, January 9, 2024 12:58 PM To: usnistgov/OSCAL ***@***.***> Cc: Brian Ruf ***@***.***>; Mention ***@***.***> Subject: Re: [usnistgov/OSCAL] Interpreting some XPath (Discussion #1968) This was apparently changed to be an RMF extension at some point after AJ's example was created. Simply add the RFM namespace to the example above and that error will no longer be flagged. @brian-comply0<https://github.com/brian-comply0> - Thank you fro your suggestion. Unfortunately, the RMF namespace is not applicable to an example like the one I created since the example is not RMF specific. It is part of the core OSCAL model, and if those constraints are needed in the OSCAL AP, then we need to properly define them and fix the bug revealed while working on an OSCAL Assessment Plan (AP) example. The bug might appear in other models as well, and further investigation is needed. ***@***.*** and ***@***.*** constraint issues are also identified in #1972<#1972>. — Reply to this email directly, view it on GitHub<#1968 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AZJTI4ZPJRP2YW4LUIUO2PDYNWAK7AVCNFSM6AAAAABAU6IHQ6VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DANZQGE4TI>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[iMichaela]

If others are interested in better understanding the conversation and what to try it, here is the AP example:

<?xml version="1.0" encoding="UTF-8"?>
<assessment-plan 
    uuid="60077e84-e62f-4375-8c6c-b0e0d4560c5f"
    xmlns="http://csrc.nist.gov/ns/oscal/1.0">
    <metadata>
        <title>IFA GoodRead Assessment Plan</title>
        <last-modified>2023-05-18T13:57:28.355446-04:00</last-modified>
        <version>1.0</version>
        <oscal-version>1.0.4</oscal-version>
        <role id="assessor">
            <title>IFA Security Control Assessor</title>
        </role>
        <party uuid="e7730080-71ce-4b20-bec4-84f33136fd58" type="person">
            <name>Amy Assessor</name>
            <member-of-organization>3a675986-b4ff-4030-b178-e953c2e55d64</member-of-organization>
        </party>
        <party uuid="3a675986-b4ff-4030-b178-e953c2e55d64" type="organization">
            <name>Important Federal Agency</name>
            <short-name>IFA</short-name>
            <link href="https://www.ifa.gov" rel="website"/>
        </party>
        <responsible-party role-id="assessor">
            <party-uuid>e7730080-71ce-4b20-bec4-84f33136fd58</party-uuid>
        </responsible-party>
    </metadata>
    <import-ssp href="../3-implementation/ssp.oscal.xml"/>
    <local-definitions>
        <objectives-and-methods control-id="ac-6.1">
            <description>
                <p>docuemnting AC-06(01) assessment objectives and methods</p>
            </description>
            <prop name="marking" value="ac-6.1_om"/>
            <part id="ac-6.1_mtd" name="assessment-method">
                <prop name="method" value="TEST">
                    <remarks>
                        <p>some text</p>
                    </remarks>
                </prop>
                <p>some more info</p> 
                <part name="assessment-objects" id="ac-6.1_objects"/>
                <link href="https://www.ifa.gov/assessment_objects"/>              
            </part>
            <part id="ac-6.1_obj" name="assessment-objective">
                <prop name="method-id" value="some-objective-value">
                    <remarks>
                        <p>some obj text</p>
                    </remarks>
                </prop>
                <p>some more obj info</p>
            </part>
        </objectives-and-methods>
        <activity uuid="52277182-1ba3-4cb6-8d96-b1b97aaf9d6b">
            <title>Examine System Elements for Least Privilege Design and Implementation</title>
            <description>
                <p>The activity and it steps will be performed by the assessor and facilitated by owner, ISSO, and product team for the IFA GoodRead system with necessary information and access about least privilege design and implementation of the system's elements: the application, web framework, server, and cloud account infrastructure.</p>
            </description>
            <prop name="method" value="EXAMINE"/>
            <step uuid="733e3cbf-e398-46b6-9c02-a2cb534c341e">
                <title>Obtain Network Access via VPN to IFA GoodRead Environment</title>
                <description>
                    <p>The assessor will obtain network access with appropriately configured VPN account to see admin frontend to the application for PAO staff, which is only accessible via VPN with an appropriately configured role for PAO staff accounts.</p>
                </description>
            </step>
            <step uuid="4ce7e0b4-d69e-4b80-a700-8600b4d4d933">
                <title>Obtain Credentials and Access to AwesomeCloud Account for IFA GoodRead System</title>
                <description>
                    <p>The assessor will obtain access to the GoodRead Product Team's AwesomeCloud account with their single sign-on credentials to a read-only assessor role.</p>
                </description>
            </step>
            <step uuid="3d0297de-e47b-4360-b9c3-cf5c425f86cd">
                <title>Obtain Applcation Access Provided by Product Team</title>
                <description>
                    <p>The assessor will obtain non-privileged account credentials with the PAO staff role to test this role in the application does not permit excessive administrative operations.</p>
                </description>
            </step>
            <step uuid="64ca1ef6-3ad4-4747-97c6-40890222463f">
                <title>Confirm Load Balancer Blocks Access to Admin Frontend from Internet</title>
                <description>
                    <p>The assessor will confirm that the load balancer for public access does not allow access to Admin Frontend of the application from the Internet.</p>
                </description>
            </step>
            <step uuid="715f0592-166f-44f6-bb66-d99623e035dc">
                <title>Confirm GoodRead's PAO Role Cannot Manage Users</title>
                <description>
                    <p>The assessor will confirm that user's logged into the GoodRead Application with the PAO staff role cannot add, modify, or disable users from the system.</p>
                </description>
            </step>
            <step uuid="4641957b-a0fa-4c61-af1a-d3e9101efe40">
                <title>Confirm Django Admin Panel Not Available</title>
                <description>
                    <p>The assessor will confirm with web-based interface and API methods users with the PAO Staff role cannot access the Django admin panel functions and interactively change application's database records.</p>
                </description>
            </step>
            <related-controls>
                <control-selection>
                    <include-control control-id="ac-6.1"/>
                </control-selection>
            </related-controls>
            <responsible-role role-id="assessor">
                <party-uuid>e7730080-71ce-4b20-bec4-84f33136fd58</party-uuid>
            </responsible-role>
        </activity>
    </local-definitions>
    <reviewed-controls>
        <control-selection>
            <include-control control-id="ac-6.1"/>
        </control-selection>
        <control-objective-selection>
            <include-all/>
        </control-objective-selection>
    </reviewed-controls>
    <assessment-subject type="component">
        <description>
            <p>The assessor for the IFA GoodRead Project, including the application and infrastructure for this information system, are within scope of this assessment.</p>
        </description>
        <include-all/>
    </assessment-subject>
    <task uuid="b3504d22-0e75-4dd7-9247-618661beba4e" type="action">
        <title>Examine Least Privilege Design and Implementation</title>
        <associated-activity activity-uuid="0d243b23-a889-478f-9716-6d4870e56209">
            <subject type="component">
                <include-all/>
            </subject>
        </associated-activity>
        <responsible-role role-id="assessor"/>
        <remarks>
            <p>Per IFA's use of NIST SP-800 53A, the assessor, with the support of the owner, information system security officer, and product team for the IFA GoodRead project, will examine least privilege design and implementation with the following:</p>
            <ul>
                <li>list of security functions (deployed in hardware, software, and firmware) and security-relevant information for which access must be explicitly authorized;</li>
                <li>system configuration settings and associated documentation;</li>
            </ul>
        </remarks>
    </task>
</assessment-plan>

Validation with oscal-cli produces:

Validating 'AP/ap.xml' as XML.
Validation identified the following in file 'AP/ap.xml'.
[ERROR] [/assessment-plan/local-definitions[1]/objectives-and-methods[1]/part[1]/prop[1]/@name] Value 'method' doesn't match one of 'alt-identifier, label, marking, or sort-id' at path '/assessment-plan/local-definitions[1]/objectives-and-methods[1]/part[1]/prop[1]/@name'
[ERROR] [/assessment-plan/local-definitions[1]/objectives-and-methods[1]/part[2]/prop[1]/@name] Value 'method-id' doesn't match one of 'alt-identifier, label, marking, or sort-id' at path '/assessment-plan/local-definitions[1]/objectives-and-methods[1]/part[2]/prop[1]/@name'

NOTE A:
/assessment-plan/local-definitions/objectives-and-methods/part/prop/@name='method' is a constraint defined but not properly implemented (allowed) in core OSCAL.

/part/prop/@name='method' is used in the NIST 800-53 catalog (for the 800-53A) and it is passing the oscal-cli validation because the RMF name space is added. For example:

NOTE B:
/assessment-plan/local-definitions/objectives-and-methods/part/prop/@name='method-id' is another constraint defined but not properly implemented (allowed) in core OSCAL.
@brian-comply0 and I debated if:

1. its cardinality 1 is a must and
2. the the constraint prop/@name='method-id' is needed

[GaryGapinski]

Is there a (concise) report of all OSCAL constraints (and the locus/focus of each)?

[iMichaela]

No, there is no report of the constraints. I think we need to have a way of extracting and validating the constraints from the oscal metaschema, because there are other areas where the constraints have issues. The extraction will have to account for how the oscal metaschema definition files are imported into each other. I am open to any suggestion you might have.

[GaryGapinski]

Is there a concise and comprehensive locus of OSCAL constraint declarations (despite the absence of documentation) which are cast in a domain specific language suitable for automated analysis?
I ask these questions to determine the chances of finding approachable ways to detect errors and absurdities in the declarations with an eye to finding and ultimately preventing such defects, as well as additionally enabling documentation.

[david-waltermire]

I have been thinking about this too for a while now. I plan to work this as a generalized approach in the metaschema java project in the near future. I'll post an issue soon there and link it here.

[iMichaela]

Thank you both @GaryGapinski and @david-waltermire . An approach of extracting and analyzing the constraints for accuracy, consistency and to detect errors like this one discussed here is very much needed. I also discussed this issue with the OSCAL team .

[wendellpiez]

We have been discussing the need for better indexing and access to constraint definitions for some time now.

Additionally, as @GaryGapinski and @david-waltermire also know (part of why this is frustrating to those of use who have been looking at this), some skills using XML-capable query technologies could rapidly mow this lawn, if used carefully.

However, such developers are apparently scarce and those that are available are already oversubscribed - while at the same time even XQuery is useless if you don't know first what to look for, or can't confirm it when you think you've found it.

To make matters worse, as Metaschema rolls out new features for supporting out-of-line constraints definitions, any indexing of constraints definitions (static or dynamic) may have to be realigned.