Automatus struggles with profile-specific test scenarios

[jan-cerny]

Description of problem:

During his work on #10348, @maage has discovered that with some rules there are unexpected errors when Automatus runs in rule mode. This happens when testing rule sssd_ldap_configure_tls_ca_dir on a RHEL 8 virtual machine.

The trigger seems to be that the test scenarios for this rule have profile limitation in their headers but the rule isn't part of the given profile. In rule sssd_ldap_configure_tls_ca_dir all the test scenarios have # profiles = xccdf_org.ssgproject.content_profile_stig but the rule isn't part of RHEL 8 STIG profile. It is only part of RHEL 7 STIG profile and Oracle Linux 7 profile. But still the rule is present in RHEL 8 data stream.

As a consequence of the rule not being present in the profile, for the .fail.sh test scenarios, Automatus hits the issue in OpenSCAP OpenSCAP/openscap#1963 that means that it generates an empty Ansible Playbook and then Automatus tries to apply this empty Ansible Playbook which doesn't remediate anything and then the final scan isn't fine.

The main question is the expected behavior for the profile-specific scenarios in a situation when the given rule isn't part of the profile but exists in the given SCAP source data stream.

SCAP Security Guide Version:

current upstream as of 2022-03-27 as of HEAD 2c18502

Operating System Version:

F 37

Steps to Reproduce:

1. python3 tests/automatus.py rule --libvirt qemu:///system ssgts_rhel8 --remediate-using ansible --dontclean sssd_ldap_configure_tls_ca_dir

Actual Results:

INFO - The base image option has not been specified, choosing libvirt-based test environment.
INFO - Logging into /home/jcerny/work/git/scap-security-guide/logs/rule-custom-2023-03-27-1320/test_suite.log
INFO - xccdf_org.ssgproject.content_rule_sssd_ldap_configure_tls_ca_dir
INFO - Script domain_not_there.fail.sh using profile xccdf_org.ssgproject.content_profile_stig OK
ERROR - Rule evaluation resulted in fail, instead of expected pass during final stage 
ERROR - The check after remediation failed for rule 'xccdf_org.ssgproject.content_rule_sssd_ldap_configure_tls_ca_dir'.
INFO - Script ldap_tls_cacertdir.pass.sh using profile xccdf_org.ssgproject.content_profile_stig OK
INFO - Script ldap_tls_cacertdir_bad_value.fail.sh using profile xccdf_org.ssgproject.content_profile_stig OK
ERROR - Rule evaluation resulted in fail, instead of expected pass during final stage 
ERROR - The check after remediation failed for rule 'xccdf_org.ssgproject.content_rule_sssd_ldap_configure_tls_ca_dir'.
INFO - Script ldap_tls_cacertdir_not_absolute_path.fail.sh using profile xccdf_org.ssgproject.content_profile_stig OK
ERROR - Rule evaluation resulted in fail, instead of expected pass during final stage 
ERROR - The check after remediation failed for rule 'xccdf_org.ssgproject.content_rule_sssd_ldap_configure_tls_ca_dir'.
INFO - Script ldap_tls_cacertdir_not_there.fail.sh using profile xccdf_org.ssgproject.content_profile_stig OK
ERROR - Rule evaluation resulted in fail, instead of expected pass during final stage 
ERROR - The check after remediation failed for rule 'xccdf_org.ssgproject.content_rule_sssd_ldap_configure_tls_ca_dir'.

Expected Results:

Probably no errors. But the expected results is a question that needs to be answered.

Additional Information/Debugging Steps:

• notice that the Playbook logs/rule-custom-2023-03-27-1320/xccdf_org.ssgproject.content_rule_sssd_ldap_configure_tls_ca_dir.yml doesn't contain any tasks
• notice how the # profile header is handled by Automatus
• notice that Automatus evaluates TS even the rule isn't a part of the STIG profile