Protected memory failure.

src/main/java/org/github/krashpad/domain/jdk/FatalErrorLog.java

@@ -941,8 +941,7 @@ && getStackFrameTop().matches("^V  \\[(libjvm\\.so|jvm\\.dll).+\\]  (ModuleEntry
                 && JdkUtil.getJdk8UpdateNumber(getJdkReleaseString()) < 282 && getStackFrameTop() != null
                 && getStackFrameTop().matches("^V.+JfrEventClassTransformer::on_klass_creation.+$")) {
             analysis.add(Analysis.ERROR_JDK8_JFR_CLASS_TRANSFORMED);
-        } else if (getStackFrameTop() != null
-                && !isError("There is insufficient memory for the Java Runtime Environment to continue")
+        } else if (getStackFrameTop() != null && !isMemoryAllocationFail()
                 && !isError("#  fatal error: OutOfMemory encountered: Java heap space")) {
             // Other libjvm.so/jvm.dll analysis
             if (getStackFrameTop().matches("^V  \\[libjvm\\.so.+\\](.+)?$")) {
@@ -2838,7 +2837,7 @@ public long getFailedMemoryAllocation() {
         long memoryAllocation = Long.MIN_VALUE;
         if (!headers.isEmpty()) {
             Iterator<Header> iterator = headers.iterator();
-            String regex = "^.+failed to (allocate|map) (\\d{1,}) bytes.+$";
+            String regex = "^.+failed to (allocate|map|protect) (\\d{1,}) bytes.+$";
             while (iterator.hasNext()) {
                 Header he = iterator.next();
                 if (he.getLogEntry().matches(regex)) {
@@ -6335,7 +6334,7 @@ public boolean isJnaCrash() {
     public boolean isMemoryAllocationFail() {
         boolean isMemoryAllocationFail = false;
         if (isError("There is insufficient memory for the Java Runtime Environment to continue.")
-                || isError("Failed to map memory")) {
+                || isError("Failed to map memory") || isError("failed to protect")) {
             isMemoryAllocationFail = true;
         }
         return isMemoryAllocationFail;

src/main/resources/org/github/krashpad/analysis.properties

@@ -58,24 +58,24 @@ error.oome.external.startup=It appears there is not enough physical memory for t
 error.oome.jboss.version=JBoss version check (e.g. standalone.sh --version) crash due to insufficient memory. The version check launches a full JVM with JBoss heap settings, so it's a very resource-intensive way to check the version string. A more lightweight approach would be to use the JBoss CLI and run "patch info".
 error.oome.jvm=Insufficient physical memory when the JVM heap + metaspace + class pointers + thread stack is >=95% physical memory. Either increase physical memory or slim the JVM process (heap, metaspace, native memory, thread stack size, number of threads, etc.).
 error.oome.jvm.startup=It appears there is not enough physical memory for the JVM to start. Either reduce the JVM memory needs (e.g. decrease the heap size), eliminate other processes outside the JVM competing for memory, and/or increase the available physical memory.
-error.oome.rlimit=The Out of Memory Error appears to be caused by a resource limit (rlimit) being reached. Check if the max user processes (nproc), kernel max number of threads (kernel.pid_max), open file limit (nofile), etc. is being reached. Reference: https://access.redhat.com/solutions/46410.
-error.oome.rlimit.max.map.count=The memory allocation failure appears to be caused by max_map_count being reached.
-error.oome.rlimit.oops=The Out of Memory Error appears to be caused by one of the following limits being reached: (1) The native heap reached the java heap base address. Set -XX:HeapBaseMinAddress=n higher or disable CompressedOops with -XX:-UseCompressedOops. Reference: https://access.redhat.com/solutions/2697961. (2) A resource limit (rlimit) is reached. Check if the max user processes (nproc), kernel max number of threads (kernel.pid_max), open file limit (nofile), etc. is being reached. Reference: https://access.redhat.com/solutions/46410.
-error.oome.rlimit.oops.startup=The Out of Memory Error preventing the JVM from starting appears to be caused by one of the following limits being reached: (1) The native heap reached the java heap base address. Set -XX:HeapBaseMinAddress=n higher or disable CompressedOops with -XX:-UseCompressedOops. Reference: https://access.redhat.com/solutions/2697961. (2) A resource limit (rlimit) is reached. Check if the max user processes (nproc), kernel max number of threads (kernel.pid_max), open file limit (nofile), etc. is being reached. Reference: https://access.redhat.com/solutions/46410.
-error.oome.rlimit.overcommit=The Out of Memory Error appears to be caused by one of the following limits being reached: (1) Unused memory committed to other processes is inaccessible. Check if vm.overcommit_memory=2 is causing the kernel to deny requests for memory. (2) A resource limit (rlimit) is reached. Check if the max user processes (nproc), kernel max number of threads (kernel.pid_max), open file limit (nofile), etc. is being reached.
-error.oome.rlimit.overcommit.startup=The Out of Memory Error preventing the JVM from starting appears to be caused by one of the following limits being reached: (1) Unused memory committed to other processes is inaccessible. Check if vm.overcommit_memory=2 is causing the kernel to deny requests for memory. (2) A resource limit (rlimit) is reached. Check if the max user processes (nproc), kernel max number of threads (kernel.pid_max), open file limit (nofile), etc. is being reached. Reference: https://access.redhat.com/solutions/46410.
-error.oome.rlimit.startup=The Out of Memory Error preventing the JVM from starting appears to be caused by a resource limit (rlimit) being reached. Check if the max user processes (nproc), kernel max number of threads (kernel.pid_max), open file limit (nofile), etc. is being reached. Reference: https://access.redhat.com/solutions/46410.
-error.oome.wlimit=The Out of Memory Error appears to be caused by some Windows limit being reached.
-error.oome.wlimit.page.file=The Out of Memory Error appears to be caused by Windows page file (swap) exhaustion.
-error.oome.wlimit.page.file.startup=The Out of Memory Error preventing the JVM from starting appears to be caused by Windows page file (swap) exhaustion.
-error.oome.wlimit.oops=The Out of Memory Error preventing the JVM from starting appears to be caused by one of the following limits being reached: (1) The native heap reached the java heap base address. Set -XX:HeapBaseMinAddress=n higher or disable CompressedOops with -XX:-UseCompressedOops. Reference: https://access.redhat.com/solutions/2697961. (2) Some Windows limit is reached. Reference: https://access.redhat.com/solutions/46410.
-error.oome.wlimit.startup=The Out of Memory Error preventing the JVM from starting appears to be caused by some Windows limit being reached.
+error.oome.rlimit=The memory allocation error appears to be caused by a resource limit (rlimit) being reached. Check if the max user processes (nproc), kernel max number of threads (kernel.pid_max), open file limit (nofile), etc. is being reached. Reference: https://access.redhat.com/solutions/46410.
+error.oome.rlimit.max.map.count=The memory allocation error appears to be caused by max_map_count being reached.
+error.oome.rlimit.oops=The memory allocation error appears to be caused by one of the following limits being reached: (1) The native heap reached the java heap base address. Set -XX:HeapBaseMinAddress=n higher or disable CompressedOops with -XX:-UseCompressedOops. Reference: https://access.redhat.com/solutions/2697961. (2) A resource limit (rlimit) is reached. Check if the max user processes (nproc), kernel max number of threads (kernel.pid_max), open file limit (nofile), etc. is being reached. Reference: https://access.redhat.com/solutions/46410.
+error.oome.rlimit.oops.startup=The memory allocation error preventing the JVM from starting appears to be caused by one of the following limits being reached: (1) The native heap reached the java heap base address. Set -XX:HeapBaseMinAddress=n higher or disable CompressedOops with -XX:-UseCompressedOops. Reference: https://access.redhat.com/solutions/2697961. (2) A resource limit (rlimit) is reached. Check if the max user processes (nproc), kernel max number of threads (kernel.pid_max), open file limit (nofile), etc. is being reached. Reference: https://access.redhat.com/solutions/46410.
+error.oome.rlimit.overcommit=The memory allocation error appears to be caused by one of the following limits being reached: (1) Unused memory committed to other processes is inaccessible. Check if vm.overcommit_memory=2 is causing the kernel to deny requests for memory. (2) A resource limit (rlimit) is reached. Check if the max user processes (nproc), kernel max number of threads (kernel.pid_max), open file limit (nofile), etc. is being reached.
+error.oome.rlimit.overcommit.startup=The memory allocation error preventing the JVM from starting appears to be caused by one of the following limits being reached: (1) Unused memory committed to other processes is inaccessible. Check if vm.overcommit_memory=2 is causing the kernel to deny requests for memory. (2) A resource limit (rlimit) is reached. Check if the max user processes (nproc), kernel max number of threads (kernel.pid_max), open file limit (nofile), etc. is being reached. Reference: https://access.redhat.com/solutions/46410.
+error.oome.rlimit.startup=The memory allocation error preventing the JVM from starting appears to be caused by a resource limit (rlimit) being reached. Check if the max user processes (nproc), kernel max number of threads (kernel.pid_max), open file limit (nofile), etc. is being reached. Reference: https://access.redhat.com/solutions/46410.
+error.oome.wlimit=The memory allocation error appears to be caused by some Windows limit being reached.
+error.oome.wlimit.page.file=The memory allocation error appears to be caused by Windows page file (swap) exhaustion.
+error.oome.wlimit.page.file.startup=The memory allocation error preventing the JVM from starting appears to be caused by Windows page file (swap) exhaustion.
+error.oome.wlimit.oops=The memory allocation error preventing the JVM from starting appears to be caused by one of the following limits being reached: (1) The native heap reached the java heap base address. Set -XX:HeapBaseMinAddress=n higher or disable CompressedOops with -XX:-UseCompressedOops. Reference: https://access.redhat.com/solutions/2697961. (2) Some Windows limit is reached. Reference: https://access.redhat.com/solutions/46410.
+error.oome.wlimit.startup=The memory allocation error preventing the JVM from starting appears to be caused by some Windows limit being reached.
 error.oome.native.or.external=Insufficient physical memory when the JVM heap + metaspace + class pointers + thread stack is <95% physical memory. Determine which of the following is consuming the unaccounted memory: (1) JVM native memory, (2) External process(es), (3) Hypervisor memory ballooning. Possible resolutions: (1) Increase physical memory. (2) Slim JVM (heap, metaspace, native memory, thread stack size, number of threads, etc.). (3) Slim external processes. (4) Do hypervisor tuning to ensure memory swapping does not happen (e.g. reserve virtual machine memory).
 error.oome.oops=Insufficient physical memory due to one of the following reasons: (1) The JVM or external processes are consuming significant amounts of memory. (2) The native heap reached the java heap base address. Set -XX:HeapBaseMinAddress=n higher or disable CompressedOops with -XX:-UseCompressedOops. Reference: https://access.redhat.com/solutions/2697961.
-error.oome.overcommit.rlimit=The Out of Memory Error appears to be due to vm.overcommit_memory=2 causing the kernel to deny the requested allocation because it is larger than CommitLimit - Committed_AS - user_reserve_kbytes - admin_reserve_kbytes. Verify vm.overcommit_memory=2. If it does not, the crash is due to a resource limit (rlimit) being reached. Check if the max user processes (nproc), kernel max number of threads (kernel.pid_max), open file limit (nofile), etc. is being reached.
-error.oome.overcommit.rlimit.startup=The Out of Memory Error preventing the JVM from starting appears to be due to vm.overcommit_memory=2 causing the kernel to deny the requested allocation because it is larger than CommitLimit - Committed_AS - user_reserve_kbytes - admin_reserve_kbytes. Verify vm.overcommit_memory=2. If it does not, the crash is due to a resource limit (rlimit) being reached. Check if the max user processes (nproc), kernel max number of threads (kernel.pid_max), open file limit (nofile), etc. is being reached.
-error.oome.thread.leak=The Out of Memory Error appears to be caused by a thread leak causing a limit to be reached (e.g. the max user processes [nproc] resource limit [rlimit]).
-error.oome.thread.leak.eap.executor.pool=The Out of Memory Error appears to be caused by a Wildfly or JBoss EAP executor pool leak causing a limit to be reached (e.g. the max user processes [nproc] resource limit [rlimit]). Reference: https://access.redhat.com/solutions/68039.
+error.oome.overcommit.rlimit=The memory allocation error appears to be due to vm.overcommit_memory=2 causing the kernel to deny the requested allocation because it is larger than CommitLimit - Committed_AS - user_reserve_kbytes - admin_reserve_kbytes. Verify vm.overcommit_memory=2. If it does not, the crash is due to a resource limit (rlimit) being reached. Check if the max user processes (nproc), kernel max number of threads (kernel.pid_max), open file limit (nofile), etc. is being reached.
+error.oome.overcommit.rlimit.startup=The memory allocation error preventing the JVM from starting appears to be due to vm.overcommit_memory=2 causing the kernel to deny the requested allocation because it is larger than CommitLimit - Committed_AS - user_reserve_kbytes - admin_reserve_kbytes. Verify vm.overcommit_memory=2. If it does not, the crash is due to a resource limit (rlimit) being reached. Check if the max user processes (nproc), kernel max number of threads (kernel.pid_max), open file limit (nofile), etc. is being reached.
+error.oome.thread.leak=The memory allocation error appears to be caused by a thread leak causing a limit to be reached.
+error.oome.thread.leak.eap.executor.pool=The memory allocation error appears to be caused by as Wildfly or JBoss EAP executor pool leak causing a limit to be reached. Reference: https://access.redhat.com/solutions/68039.
 error.oome.thrown.java.heap=OutOfMemoryError other than "Metaspace" or "Compressed class space" is caught and thrown. OutOfMemoryError should not be caught, as it leaves the JVM in an undetermined state. Check the relevant JDK or application code to determine the underlying cause.
 error.oome.thrown.metaspace="OutOfMemoryError: Metaspace" is caught and thrown. OutOfMemoryError should not be caught, as it leaves the JVM in an undetermined state. Either there is unintended metaspace retention, or the metaspace is too small. Reference: https://access.redhat.com/solutions/2038983.
 error.oome.thrown.comp.class.space="OutOfMemoryError: Compressed class space" is caught and thrown. OutOfMemoryError should not be caught, as it leaves the JVM in an undetermined state. The compressed class space is too small. Reference: https://access.redhat.com/solutions/2132101.

src/test/java/org/github/krashpad/domain/jdk/TestFatalErrorLog.java

@@ -547,6 +547,40 @@ void testFailedToMapMemory() {
                 Analysis.ERROR_OOME_RLIMIT_MAX_MAP_COUNT + " analysis not identified.");
     }
 
+    @Test
+    void testFailedToProtectMemory() {
+        FatalErrorLog fel = new FatalErrorLog();
+        String header1 = "# Native memory allocation (mprotect) failed to protect 16384 bytes for memory to guard "
+                + "stack page";
+        Header headerEvent1 = new Header(header1);
+        fel.getHeaders().add(headerEvent1);
+        String header2 = "#  Error: memory to guard stack pages";
+        Header headerEvent2 = new Header(header2);
+        fel.getHeaders().add(headerEvent2);
+        String os = "OS:Red Hat Enterprise Linux release 9.4 (Plow)";
+        OsInfo osEvent = new OsInfo(os);
+        fel.getOsInfos().add(osEvent);
+        Memory memory = new Memory("Memory: 4k page, physical 32604624k(246236k free), swap 9768956k(8337916k free)");
+        fel.getMemories().add(memory);
+        String maxMapCount1 = "/proc/sys/vm/max_map_count (maximum number of memory map areas a process may have)";
+        MaxMapCount maxMapCountEvent1 = new MaxMapCount(maxMapCount1);
+        fel.getMaxMapCounts().add(maxMapCountEvent1);
+        String maxMapCount2 = "65530";
+        MaxMapCount maxMapCountEvent2 = new MaxMapCount(maxMapCount2);
+        fel.getMaxMapCounts().add(maxMapCountEvent2);
+        fel.setDynamicLibrariesMappingCount(65532);
+        assertEquals(32604624L * 1024, fel.getMemoryTotal(), "Physical memory not correct.");
+        assertEquals(246236L * 1024, fel.getMemoryFree(), "Physical memory free not correct.");
+        assertEquals(9768956L * 1024, fel.getSwapTotal(), "Swap not correct.");
+        assertEquals(8337916L * 1024, fel.getSwapFree(), "Swap free not correct.");
+        assertFalse(fel.isCrashOnStartup(), "Crash on startup incorrectly identified.");
+        fel.doAnalysis();
+        assertEquals(header1 + Constants.LINE_SEPARATOR + header2, fel.getError(), "Error not identified.");
+        assertTrue(fel.isMemoryAllocationFail(), "Memory allocation failure not identified.");
+        assertTrue(fel.hasAnalysis(Analysis.ERROR_OOME_RLIMIT.getKey()),
+                Analysis.ERROR_OOME_RLIMIT + " analysis not identified.");
+    }
+
     @Test
     void testG1() {
         File testFile = new File(Constants.TEST_DATA_DIR + "dataset32.txt");

src/test/java/org/github/krashpad/domain/jdk/TestHeader.java

@@ -74,7 +74,7 @@ void testIdentity() {
     }
 
     @Test
-    void testInsufficientMemoroy() {
+    void testInsufficientMemory() {
         String logLine = "# There is insufficient memory for the Java Runtime Environment to continue.";
         assertTrue(JdkUtil.identifyEventType(logLine, null) == JdkUtil.LogEventType.HEADER,
                 JdkUtil.LogEventType.HEADER.toString() + " not identified.");
@@ -111,7 +111,7 @@ void testJreVersion() {
     }
 
     @Test
-    void testNativeMemoryAllocationFailed() {
+    void testNativeMemoryAllocationFailedToMap() {
         String logLine = "# Native memory allocation (mmap) failed to map 754974720 bytes for committing reserved "
                 + "memory.";
         assertTrue(JdkUtil.identifyEventType(logLine, null) == JdkUtil.LogEventType.HEADER,
@@ -120,6 +120,16 @@ void testNativeMemoryAllocationFailed() {
         assertTrue(headerEvent.isFailed(), "Failed not identified.");
     }
 
+    @Test
+    void testNativeMemoryAllocationFailedToProtect() {
+        String logLine = "# Native memory allocation (mprotect) failed to protect 16384 bytes for memory to guard "
+                + "stack pages";
+        assertTrue(JdkUtil.identifyEventType(logLine, null) == JdkUtil.LogEventType.HEADER,
+                JdkUtil.LogEventType.HEADER.toString() + " not identified.");
+        Header headerEvent = new Header(logLine);
+        assertTrue(headerEvent.isFailed(), "Failed not identified.");
+    }
+
     @Test
     void testNoCoreDumpWillBeWritten() {
         String logLine = "# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "