Troubleshooting a virtual machine that has stopped responding

on

      Troubleshooting a virtual machine that has stopped responding

                You see the error:
                Unable to connect to the MKS: Error connecting to /bin/vmx process.
  • Virtual machine screen is black and does not refresh.            
                 You may see one or more of the following errors:
                 /init: /init: 151: Syntax error: 0xforce=panic
                 Kernel panic - not syncing: Attempted to kill init!
                 PAGE_FAULT_IN_NONPAGED_AREA
                 Error codes 6005, 6008 in Windows Event Viewer.
 Purpose

This article provides steps to isolate possible causes of a vSphere virtual machine becoming unresponsive.

An unresponsive virtual machine does not respond to any connection attempts and may be unable to respond to any attempts to power cycle it. There are a variety of reasons a virtual machine can end up in an unresponsive state. This article enables you to identify and resolve these common causes, and, when resolved, return the virtual machine to an operational state.

It is possible to hard power off a virtual machine without troubleshooting the cause, but this will prevent collection and analysis of information which could assist with determining the root cause of the outage. For more information about shutting down the virtual machine, see Powering off a virtual machine on an ESXi host (1014165) and Powering off an unresponsive virtual machine on an ESX host (1004340).

This article assumes that the issue is currently occurring. If troubleshooting an issue that occurred in the past, some required information may be unavailable.

 Cause
Causes:
  • Virtual Machine backups and schedules may create heavy I/O load related tasks, may lead to unresponsive issues.
  • Virtual machine may become unresponsive when the hard disk runs out of space.
  • Guest Operating system on the Virtual machine may cause VM to appear unresponsive.  
  • The virtual machine disk  controllers not set as per best practices.
  • If your network contains a firewall between the ESXi/ESX host and the vSphere Client.
 Resolution

The services a virtual machine provides may become unresponsive or unreachable due to a number of causes, including problems with the applications or guest OS within the virtual machine, problems with the virtual machine monitor or virtual devices, resource contention on the host, or issues with underlying storage or networking infrastructure.

If the guest OS is producing any activity, it is successfully running. In this case, unresponsiveness is likely due to a connectivity problem or resource contention, or is specific to a higher-level component such as an application or service running within the guest OS.

Validate the scope

It is important to have accurate symptoms and an understanding of the scope of a problem. To confirm the scope of the problem, work through these checks:

  1. Confirm that the virtual machine is actually unresponsive. It is possible that the virtual machine is not responding via one interface, but is functioning correctly on others. For more information on testing whether a virtual machine is genuinely unresponsive, see Confirming whether virtual machine is unresponsive (1007802).

    If a virtual machine is responsive, but performing poorly, see Troubleshooting ESX virtual machine performance issues (2001003).
     
  2. Verify that the virtual machine is powered on. If the virtual machine has been powered off unexpectedly, power it back on and then troubleshoot the cause of the unexpected shutdown. For more information, see:
     
    Note: If a virtual machine is powered off and cannot be powered back on, see Troubleshooting a virtual machine that is unable to power on (2001005).
     
  3. Determine whether this issue is affecting multiple virtual machines or just one. If multiple virtual machines are affected, consider the similarities between the affected virtual machines when attempting to narrow the potential scope. In particular, focus on shared infrastructure which the group of affected virtual machines depend on, and whether all virtual machines depending on that common infrastructure are affected. For more information, see Assessing commonalities of an outage affecting multiple virtual machines (1019000).
     
  4. Determine whether the guest OS is responsive to interaction at the virtual machine console. If an issue has been isolated to the guest OS or applications within the virtual machine, and the guest OS is responsive at the console, interact with the guest OS at the console to address the problem. For more information, see Troubleshooting virtual machine network connection issues (1003893).
     
  5. Determine whether the guest OS or its application services are responsive to interaction via the network. If the guest OS or services respond to network communication but the console is unresponsive or non-functional, see Troubleshooting virtual machine console and MKS issues in the vSphere Client (749640) or Ensuring that a virtual machine is not inaccessible due to a VMware vCenter or VirtualCenter issue (1007808).
     
  6. Determine whether the guest OS has reported any critical errors to the console, and is sitting in a halted state. For more information, see Identifying critical Guest OS failures within virtual machines (1003999).
     
  7. Determine whether the ESX/ESXi host is unresponsive too. If the host is unresponsive as well, the scope is larger than initially assumed. For more information, see Determining why an ESX/ESXi host does not respond to user interaction at the console (1017135).

Identify the cause

At this point, you have established that one or more virtual machines are unresponsive at both the virtual console and via the network. The host itself is responsive. A problem may exist with resource accessibility or contention, or with underlying storage or networking infrastructure.

To identify the cause:

  1. Determine whether the problem is triggered by an operation or task being performed on the virtual machine. For example, snapshot and vMotion operations both stun a virtual machine for brief periods of time while memory state is copied across the network or to disk. For more information, see Taking a snapshot with virtual machine memory stuns the virtual machine while the memory is written to disk (1013163).
     
  2. Some common configuration errors can lead to a virtual machine becoming unresponsive, such as while waiting for a resource. Review the virtual machine and host configuration. For more information, see:
     
  3. Virtual machines depend on functional backing infrastructure. If there is an issue with the backing storage or networking infrastructure which the virtual machine depends on, the virtual hardware which a virtual machine presents to the guest OS may be impacted. Address the underlying storage or networking issue. For more information, see:
     
  4. Virtual machines depend on available host resources (CPU, Memory), and the guest OS consumes those resources. A problem with resource availability or scheduling inside or outside the virtual machine may cause it to become unresponsive. The virtual machine may also be blocking on unavailable resources or spinning at 100% vCPU utilization. For more information, see Troubleshooting a virtual machine that has stopped responding: VMM and Guest CPU usage comparison (1017926).

Action Plan

At this point, you have established that the host running the virtual machine(s) is both responsive and not encountering any shared storage or networking infrastructure issues. The guest OS has not failed with a critical error, but remains unresponsive at the virtual machine console and via the network.

Take action to recover or collect information about the unresponsive virtual machine based on the architectural layer which is suspect:

Cheers 😉😉😉😉😉
Happy Learning 

Comments

Post a Comment