Troubleshooting

This page covers common problems when running Xen and how to diagnose them.

Tip

If you cannot resolve a problem on your own, the xen-users mailing list is the best place to ask for help. Include as much diagnostic information as possible.

Diagnosing problems

Start with xl list to see which domains are running and their state. The state column uses single letters: r (running), b (blocked/idle), p (paused), s (shutdown), c (crashed).

For an ongoing view instead of a snapshot, xentop behaves like top for Xen — live per-domain CPU, memory, network, and disk stats, updated every few seconds.

Check the Xen log for errors:

xl dmesg

Also check dom0 kernel messages:

dmesg | grep -i xen

Useful log files:

  • /var/log/xen/ — Xen-specific logs

  • /var/log/xen/xen-hotplug.log — device hotplug messages

  • /var/log/xen/qemu-dm-<domname>.log — QEMU device model (HVM guests)

  • /var/log/xen/xl-<domname>.log — xl command logs

Guest console logs

For guest boot issues, enable console logging by setting XENCONSOLED_TRACE=guest in your distribution’s Xen configuration file (/etc/sysconfig/xencommons or /etc/defaults/xencommons). Restart xenconsoled and start a new guest. Console output will appear in /var/log/xen/console/guest-<domname>.

If you suspect that the wrong values are being written to or read from Xenstore, then you may turn on xenstore tracing to help debug the problem:

  1. Set XENSTORED_TRACE=1 into /etc/{sysconfig,default}/xencommons.

  2. Reboot.

  3. xenstored will then write a trace file, giving every read and write on the store. Use xen-bugtool to attach that and all your other logs to a Bugzilla entry.

Host boot problems: serial console

If the host itself fails to boot, or crashes before you can reach a shell to run xl dmesg, a serial console captures the hypervisor and dom0 boot output independently — over a physical serial port, or a virtual one such as IPMI Serial-over-LAN, which most server hardware and cloud providers expose without any extra cabling.

Add serial output to the hypervisor’s boot parameters, alongside the Xen entry in your bootloader config:

com1=115200,8n1 console=com1,vga

dom0’s kernel parameters need console=hvc0 too (see Getting Started for the same option in a guest context) so its messages reach the same serial line as the hypervisor’s.

Connecting to the other end depends entirely on the hardware — a null-modem cable to another machine’s serial port, an IPMI/BMC’s own web console or ipmitool -I lanplus -H <host> -U <user> sol activate, or your cloud provider’s serial/console access feature. See the wiki’s Xen Serial Console page for hardware-specific setup.

Common problems

Guest fails to start

Run xl create -p /etc/xen/myguest.cfg to create the guest in a paused state. If creation fails, the error message will indicate the cause.

Common causes:

  • Wrong path to kernel or disk image

  • Insufficient memory (check xl info for free memory)

  • Missing bridge interface (check brctl show)

If xl info and free both show plenty of memory but Xen still reports an out-of-memory error, the hypervisor’s own free memory is what matters, not dom0’s — dom0 is a guest itself and can be holding onto memory that was never handed back to the hypervisor. Compare the “free memory” line in xl info against the total your host has; if dom0 is consuming most of it, reduce dom0’s memory allocation (or set a dom0_mem cap in the Xen boot options) to leave room for other guests. See Getting Started for how to set this up from the start rather than fixing it after the fact.

Guest crashes or reboots unexpectedly

Check /var/log/xen/ for error messages. If a crash occurred, also look at the host console output and enable noreboot in the Xen boot options to pause on crash.

For a PV or PVH guest kernel crash specifically, xenctx gives you a stack trace without needing anything set up in advance inside the guest. Set on_crash = "preserve" in the guest’s config file so it isn’t torn down automatically, then after it crashes, run in dom0 (once per vCPU):

xenctx -s /path/to/System.map <domid> <vcpu>

The System.map must match the exact kernel build the guest is running, or the symbol names in the trace will be wrong.

Can’t connect to, or nothing shows on, the guest console

Connect to a running guest’s console with xl console myguest, or attach immediately at creation with xl create -c. Detach without stopping the guest with Ctrl-].

If the console won’t connect at all, only one console session per guest is allowed at a time — a stale session can block new ones. Restarting xenconsoled in dom0 usually clears it.

If the console connects but shows nothing at all — not even boot messages — the guest kernel likely isn’t sending output to the console Xen expects. Modern (pvops) kernels use hvc0; add console=hvc0 to the guest’s kernel command line (the extra option in its config file, or its own bootloader config if it uses pygrub) and restart it.

No network connectivity in guest

Verify the bridge is set up in dom0:

brctl show
ip link show xenbr0

Confirm the vif appears in xl network-list <domname> and that the interface is up inside the guest.

Reporting bugs

If you are not sure whether what you discovered is a bug or a configuration issue, ask on the xen-users mailing list first.

Once you have confirmed a reproducible bug, post to the xen-devel mailing list with [BUG] in the subject line. You do not need to be subscribed to post. See the Contributor Handbook for more guidance on how to report a bug effectively, including what information to include and where to send it.

Note

Security vulnerabilities should be reported privately to security@xenproject.org rather than on the public mailing list.