VMware ESX/ESXi and Syslog
We all know that log files provides wealth of information when there is a need to troubleshoot an issue or while investigating a security incident to give you some examples. There is already lot written about VMware ESXi hosts and vCenter log files. However in this post, I am going to explore this topic from a very basic point of view. So let’s get started.
Q1. What are the different log files that exists on ESXi hosts? And where are they stored?
A1. By default, the common log files are stored as specified below:
- The VMkernel, vmkwarning, and hostd logs are located at /var/log/messages.
- The Host Management service (hostd = Host daemon) log is located at /var/log/vmware/hostd.log.
- The VirtualCenter Agent log is located at /var/log/vmware/vpx/vpxa.log.
- The System boot log is located at /var/log/sysboot.log.
Please read this article for more details.
Q2. Why do we need to send the VMware ESX/ESXi log files to another server such as a Syslog server?
A2. By default the VMware ESXi server stores log files in a scratch partition on the local disk. The scratch partition is non-persistent by default, meaning the contents it holds will be lost when the host server is rebooted or powered-off.
Q3. Is there a way to make scratch partition persistent?
A3. Yes, navigate to this article for more details.
Q4. When is the scratch partition created?
A4. During installation of ESX/ESXi, upon first reboot.
Q.5 What happens if there is no sufficient space on local disk to create a scratch partition?
A5. A portion of RAM will be used as scratch partition. This portion is called RAM disk.
Q6. What does the scratch partition contain?
A6. It contains log files, diagnostic info and system swap.
Q7. So how to send the log files to remote Syslog server?
A7. This is a two step process, usually.
- Make sure a Syslog server exists in your network. Few examples of Syslog server are VMware vMA vilogger, splunk, etc.
- Configure your ESXi hosts to send the log files to the Syslog server.
Note: The procedure on how to configure the above two components depends on the software you are using. Below, are few references:
ESXi 4.1 remote log collection with VMware vMA 4.1
Using vMA as a syslog collector
Additionally, the recently previewed vSphere 5 includes a syslogger component that appears to be very easy to setup. Jason Boche has blogged about this new feature on his blog which is here. As you have read above there are various options to record and maintained vSphere log files that will be handy while troubleshooting your vSphere environment.
