Kernel Crash Trace Log: Difference between revisions
Jump to navigation
Jump to search
| (10 intermediate revisions by 3 users not shown) | |||
| Line 1: | Line 1: | ||
== | == Introduction == | ||
* Sometimes you are not able to get a trace log of crashed kernel because it is not written into any file in a filesystem. This guide is aimed to help with getting a kernel crash log. | * Sometimes you are not able to get a trace log of crashed kernel because it is not written into any file in a filesystem. This guide is aimed to help with getting a kernel crash log. | ||
* The easiest way to archive the goal is to utilize a remote system through network. This example shows the case with two Proxmox VE | * The easiest way to archive the goal is to utilize a remote system through network. This example shows the case with two Proxmox VE 4.X hosts. One of them (server1) is the host to debug and another one (server2) is the host to catch a log. | ||
* Server1 IP: 10.10.10.1 | * Server1 IP: 10.10.10.1 | ||
* Server2 IP: 10.10.10.2 | * Server2 IP: 10.10.10.2 | ||
| Line 8: | Line 8: | ||
== Server1 configuration == | == Server1 configuration == | ||
Insert the following line to GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub | |||
= | netconsole=<port>@<your ip>/eth0,<port>@<remote logging ip>/<mac address of logging pc> loglevel=7 | ||
The hole line locks like this. | |||
GRUB_CMDLINE_LINUX_DEFAULT="quiet netconsole=5555@10.10.10.1/eth0,5555@10.10.10.2/0c:c4:7a:44:1e:fe loglevel=7" | |||
* Check https://www.kernel.org/doc/Documentation/networking/netconsole.txt for more information | |||
Note: it's good idea to disable this configuration after you have done with debugging. | |||
To find out your MAC address use on the logging Server/Laptop | |||
ip a | |||
== Server2 configuration == | == Server2 configuration == | ||
Now you can setup a rsyslog service to log the other server or you get the log with nc direct to a console on a remote device. | |||
=== Set up a rsyslog === | |||
<ol> | <ol> | ||
<li> Create a new file '''/etc/rsyslog.d/01-netconsole-collector.conf''' with the following content: | <li> Create a new file '''/etc/rsyslog.d/01-netconsole-collector.conf''' with the following content: | ||
<pre> | <pre> | ||
# Start UDP server on port | # Start UDP server on port 5555 | ||
$ModLoad imudp | $ModLoad imudp | ||
$UDPServerRun | $UDPServerRun 5555 | ||
# Define templates | # Define templates | ||
| Line 45: | Line 50: | ||
<li> Restart rsyslog | <li> Restart rsyslog | ||
<pre> | <pre> | ||
systemctl restart rsyslog | |||
</pre> | </pre> | ||
</li> | </li> | ||
| Line 51: | Line 56: | ||
* Note: it's good idea to disable this configuration after you have done with debugging because $EscapeControlCharactersOnReceive and $DropTrailingLFOnReception are global options and they change default behaviour of rsyslog. | * Note: it's good idea to disable this configuration after you have done with debugging because $EscapeControlCharactersOnReceive and $DropTrailingLFOnReception are global options and they change default behaviour of rsyslog. | ||
=== Output on a remote conlsole === | |||
nc -l -p 5555 -u | |||
== Examination == | == Examination == | ||
| Line 64: | Line 69: | ||
* Server1 will crash and you should get a crash log in /var/log/netconsole/10.10.10.1.log on server2 | * Server1 will crash and you should get a crash log in /var/log/netconsole/10.10.10.1.log on server2 | ||
[[Category:Troubleshooting]] | [[Category:Troubleshooting]] [[Category:HOWTO]] | ||
Latest revision as of 06:44, 13 February 2019
Introduction
- Sometimes you are not able to get a trace log of crashed kernel because it is not written into any file in a filesystem. This guide is aimed to help with getting a kernel crash log.
- The easiest way to archive the goal is to utilize a remote system through network. This example shows the case with two Proxmox VE 4.X hosts. One of them (server1) is the host to debug and another one (server2) is the host to catch a log.
- Server1 IP: 10.10.10.1
- Server2 IP: 10.10.10.2
Server1 configuration
Insert the following line to GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub
netconsole=<port>@<your ip>/eth0,<port>@<remote logging ip>/<mac address of logging pc> loglevel=7
The hole line locks like this. GRUB_CMDLINE_LINUX_DEFAULT="quiet netconsole=5555@10.10.10.1/eth0,5555@10.10.10.2/0c:c4:7a:44:1e:fe loglevel=7"
- Check https://www.kernel.org/doc/Documentation/networking/netconsole.txt for more information
Note: it's good idea to disable this configuration after you have done with debugging.
To find out your MAC address use on the logging Server/Laptop
ip a
Server2 configuration
Now you can setup a rsyslog service to log the other server or you get the log with nc direct to a console on a remote device.
Set up a rsyslog
- Create a new file /etc/rsyslog.d/01-netconsole-collector.conf with the following content:
# Start UDP server on port 5555 $ModLoad imudp $UDPServerRun 5555 # Define templates $template NetconsoleFile,"/var/log/netconsole/%fromhost-ip%.log" $template NetconsoleFormat,"%rawmsg%" # Accept endline characters (unfortunatelly these options are global) $EscapeControlCharactersOnReceive off $DropTrailingLFOnReception off # Store collected logs using templates without local ones :fromhost-ip, !isequal, "127.0.0.1" ?NetconsoleFile;NetconsoleFormat # Discard logs match the rule above & ~
- Restart rsyslog
systemctl restart rsyslog
- Note: it's good idea to disable this configuration after you have done with debugging because $EscapeControlCharactersOnReceive and $DropTrailingLFOnReception are global options and they change default behaviour of rsyslog.
Output on a remote conlsole
nc -l -p 5555 -u
Examination
- We can check if everything works by causing kernel crash intentionally (be careful!, it's going to be a real crash), type the following command on server1
echo c > /proc/sysrq-trigger
- Server1 will crash and you should get a crash log in /var/log/netconsole/10.10.10.1.log on server2