CPU/Kernel/MB/RAID problem?
Jan 5 12:45:05 testbox kernel: [653298.890004] BUG: soft lockup - CPU#0 stuck for 61s! [hal-acl-tool:4168]
Jan 5 12:45:05 testbox kernel: [653298.890005] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:45:05 testbox kernel: [653298.891687]
Jan 5 12:45:05 testbox kernel: [653298.891687] Pid: 4168, comm: hal-acl-tool Tainted: G D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:45:05 testbox kernel: [653298.891687] EIP: 0060:[
Jan 5 12:45:05 testbox kernel: [653298.891687] EIP is at __ticket_spin_lock+0x15/0x20
Jan 5 12:45:05 testbox kernel: [653298.891687] EAX: f5d962b8 EBX: d6ba3b00 ECX: f5d96200 EDX: 00004f4d
Jan 5 12:45:05 testbox kernel: [653298.891687] ESI: d20d3000 EDI: f5d962b8 EBP: e0491c60 ESP: e0491c60
Jan 5 12:45:05 testbox kernel: [653298.891687] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:45:05 testbox kernel: [653298.891687] CR0: 8005003b CR2: 09202568 CR3: 2a445000 CR4: 000006b0
Jan 5 12:45:05 testbox kernel: [653298.891687] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:45:05 testbox kernel: [653298.891687] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:45:05 testbox kernel: [653298.891687] Call Trace:
Jan 5 12:45:05 testbox kernel: [653298.891687] [
Jan 5 12:45:05 testbox kernel: [653298.891687] [
Jan 5 12:45:05 testbox kernel: [653298.891687] [
Jan 5 12:45:05 testbox kernel: [653298.891687] [
Jan 5 12:45:05 testbox kernel: [653298.891687] [
Jan 5 12:45:05 testbox kernel: [653298.891687] [
Jan 5 12:45:05 testbox kernel: [653298.891687] [
Jan 5 12:45:05 testbox kernel: [653298.891687] [
Jan 5 12:45:05 testbox kernel: [653298.891687] [
Jan 5 12:45:05 testbox kernel: [653298.891687] [
Jan 5 12:45:05 testbox kernel: [653298.891687] [
Jan 5 12:45:05 testbox kernel: [653298.891687] [
Jan 5 12:45:05 testbox kernel: [653298.891687] [
Jan 5 12:45:05 testbox kernel: [653298.891687] [
Jan 5 12:45:05 testbox kernel: [653298.891687] [
Jan 5 12:45:05 testbox kernel: [653298.891687] [
Jan 5 12:45:05 testbox kernel: [653298.891687] [
Jan 5 12:45:05 testbox kernel: [653298.891687] [
Jan 5 12:45:05 testbox kernel: [653298.891687] [
Jan 5 12:45:05 testbox kernel: [653298.891687] [
Jan 5 12:45:05 testbox kernel: [653298.891687] [
Jan 5 12:45:05 testbox kernel: [653298.891687] [
Jan 5 12:45:05 testbox kernel: [653298.891687] [
Jan 5 12:45:05 testbox kernel: [653298.891687] [
Jan 5 12:45:05 testbox kernel: [653298.891687] [
Jan 5 12:45:05 testbox kernel: [653298.891687] [
Jan 5 12:45:05 testbox kernel: [653298.891687] [
Jan 5 12:45:05 testbox kernel: [653298.891687] [
Jan 5 12:45:05 testbox kernel: [653298.891687] [
Jan 5 12:45:37 testbox kernel: [653330.032503] BUG: soft lockup - CPU#3 stuck for 61s! [rsync:30532]
Jan 5 12:45:37 testbox kernel: [653330.032503] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:45:37 testbox kernel: [653330.032503]
Jan 5 12:45:37 testbox kernel: [653330.032503] Pid: 30532, comm: rsync Tainted: G D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:45:37 testbox kernel: [653330.032503] EIP: 0060:[
Jan 5 12:45:37 testbox kernel: [653330.032503] EIP is at __ticket_spin_lock+0x13/0x20
Jan 5 12:45:37 testbox kernel: [653330.032503] EAX: f5d962b8 EBX: e0c8de00 ECX: 00000000 EDX: 00004e4d
Jan 5 12:45:37 testbox kernel: [653330.032503] ESI: f5d96200 EDI: f6d00450 EBP: db0f1cc8 ESP: db0f1cc8
Jan 5 12:45:37 testbox kernel: [653330.032503] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:45:37 testbox kernel: [653330.032503] CR0: 8005003b CR2: 00000000 CR3: 1b0d9000 CR4: 000006b0
Jan 5 12:45:37 testbox kernel: [653330.032503] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:45:37 testbox kernel: [653330.032503] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:45:37 testbox kernel: [653330.032503] Call Trace:
Jan 5 12:45:37 testbox kernel: [653330.032503] [
Jan 5 12:45:37 testbox kernel: [653330.032503] [
Jan 5 12:45:37 testbox kernel: [653330.032503] [
Jan 5 12:45:37 testbox kernel: [653330.032503] [
Jan 5 12:45:37 testbox kernel: [653330.032503] [
Jan 5 12:45:37 testbox kernel: [653330.032503] [
Jan 5 12:45:37 testbox kernel: [653330.032503] [
Jan 5 12:45:37 testbox kernel: [653330.032503] [
Jan 5 12:45:37 testbox kernel: [653330.032503] [
Jan 5 12:45:37 testbox kernel: [653330.032503] [
Jan 5 12:45:37 testbox kernel: [653330.032503] [
Jan 5 12:45:37 testbox kernel: [653330.032503] [
Jan 5 12:45:37 testbox kernel: [653330.032503] [
Jan 5 12:45:37 testbox kernel: [653330.032503] [
Jan 5 12:45:37 testbox kernel: [653330.032503] [
Jan 5 12:45:37 testbox kernel: [653330.032503] [
Jan 5 12:45:37 testbox kernel: [653330.032503] [
Jan 5 12:45:37 testbox kernel: [653330.032503] [
Jan 5 12:45:37 testbox kernel: [653330.032503] [
Jan 5 12:45:37 testbox kernel: [653330.032503] [
Jan 5 12:45:37 testbox kernel: [653330.032503] [
Jan 5 12:45:37 testbox kernel: [653330.032503] [
Jan 5 12:45:37 testbox kernel: [653330.032503] [
Jan 5 12:45:37 testbox kernel: [653330.035919] hpet1: lost 3 rtc interrupts
Jan 5 12:46:11 testbox kernel: [653364.390003] BUG: soft lockup - CPU#0 stuck for 61s! [hal-acl-tool:4168]
Jan 5 12:46:11 testbox kernel: [653364.390005] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:46:11 testbox kernel: [653364.391694]
Jan 5 12:46:11 testbox kernel: [653364.391694] Pid: 4168, comm: hal-acl-tool Tainted: G D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:46:11 testbox kernel: [653364.391694] EIP: 0060:[
Jan 5 12:46:11 testbox kernel: [653364.391694] EIP is at __ticket_spin_lock+0x13/0x20
Jan 5 12:46:11 testbox kernel: [653364.391694] EAX: f5d962b8 EBX: d6ba3b00 ECX: f5d96200 EDX: 00004f4d
Jan 5 12:46:11 testbox kernel: [653364.391694] ESI: d20d3000 EDI: f5d962b8 EBP: e0491c60 ESP: e0491c60
Jan 5 12:46:11 testbox kernel: [653364.391694] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:46:11 testbox kernel: [653364.391694] CR0: 8005003b CR2: 09202568 CR3: 2a445000 CR4: 000006b0
Jan 5 12:46:11 testbox kernel: [653364.391694] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:46:11 testbox kernel: [653364.391694] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:46:11 testbox kernel: [653364.391694] Call Trace:
Jan 5 12:46:11 testbox kernel: [653364.391694] [
Jan 5 12:46:11 testbox kernel: [653364.391694] [
Jan 5 12:46:11 testbox kernel: [653364.391694] [
Jan 5 12:46:11 testbox kernel: [653364.391694] [
Jan 5 12:46:11 testbox kernel: [653364.391694] [
Jan 5 12:46:11 testbox kernel: [653364.391694] [
Jan 5 12:46:11 testbox kernel: [653364.391694] [
Jan 5 12:46:11 testbox kernel: [653364.391694] [
Jan 5 12:46:11 testbox kernel: [653364.391694] [
Jan 5 12:46:11 testbox kernel: [653364.391694] [
Jan 5 12:46:11 testbox kernel: [653364.391694] [
Jan 5 12:46:11 testbox kernel: [653364.391694] [
Jan 5 12:46:11 testbox kernel: [653364.391694] [
Jan 5 12:46:11 testbox kernel: [653364.391694] [
Jan 5 12:46:11 testbox kernel: [653364.391694] [
Jan 5 12:46:11 testbox kernel: [653364.391694] [
Jan 5 12:46:11 testbox kernel: [653364.391694] [
Jan 5 12:46:11 testbox kernel: [653364.391694] [
Jan 5 12:46:11 testbox kernel: [653364.391694] [
Jan 5 12:46:11 testbox kernel: [653364.391694] [
Jan 5 12:46:11 testbox kernel: [653364.391694] [
Jan 5 12:46:11 testbox kernel: [653364.391694] [
Jan 5 12:46:11 testbox kernel: [653364.391694] [
Jan 5 12:46:11 testbox kernel: [653364.391694] [
Jan 5 12:46:11 testbox kernel: [653364.391694] [
Jan 5 12:46:11 testbox kernel: [653364.391694] [
Jan 5 12:46:11 testbox kernel: [653364.391694] [
Jan 5 12:46:11 testbox kernel: [653364.391694] [
Jan 5 12:46:11 testbox kernel: [653364.391694] [
Jan 5 12:46:42 testbox kernel: [653395.522503] BUG: soft lockup - CPU#3 stuck for 61s! [rsync:30532]
Jan 5 12:46:42 testbox kernel: [653395.522503] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:46:42 testbox kernel: [653395.522503]
Jan 5 12:46:42 testbox kernel: [653395.522503] Pid: 30532, comm: rsync Tainted: G D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:46:42 testbox kernel: [653395.522503] EIP: 0060:[
Jan 5 12:46:42 testbox kernel: [653395.522503] EIP is at __ticket_spin_lock+0x15/0x20
Jan 5 12:46:42 testbox kernel: [653395.522503] EAX: f5d962b8 EBX: e0c8de00 ECX: 00000000 EDX: 00004e4d
Jan 5 12:46:42 testbox kernel: [653395.522503] ESI: f5d96200 EDI: f6d00450 EBP: db0f1cc8 ESP: db0f1cc8
Jan 5 12:46:42 testbox kernel: [653395.522503] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:46:42 testbox kernel: [653395.522503] CR0: 8005003b CR2: 00000000 CR3: 1b0d9000 CR4: 000006b0
Jan 5 12:46:42 testbox kernel: [653395.522503] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:46:42 testbox kernel: [653395.522503] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:46:42 testbox kernel: [653395.522503] Call Trace:
Jan 5 12:46:42 testbox kernel: [653395.522503] [
Jan 5 12:46:42 testbox kernel: [653395.522503] [
Jan 5 12:46:42 testbox kernel: [653395.522503] [
Jan 5 12:46:42 testbox kernel: [653395.522503] [
Jan 5 12:46:42 testbox kernel: [653395.522503] [
Jan 5 12:46:42 testbox kernel: [653395.522503] [
Jan 5 12:46:42 testbox kernel: [653395.522503] [
Jan 5 12:46:42 testbox kernel: [653395.522503] [
Jan 5 12:46:42 testbox kernel: [653395.522503] [
Jan 5 12:46:42 testbox kernel: [653395.522503] [
Jan 5 12:46:42 testbox kernel: [653395.522503] [
Jan 5 12:46:42 testbox kernel: [653395.522503] [
Jan 5 12:46:42 testbox kernel: [653395.522503] [
Jan 5 12:46:42 testbox kernel: [653395.522503] [
Jan 5 12:46:42 testbox kernel: [653395.522503] [
Jan 5 12:46:42 testbox kernel: [653395.522503] [
Jan 5 12:46:42 testbox kernel: [653395.522503] [
Jan 5 12:46:42 testbox kernel: [653395.522503] [
Jan 5 12:46:42 testbox kernel: [653395.522503] [
Jan 5 12:46:42 testbox kernel: [653395.522503] [
Jan 5 12:46:42 testbox kernel: [653395.522503] [
Jan 5 12:46:42 testbox kernel: [653395.522503] [
Jan 5 12:46:42 testbox kernel: [653395.522503] [
Jan 5 12:46:42 testbox kernel: [653395.526521] hpet1: lost 3 rtc interrupts
Jan 5 12:47:16 testbox kernel: [653429.890004] BUG: soft lockup - CPU#0 stuck for 61s! [hal-acl-tool:4168]
Jan 5 12:47:16 testbox kernel: [653429.890005] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:47:16 testbox kernel: [653429.891694]
Jan 5 12:47:16 testbox kernel: [653429.891694] Pid: 4168, comm: hal-acl-tool Tainted: G D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:47:16 testbox kernel: [653429.891694] EIP: 0060:[
Jan 5 12:47:16 testbox kernel: [653429.891694] EIP is at __ticket_spin_lock+0xf/0x20
Jan 5 12:47:16 testbox kernel: [653429.891694] EAX: f5d962b8 EBX: d6ba3b00 ECX: f5d96200 EDX: 00004f4d
Jan 5 12:47:16 testbox kernel: [653429.891694] ESI: d20d3000 EDI: f5d962b8 EBP: e0491c60 ESP: e0491c60
Jan 5 12:47:16 testbox kernel: [653429.891694] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:47:16 testbox kernel: [653429.891694] CR0: 8005003b CR2: 09202568 CR3: 2a445000 CR4: 000006b0
Jan 5 12:47:16 testbox kernel: [653429.891694] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:47:16 testbox kernel: [653429.891694] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:47:16 testbox kernel: [653429.891694] Call Trace:
Jan 5 12:47:16 testbox kernel: [653429.891694] [
Jan 5 12:47:16 testbox kernel: [653429.891694] [
Jan 5 12:47:16 testbox kernel: [653429.891694] [
Jan 5 12:47:16 testbox kernel: [653429.891694] [
Jan 5 12:47:16 testbox kernel: [653429.891694] [
Jan 5 12:47:16 testbox kernel: [653429.891694] [
Jan 5 12:47:16 testbox kernel: [653429.891694] [
Jan 5 12:47:16 testbox kernel: [653429.891694] [
Jan 5 12:47:16 testbox kernel: [653429.891694] [
Jan 5 12:47:16 testbox kernel: [653429.891694] [
Jan 5 12:47:16 testbox kernel: [653429.891694] [
Jan 5 12:47:16 testbox kernel: [653429.891694] [
Jan 5 12:47:16 testbox kernel: [653429.891694] [
Jan 5 12:47:16 testbox kernel: [653429.891694] [
Jan 5 12:47:16 testbox kernel: [653429.891694] [
Jan 5 12:47:16 testbox kernel: [653429.891694] [
Jan 5 12:47:16 testbox kernel: [653429.891694] [
Jan 5 12:47:16 testbox kernel: [653429.891694] [
Jan 5 12:47:16 testbox kernel: [653429.891694] [
Jan 5 12:47:16 testbox kernel: [653429.891694] [
Jan 5 12:47:16 testbox kernel: [653429.891694] [
Jan 5 12:47:16 testbox kernel: [653429.891694] [
Jan 5 12:47:16 testbox kernel: [653429.891694] [
Jan 5 12:47:16 testbox kernel: [653429.891694] [
Jan 5 12:47:16 testbox kernel: [653429.891694] [
Jan 5 12:47:16 testbox kernel: [653429.891694] [
Jan 5 12:47:16 testbox kernel: [653429.891694] [
Jan 5 12:47:16 testbox kernel: [653429.891694] [
Jan 5 12:47:16 testbox kernel: [653429.891694] [
Jan 5 12:47:48 testbox kernel: [653461.022504] BUG: soft lockup - CPU#3 stuck for 61s! [rsync:30532]
Jan 5 12:47:48 testbox kernel: [653461.022504] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:47:48 testbox kernel: [653461.022504]
Jan 5 12:47:48 testbox kernel: [653461.022504] Pid: 30532, comm: rsync Tainted: G D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:47:48 testbox kernel: [653461.022504] EIP: 0060:[
Jan 5 12:47:48 testbox kernel: [653461.022504] EIP is at __ticket_spin_lock+0x15/0x20
Jan 5 12:47:48 testbox kernel: [653461.022504] EAX: f5d962b8 EBX: e0c8de00 ECX: 00000000 EDX: 00004e4d
Jan 5 12:47:48 testbox kernel: [653461.022504] ESI: f5d96200 EDI: f6d00450 EBP: db0f1cc8 ESP: db0f1cc8
Jan 5 12:47:48 testbox kernel: [653461.022504] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:47:48 testbox kernel: [653461.022504] CR0: 8005003b CR2: 00000000 CR3: 1b0d9000 CR4: 000006b0
Jan 5 12:47:48 testbox kernel: [653461.022504] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:47:48 testbox kernel: [653461.022504] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:47:48 testbox kernel: [653461.022504] Call Trace:
Jan 5 12:47:48 testbox kernel: [653461.022504] [
Jan 5 12:47:48 testbox kernel: [653461.022504] [
Jan 5 12:47:48 testbox kernel: [653461.022504] [
Jan 5 12:47:48 testbox kernel: [653461.022504] [
Jan 5 12:47:48 testbox kernel: [653461.022504] [
Jan 5 12:47:48 testbox kernel: [653461.022504] [
Jan 5 12:47:48 testbox kernel: [653461.022504] [
Jan 5 12:47:48 testbox kernel: [653461.022504] [
Jan 5 12:47:48 testbox kernel: [653461.022504] [
Jan 5 12:47:48 testbox kernel: [653461.022504] [
Jan 5 12:47:48 testbox kernel: [653461.022504] [
Jan 5 12:47:48 testbox kernel: [653461.022504] [
Jan 5 12:47:48 testbox kernel: [653461.022504] [
Jan 5 12:47:48 testbox kernel: [653461.022504] [
Jan 5 12:47:48 testbox kernel: [653461.022504] [
Jan 5 12:47:48 testbox kernel: [653461.022504] [
Jan 5 12:47:48 testbox kernel: [653461.022504] [
Jan 5 12:47:48 testbox kernel: [653461.022504] [
Jan 5 12:47:48 testbox kernel: [653461.022504] [
Jan 5 12:47:48 testbox kernel: [653461.022504] [
Jan 5 12:47:48 testbox kernel: [653461.022504] [
Jan 5 12:47:48 testbox kernel: [653461.022504] [
Jan 5 12:47:48 testbox kernel: [653461.022504] [
Jan 5 12:47:48 testbox kernel: [653461.025921] hpet1: lost 2 rtc interrupts
Jan 5 12:48:22 testbox kernel: [653495.390004] BUG: soft lockup - CPU#0 stuck for 61s! [hal-acl-tool:4168]
Jan 5 12:48:22 testbox kernel: [653495.390005] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:48:22 testbox kernel: [653495.391695]
Jan 5 12:48:22 testbox kernel: [653495.391695] Pid: 4168, comm: hal-acl-tool Tainted: G D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:48:22 testbox kernel: [653495.391695] EIP: 0060:[
Jan 5 12:48:22 testbox kernel: [653495.391695] EIP is at __ticket_spin_lock+0x13/0x20
Jan 5 12:48:22 testbox kernel: [653495.391695] EAX: f5d962b8 EBX: d6ba3b00 ECX: f5d96200 EDX: 00004f4d
Jan 5 12:48:22 testbox kernel: [653495.391695] ESI: d20d3000 EDI: f5d962b8 EBP: e0491c60 ESP: e0491c60
Jan 5 12:48:22 testbox kernel: [653495.391695] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:48:22 testbox kernel: [653495.391695] CR0: 8005003b CR2: 09202568 CR3: 2a445000 CR4: 000006b0
Jan 5 12:48:22 testbox kernel: [653495.391695] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:48:22 testbox kernel: [653495.391695] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:48:22 testbox kernel: [653495.391695] Call Trace:
Jan 5 12:48:22 testbox kernel: [653495.391695] [
Jan 5 12:48:22 testbox kernel: [653495.391695] [
Jan 5 12:48:22 testbox kernel: [653495.391695] [
Jan 5 12:48:22 testbox kernel: [653495.391695] [
Jan 5 12:48:22 testbox kernel: [653495.391695] [
Jan 5 12:48:22 testbox kernel: [653495.391695] [
Jan 5 12:48:22 testbox kernel: [653495.391695] [
Jan 5 12:48:22 testbox kernel: [653495.391695] [
Jan 5 12:48:22 testbox kernel: [653495.391695] [
Jan 5 12:48:22 testbox kernel: [653495.391695] [
Jan 5 12:48:22 testbox kernel: [653495.391695] [
Jan 5 12:48:22 testbox kernel: [653495.391695] [
Jan 5 12:48:22 testbox kernel: [653495.391695] [
Jan 5 12:48:22 testbox kernel: [653495.391695] [
Jan 5 12:48:22 testbox kernel: [653495.391695] [
Jan 5 12:48:22 testbox kernel: [653495.391695] [
Jan 5 12:48:22 testbox kernel: [653495.391695] [
Jan 5 12:48:22 testbox kernel: [653495.391695] [
Jan 5 12:48:22 testbox kernel: [653495.391695] [
Jan 5 12:48:22 testbox kernel: [653495.391695] [
Jan 5 12:48:22 testbox kernel: [653495.391695] [
Jan 5 12:48:22 testbox kernel: [653495.391695] [
Jan 5 12:48:22 testbox kernel: [653495.391695] [
Jan 5 12:48:22 testbox kernel: [653495.391695] [
Jan 5 12:48:22 testbox kernel: [653495.391695] [
Jan 5 12:48:22 testbox kernel: [653495.391695] [
Jan 5 12:48:22 testbox kernel: [653495.391695] [
Jan 5 12:48:22 testbox kernel: [653495.391695] [
Jan 5 12:48:22 testbox kernel: [653495.391695] [
Jan 5 12:48:53 testbox kernel: [653526.522503] BUG: soft lockup - CPU#3 stuck for 61s! [rsync:30532]
Jan 5 12:48:53 testbox kernel: [653526.522504] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:48:53 testbox kernel: [653526.522504]
Jan 5 12:48:53 testbox kernel: [653526.522504] Pid: 30532, comm: rsync Tainted: G D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:48:53 testbox kernel: [653526.522504] EIP: 0060:[
Jan 5 12:48:53 testbox kernel: [653526.522504] EIP is at __ticket_spin_lock+0x15/0x20
Jan 5 12:48:53 testbox kernel: [653526.522504] EAX: f5d962b8 EBX: e0c8de00 ECX: 00000000 EDX: 00004e4d
Jan 5 12:48:53 testbox kernel: [653526.522504] ESI: f5d96200 EDI: f6d00450 EBP: db0f1cc8 ESP: db0f1cc8
Jan 5 12:48:53 testbox kernel: [653526.522504] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:48:53 testbox kernel: [653526.522504] CR0: 8005003b CR2: 00000000 CR3: 1b0d9000 CR4: 000006b0
Jan 5 12:48:53 testbox kernel: [653526.522504] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:48:53 testbox kernel: [653526.522504] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:48:53 testbox kernel: [653526.522504] Call Trace:
Jan 5 12:48:53 testbox kernel: [653526.522504] [
Jan 5 12:48:53 testbox kernel: [653526.522504] [
Jan 5 12:48:53 testbox kernel: [653526.522504] [
Jan 5 12:48:53 testbox kernel: [653526.522504] [
Jan 5 12:48:53 testbox kernel: [653526.522504] [
Jan 5 12:48:53 testbox kernel: [653526.522504] [
Jan 5 12:48:53 testbox kernel: [653526.522504] [
Jan 5 12:48:53 testbox kernel: [653526.522504] [
Jan 5 12:48:53 testbox kernel: [653526.522504] [
Jan 5 12:48:53 testbox kernel: [653526.522504] [
Jan 5 12:48:53 testbox kernel: [653526.522504] [
Jan 5 12:48:53 testbox kernel: [653526.522504] [
Jan 5 12:48:53 testbox kernel: [653526.522504] [
Jan 5 12:48:53 testbox kernel: [653526.522504] [
Jan 5 12:48:53 testbox kernel: [653526.522504] [
Jan 5 12:48:53 testbox kernel: [653526.522504] [
Jan 5 12:48:53 testbox kernel: [653526.522504] [
Jan 5 12:48:53 testbox kernel: [653526.522504] [
Jan 5 12:48:53 testbox kernel: [653526.522504] [
Jan 5 12:48:53 testbox kernel: [653526.522504] [
Jan 5 12:48:53 testbox kernel: [653526.522504] [
Jan 5 12:48:53 testbox kernel: [653526.522504] [
Jan 5 12:48:53 testbox kernel: [653526.522504] [
Jan 5 12:48:53 testbox kernel: [653526.526518] hpet1: lost 3 rtc interrupts
Jan 5 12:49:27 testbox kernel: [653560.880003] BUG: soft lockup - CPU#0 stuck for 61s! [hal-acl-tool:4168]
Jan 5 12:49:27 testbox kernel: [653560.880005] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:49:27 testbox kernel: [653560.881694]
Jan 5 12:49:27 testbox kernel: [653560.881694] Pid: 4168, comm: hal-acl-tool Tainted: G D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:49:27 testbox kernel: [653560.881694] EIP: 0060:[
Jan 5 12:49:27 testbox kernel: [653560.881694] EIP is at __ticket_spin_lock+0xf/0x20
Jan 5 12:49:27 testbox kernel: [653560.881694] EAX: f5d962b8 EBX: d6ba3b00 ECX: f5d96200 EDX: 00004f4d
Jan 5 12:49:27 testbox kernel: [653560.881694] ESI: d20d3000 EDI: f5d962b8 EBP: e0491c60 ESP: e0491c60
Jan 5 12:49:27 testbox kernel: [653560.881694] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:49:27 testbox kernel: [653560.881694] CR0: 8005003b CR2: 09202568 CR3: 2a445000 CR4: 000006b0
Jan 5 12:49:27 testbox kernel: [653560.881694] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:49:27 testbox kernel: [653560.881694] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:49:27 testbox kernel: [653560.881694] Call Trace:
Jan 5 12:49:27 testbox kernel: [653560.881694] [
Jan 5 12:49:27 testbox kernel: [653560.881694] [
Jan 5 12:49:27 testbox kernel: [653560.881694] [
Jan 5 12:49:27 testbox kernel: [653560.881694] [
Jan 5 12:49:27 testbox kernel: [653560.881694] [
Jan 5 12:49:27 testbox kernel: [653560.881694] [
Jan 5 12:49:27 testbox kernel: [653560.881694] [
Jan 5 12:49:27 testbox kernel: [653560.881694] [
Jan 5 12:49:27 testbox kernel: [653560.881694] [
Jan 5 12:49:27 testbox kernel: [653560.881694] [
Jan 5 12:49:27 testbox kernel: [653560.881694] [
Jan 5 12:49:27 testbox kernel: [653560.881694] [
Jan 5 12:49:27 testbox kernel: [653560.881694] [
Jan 5 12:49:27 testbox kernel: [653560.881694] [
Jan 5 12:49:27 testbox kernel: [653560.881694] [
Jan 5 12:49:27 testbox kernel: [653560.881694] [
Jan 5 12:49:27 testbox kernel: [653560.881694] [
Jan 5 12:49:27 testbox kernel: [653560.881694] [
Jan 5 12:49:27 testbox kernel: [653560.881694] [
Jan 5 12:49:27 testbox kernel: [653560.881694] [
Jan 5 12:49:27 testbox kernel: [653560.881694] [
Jan 5 12:49:27 testbox kernel: [653560.881694] [
Jan 5 12:49:27 testbox kernel: [653560.881694] [
Jan 5 12:49:27 testbox kernel: [653560.881694] [
Jan 5 12:49:27 testbox kernel: [653560.881694] [
Jan 5 12:49:27 testbox kernel: [653560.881694] [
Jan 5 12:49:27 testbox kernel: [653560.881694] [
Jan 5 12:49:27 testbox kernel: [653560.881694] [
Jan 5 12:49:27 testbox kernel: [653560.881694] [
Jan 5 12:49:59 testbox kernel: [653592.022503] BUG: soft lockup - CPU#3 stuck for 61s! [rsync:30532]
Jan 5 12:49:59 testbox kernel: [653592.022504] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:49:59 testbox kernel: [653592.022504]
Jan 5 12:49:59 testbox kernel: [653592.022504] Pid: 30532, comm: rsync Tainted: G D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:49:59 testbox kernel: [653592.022504] EIP: 0060:[
Jan 5 12:49:59 testbox kernel: [653592.022504] EIP is at __ticket_spin_lock+0x15/0x20
Jan 5 12:49:59 testbox kernel: [653592.022504] EAX: f5d962b8 EBX: e0c8de00 ECX: 00000000 EDX: 00004e4d
Jan 5 12:49:59 testbox kernel: [653592.022504] ESI: f5d96200 EDI: f6d00450 EBP: db0f1cc8 ESP: db0f1cc8
Jan 5 12:49:59 testbox kernel: [653592.022504] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:49:59 testbox kernel: [653592.022504] CR0: 8005003b CR2: 00000000 CR3: 1b0d9000 CR4: 000006b0
Jan 5 12:49:59 testbox kernel: [653592.022504] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:49:59 testbox kernel: [653592.022504] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:49:59 testbox kernel: [653592.022504] Call Trace:
Jan 5 12:49:59 testbox kernel: [653592.022504] [
Jan 5 12:49:59 testbox kernel: [653592.022504] [