Ubuntu 9.04 Crash

CPU/Kernel/MB/RAID problem?

Jan 5 12:45:05 testbox kernel: [653298.890004] BUG: soft lockup - CPU#0 stuck for 61s! [hal-acl-tool:4168]
Jan 5 12:45:05 testbox kernel: [653298.890005] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:45:05 testbox kernel: [653298.891687]
Jan 5 12:45:05 testbox kernel: [653298.891687] Pid: 4168, comm: hal-acl-tool Tainted: G D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:45:05 testbox kernel: [653298.891687] EIP: 0060:[] EFLAGS: 00000293 CPU: 0
Jan 5 12:45:05 testbox kernel: [653298.891687] EIP is at __ticket_spin_lock+0x15/0x20
Jan 5 12:45:05 testbox kernel: [653298.891687] EAX: f5d962b8 EBX: d6ba3b00 ECX: f5d96200 EDX: 00004f4d
Jan 5 12:45:05 testbox kernel: [653298.891687] ESI: d20d3000 EDI: f5d962b8 EBP: e0491c60 ESP: e0491c60
Jan 5 12:45:05 testbox kernel: [653298.891687] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:45:05 testbox kernel: [653298.891687] CR0: 8005003b CR2: 09202568 CR3: 2a445000 CR4: 000006b0
Jan 5 12:45:05 testbox kernel: [653298.891687] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:45:05 testbox kernel: [653298.891687] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:45:05 testbox kernel: [653298.891687] Call Trace:
Jan 5 12:45:05 testbox kernel: [653298.891687] [] _spin_lock+0x8/0x10
Jan 5 12:45:05 testbox kernel: [653298.891687] [] __journal_try_to_free_buffer+0x4d/0xc0
Jan 5 12:45:05 testbox kernel: [653298.891687] [] journal_try_to_free_buffers+0x62/0x100
Jan 5 12:45:05 testbox kernel: [653298.891687] [] ext3_releasepage+0x45/0x70
Jan 5 12:45:05 testbox kernel: [653298.891687] [] try_to_release_page+0x2e/0x50
Jan 5 12:45:05 testbox kernel: [653298.891687] [] __invalidate_mapping_pages+0x136/0x150
Jan 5 12:45:05 testbox kernel: [653298.891687] [] invalidate_mapping_pages+0x12/0x20
Jan 5 12:45:05 testbox kernel: [653298.891687] [] prune_icache+0x21a/0x260
Jan 5 12:45:05 testbox kernel: [653298.891687] [] shrink_icache_memory+0x3e/0x50
Jan 5 12:45:05 testbox kernel: [653298.891687] [] shrink_slab+0x11f/0x180
Jan 5 12:45:05 testbox kernel: [653298.891687] [] do_try_to_free_pages+0x155/0x2c0
Jan 5 12:45:05 testbox kernel: [653298.891687] [] ? autoremove_wake_function+0x1b/0x50
Jan 5 12:45:05 testbox kernel: [653298.891687] [] try_to_free_pages+0x6c/0x80
Jan 5 12:45:05 testbox kernel: [653298.891687] [] ? isolate_pages_global+0x0/0x70
Jan 5 12:45:05 testbox kernel: [653298.891687] [] __alloc_pages_internal+0x1cb/0x450
Jan 5 12:45:05 testbox kernel: [653298.891687] [] new_slab+0x7b/0x1e0
Jan 5 12:45:05 testbox kernel: [653298.891687] [] ? unfreeze_slab+0x75/0xb0
Jan 5 12:45:05 testbox kernel: [653298.891687] [] __slab_alloc+0x20b/0x260
Jan 5 12:45:05 testbox kernel: [653298.891687] [] kmem_cache_alloc+0xa6/0xb0
Jan 5 12:45:05 testbox kernel: [653298.891687] [] ? dup_task_struct+0x1c/0x110
Jan 5 12:45:05 testbox kernel: [653298.891687] [] dup_task_struct+0x1c/0x110
Jan 5 12:45:05 testbox kernel: [653298.891687] [] copy_process+0x6e/0xcf0
Jan 5 12:45:05 testbox kernel: [653298.891687] [] ? alloc_fd+0xe0/0x100
Jan 5 12:45:05 testbox kernel: [653298.891687] [] ? get_empty_filp+0x6c/0x170
Jan 5 12:45:05 testbox kernel: [653298.891687] [] ? _spin_lock+0x8/0x10
Jan 5 12:45:05 testbox kernel: [653298.891687] [] do_fork+0x62/0x300
Jan 5 12:45:05 testbox kernel: [653298.891687] [] ? sys_pipe2+0x3f/0x60
Jan 5 12:45:05 testbox kernel: [653298.891687] [] sys_clone+0x2f/0x40
Jan 5 12:45:05 testbox kernel: [653298.891687] [] sysenter_do_call+0x12/0x2f
Jan 5 12:45:37 testbox kernel: [653330.032503] BUG: soft lockup - CPU#3 stuck for 61s! [rsync:30532]
Jan 5 12:45:37 testbox kernel: [653330.032503] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:45:37 testbox kernel: [653330.032503]
Jan 5 12:45:37 testbox kernel: [653330.032503] Pid: 30532, comm: rsync Tainted: G D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:45:37 testbox kernel: [653330.032503] EIP: 0060:[] EFLAGS: 00000297 CPU: 3
Jan 5 12:45:37 testbox kernel: [653330.032503] EIP is at __ticket_spin_lock+0x13/0x20
Jan 5 12:45:37 testbox kernel: [653330.032503] EAX: f5d962b8 EBX: e0c8de00 ECX: 00000000 EDX: 00004e4d
Jan 5 12:45:37 testbox kernel: [653330.032503] ESI: f5d96200 EDI: f6d00450 EBP: db0f1cc8 ESP: db0f1cc8
Jan 5 12:45:37 testbox kernel: [653330.032503] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:45:37 testbox kernel: [653330.032503] CR0: 8005003b CR2: 00000000 CR3: 1b0d9000 CR4: 000006b0
Jan 5 12:45:37 testbox kernel: [653330.032503] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:45:37 testbox kernel: [653330.032503] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:45:37 testbox kernel: [653330.032503] Call Trace:
Jan 5 12:45:37 testbox kernel: [653330.032503] [] _spin_lock+0x8/0x10
Jan 5 12:45:37 testbox kernel: [653330.032503] [] journal_dirty_data+0x5e/0x1f0
Jan 5 12:45:37 testbox kernel: [653330.032503] [] ext3_journal_dirty_data+0x18/0x50
Jan 5 12:45:37 testbox kernel: [653330.032503] [] walk_page_buffers+0x2f/0x80
Jan 5 12:45:37 testbox kernel: [653330.032503] [] ext3_ordered_write_end+0x63/0x1b0
Jan 5 12:45:37 testbox kernel: [653330.032503] [] ? ext3_journal_dirty_data+0x0/0x50
Jan 5 12:45:37 testbox kernel: [653330.032503] [] ? iov_iter_copy_from_user_atomic+0x5d/0xc0
Jan 5 12:45:37 testbox kernel: [653330.032503] [] generic_perform_write+0x10f/0x190
Jan 5 12:45:37 testbox kernel: [653330.032503] [] generic_file_buffered_write+0x5d/0x120
Jan 5 12:45:37 testbox kernel: [653330.032503] [] __generic_file_aio_write_nolock+0x19b/0x520
Jan 5 12:45:37 testbox kernel: [653330.032503] [] ? resched_task+0x1e/0x70
Jan 5 12:45:37 testbox kernel: [653330.032503] [] generic_file_aio_write+0x5e/0xd0
Jan 5 12:45:37 testbox kernel: [653330.032503] [] ext3_file_write+0x2d/0xc0
Jan 5 12:45:37 testbox kernel: [653330.032503] [] do_sync_write+0xd1/0x110
Jan 5 12:45:37 testbox kernel: [653330.032503] [] ? rtc_update_irq+0x92/0xa0
Jan 5 12:45:37 testbox kernel: [653330.032503] [] ? autoremove_wake_function+0x0/0x50
Jan 5 12:45:37 testbox kernel: [653330.032503] [] ? apparmor_file_permission+0x20/0x30
Jan 5 12:45:37 testbox kernel: [653330.032503] [] ? security_file_permission+0xf/0x20
Jan 5 12:45:37 testbox kernel: [653330.032503] [] ? rw_verify_area+0x54/0xd0
Jan 5 12:45:37 testbox kernel: [653330.032503] [] vfs_write+0x98/0x110
Jan 5 12:45:37 testbox kernel: [653330.032503] [] ? do_sync_write+0x0/0x110
Jan 5 12:45:37 testbox kernel: [653330.032503] [] sys_write+0x3d/0x70
Jan 5 12:45:37 testbox kernel: [653330.032503] [] sysenter_do_call+0x12/0x2f
Jan 5 12:45:37 testbox kernel: [653330.035919] hpet1: lost 3 rtc interrupts
Jan 5 12:46:11 testbox kernel: [653364.390003] BUG: soft lockup - CPU#0 stuck for 61s! [hal-acl-tool:4168]
Jan 5 12:46:11 testbox kernel: [653364.390005] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:46:11 testbox kernel: [653364.391694]
Jan 5 12:46:11 testbox kernel: [653364.391694] Pid: 4168, comm: hal-acl-tool Tainted: G D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:46:11 testbox kernel: [653364.391694] EIP: 0060:[] EFLAGS: 00000293 CPU: 0
Jan 5 12:46:11 testbox kernel: [653364.391694] EIP is at __ticket_spin_lock+0x13/0x20
Jan 5 12:46:11 testbox kernel: [653364.391694] EAX: f5d962b8 EBX: d6ba3b00 ECX: f5d96200 EDX: 00004f4d
Jan 5 12:46:11 testbox kernel: [653364.391694] ESI: d20d3000 EDI: f5d962b8 EBP: e0491c60 ESP: e0491c60
Jan 5 12:46:11 testbox kernel: [653364.391694] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:46:11 testbox kernel: [653364.391694] CR0: 8005003b CR2: 09202568 CR3: 2a445000 CR4: 000006b0
Jan 5 12:46:11 testbox kernel: [653364.391694] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:46:11 testbox kernel: [653364.391694] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:46:11 testbox kernel: [653364.391694] Call Trace:
Jan 5 12:46:11 testbox kernel: [653364.391694] [] _spin_lock+0x8/0x10
Jan 5 12:46:11 testbox kernel: [653364.391694] [] __journal_try_to_free_buffer+0x4d/0xc0
Jan 5 12:46:11 testbox kernel: [653364.391694] [] journal_try_to_free_buffers+0x62/0x100
Jan 5 12:46:11 testbox kernel: [653364.391694] [] ext3_releasepage+0x45/0x70
Jan 5 12:46:11 testbox kernel: [653364.391694] [] try_to_release_page+0x2e/0x50
Jan 5 12:46:11 testbox kernel: [653364.391694] [] __invalidate_mapping_pages+0x136/0x150
Jan 5 12:46:11 testbox kernel: [653364.391694] [] invalidate_mapping_pages+0x12/0x20
Jan 5 12:46:11 testbox kernel: [653364.391694] [] prune_icache+0x21a/0x260
Jan 5 12:46:11 testbox kernel: [653364.391694] [] shrink_icache_memory+0x3e/0x50
Jan 5 12:46:11 testbox kernel: [653364.391694] [] shrink_slab+0x11f/0x180
Jan 5 12:46:11 testbox kernel: [653364.391694] [] do_try_to_free_pages+0x155/0x2c0
Jan 5 12:46:11 testbox kernel: [653364.391694] [] ? autoremove_wake_function+0x1b/0x50
Jan 5 12:46:11 testbox kernel: [653364.391694] [] try_to_free_pages+0x6c/0x80
Jan 5 12:46:11 testbox kernel: [653364.391694] [] ? isolate_pages_global+0x0/0x70
Jan 5 12:46:11 testbox kernel: [653364.391694] [] __alloc_pages_internal+0x1cb/0x450
Jan 5 12:46:11 testbox kernel: [653364.391694] [] new_slab+0x7b/0x1e0
Jan 5 12:46:11 testbox kernel: [653364.391694] [] ? unfreeze_slab+0x75/0xb0
Jan 5 12:46:11 testbox kernel: [653364.391694] [] __slab_alloc+0x20b/0x260
Jan 5 12:46:11 testbox kernel: [653364.391694] [] kmem_cache_alloc+0xa6/0xb0
Jan 5 12:46:11 testbox kernel: [653364.391694] [] ? dup_task_struct+0x1c/0x110
Jan 5 12:46:11 testbox kernel: [653364.391694] [] dup_task_struct+0x1c/0x110
Jan 5 12:46:11 testbox kernel: [653364.391694] [] copy_process+0x6e/0xcf0
Jan 5 12:46:11 testbox kernel: [653364.391694] [] ? alloc_fd+0xe0/0x100
Jan 5 12:46:11 testbox kernel: [653364.391694] [] ? get_empty_filp+0x6c/0x170
Jan 5 12:46:11 testbox kernel: [653364.391694] [] ? _spin_lock+0x8/0x10
Jan 5 12:46:11 testbox kernel: [653364.391694] [] do_fork+0x62/0x300
Jan 5 12:46:11 testbox kernel: [653364.391694] [] ? sys_pipe2+0x3f/0x60
Jan 5 12:46:11 testbox kernel: [653364.391694] [] sys_clone+0x2f/0x40
Jan 5 12:46:11 testbox kernel: [653364.391694] [] sysenter_do_call+0x12/0x2f
Jan 5 12:46:42 testbox kernel: [653395.522503] BUG: soft lockup - CPU#3 stuck for 61s! [rsync:30532]
Jan 5 12:46:42 testbox kernel: [653395.522503] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:46:42 testbox kernel: [653395.522503]
Jan 5 12:46:42 testbox kernel: [653395.522503] Pid: 30532, comm: rsync Tainted: G D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:46:42 testbox kernel: [653395.522503] EIP: 0060:[] EFLAGS: 00000297 CPU: 3
Jan 5 12:46:42 testbox kernel: [653395.522503] EIP is at __ticket_spin_lock+0x15/0x20
Jan 5 12:46:42 testbox kernel: [653395.522503] EAX: f5d962b8 EBX: e0c8de00 ECX: 00000000 EDX: 00004e4d
Jan 5 12:46:42 testbox kernel: [653395.522503] ESI: f5d96200 EDI: f6d00450 EBP: db0f1cc8 ESP: db0f1cc8
Jan 5 12:46:42 testbox kernel: [653395.522503] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:46:42 testbox kernel: [653395.522503] CR0: 8005003b CR2: 00000000 CR3: 1b0d9000 CR4: 000006b0
Jan 5 12:46:42 testbox kernel: [653395.522503] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:46:42 testbox kernel: [653395.522503] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:46:42 testbox kernel: [653395.522503] Call Trace:
Jan 5 12:46:42 testbox kernel: [653395.522503] [] _spin_lock+0x8/0x10
Jan 5 12:46:42 testbox kernel: [653395.522503] [] journal_dirty_data+0x5e/0x1f0
Jan 5 12:46:42 testbox kernel: [653395.522503] [] ext3_journal_dirty_data+0x18/0x50
Jan 5 12:46:42 testbox kernel: [653395.522503] [] walk_page_buffers+0x2f/0x80
Jan 5 12:46:42 testbox kernel: [653395.522503] [] ext3_ordered_write_end+0x63/0x1b0
Jan 5 12:46:42 testbox kernel: [653395.522503] [] ? ext3_journal_dirty_data+0x0/0x50
Jan 5 12:46:42 testbox kernel: [653395.522503] [] ? iov_iter_copy_from_user_atomic+0x5d/0xc0
Jan 5 12:46:42 testbox kernel: [653395.522503] [] generic_perform_write+0x10f/0x190
Jan 5 12:46:42 testbox kernel: [653395.522503] [] generic_file_buffered_write+0x5d/0x120
Jan 5 12:46:42 testbox kernel: [653395.522503] [] __generic_file_aio_write_nolock+0x19b/0x520
Jan 5 12:46:42 testbox kernel: [653395.522503] [] ? resched_task+0x1e/0x70
Jan 5 12:46:42 testbox kernel: [653395.522503] [] generic_file_aio_write+0x5e/0xd0
Jan 5 12:46:42 testbox kernel: [653395.522503] [] ext3_file_write+0x2d/0xc0
Jan 5 12:46:42 testbox kernel: [653395.522503] [] do_sync_write+0xd1/0x110
Jan 5 12:46:42 testbox kernel: [653395.522503] [] ? rtc_update_irq+0x92/0xa0
Jan 5 12:46:42 testbox kernel: [653395.522503] [] ? autoremove_wake_function+0x0/0x50
Jan 5 12:46:42 testbox kernel: [653395.522503] [] ? apparmor_file_permission+0x20/0x30
Jan 5 12:46:42 testbox kernel: [653395.522503] [] ? security_file_permission+0xf/0x20
Jan 5 12:46:42 testbox kernel: [653395.522503] [] ? rw_verify_area+0x54/0xd0
Jan 5 12:46:42 testbox kernel: [653395.522503] [] vfs_write+0x98/0x110
Jan 5 12:46:42 testbox kernel: [653395.522503] [] ? do_sync_write+0x0/0x110
Jan 5 12:46:42 testbox kernel: [653395.522503] [] sys_write+0x3d/0x70
Jan 5 12:46:42 testbox kernel: [653395.522503] [] sysenter_do_call+0x12/0x2f
Jan 5 12:46:42 testbox kernel: [653395.526521] hpet1: lost 3 rtc interrupts
Jan 5 12:47:16 testbox kernel: [653429.890004] BUG: soft lockup - CPU#0 stuck for 61s! [hal-acl-tool:4168]
Jan 5 12:47:16 testbox kernel: [653429.890005] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:47:16 testbox kernel: [653429.891694]
Jan 5 12:47:16 testbox kernel: [653429.891694] Pid: 4168, comm: hal-acl-tool Tainted: G D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:47:16 testbox kernel: [653429.891694] EIP: 0060:[] EFLAGS: 00000293 CPU: 0
Jan 5 12:47:16 testbox kernel: [653429.891694] EIP is at __ticket_spin_lock+0xf/0x20
Jan 5 12:47:16 testbox kernel: [653429.891694] EAX: f5d962b8 EBX: d6ba3b00 ECX: f5d96200 EDX: 00004f4d
Jan 5 12:47:16 testbox kernel: [653429.891694] ESI: d20d3000 EDI: f5d962b8 EBP: e0491c60 ESP: e0491c60
Jan 5 12:47:16 testbox kernel: [653429.891694] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:47:16 testbox kernel: [653429.891694] CR0: 8005003b CR2: 09202568 CR3: 2a445000 CR4: 000006b0
Jan 5 12:47:16 testbox kernel: [653429.891694] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:47:16 testbox kernel: [653429.891694] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:47:16 testbox kernel: [653429.891694] Call Trace:
Jan 5 12:47:16 testbox kernel: [653429.891694] [] _spin_lock+0x8/0x10
Jan 5 12:47:16 testbox kernel: [653429.891694] [] __journal_try_to_free_buffer+0x4d/0xc0
Jan 5 12:47:16 testbox kernel: [653429.891694] [] journal_try_to_free_buffers+0x62/0x100
Jan 5 12:47:16 testbox kernel: [653429.891694] [] ext3_releasepage+0x45/0x70
Jan 5 12:47:16 testbox kernel: [653429.891694] [] try_to_release_page+0x2e/0x50
Jan 5 12:47:16 testbox kernel: [653429.891694] [] __invalidate_mapping_pages+0x136/0x150
Jan 5 12:47:16 testbox kernel: [653429.891694] [] invalidate_mapping_pages+0x12/0x20
Jan 5 12:47:16 testbox kernel: [653429.891694] [] prune_icache+0x21a/0x260
Jan 5 12:47:16 testbox kernel: [653429.891694] [] shrink_icache_memory+0x3e/0x50
Jan 5 12:47:16 testbox kernel: [653429.891694] [] shrink_slab+0x11f/0x180
Jan 5 12:47:16 testbox kernel: [653429.891694] [] do_try_to_free_pages+0x155/0x2c0
Jan 5 12:47:16 testbox kernel: [653429.891694] [] ? autoremove_wake_function+0x1b/0x50
Jan 5 12:47:16 testbox kernel: [653429.891694] [] try_to_free_pages+0x6c/0x80
Jan 5 12:47:16 testbox kernel: [653429.891694] [] ? isolate_pages_global+0x0/0x70
Jan 5 12:47:16 testbox kernel: [653429.891694] [] __alloc_pages_internal+0x1cb/0x450
Jan 5 12:47:16 testbox kernel: [653429.891694] [] new_slab+0x7b/0x1e0
Jan 5 12:47:16 testbox kernel: [653429.891694] [] ? unfreeze_slab+0x75/0xb0
Jan 5 12:47:16 testbox kernel: [653429.891694] [] __slab_alloc+0x20b/0x260
Jan 5 12:47:16 testbox kernel: [653429.891694] [] kmem_cache_alloc+0xa6/0xb0
Jan 5 12:47:16 testbox kernel: [653429.891694] [] ? dup_task_struct+0x1c/0x110
Jan 5 12:47:16 testbox kernel: [653429.891694] [] dup_task_struct+0x1c/0x110
Jan 5 12:47:16 testbox kernel: [653429.891694] [] copy_process+0x6e/0xcf0
Jan 5 12:47:16 testbox kernel: [653429.891694] [] ? alloc_fd+0xe0/0x100
Jan 5 12:47:16 testbox kernel: [653429.891694] [] ? get_empty_filp+0x6c/0x170
Jan 5 12:47:16 testbox kernel: [653429.891694] [] ? _spin_lock+0x8/0x10
Jan 5 12:47:16 testbox kernel: [653429.891694] [] do_fork+0x62/0x300
Jan 5 12:47:16 testbox kernel: [653429.891694] [] ? sys_pipe2+0x3f/0x60
Jan 5 12:47:16 testbox kernel: [653429.891694] [] sys_clone+0x2f/0x40
Jan 5 12:47:16 testbox kernel: [653429.891694] [] sysenter_do_call+0x12/0x2f
Jan 5 12:47:48 testbox kernel: [653461.022504] BUG: soft lockup - CPU#3 stuck for 61s! [rsync:30532]
Jan 5 12:47:48 testbox kernel: [653461.022504] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:47:48 testbox kernel: [653461.022504]
Jan 5 12:47:48 testbox kernel: [653461.022504] Pid: 30532, comm: rsync Tainted: G D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:47:48 testbox kernel: [653461.022504] EIP: 0060:[] EFLAGS: 00000297 CPU: 3
Jan 5 12:47:48 testbox kernel: [653461.022504] EIP is at __ticket_spin_lock+0x15/0x20
Jan 5 12:47:48 testbox kernel: [653461.022504] EAX: f5d962b8 EBX: e0c8de00 ECX: 00000000 EDX: 00004e4d
Jan 5 12:47:48 testbox kernel: [653461.022504] ESI: f5d96200 EDI: f6d00450 EBP: db0f1cc8 ESP: db0f1cc8
Jan 5 12:47:48 testbox kernel: [653461.022504] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:47:48 testbox kernel: [653461.022504] CR0: 8005003b CR2: 00000000 CR3: 1b0d9000 CR4: 000006b0
Jan 5 12:47:48 testbox kernel: [653461.022504] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:47:48 testbox kernel: [653461.022504] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:47:48 testbox kernel: [653461.022504] Call Trace:
Jan 5 12:47:48 testbox kernel: [653461.022504] [] _spin_lock+0x8/0x10
Jan 5 12:47:48 testbox kernel: [653461.022504] [] journal_dirty_data+0x5e/0x1f0
Jan 5 12:47:48 testbox kernel: [653461.022504] [] ext3_journal_dirty_data+0x18/0x50
Jan 5 12:47:48 testbox kernel: [653461.022504] [] walk_page_buffers+0x2f/0x80
Jan 5 12:47:48 testbox kernel: [653461.022504] [] ext3_ordered_write_end+0x63/0x1b0
Jan 5 12:47:48 testbox kernel: [653461.022504] [] ? ext3_journal_dirty_data+0x0/0x50
Jan 5 12:47:48 testbox kernel: [653461.022504] [] ? iov_iter_copy_from_user_atomic+0x5d/0xc0
Jan 5 12:47:48 testbox kernel: [653461.022504] [] generic_perform_write+0x10f/0x190
Jan 5 12:47:48 testbox kernel: [653461.022504] [] generic_file_buffered_write+0x5d/0x120
Jan 5 12:47:48 testbox kernel: [653461.022504] [] __generic_file_aio_write_nolock+0x19b/0x520
Jan 5 12:47:48 testbox kernel: [653461.022504] [] ? resched_task+0x1e/0x70
Jan 5 12:47:48 testbox kernel: [653461.022504] [] generic_file_aio_write+0x5e/0xd0
Jan 5 12:47:48 testbox kernel: [653461.022504] [] ext3_file_write+0x2d/0xc0
Jan 5 12:47:48 testbox kernel: [653461.022504] [] do_sync_write+0xd1/0x110
Jan 5 12:47:48 testbox kernel: [653461.022504] [] ? rtc_update_irq+0x92/0xa0
Jan 5 12:47:48 testbox kernel: [653461.022504] [] ? autoremove_wake_function+0x0/0x50
Jan 5 12:47:48 testbox kernel: [653461.022504] [] ? apparmor_file_permission+0x20/0x30
Jan 5 12:47:48 testbox kernel: [653461.022504] [] ? security_file_permission+0xf/0x20
Jan 5 12:47:48 testbox kernel: [653461.022504] [] ? rw_verify_area+0x54/0xd0
Jan 5 12:47:48 testbox kernel: [653461.022504] [] vfs_write+0x98/0x110
Jan 5 12:47:48 testbox kernel: [653461.022504] [] ? do_sync_write+0x0/0x110
Jan 5 12:47:48 testbox kernel: [653461.022504] [] sys_write+0x3d/0x70
Jan 5 12:47:48 testbox kernel: [653461.022504] [] sysenter_do_call+0x12/0x2f
Jan 5 12:47:48 testbox kernel: [653461.025921] hpet1: lost 2 rtc interrupts
Jan 5 12:48:22 testbox kernel: [653495.390004] BUG: soft lockup - CPU#0 stuck for 61s! [hal-acl-tool:4168]
Jan 5 12:48:22 testbox kernel: [653495.390005] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:48:22 testbox kernel: [653495.391695]
Jan 5 12:48:22 testbox kernel: [653495.391695] Pid: 4168, comm: hal-acl-tool Tainted: G D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:48:22 testbox kernel: [653495.391695] EIP: 0060:[] EFLAGS: 00000293 CPU: 0
Jan 5 12:48:22 testbox kernel: [653495.391695] EIP is at __ticket_spin_lock+0x13/0x20
Jan 5 12:48:22 testbox kernel: [653495.391695] EAX: f5d962b8 EBX: d6ba3b00 ECX: f5d96200 EDX: 00004f4d
Jan 5 12:48:22 testbox kernel: [653495.391695] ESI: d20d3000 EDI: f5d962b8 EBP: e0491c60 ESP: e0491c60
Jan 5 12:48:22 testbox kernel: [653495.391695] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:48:22 testbox kernel: [653495.391695] CR0: 8005003b CR2: 09202568 CR3: 2a445000 CR4: 000006b0
Jan 5 12:48:22 testbox kernel: [653495.391695] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:48:22 testbox kernel: [653495.391695] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:48:22 testbox kernel: [653495.391695] Call Trace:
Jan 5 12:48:22 testbox kernel: [653495.391695] [] _spin_lock+0x8/0x10
Jan 5 12:48:22 testbox kernel: [653495.391695] [] __journal_try_to_free_buffer+0x4d/0xc0
Jan 5 12:48:22 testbox kernel: [653495.391695] [] journal_try_to_free_buffers+0x62/0x100
Jan 5 12:48:22 testbox kernel: [653495.391695] [] ext3_releasepage+0x45/0x70
Jan 5 12:48:22 testbox kernel: [653495.391695] [] try_to_release_page+0x2e/0x50
Jan 5 12:48:22 testbox kernel: [653495.391695] [] __invalidate_mapping_pages+0x136/0x150
Jan 5 12:48:22 testbox kernel: [653495.391695] [] invalidate_mapping_pages+0x12/0x20
Jan 5 12:48:22 testbox kernel: [653495.391695] [] prune_icache+0x21a/0x260
Jan 5 12:48:22 testbox kernel: [653495.391695] [] shrink_icache_memory+0x3e/0x50
Jan 5 12:48:22 testbox kernel: [653495.391695] [] shrink_slab+0x11f/0x180
Jan 5 12:48:22 testbox kernel: [653495.391695] [] do_try_to_free_pages+0x155/0x2c0
Jan 5 12:48:22 testbox kernel: [653495.391695] [] ? autoremove_wake_function+0x1b/0x50
Jan 5 12:48:22 testbox kernel: [653495.391695] [] try_to_free_pages+0x6c/0x80
Jan 5 12:48:22 testbox kernel: [653495.391695] [] ? isolate_pages_global+0x0/0x70
Jan 5 12:48:22 testbox kernel: [653495.391695] [] __alloc_pages_internal+0x1cb/0x450
Jan 5 12:48:22 testbox kernel: [653495.391695] [] new_slab+0x7b/0x1e0
Jan 5 12:48:22 testbox kernel: [653495.391695] [] ? unfreeze_slab+0x75/0xb0
Jan 5 12:48:22 testbox kernel: [653495.391695] [] __slab_alloc+0x20b/0x260
Jan 5 12:48:22 testbox kernel: [653495.391695] [] kmem_cache_alloc+0xa6/0xb0
Jan 5 12:48:22 testbox kernel: [653495.391695] [] ? dup_task_struct+0x1c/0x110
Jan 5 12:48:22 testbox kernel: [653495.391695] [] dup_task_struct+0x1c/0x110
Jan 5 12:48:22 testbox kernel: [653495.391695] [] copy_process+0x6e/0xcf0
Jan 5 12:48:22 testbox kernel: [653495.391695] [] ? alloc_fd+0xe0/0x100
Jan 5 12:48:22 testbox kernel: [653495.391695] [] ? get_empty_filp+0x6c/0x170
Jan 5 12:48:22 testbox kernel: [653495.391695] [] ? _spin_lock+0x8/0x10
Jan 5 12:48:22 testbox kernel: [653495.391695] [] do_fork+0x62/0x300
Jan 5 12:48:22 testbox kernel: [653495.391695] [] ? sys_pipe2+0x3f/0x60
Jan 5 12:48:22 testbox kernel: [653495.391695] [] sys_clone+0x2f/0x40
Jan 5 12:48:22 testbox kernel: [653495.391695] [] sysenter_do_call+0x12/0x2f
Jan 5 12:48:53 testbox kernel: [653526.522503] BUG: soft lockup - CPU#3 stuck for 61s! [rsync:30532]
Jan 5 12:48:53 testbox kernel: [653526.522504] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:48:53 testbox kernel: [653526.522504]
Jan 5 12:48:53 testbox kernel: [653526.522504] Pid: 30532, comm: rsync Tainted: G D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:48:53 testbox kernel: [653526.522504] EIP: 0060:[] EFLAGS: 00000297 CPU: 3
Jan 5 12:48:53 testbox kernel: [653526.522504] EIP is at __ticket_spin_lock+0x15/0x20
Jan 5 12:48:53 testbox kernel: [653526.522504] EAX: f5d962b8 EBX: e0c8de00 ECX: 00000000 EDX: 00004e4d
Jan 5 12:48:53 testbox kernel: [653526.522504] ESI: f5d96200 EDI: f6d00450 EBP: db0f1cc8 ESP: db0f1cc8
Jan 5 12:48:53 testbox kernel: [653526.522504] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:48:53 testbox kernel: [653526.522504] CR0: 8005003b CR2: 00000000 CR3: 1b0d9000 CR4: 000006b0
Jan 5 12:48:53 testbox kernel: [653526.522504] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:48:53 testbox kernel: [653526.522504] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:48:53 testbox kernel: [653526.522504] Call Trace:
Jan 5 12:48:53 testbox kernel: [653526.522504] [] _spin_lock+0x8/0x10
Jan 5 12:48:53 testbox kernel: [653526.522504] [] journal_dirty_data+0x5e/0x1f0
Jan 5 12:48:53 testbox kernel: [653526.522504] [] ext3_journal_dirty_data+0x18/0x50
Jan 5 12:48:53 testbox kernel: [653526.522504] [] walk_page_buffers+0x2f/0x80
Jan 5 12:48:53 testbox kernel: [653526.522504] [] ext3_ordered_write_end+0x63/0x1b0
Jan 5 12:48:53 testbox kernel: [653526.522504] [] ? ext3_journal_dirty_data+0x0/0x50
Jan 5 12:48:53 testbox kernel: [653526.522504] [] ? iov_iter_copy_from_user_atomic+0x5d/0xc0
Jan 5 12:48:53 testbox kernel: [653526.522504] [] generic_perform_write+0x10f/0x190
Jan 5 12:48:53 testbox kernel: [653526.522504] [] generic_file_buffered_write+0x5d/0x120
Jan 5 12:48:53 testbox kernel: [653526.522504] [] __generic_file_aio_write_nolock+0x19b/0x520
Jan 5 12:48:53 testbox kernel: [653526.522504] [] ? resched_task+0x1e/0x70
Jan 5 12:48:53 testbox kernel: [653526.522504] [] generic_file_aio_write+0x5e/0xd0
Jan 5 12:48:53 testbox kernel: [653526.522504] [] ext3_file_write+0x2d/0xc0
Jan 5 12:48:53 testbox kernel: [653526.522504] [] do_sync_write+0xd1/0x110
Jan 5 12:48:53 testbox kernel: [653526.522504] [] ? rtc_update_irq+0x92/0xa0
Jan 5 12:48:53 testbox kernel: [653526.522504] [] ? autoremove_wake_function+0x0/0x50
Jan 5 12:48:53 testbox kernel: [653526.522504] [] ? apparmor_file_permission+0x20/0x30
Jan 5 12:48:53 testbox kernel: [653526.522504] [] ? security_file_permission+0xf/0x20
Jan 5 12:48:53 testbox kernel: [653526.522504] [] ? rw_verify_area+0x54/0xd0
Jan 5 12:48:53 testbox kernel: [653526.522504] [] vfs_write+0x98/0x110
Jan 5 12:48:53 testbox kernel: [653526.522504] [] ? do_sync_write+0x0/0x110
Jan 5 12:48:53 testbox kernel: [653526.522504] [] sys_write+0x3d/0x70
Jan 5 12:48:53 testbox kernel: [653526.522504] [] sysenter_do_call+0x12/0x2f
Jan 5 12:48:53 testbox kernel: [653526.526518] hpet1: lost 3 rtc interrupts
Jan 5 12:49:27 testbox kernel: [653560.880003] BUG: soft lockup - CPU#0 stuck for 61s! [hal-acl-tool:4168]
Jan 5 12:49:27 testbox kernel: [653560.880005] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:49:27 testbox kernel: [653560.881694]
Jan 5 12:49:27 testbox kernel: [653560.881694] Pid: 4168, comm: hal-acl-tool Tainted: G D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:49:27 testbox kernel: [653560.881694] EIP: 0060:[] EFLAGS: 00000293 CPU: 0
Jan 5 12:49:27 testbox kernel: [653560.881694] EIP is at __ticket_spin_lock+0xf/0x20
Jan 5 12:49:27 testbox kernel: [653560.881694] EAX: f5d962b8 EBX: d6ba3b00 ECX: f5d96200 EDX: 00004f4d
Jan 5 12:49:27 testbox kernel: [653560.881694] ESI: d20d3000 EDI: f5d962b8 EBP: e0491c60 ESP: e0491c60
Jan 5 12:49:27 testbox kernel: [653560.881694] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:49:27 testbox kernel: [653560.881694] CR0: 8005003b CR2: 09202568 CR3: 2a445000 CR4: 000006b0
Jan 5 12:49:27 testbox kernel: [653560.881694] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:49:27 testbox kernel: [653560.881694] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:49:27 testbox kernel: [653560.881694] Call Trace:
Jan 5 12:49:27 testbox kernel: [653560.881694] [] _spin_lock+0x8/0x10
Jan 5 12:49:27 testbox kernel: [653560.881694] [] __journal_try_to_free_buffer+0x4d/0xc0
Jan 5 12:49:27 testbox kernel: [653560.881694] [] journal_try_to_free_buffers+0x62/0x100
Jan 5 12:49:27 testbox kernel: [653560.881694] [] ext3_releasepage+0x45/0x70
Jan 5 12:49:27 testbox kernel: [653560.881694] [] try_to_release_page+0x2e/0x50
Jan 5 12:49:27 testbox kernel: [653560.881694] [] __invalidate_mapping_pages+0x136/0x150
Jan 5 12:49:27 testbox kernel: [653560.881694] [] invalidate_mapping_pages+0x12/0x20
Jan 5 12:49:27 testbox kernel: [653560.881694] [] prune_icache+0x21a/0x260
Jan 5 12:49:27 testbox kernel: [653560.881694] [] shrink_icache_memory+0x3e/0x50
Jan 5 12:49:27 testbox kernel: [653560.881694] [] shrink_slab+0x11f/0x180
Jan 5 12:49:27 testbox kernel: [653560.881694] [] do_try_to_free_pages+0x155/0x2c0
Jan 5 12:49:27 testbox kernel: [653560.881694] [] ? autoremove_wake_function+0x1b/0x50
Jan 5 12:49:27 testbox kernel: [653560.881694] [] try_to_free_pages+0x6c/0x80
Jan 5 12:49:27 testbox kernel: [653560.881694] [] ? isolate_pages_global+0x0/0x70
Jan 5 12:49:27 testbox kernel: [653560.881694] [] __alloc_pages_internal+0x1cb/0x450
Jan 5 12:49:27 testbox kernel: [653560.881694] [] new_slab+0x7b/0x1e0
Jan 5 12:49:27 testbox kernel: [653560.881694] [] ? unfreeze_slab+0x75/0xb0
Jan 5 12:49:27 testbox kernel: [653560.881694] [] __slab_alloc+0x20b/0x260
Jan 5 12:49:27 testbox kernel: [653560.881694] [] kmem_cache_alloc+0xa6/0xb0
Jan 5 12:49:27 testbox kernel: [653560.881694] [] ? dup_task_struct+0x1c/0x110
Jan 5 12:49:27 testbox kernel: [653560.881694] [] dup_task_struct+0x1c/0x110
Jan 5 12:49:27 testbox kernel: [653560.881694] [] copy_process+0x6e/0xcf0
Jan 5 12:49:27 testbox kernel: [653560.881694] [] ? alloc_fd+0xe0/0x100
Jan 5 12:49:27 testbox kernel: [653560.881694] [] ? get_empty_filp+0x6c/0x170
Jan 5 12:49:27 testbox kernel: [653560.881694] [] ? _spin_lock+0x8/0x10
Jan 5 12:49:27 testbox kernel: [653560.881694] [] do_fork+0x62/0x300
Jan 5 12:49:27 testbox kernel: [653560.881694] [] ? sys_pipe2+0x3f/0x60
Jan 5 12:49:27 testbox kernel: [653560.881694] [] sys_clone+0x2f/0x40
Jan 5 12:49:27 testbox kernel: [653560.881694] [] sysenter_do_call+0x12/0x2f
Jan 5 12:49:59 testbox kernel: [653592.022503] BUG: soft lockup - CPU#3 stuck for 61s! [rsync:30532]
Jan 5 12:49:59 testbox kernel: [653592.022504] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:49:59 testbox kernel: [653592.022504]
Jan 5 12:49:59 testbox kernel: [653592.022504] Pid: 30532, comm: rsync Tainted: G D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:49:59 testbox kernel: [653592.022504] EIP: 0060:[] EFLAGS: 00000297 CPU: 3
Jan 5 12:49:59 testbox kernel: [653592.022504] EIP is at __ticket_spin_lock+0x15/0x20
Jan 5 12:49:59 testbox kernel: [653592.022504] EAX: f5d962b8 EBX: e0c8de00 ECX: 00000000 EDX: 00004e4d
Jan 5 12:49:59 testbox kernel: [653592.022504] ESI: f5d96200 EDI: f6d00450 EBP: db0f1cc8 ESP: db0f1cc8
Jan 5 12:49:59 testbox kernel: [653592.022504] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:49:59 testbox kernel: [653592.022504] CR0: 8005003b CR2: 00000000 CR3: 1b0d9000 CR4: 000006b0
Jan 5 12:49:59 testbox kernel: [653592.022504] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:49:59 testbox kernel: [653592.022504] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:49:59 testbox kernel: [653592.022504] Call Trace:
Jan 5 12:49:59 testbox kernel: [653592.022504] [] _spin_lock+0x8/0x10
Jan 5 12:49:59 testbox kernel: [653592.022504] [] journal_dirty_data+0x5e/0x1f0
Jan 5 12:49:59 testbox kernel: [653592.022504] [] ext3_journal_dirty_data+0x18/0x50
Jan 5 12:49:59 testbox kernel: [653592.022504] [] walk_page_buffers+0x2f/0x80
Jan 5 12:49:59 testbox kernel: [653592.022504] [] ext3_ordered_write_end+0x63/0x1b0
Jan 5 12:49:59 testbox kernel: [653592.022504] [] ? ext3_journal_dirty_data+0x0/0x50
Jan 5 12:49:59 testbox kernel: [653592.022504] [] ? iov_iter_copy_from_user_atomic+0x5d/0xc0
Jan 5 12:49:59 testbox kernel: [653592.022504] [] generic_perform_write+0x10f/0x190
Jan 5 12:49:59 testbox kernel: [653592.022504] [] generic_file_buffered_write+0x5d/0x120
Jan 5 12:49:59 testbox kernel: [653592.022504] [] __generic_file_aio_write_nolock+0x19b/0x520
Jan 5 12:49:59 testbox kernel: [653592.022504] [] ? resched_task+0x1e/0x70
Jan 5 12:49:59 testbox kernel: [653592.022504] [] generic_file_aio_write+0x5e/0xd0
Jan 5 12:49:59 testbox kernel: [653592.022504] [] ext3_file_write+0x2d/0xc0
Jan 5 12:49:59 testbox kernel: [653592.022504] [] do_sync_write+0xd1/0x110
Jan 5 12:49:59 testbox kernel: [653592.022504] [] ? rtc_update_irq+0x92/0xa0
Jan 5 12:49:59 testbox kernel: [653592.022504] [] ? autoremove_wake_function+0x0/0x50
Jan 5 12:49:59 testbox kernel: [653592.022504] [] ? apparmor_file_permission+0x20/0x30
Jan 5 12:49:59 testbox kernel: [653592.022504] [] ? security_file_permission+0xf/0x20
Jan 5 12:49:59 testbox kernel: [653592.022504] [] ? rw_verify_area+0x54/0xd0
Jan 5 12:49:59 testbox kernel: [653592.022504] [] vfs_write+0x98/0x110
Jan 5 12:49:59 testbox kernel: [653592.022504] [] ? do_sync_write+0x0/0x110
Jan 5 12:49:59 testbox kernel: [653592.022504] [] sys_write+0x3d/0x70
Jan 5 12:49:59 testbox kernel: [653592.022504] [] sysenter_do_call+0x12/0x2f
Jan 5 12:49:59 testbox kernel: [653592.025918] hpet1: lost 2 rtc interrupts
Jan 5 12:50:33 testbox kernel: [653626.380003] BUG: soft lockup - CPU#0 stuck for 61s! [hal-acl-tool:4168]
Jan 5 12:50:33 testbox kernel: [653626.380005] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:50:33 testbox kernel: [653626.381684]
Jan 5 12:50:33 testbox kernel: [653626.381684] Pid: 4168, comm: hal-acl-tool Tainted: G D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:50:33 testbox kernel: [653626.381684] EIP: 0060:[] EFLAGS: 00000293 CPU: 0
Jan 5 12:50:33 testbox kernel: [653626.381684] EIP is at __ticket_spin_lock+0x11/0x20
Jan 5 12:50:33 testbox kernel: [653626.381684] EAX: f5d962b8 EBX: d6ba3b00 ECX: f5d96200 EDX: 00004f4d
Jan 5 12:50:33 testbox kernel: [653626.381684] ESI: d20d3000 EDI: f5d962b8 EBP: e0491c60 ESP: e0491c60
Jan 5 12:50:33 testbox kernel: [653626.381684] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:50:33 testbox kernel: [653626.381684] CR0: 8005003b CR2: 09202568 CR3: 2a445000 CR4: 000006b0
Jan 5 12:50:33 testbox kernel: [653626.381684] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:50:33 testbox kernel: [653626.381684] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:50:33 testbox kernel: [653626.381684] Call Trace:
Jan 5 12:50:33 testbox kernel: [653626.381684] [] _spin_lock+0x8/0x10
Jan 5 12:50:33 testbox kernel: [653626.381684] [] __journal_try_to_free_buffer+0x4d/0xc0
Jan 5 12:50:33 testbox kernel: [653626.381684] [] journal_try_to_free_buffers+0x62/0x100
Jan 5 12:50:33 testbox kernel: [653626.381684] [] ext3_releasepage+0x45/0x70
Jan 5 12:50:33 testbox kernel: [653626.381684] [] try_to_release_page+0x2e/0x50
Jan 5 12:50:33 testbox kernel: [653626.381684] [] __invalidate_mapping_pages+0x136/0x150
Jan 5 12:50:33 testbox kernel: [653626.381684] [] invalidate_mapping_pages+0x12/0x20
Jan 5 12:50:33 testbox kernel: [653626.381684] [] prune_icache+0x21a/0x260
Jan 5 12:50:33 testbox kernel: [653626.381684] [] shrink_icache_memory+0x3e/0x50
Jan 5 12:50:33 testbox kernel: [653626.381684] [] shrink_slab+0x11f/0x180
Jan 5 12:50:33 testbox kernel: [653626.381684] [] do_try_to_free_pages+0x155/0x2c0
Jan 5 12:50:33 testbox kernel: [653626.381684] [] ? autoremove_wake_function+0x1b/0x50
Jan 5 12:50:33 testbox kernel: [653626.381684] [] try_to_free_pages+0x6c/0x80
Jan 5 12:50:33 testbox kernel: [653626.381684] [] ? isolate_pages_global+0x0/0x70
Jan 5 12:50:33 testbox kernel: [653626.381684] [] __alloc_pages_internal+0x1cb/0x450
Jan 5 12:50:33 testbox kernel: [653626.381684] [] new_slab+0x7b/0x1e0
Jan 5 12:50:33 testbox kernel: [653626.381684] [] ? unfreeze_slab+0x75/0xb0
Jan 5 12:50:33 testbox kernel: [653626.381684] [] __slab_alloc+0x20b/0x260
Jan 5 12:50:33 testbox kernel: [653626.381684] [] kmem_cache_alloc+0xa6/0xb0
Jan 5 12:50:33 testbox kernel: [653626.381684] [] ? dup_task_struct+0x1c/0x110
Jan 5 12:50:33 testbox kernel: [653626.381684] [] dup_task_struct+0x1c/0x110
Jan 5 12:50:33 testbox kernel: [653626.381684] [] copy_process+0x6e/0xcf0
Jan 5 12:50:33 testbox kernel: [653626.381684] [] ? alloc_fd+0xe0/0x100
Jan 5 12:50:33 testbox kernel: [653626.381684] [] ? get_empty_filp+0x6c/0x170
Jan 5 12:50:33 testbox kernel: [653626.381684] [] ? _spin_lock+0x8/0x10
Jan 5 12:50:33 testbox kernel: [653626.381684] [] do_fork+0x62/0x300
Jan 5 12:50:33 testbox kernel: [653626.381684] [] ? sys_pipe2+0x3f/0x60
Jan 5 12:50:33 testbox kernel: [653626.381684] [] sys_clone+0x2f/0x40
Jan 5 12:50:33 testbox kernel: [653626.381684] [] sysenter_do_call+0x12/0x2f
Jan 5 12:51:04 testbox kernel: [653657.522503] BUG: soft lockup - CPU#3 stuck for 61s! [rsync:30532]
Jan 5 12:51:04 testbox kernel: [653657.522504] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:51:04 testbox kernel: [653657.522504]
Jan 5 12:51:04 testbox kernel: [653657.522504] Pid: 30532, comm: rsync Tainted: G D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:51:04 testbox kernel: [653657.522504] EIP: 0060:[] EFLAGS: 00000297 CPU: 3
Jan 5 12:51:04 testbox kernel: [653657.522504] EIP is at __ticket_spin_lock+0x13/0x20
Jan 5 12:51:04 testbox kernel: [653657.522504] EAX: f5d962b8 EBX: e0c8de00 ECX: 00000000 EDX: 00004e4d
Jan 5 12:51:04 testbox kernel: [653657.522504] ESI: f5d96200 EDI: f6d00450 EBP: db0f1cc8 ESP: db0f1cc8
Jan 5 12:51:04 testbox kernel: [653657.522504] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:51:04 testbox kernel: [653657.522504] CR0: 8005003b CR2: 00000000 CR3: 1b0d9000 CR4: 000006b0
Jan 5 12:51:04 testbox kernel: [653657.522504] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:51:04 testbox kernel: [653657.522504] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:51:04 testbox kernel: [653657.522504] Call Trace:
Jan 5 12:51:04 testbox kernel: [653657.522504] [] _spin_lock+0x8/0x10
Jan 5 12:51:04 testbox kernel: [653657.522504] [] journal_dirty_data+0x5e/0x1f0
Jan 5 12:51:04 testbox kernel: [653657.522504] [] ext3_journal_dirty_data+0x18/0x50
Jan 5 12:51:04 testbox kernel: [653657.522504] [] walk_page_buffers+0x2f/0x80
Jan 5 12:51:04 testbox kernel: [653657.522504] [] ext3_ordered_write_end+0x63/0x1b0
Jan 5 12:51:04 testbox kernel: [653657.522504] [] ? ext3_journal_dirty_data+0x0/0x50
Jan 5 12:51:04 testbox kernel: [653657.522504] [] ? iov_iter_copy_from_user_atomic+0x5d/0xc0
Jan 5 12:51:04 testbox kernel: [653657.522504] [] generic_perform_write+0x10f/0x190
Jan 5 12:51:04 testbox kernel: [653657.522504] [] generic_file_buffered_write+0x5d/0x120
Jan 5 12:51:04 testbox kernel: [653657.522504] [] __generic_file_aio_write_nolock+0x19b/0x520
Jan 5 12:51:04 testbox kernel: [653657.522504] [] ? resched_task+0x1e/0x70
Jan 5 12:51:04 testbox kernel: [653657.522504] [] generic_file_aio_write+0x5e/0xd0
Jan 5 12:51:04 testbox kernel: [653657.522504] [] ext3_file_write+0x2d/0xc0
Jan 5 12:51:04 testbox kernel: [653657.522504] [] do_sync_write+0xd1/0x110
Jan 5 12:51:04 testbox kernel: [653657.522504] [] ? rtc_update_irq+0x92/0xa0
Jan 5 12:51:04 testbox kernel: [653657.522504] [] ? autoremove_wake_function+0x0/0x50
Jan 5 12:51:04 testbox kernel: [653657.522504] [] ? apparmor_file_permission+0x20/0x30
Jan 5 12:51:04 testbox kernel: [653657.522504] [] ? security_file_permission+0xf/0x20
Jan 5 12:51:04 testbox kernel: [653657.522504] [] ? rw_verify_area+0x54/0xd0
Jan 5 12:51:04 testbox kernel: [653657.522504] [] vfs_write+0x98/0x110
Jan 5 12:51:04 testbox kernel: [653657.522504] [] ? do_sync_write+0x0/0x110
Jan 5 12:51:04 testbox kernel: [653657.522504] [] sys_write+0x3d/0x70
Jan 5 12:51:04 testbox kernel: [653657.522504] [] sysenter_do_call+0x12/0x2f
Jan 5 12:51:04 testbox kernel: [653657.526512] hpet1: lost 3 rtc interrupts
Jan 5 12:51:38 testbox kernel: [653691.880004] BUG: soft lockup - CPU#0 stuck for 61s! [hal-acl-tool:4168]
Jan 5 12:51:38 testbox kernel: [653691.880005] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:51:38 testbox kernel: [653691.880859]
Jan 5 12:51:38 testbox kernel: [653691.880859] Pid: 4168, comm: hal-acl-tool Tainted: G D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:51:38 testbox kernel: [653691.880859] EIP: 0060:[] EFLAGS: 00000293 CPU: 0
Jan 5 12:51:38 testbox kernel: [653691.880859] EIP is at __ticket_spin_lock+0x15/0x20
Jan 5 12:51:38 testbox kernel: [653691.880859] EAX: f5d962b8 EBX: d6ba3b00 ECX: f5d96200 EDX: 00004f4d
Jan 5 12:51:38 testbox kernel: [653691.880859] ESI: d20d3000 EDI: f5d962b8 EBP: e0491c60 ESP: e0491c60
Jan 5 12:51:38 testbox kernel: [653691.880859] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:51:38 testbox kernel: [653691.880859] CR0: 8005003b CR2: 09202568 CR3: 2a445000 CR4: 000006b0
Jan 5 12:51:38 testbox kernel: [653691.880859] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:51:38 testbox kernel: [653691.880859] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:51:38 testbox kernel: [653691.880859] Call Trace:
Jan 5 12:51:38 testbox kernel: [653691.880859] [] _spin_lock+0x8/0x10
Jan 5 12:51:38 testbox kernel: [653691.880859] [] __journal_try_to_free_buffer+0x4d/0xc0
Jan 5 12:51:38 testbox kernel: [653691.880859] [] journal_try_to_free_buffers+0x62/0x100
Jan 5 12:51:38 testbox kernel: [653691.880859] [] ext3_releasepage+0x45/0x70
Jan 5 12:51:38 testbox kernel: [653691.880859] [] try_to_release_page+0x2e/0x50
Jan 5 12:51:38 testbox kernel: [653691.880859] [] __invalidate_mapping_pages+0x136/0x150
Jan 5 12:51:38 testbox kernel: [653691.880859] [] invalidate_mapping_pages+0x12/0x20
Jan 5 12:51:38 testbox kernel: [653691.880859] [] prune_icache+0x21a/0x260
Jan 5 12:51:38 testbox kernel: [653691.880859] [] shrink_icache_memory+0x3e/0x50
Jan 5 12:51:38 testbox kernel: [653691.880859] [] shrink_slab+0x11f/0x180
Jan 5 12:51:38 testbox kernel: [653691.880859] [] do_try_to_free_pages+0x155/0x2c0
Jan 5 12:51:38 testbox kernel: [653691.880859] [] ? autoremove_wake_function+0x1b/0x50
Jan 5 12:51:38 testbox kernel: [653691.880859] [] try_to_free_pages+0x6c/0x80
Jan 5 12:51:38 testbox kernel: [653691.880859] [] ? isolate_pages_global+0x0/0x70
Jan 5 12:51:38 testbox kernel: [653691.880859] [] __alloc_pages_internal+0x1cb/0x450
Jan 5 12:51:38 testbox kernel: [653691.880859] [] new_slab+0x7b/0x1e0
Jan 5 12:51:38 testbox kernel: [653691.880859] [] ? unfreeze_slab+0x75/0xb0
Jan 5 12:51:38 testbox kernel: [653691.880859] [] __slab_alloc+0x20b/0x260
Jan 5 12:51:38 testbox kernel: [653691.880859] [] kmem_cache_alloc+0xa6/0xb0
Jan 5 12:51:38 testbox kernel: [653691.880859] [] ? dup_task_struct+0x1c/0x110
Jan 5 12:51:38 testbox kernel: [653691.880859] [] dup_task_struct+0x1c/0x110
Jan 5 12:51:38 testbox kernel: [653691.880859] [] copy_process+0x6e/0xcf0
Jan 5 12:51:38 testbox kernel: [653691.880859] [] ? alloc_fd+0xe0/0x100
Jan 5 12:51:38 testbox kernel: [653691.880859] [] ? get_empty_filp+0x6c/0x170
Jan 5 12:51:38 testbox kernel: [653691.880859] [] ? _spin_lock+0x8/0x10
Jan 5 12:51:38 testbox kernel: [653691.880859] [] do_fork+0x62/0x300
Jan 5 12:51:38 testbox kernel: [653691.880859] [] ? sys_pipe2+0x3f/0x60
Jan 5 12:51:38 testbox kernel: [653691.880859] [] sys_clone+0x2f/0x40
Jan 5 12:51:38 testbox kernel: [653691.880859] [] sysenter_do_call+0x12/0x2f
Jan 5 12:52:10 testbox kernel: [653723.022504] BUG: soft lockup - CPU#3 stuck for 61s! [rsync:30532]
Jan 5 12:52:10 testbox kernel: [653723.022504] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:52:10 testbox kernel: [653723.022504]
Jan 5 12:52:10 testbox kernel: [653723.022504] Pid: 30532, comm: rsync Tainted: G D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:52:10 testbox kernel: [653723.022504] EIP: 0060:[] EFLAGS: 00000297 CPU: 3
Jan 5 12:52:10 testbox kernel: [653723.022504] EIP is at __ticket_spin_lock+0x13/0x20
Jan 5 12:52:10 testbox kernel: [653723.022504] EAX: f5d962b8 EBX: e0c8de00 ECX: 00000000 EDX: 00004e4d
Jan 5 12:52:10 testbox kernel: [653723.022504] ESI: f5d96200 EDI: f6d00450 EBP: db0f1cc8 ESP: db0f1cc8
Jan 5 12:52:10 testbox kernel: [653723.022504] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:52:10 testbox kernel: [653723.022504] CR0: 8005003b CR2: 00000000 CR3: 1b0d9000 CR4: 000006b0
Jan 5 12:52:10 testbox kernel: [653723.022504] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:52:10 testbox kernel: [653723.022504] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:52:10 testbox kernel: [653723.022504] Call Trace:
Jan 5 12:52:10 testbox kernel: [653723.022504] [] _spin_lock+0x8/0x10
Jan 5 12:52:10 testbox kernel: [653723.022504] [] journal_dirty_data+0x5e/0x1f0
Jan 5 12:52:10 testbox kernel: [653723.022504] [] ext3_journal_dirty_data+0x18/0x50
Jan 5 12:52:10 testbox kernel: [653723.022504] [] walk_page_buffers+0x2f/0x80
Jan 5 12:52:10 testbox kernel: [653723.022504] [] ext3_ordered_write_end+0x63/0x1b0
Jan 5 12:52:10 testbox kernel: [653723.022504] [] ? ext3_journal_dirty_data+0x0/0x50
Jan 5 12:52:10 testbox kernel: [653723.022504] [] ? iov_iter_copy_from_user_atomic+0x5d/0xc0
Jan 5 12:52:10 testbox kernel: [653723.022504] [] generic_perform_write+0x10f/0x190
Jan 5 12:52:10 testbox kernel: [653723.022504] [] generic_file_buffered_write+0x5d/0x120
Jan 5 12:52:10 testbox kernel: [653723.022504] [] __generic_file_aio_write_nolock+0x19b/0x520
Jan 5 12:52:10 testbox kernel: [653723.022504] [] ? resched_task+0x1e/0x70
Jan 5 12:52:10 testbox kernel: [653723.022504] [] generic_file_aio_write+0x5e/0xd0
Jan 5 12:52:10 testbox kernel: [653723.022504] [] ext3_file_write+0x2d/0xc0
Jan 5 12:52:10 testbox kernel: [653723.022504] [] do_sync_write+0xd1/0x110
Jan 5 12:52:10 testbox kernel: [653723.022504] [] ? rtc_update_irq+0x92/0xa0
Jan 5 12:52:10 testbox kernel: [653723.022504] [] ? autoremove_wake_function+0x0/0x50
Jan 5 12:52:10 testbox kernel: [653723.022504] [] ? apparmor_file_permission+0x20/0x30
Jan 5 12:52:10 testbox kernel: [653723.022504] [] ? security_file_permission+0xf/0x20
Jan 5 12:52:10 testbox kernel: [653723.022504] [] ? rw_verify_area+0x54/0xd0
Jan 5 12:52:10 testbox kernel: [653723.022504] [] vfs_write+0x98/0x110
Jan 5 12:52:10 testbox kernel: [653723.022504] [] ? do_sync_write+0x0/0x110
Jan 5 12:52:10 testbox kernel: [653723.022504] [] sys_write+0x3d/0x70
Jan 5 12:52:10 testbox kernel: [653723.022504] [] sysenter_do_call+0x12/0x2f
Jan 5 12:52:10 testbox kernel: [653723.025920] hpet1: lost 3 rtc interrupts
Jan 5 12:52:44 testbox kernel: [653757.380003] BUG: soft lockup - CPU#0 stuck for 61s! [hal-acl-tool:4168]
Jan 5 12:52:44 testbox kernel: [653757.380005] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:52:44 testbox kernel: [653757.381694]
Jan 5 12:52:44 testbox kernel: [653757.381694] Pid: 4168, comm: hal-acl-tool Tainted: G D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:52:44 testbox kernel: [653757.381694] EIP: 0060:[] EFLAGS: 00000293 CPU: 0
Jan 5 12:52:44 testbox kernel: [653757.381694] EIP is at __ticket_spin_lock+0xf/0x20
Jan 5 12:52:44 testbox kernel: [653757.381694] EAX: f5d962b8 EBX: d6ba3b00 ECX: f5d96200 EDX: 00004f4d
Jan 5 12:52:44 testbox kernel: [653757.381694] ESI: d20d3000 EDI: f5d962b8 EBP: e0491c60 ESP: e0491c60
Jan 5 12:52:44 testbox kernel: [653757.381694] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:52:44 testbox kernel: [653757.381694] CR0: 8005003b CR2: 09202568 CR3: 2a445000 CR4: 000006b0
Jan 5 12:52:44 testbox kernel: [653757.381694] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:52:44 testbox kernel: [653757.381694] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:52:44 testbox kernel: [653757.381694] Call Trace:
Jan 5 12:52:44 testbox kernel: [653757.381694] [] _spin_lock+0x8/0x10
Jan 5 12:52:44 testbox kernel: [653757.381694] [] __journal_try_to_free_buffer+0x4d/0xc0
Jan 5 12:52:44 testbox kernel: [653757.381694] [] journal_try_to_free_buffers+0x62/0x100
Jan 5 12:52:44 testbox kernel: [653757.381694] [] ext3_releasepage+0x45/0x70
Jan 5 12:52:44 testbox kernel: [653757.381694] [] try_to_release_page+0x2e/0x50
Jan 5 12:52:44 testbox kernel: [653757.381694] [] __invalidate_mapping_pages+0x136/0x150
Jan 5 12:52:44 testbox kernel: [653757.381694] [] invalidate_mapping_pages+0x12/0x20
Jan 5 12:52:44 testbox kernel: [653757.381694] [] prune_icache+0x21a/0x260
Jan 5 12:52:44 testbox kernel: [653757.381694] [] shrink_icache_memory+0x3e/0x50
Jan 5 12:52:44 testbox kernel: [653757.381694] [] shrink_slab+0x11f/0x180
Jan 5 12:52:44 testbox kernel: [653757.381694] [] do_try_to_free_pages+0x155/0x2c0
Jan 5 12:52:44 testbox kernel: [653757.381694] [] ? autoremove_wake_function+0x1b/0x50
Jan 5 12:52:44 testbox kernel: [653757.381694] [] try_to_free_pages+0x6c/0x80
Jan 5 12:52:44 testbox kernel: [653757.381694] [] ? isolate_pages_global+0x0/0x70
Jan 5 12:52:44 testbox kernel: [653757.381694] [] __alloc_pages_internal+0x1cb/0x450
Jan 5 12:52:44 testbox kernel: [653757.381694] [] new_slab+0x7b/0x1e0
Jan 5 12:52:44 testbox kernel: [653757.381694] [] ? unfreeze_slab+0x75/0xb0
Jan 5 12:52:44 testbox kernel: [653757.381694] [] __slab_alloc+0x20b/0x260
Jan 5 12:52:44 testbox kernel: [653757.381694] [] kmem_cache_alloc+0xa6/0xb0
Jan 5 12:52:44 testbox kernel: [653757.381694] [] ? dup_task_struct+0x1c/0x110
Jan 5 12:52:44 testbox kernel: [653757.381694] [] dup_task_struct+0x1c/0x110
Jan 5 12:52:44 testbox kernel: [653757.381694] [] copy_process+0x6e/0xcf0
Jan 5 12:52:44 testbox kernel: [653757.381694] [] ? alloc_fd+0xe0/0x100
Jan 5 12:52:44 testbox kernel: [653757.381694] [] ? get_empty_filp+0x6c/0x170
Jan 5 12:52:44 testbox kernel: [653757.381694] [] ? _spin_lock+0x8/0x10
Jan 5 12:52:44 testbox kernel: [653757.381694] [] do_fork+0x62/0x300
Jan 5 12:52:44 testbox kernel: [653757.381694] [] ? sys_pipe2+0x3f/0x60
Jan 5 12:52:44 testbox kernel: [653757.381694] [] sys_clone+0x2f/0x40
Jan 5 12:52:44 testbox kernel: [653757.381694] [] sysenter_do_call+0x12/0x2f
Jan 5 12:53:15 testbox kernel: [653788.512503] BUG: soft lockup - CPU#3 stuck for 61s! [rsync:30532]
Jan 5 12:53:15 testbox kernel: [653788.512504] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:53:15 testbox kernel: [653788.512504]
Jan 5 12:53:15 testbox kernel: [653788.512504] Pid: 30532, comm: rsync Tainted: G D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:53:15 testbox kernel: [653788.512504] EIP: 0060:[] EFLAGS: 00000297 CPU: 3
Jan 5 12:53:15 testbox kernel: [653788.512504] EIP is at __ticket_spin_lock+0x11/0x20
Jan 5 12:53:15 testbox kernel: [653788.512504] EAX: f5d962b8 EBX: e0c8de00 ECX: 00000000 EDX: 00004e4d
Jan 5 12:53:15 testbox kernel: [653788.512504] ESI: f5d96200 EDI: f6d00450 EBP: db0f1cc8 ESP: db0f1cc8
Jan 5 12:53:15 testbox kernel: [653788.512504] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:53:15 testbox kernel: [653788.512504] CR0: 8005003b CR2: 00000000 CR3: 1b0d9000 CR4: 000006b0
Jan 5 12:53:15 testbox kernel: [653788.512504] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:53:15 testbox kernel: [653788.512504] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:53:15 testbox kernel: [653788.512504] Call Trace:
Jan 5 12:53:15 testbox kernel: [653788.512504] [] _spin_lock+0x8/0x10
Jan 5 12:53:15 testbox kernel: [653788.512504] [] journal_dirty_data+0x5e/0x1f0
Jan 5 12:53:15 testbox kernel: [653788.512504] [] ext3_journal_dirty_data+0x18/0x50
Jan 5 12:53:15 testbox kernel: [653788.512504] [] walk_page_buffers+0x2f/0x80
Jan 5 12:53:15 testbox kernel: [653788.512504] [] ext3_ordered_write_end+0x63/0x1b0
Jan 5 12:53:15 testbox kernel: [653788.512504] [] ? ext3_journal_dirty_data+0x0/0x50
Jan 5 12:53:15 testbox kernel: [653788.512504] [] ? iov_iter_copy_from_user_atomic+0x5d/0xc0
Jan 5 12:53:15 testbox kernel: [653788.512504] [] generic_perform_write+0x10f/0x190
Jan 5 12:53:15 testbox kernel: [653788.512504] [] generic_file_buffered_write+0x5d/0x120
Jan 5 12:53:15 testbox kernel: [653788.512504] [] __generic_file_aio_write_nolock+0x19b/0x520
Jan 5 12:53:15 testbox kernel: [653788.512504] [] ? resched_task+0x1e/0x70
Jan 5 12:53:15 testbox kernel: [653788.512504] [] generic_file_aio_write+0x5e/0xd0
Jan 5 12:53:15 testbox kernel: [653788.512504] [] ext3_file_write+0x2d/0xc0
Jan 5 12:53:15 testbox kernel: [653788.512504] [] do_sync_write+0xd1/0x110
Jan 5 12:53:15 testbox kernel: [653788.512504] [] ? rtc_update_irq+0x92/0xa0
Jan 5 12:53:15 testbox kernel: [653788.512504] [] ? autoremove_wake_function+0x0/0x50
Jan 5 12:53:15 testbox kernel: [653788.512504] [] ? apparmor_file_permission+0x20/0x30
Jan 5 12:53:15 testbox kernel: [653788.512504] [] ? security_file_permission+0xf/0x20
Jan 5 12:53:15 testbox kernel: [653788.512504] [] ? rw_verify_area+0x54/0xd0
Jan 5 12:53:15 testbox kernel: [653788.512504] [] vfs_write+0x98/0x110
Jan 5 12:53:15 testbox kernel: [653788.512504] [] ? do_sync_write+0x0/0x110
Jan 5 12:53:15 testbox kernel: [653788.512504] [] sys_write+0x3d/0x70
Jan 5 12:53:15 testbox kernel: [653788.512504] [] sysenter_do_call+0x12/0x2f
Jan 5 12:53:15 testbox kernel: [653788.516515] hpet1: lost 3 rtc interrupts
Jan 5 12:53:49 testbox kernel: [653822.880003] BUG: soft lockup - CPU#0 stuck for 61s! [hal-acl-tool:4168]
Jan 5 12:53:49 testbox kernel: [653822.880005] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:53:49 testbox kernel: [653822.881681]
Jan 5 12:53:49 testbox kernel: [653822.881681] Pid: 4168, comm: hal-acl-tool Tainted: G D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:53:49 testbox kernel: [653822.881681] EIP: 0060:[] EFLAGS: 00000293 CPU: 0
Jan 5 12:53:49 testbox kernel: [653822.881681] EIP is at __ticket_spin_lock+0xf/0x20
Jan 5 12:53:49 testbox kernel: [653822.881681] EAX: f5d962b8 EBX: d6ba3b00 ECX: f5d96200 EDX: 00004f4d
Jan 5 12:53:49 testbox kernel: [653822.881681] ESI: d20d3000 EDI: f5d962b8 EBP: e0491c60 ESP: e0491c60
Jan 5 12:53:49 testbox kernel: [653822.881681] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:53:49 testbox kernel: [653822.881681] CR0: 8005003b CR2: 09202568 CR3: 2a445000 CR4: 000006b0
Jan 5 12:53:49 testbox kernel: [653822.881681] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:53:49 testbox kernel: [653822.881681] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:53:49 testbox kernel: [653822.881681] Call Trace:
Jan 5 12:53:49 testbox kernel: [653822.881681] [] _spin_lock+0x8/0x10
Jan 5 12:53:49 testbox kernel: [653822.881681] [] __journal_try_to_free_buffer+0x4d/0xc0
Jan 5 12:53:49 testbox kernel: [653822.881681] [] journal_try_to_free_buffers+0x62/0x100
Jan 5 12:53:49 testbox kernel: [653822.881681] [] ext3_releasepage+0x45/0x70
Jan 5 12:53:49 testbox kernel: [653822.881681] [] try_to_release_page+0x2e/0x50
Jan 5 12:53:49 testbox kernel: [653822.881681] [] __invalidate_mapping_pages+0x136/0x150
Jan 5 12:53:49 testbox kernel: [653822.881681] [] invalidate_mapping_pages+0x12/0x20
Jan 5 12:53:49 testbox kernel: [653822.881681] [] prune_icache+0x21a/0x260
Jan 5 12:53:49 testbox kernel: [653822.881681] [] shrink_icache_memory+0x3e/0x50
Jan 5 12:53:49 testbox kernel: [653822.881681] [] shrink_slab+0x11f/0x180
Jan 5 12:53:49 testbox kernel: [653822.881681] [] do_try_to_free_pages+0x155/0x2c0
Jan 5 12:53:49 testbox kernel: [653822.881681] [] ? autoremove_wake_function+0x1b/0x50
Jan 5 12:53:49 testbox kernel: [653822.881681] [] try_to_free_pages+0x6c/0x80
Jan 5 12:53:49 testbox kernel: [653822.881681] [] ? isolate_pages_global+0x0/0x70
Jan 5 12:53:50 testbox kernel: [653822.881681] [] __alloc_pages_internal+0x1cb/0x450
Jan 5 12:53:50 testbox kernel: [653822.881681] [] new_slab+0x7b/0x1e0
Jan 5 12:53:50 testbox kernel: [653822.881681] [] ? unfreeze_slab+0x75/0xb0
Jan 5 12:53:50 testbox kernel: [653822.881681] [] __slab_alloc+0x20b/0x260
Jan 5 12:53:50 testbox kernel: [653822.881681] [] kmem_cache_alloc+0xa6/0xb0
Jan 5 12:53:50 testbox kernel: [653822.881681] [] ? dup_task_struct+0x1c/0x110
Jan 5 12:53:50 testbox kernel: [653822.881681] [] dup_task_struct+0x1c/0x110
Jan 5 12:53:50 testbox kernel: [653822.881681] [] copy_process+0x6e/0xcf0
Jan 5 12:53:50 testbox kernel: [653822.881681] [] ? alloc_fd+0xe0/0x100
Jan 5 12:53:50 testbox kernel: [653822.881681] [] ? get_empty_filp+0x6c/0x170
Jan 5 12:53:50 testbox kernel: [653822.881681] [] ? _spin_lock+0x8/0x10
Jan 5 12:53:50 testbox kernel: [653822.881681] [] do_fork+0x62/0x300
Jan 5 12:53:50 testbox kernel: [653822.881681] [] ? sys_pipe2+0x3f/0x60
Jan 5 12:53:50 testbox kernel: [653822.881681] [] sys_clone+0x2f/0x40
Jan 5 12:53:50 testbox kernel: [653822.881681] [] sysenter_do_call+0x12/0x2f
Jan 5 12:54:21 testbox kernel: [653854.012503] BUG: soft lockup - CPU#3 stuck for 61s! [rsync:30532]
Jan 5 12:54:21 testbox kernel: [653854.012504] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:54:21 testbox kernel: [653854.012504]
Jan 5 12:54:21 testbox kernel: [653854.012504] Pid: 30532, comm: rsync Tainted: G D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:54:21 testbox kernel: [653854.012504] EIP: 0060:[] EFLAGS: 00000297 CPU: 3
Jan 5 12:54:21 testbox kernel: [653854.012504] EIP is at __ticket_spin_lock+0x15/0x20
Jan 5 12:54:21 testbox kernel: [653854.012504] EAX: f5d962b8 EBX: e0c8de00 ECX: 00000000 EDX: 00004e4d
Jan 5 12:54:21 testbox kernel: [653854.012504] ESI: f5d96200 EDI: f6d00450 EBP: db0f1cc8 ESP: db0f1cc8
Jan 5 12:54:21 testbox kernel: [653854.012504] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:54:21 testbox kernel: [653854.012504] CR0: 8005003b CR2: 00000000 CR3: 1b0d9000 CR4: 000006b0
Jan 5 12:54:21 testbox kernel: [653854.012504] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:54:21 testbox kernel: [653854.012504] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:54:21 testbox kernel: [653854.012504] Call Trace:
Jan 5 12:54:21 testbox kernel: [653854.012504] [] _spin_lock+0x8/0x10
Jan 5 12:54:21 testbox kernel: [653854.012504] [] journal_dirty_data+0x5e/0x1f0
Jan 5 12:54:21 testbox kernel: [653854.012504] [] ext3_journal_dirty_data+0x18/0x50
Jan 5 12:54:21 testbox kernel: [653854.012504] [] walk_page_buffers+0x2f/0x80
Jan 5 12:54:21 testbox kernel: [653854.012504] [] ext3_ordered_write_end+0x63/0x1b0
Jan 5 12:54:21 testbox kernel: [653854.012504] [] ? ext3_journal_dirty_data+0x0/0x50
Jan 5 12:54:21 testbox kernel: [653854.012504] [] ? iov_iter_copy_from_user_atomic+0x5d/0xc0
Jan 5 12:54:21 testbox kernel: [653854.012504] [] generic_perform_write+0x10f/0x190
Jan 5 12:54:21 testbox kernel: [653854.012504] [] generic_file_buffered_write+0x5d/0x120
Jan 5 12:54:21 testbox kernel: [653854.012504] [] __generic_file_aio_write_nolock+0x19b/0x520
Jan 5 12:54:21 testbox kernel: [653854.012504] [] ? resched_task+0x1e/0x70
Jan 5 12:54:21 testbox kernel: [653854.012504] [] generic_file_aio_write+0x5e/0xd0
Jan 5 12:54:21 testbox kernel: [653854.012504] [] ext3_file_write+0x2d/0xc0
Jan 5 12:54:21 testbox kernel: [653854.012504] [] do_sync_write+0xd1/0x110
Jan 5 12:54:21 testbox kernel: [653854.012504] [] ? rtc_update_irq+0x92/0xa0
Jan 5 12:54:21 testbox kernel: [653854.012504] [] ? autoremove_wake_function+0x0/0x50
Jan 5 12:54:21 testbox kernel: [653854.012504] [] ? apparmor_file_permission+0x20/0x30
Jan 5 12:54:21 testbox kernel: [653854.012504] [] ? security_file_permission+0xf/0x20
Jan 5 12:54:21 testbox kernel: [653854.012504] [] ? rw_verify_area+0x54/0xd0
Jan 5 12:54:21 testbox kernel: [653854.012504] [] vfs_write+0x98/0x110
Jan 5 12:54:21 testbox kernel: [653854.012504] [] ? do_sync_write+0x0/0x110
Jan 5 12:54:21 testbox kernel: [653854.012504] [] sys_write+0x3d/0x70
Jan 5 12:54:21 testbox kernel: [653854.012504] [] sysenter_do_call+0x12/0x2f
Jan 5 12:54:21 testbox kernel: [653854.015940] hpet1: lost 2 rtc interrupts
Jan 5 12:54:55 testbox kernel: [653888.370004] BUG: soft lockup - CPU#0 stuck for 61s! [hal-acl-tool:4168]
Jan 5 12:54:55 testbox kernel: [653888.370005] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:54:55 testbox kernel: [653888.371683]
Jan 5 12:54:55 testbox kernel: [653888.371683] Pid: 4168, comm: hal-acl-tool Tainted: G D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:54:55 testbox kernel: [653888.371683] EIP: 0060:[] EFLAGS: 00000293 CPU: 0
Jan 5 12:54:55 testbox kernel: [653888.371683] EIP is at __ticket_spin_lock+0xf/0x20
Jan 5 12:54:55 testbox kernel: [653888.371683] EAX: f5d962b8 EBX: d6ba3b00 ECX: f5d96200 EDX: 00004f4d
Jan 5 12:54:55 testbox kernel: [653888.371683] ESI: d20d3000 EDI: f5d962b8 EBP: e0491c60 ESP: e0491c60
Jan 5 12:54:55 testbox kernel: [653888.371683] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:54:55 testbox kernel: [653888.371683] CR0: 8005003b CR2: 09202568 CR3: 2a445000 CR4: 000006b0
Jan 5 12:54:55 testbox kernel: [653888.371683] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:54:55 testbox kernel: [653888.371683] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:54:55 testbox kernel: [653888.371683] Call Trace:
Jan 5 12:54:55 testbox kernel: [653888.371683] [] _spin_lock+0x8/0x10
Jan 5 12:54:55 testbox kernel: [653888.371683] [] __journal_try_to_free_buffer+0x4d/0xc0
Jan 5 12:54:55 testbox kernel: [653888.371683] [] journal_try_to_free_buffers+0x62/0x100
Jan 5 12:54:55 testbox kernel: [653888.371683] [] ext3_releasepage+0x45/0x70
Jan 5 12:54:55 testbox kernel: [653888.371683] [] try_to_release_page+0x2e/0x50
Jan 5 12:54:55 testbox kernel: [653888.371683] [] __invalidate_mapping_pages+0x136/0x150
Jan 5 12:54:55 testbox kernel: [653888.371683] [] invalidate_mapping_pages+0x12/0x20
Jan 5 12:54:55 testbox kernel: [653888.371683] [] prune_icache+0x21a/0x260
Jan 5 12:54:55 testbox kernel: [653888.371683] [] shrink_icache_memory+0x3e/0x50
Jan 5 12:54:55 testbox kernel: [653888.371683] [] shrink_slab+0x11f/0x180
Jan 5 12:54:55 testbox kernel: [653888.371683] [] do_try_to_free_pages+0x155/0x2c0
Jan 5 12:54:55 testbox kernel: [653888.371683] [] ? autoremove_wake_function+0x1b/0x50
Jan 5 12:54:55 testbox kernel: [653888.371683] [] try_to_free_pages+0x6c/0x80
Jan 5 12:54:55 testbox kernel: [653888.371683] [] ? isolate_pages_global+0x0/0x70
Jan 5 12:54:55 testbox kernel: [653888.371683] [] __alloc_pages_internal+0x1cb/0x450
Jan 5 12:54:55 testbox kernel: [653888.371683] [] new_slab+0x7b/0x1e0
Jan 5 12:54:55 testbox kernel: [653888.371683] [] ? unfreeze_slab+0x75/0xb0
Jan 5 12:54:55 testbox kernel: [653888.371683] [] __slab_alloc+0x20b/0x260
Jan 5 12:54:55 testbox kernel: [653888.371683] [] kmem_cache_alloc+0xa6/0xb0
Jan 5 12:54:55 testbox kernel: [653888.371683] [] ? dup_task_struct+0x1c/0x110
Jan 5 12:54:55 testbox kernel: [653888.371683] [] dup_task_struct+0x1c/0x110
Jan 5 12:54:55 testbox kernel: [653888.371683] [] copy_process+0x6e/0xcf0
Jan 5 12:54:55 testbox kernel: [653888.371683] [] ? alloc_fd+0xe0/0x100
Jan 5 12:54:55 testbox kernel: [653888.371683] [] ? get_empty_filp+0x6c/0x170
Jan 5 12:54:55 testbox kernel: [653888.371683] [] ? _spin_lock+0x8/0x10
Jan 5 12:54:55 testbox kernel: [653888.371683] [] do_fork+0x62/0x300
Jan 5 12:54:55 testbox kernel: [653888.371683] [] ? sys_pipe2+0x3f/0x60
Jan 5 12:54:55 testbox kernel: [653888.371683] [] sys_clone+0x2f/0x40
Jan 5 12:54:55 testbox kernel: [653888.371683] [] sysenter_do_call+0x12/0x2f