Ubuntu 9.04 Crash

CPU/Kernel/MB/RAID problem?

Jan  5 12:45:05 testbox kernel: [653298.890004] BUG: soft lockup - CPU#0 stuck for 61s! [hal-acl-tool:4168]
Jan  5 12:45:05 testbox kernel: [653298.890005] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan  5 12:45:05 testbox kernel: [653298.891687]
Jan  5 12:45:05 testbox kernel: [653298.891687] Pid: 4168, comm: hal-acl-tool Tainted: G      D W  (2.6.28-19-server #66-Ubuntu) System Product Name
Jan  5 12:45:05 testbox kernel: [653298.891687] EIP: 0060:[<c0129085>] EFLAGS: 00000293 CPU: 0
Jan  5 12:45:05 testbox kernel: [653298.891687] EIP is at __ticket_spin_lock+0x15/0x20
Jan  5 12:45:05 testbox kernel: [653298.891687] EAX: f5d962b8 EBX: d6ba3b00 ECX: f5d96200 EDX: 00004f4d
Jan  5 12:45:05 testbox kernel: [653298.891687] ESI: d20d3000 EDI: f5d962b8 EBP: e0491c60 ESP: e0491c60
Jan  5 12:45:05 testbox kernel: [653298.891687]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan  5 12:45:05 testbox kernel: [653298.891687] CR0: 8005003b CR2: 09202568 CR3: 2a445000 CR4: 000006b0
Jan  5 12:45:05 testbox kernel: [653298.891687] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan  5 12:45:05 testbox kernel: [653298.891687] DR6: ffff0ff0 DR7: 00000400
Jan  5 12:45:05 testbox kernel: [653298.891687] Call Trace:
Jan  5 12:45:05 testbox kernel: [653298.891687]  [<c050da98>] _spin_lock+0x8/0x10
Jan  5 12:45:05 testbox kernel: [653298.891687]  [<c025345d>] __journal_try_to_free_buffer+0x4d/0xc0
Jan  5 12:45:05 testbox kernel: [653298.891687]  [<c0253532>] journal_try_to_free_buffers+0x62/0x100
Jan  5 12:45:05 testbox kernel: [653298.891687]  [<c021bda5>] ext3_releasepage+0x45/0x70
Jan  5 12:45:05 testbox kernel: [653298.891687]  [<c0196c8e>] try_to_release_page+0x2e/0x50
Jan  5 12:45:05 testbox kernel: [653298.891687]  [<c01a0346>] __invalidate_mapping_pages+0x136/0x150
Jan  5 12:45:05 testbox kernel: [653298.891687]  [<c01a0372>] invalidate_mapping_pages+0x12/0x20
Jan  5 12:45:05 testbox kernel: [653298.891687]  [<c01d98ba>] prune_icache+0x21a/0x260
Jan  5 12:45:05 testbox kernel: [653298.891687]  [<c01d993e>] shrink_icache_memory+0x3e/0x50
Jan  5 12:45:05 testbox kernel: [653298.891687]  [<c01a086f>] shrink_slab+0x11f/0x180
Jan  5 12:45:05 testbox kernel: [653298.891687]  [<c01a2995>] do_try_to_free_pages+0x155/0x2c0
Jan  5 12:45:05 testbox kernel: [653298.891687]  [<c0156a8b>] ? autoremove_wake_function+0x1b/0x50
Jan  5 12:45:05 testbox kernel: [653298.891687]  [<c01a2bec>] try_to_free_pages+0x6c/0x80
Jan  5 12:45:05 testbox kernel: [653298.891687]  [<c01a13a0>] ? isolate_pages_global+0x0/0x70
Jan  5 12:45:05 testbox kernel: [653298.891687]  [<c019bdbb>] __alloc_pages_internal+0x1cb/0x450
Jan  5 12:45:05 testbox kernel: [653298.891687]  [<c01c004b>] new_slab+0x7b/0x1e0
Jan  5 12:45:05 testbox kernel: [653298.891687]  [<c01c1165>] ? unfreeze_slab+0x75/0xb0
Jan  5 12:45:05 testbox kernel: [653298.891687]  [<c01c141b>] __slab_alloc+0x20b/0x260
Jan  5 12:45:05 testbox kernel: [653298.891687]  [<c01c1516>] kmem_cache_alloc+0xa6/0xb0
Jan  5 12:45:05 testbox kernel: [653298.891687]  [<c013ef4c>] ? dup_task_struct+0x1c/0x110
Jan  5 12:45:05 testbox kernel: [653298.891687]  [<c013ef4c>] dup_task_struct+0x1c/0x110
Jan  5 12:45:05 testbox kernel: [653298.891687]  [<c013fd7e>] copy_process+0x6e/0xcf0
Jan  5 12:45:05 testbox kernel: [653298.891687]  [<c01db240>] ? alloc_fd+0xe0/0x100
Jan  5 12:45:05 testbox kernel: [653298.891687]  [<c01c7bbc>] ? get_empty_filp+0x6c/0x170
Jan  5 12:45:05 testbox kernel: [653298.891687]  [<c050da98>] ? _spin_lock+0x8/0x10
Jan  5 12:45:05 testbox kernel: [653298.891687]  [<c0140a62>] do_fork+0x62/0x300
Jan  5 12:45:05 testbox kernel: [653298.891687]  [<c01cd6ef>] ? sys_pipe2+0x3f/0x60
Jan  5 12:45:05 testbox kernel: [653298.891687]  [<c01083df>] sys_clone+0x2f/0x40
Jan  5 12:45:05 testbox kernel: [653298.891687]  [<c0109eef>] sysenter_do_call+0x12/0x2f
Jan  5 12:45:37 testbox kernel: [653330.032503] BUG: soft lockup - CPU#3 stuck for 61s! [rsync:30532]
Jan  5 12:45:37 testbox kernel: [653330.032503] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan  5 12:45:37 testbox kernel: [653330.032503]
Jan  5 12:45:37 testbox kernel: [653330.032503] Pid: 30532, comm: rsync Tainted: G      D W  (2.6.28-19-server #66-Ubuntu) System Product Name
Jan  5 12:45:37 testbox kernel: [653330.032503] EIP: 0060:[<c0129083>] EFLAGS: 00000297 CPU: 3
Jan  5 12:45:37 testbox kernel: [653330.032503] EIP is at __ticket_spin_lock+0x13/0x20
Jan  5 12:45:37 testbox kernel: [653330.032503] EAX: f5d962b8 EBX: e0c8de00 ECX: 00000000 EDX: 00004e4d
Jan  5 12:45:37 testbox kernel: [653330.032503] ESI: f5d96200 EDI: f6d00450 EBP: db0f1cc8 ESP: db0f1cc8
Jan  5 12:45:37 testbox kernel: [653330.032503]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan  5 12:45:37 testbox kernel: [653330.032503] CR0: 8005003b CR2: 00000000 CR3: 1b0d9000 CR4: 000006b0
Jan  5 12:45:37 testbox kernel: [653330.032503] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan  5 12:45:37 testbox kernel: [653330.032503] DR6: ffff0ff0 DR7: 00000400
Jan  5 12:45:37 testbox kernel: [653330.032503] Call Trace:
Jan  5 12:45:37 testbox kernel: [653330.032503]  [<c050da98>] _spin_lock+0x8/0x10
Jan  5 12:45:37 testbox kernel: [653330.032503]  [<c025393e>] journal_dirty_data+0x5e/0x1f0
Jan  5 12:45:37 testbox kernel: [653330.032503]  [<c021b048>] ext3_journal_dirty_data+0x18/0x50
Jan  5 12:45:37 testbox kernel: [653330.032503]  [<c021b7ef>] walk_page_buffers+0x2f/0x80
Jan  5 12:45:37 testbox kernel: [653330.032503]  [<c021bbf3>] ext3_ordered_write_end+0x63/0x1b0
Jan  5 12:45:37 testbox kernel: [653330.032503]  [<c021b030>] ? ext3_journal_dirty_data+0x0/0x50
Jan  5 12:45:37 testbox kernel: [653330.032503]  [<c0195e6d>] ? iov_iter_copy_from_user_atomic+0x5d/0xc0
Jan  5 12:45:37 testbox kernel: [653330.032503]  [<c0195fdf>] generic_perform_write+0x10f/0x190
Jan  5 12:45:37 testbox kernel: [653330.032503]  [<c01970ad>] generic_file_buffered_write+0x5d/0x120
Jan  5 12:45:37 testbox kernel: [653330.032503]  [<c01986ab>] __generic_file_aio_write_nolock+0x19b/0x520
Jan  5 12:45:37 testbox kernel: [653330.032503]  [<c013949e>] ? resched_task+0x1e/0x70
Jan  5 12:45:37 testbox kernel: [653330.032503]  [<c0198b3e>] generic_file_aio_write+0x5e/0xd0
Jan  5 12:45:37 testbox kernel: [653330.032503]  [<c02194fd>] ext3_file_write+0x2d/0xc0
Jan  5 12:45:37 testbox kernel: [653330.032503]  [<c01c6831>] do_sync_write+0xd1/0x110
Jan  5 12:45:37 testbox kernel: [653330.032503]  [<c03f1082>] ? rtc_update_irq+0x92/0xa0
Jan  5 12:45:37 testbox kernel: [653330.032503]  [<c0156a70>] ? autoremove_wake_function+0x0/0x50
Jan  5 12:45:37 testbox kernel: [653330.032503]  [<c02b22f0>] ? apparmor_file_permission+0x20/0x30
Jan  5 12:45:37 testbox kernel: [653330.032503]  [<c029049f>] ? security_file_permission+0xf/0x20
Jan  5 12:45:37 testbox kernel: [653330.032503]  [<c01c69d4>] ? rw_verify_area+0x54/0xd0
Jan  5 12:45:37 testbox kernel: [653330.032503]  [<c01c6ec8>] vfs_write+0x98/0x110
Jan  5 12:45:37 testbox kernel: [653330.032503]  [<c01c6760>] ? do_sync_write+0x0/0x110
Jan  5 12:45:37 testbox kernel: [653330.032503]  [<c01c6ffd>] sys_write+0x3d/0x70
Jan  5 12:45:37 testbox kernel: [653330.032503]  [<c0109eef>] sysenter_do_call+0x12/0x2f
Jan  5 12:45:37 testbox kernel: [653330.035919] hpet1: lost 3 rtc interrupts
Jan  5 12:46:11 testbox kernel: [653364.390003] BUG: soft lockup - CPU#0 stuck for 61s! [hal-acl-tool:4168]
Jan  5 12:46:11 testbox kernel: [653364.390005] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan  5 12:46:11 testbox kernel: [653364.391694]
Jan  5 12:46:11 testbox kernel: [653364.391694] Pid: 4168, comm: hal-acl-tool Tainted: G      D W  (2.6.28-19-server #66-Ubuntu) System Product Name
Jan  5 12:46:11 testbox kernel: [653364.391694] EIP: 0060:[<c0129083>] EFLAGS: 00000293 CPU: 0
Jan  5 12:46:11 testbox kernel: [653364.391694] EIP is at __ticket_spin_lock+0x13/0x20
Jan  5 12:46:11 testbox kernel: [653364.391694] EAX: f5d962b8 EBX: d6ba3b00 ECX: f5d96200 EDX: 00004f4d
Jan  5 12:46:11 testbox kernel: [653364.391694] ESI: d20d3000 EDI: f5d962b8 EBP: e0491c60 ESP: e0491c60
Jan  5 12:46:11 testbox kernel: [653364.391694]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan  5 12:46:11 testbox kernel: [653364.391694] CR0: 8005003b CR2: 09202568 CR3: 2a445000 CR4: 000006b0
Jan  5 12:46:11 testbox kernel: [653364.391694] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan  5 12:46:11 testbox kernel: [653364.391694] DR6: ffff0ff0 DR7: 00000400
Jan  5 12:46:11 testbox kernel: [653364.391694] Call Trace:
Jan  5 12:46:11 testbox kernel: [653364.391694]  [<c050da98>] _spin_lock+0x8/0x10
Jan  5 12:46:11 testbox kernel: [653364.391694]  [<c025345d>] __journal_try_to_free_buffer+0x4d/0xc0
Jan  5 12:46:11 testbox kernel: [653364.391694]  [<c0253532>] journal_try_to_free_buffers+0x62/0x100
Jan  5 12:46:11 testbox kernel: [653364.391694]  [<c021bda5>] ext3_releasepage+0x45/0x70
Jan  5 12:46:11 testbox kernel: [653364.391694]  [<c0196c8e>] try_to_release_page+0x2e/0x50
Jan  5 12:46:11 testbox kernel: [653364.391694]  [<c01a0346>] __invalidate_mapping_pages+0x136/0x150
Jan  5 12:46:11 testbox kernel: [653364.391694]  [<c01a0372>] invalidate_mapping_pages+0x12/0x20
Jan  5 12:46:11 testbox kernel: [653364.391694]  [<c01d98ba>] prune_icache+0x21a/0x260
Jan  5 12:46:11 testbox kernel: [653364.391694]  [<c01d993e>] shrink_icache_memory+0x3e/0x50
Jan  5 12:46:11 testbox kernel: [653364.391694]  [<c01a086f>] shrink_slab+0x11f/0x180
Jan  5 12:46:11 testbox kernel: [653364.391694]  [<c01a2995>] do_try_to_free_pages+0x155/0x2c0
Jan  5 12:46:11 testbox kernel: [653364.391694]  [<c0156a8b>] ? autoremove_wake_function+0x1b/0x50
Jan  5 12:46:11 testbox kernel: [653364.391694]  [<c01a2bec>] try_to_free_pages+0x6c/0x80
Jan  5 12:46:11 testbox kernel: [653364.391694]  [<c01a13a0>] ? isolate_pages_global+0x0/0x70
Jan  5 12:46:11 testbox kernel: [653364.391694]  [<c019bdbb>] __alloc_pages_internal+0x1cb/0x450
Jan  5 12:46:11 testbox kernel: [653364.391694]  [<c01c004b>] new_slab+0x7b/0x1e0
Jan  5 12:46:11 testbox kernel: [653364.391694]  [<c01c1165>] ? unfreeze_slab+0x75/0xb0
Jan  5 12:46:11 testbox kernel: [653364.391694]  [<c01c141b>] __slab_alloc+0x20b/0x260
Jan  5 12:46:11 testbox kernel: [653364.391694]  [<c01c1516>] kmem_cache_alloc+0xa6/0xb0
Jan  5 12:46:11 testbox kernel: [653364.391694]  [<c013ef4c>] ? dup_task_struct+0x1c/0x110
Jan  5 12:46:11 testbox kernel: [653364.391694]  [<c013ef4c>] dup_task_struct+0x1c/0x110
Jan  5 12:46:11 testbox kernel: [653364.391694]  [<c013fd7e>] copy_process+0x6e/0xcf0
Jan  5 12:46:11 testbox kernel: [653364.391694]  [<c01db240>] ? alloc_fd+0xe0/0x100
Jan  5 12:46:11 testbox kernel: [653364.391694]  [<c01c7bbc>] ? get_empty_filp+0x6c/0x170
Jan  5 12:46:11 testbox kernel: [653364.391694]  [<c050da98>] ? _spin_lock+0x8/0x10
Jan  5 12:46:11 testbox kernel: [653364.391694]  [<c0140a62>] do_fork+0x62/0x300
Jan  5 12:46:11 testbox kernel: [653364.391694]  [<c01cd6ef>] ? sys_pipe2+0x3f/0x60
Jan  5 12:46:11 testbox kernel: [653364.391694]  [<c01083df>] sys_clone+0x2f/0x40
Jan  5 12:46:11 testbox kernel: [653364.391694]  [<c0109eef>] sysenter_do_call+0x12/0x2f
Jan  5 12:46:42 testbox kernel: [653395.522503] BUG: soft lockup - CPU#3 stuck for 61s! [rsync:30532]
Jan  5 12:46:42 testbox kernel: [653395.522503] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan  5 12:46:42 testbox kernel: [653395.522503]
Jan  5 12:46:42 testbox kernel: [653395.522503] Pid: 30532, comm: rsync Tainted: G      D W  (2.6.28-19-server #66-Ubuntu) System Product Name
Jan  5 12:46:42 testbox kernel: [653395.522503] EIP: 0060:[<c0129085>] EFLAGS: 00000297 CPU: 3
Jan  5 12:46:42 testbox kernel: [653395.522503] EIP is at __ticket_spin_lock+0x15/0x20
Jan  5 12:46:42 testbox kernel: [653395.522503] EAX: f5d962b8 EBX: e0c8de00 ECX: 00000000 EDX: 00004e4d
Jan  5 12:46:42 testbox kernel: [653395.522503] ESI: f5d96200 EDI: f6d00450 EBP: db0f1cc8 ESP: db0f1cc8
Jan  5 12:46:42 testbox kernel: [653395.522503]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan  5 12:46:42 testbox kernel: [653395.522503] CR0: 8005003b CR2: 00000000 CR3: 1b0d9000 CR4: 000006b0
Jan  5 12:46:42 testbox kernel: [653395.522503] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan  5 12:46:42 testbox kernel: [653395.522503] DR6: ffff0ff0 DR7: 00000400
Jan  5 12:46:42 testbox kernel: [653395.522503] Call Trace:
Jan  5 12:46:42 testbox kernel: [653395.522503]  [<c050da98>] _spin_lock+0x8/0x10
Jan  5 12:46:42 testbox kernel: [653395.522503]  [<c025393e>] journal_dirty_data+0x5e/0x1f0
Jan  5 12:46:42 testbox kernel: [653395.522503]  [<c021b048>] ext3_journal_dirty_data+0x18/0x50
Jan  5 12:46:42 testbox kernel: [653395.522503]  [<c021b7ef>] walk_page_buffers+0x2f/0x80
Jan  5 12:46:42 testbox kernel: [653395.522503]  [<c021bbf3>] ext3_ordered_write_end+0x63/0x1b0
Jan  5 12:46:42 testbox kernel: [653395.522503]  [<c021b030>] ? ext3_journal_dirty_data+0x0/0x50
Jan  5 12:46:42 testbox kernel: [653395.522503]  [<c0195e6d>] ? iov_iter_copy_from_user_atomic+0x5d/0xc0
Jan  5 12:46:42 testbox kernel: [653395.522503]  [<c0195fdf>] generic_perform_write+0x10f/0x190
Jan  5 12:46:42 testbox kernel: [653395.522503]  [<c01970ad>] generic_file_buffered_write+0x5d/0x120
Jan  5 12:46:42 testbox kernel: [653395.522503]  [<c01986ab>] __generic_file_aio_write_nolock+0x19b/0x520
Jan  5 12:46:42 testbox kernel: [653395.522503]  [<c013949e>] ? resched_task+0x1e/0x70
Jan  5 12:46:42 testbox kernel: [653395.522503]  [<c0198b3e>] generic_file_aio_write+0x5e/0xd0
Jan  5 12:46:42 testbox kernel: [653395.522503]  [<c02194fd>] ext3_file_write+0x2d/0xc0
Jan  5 12:46:42 testbox kernel: [653395.522503]  [<c01c6831>] do_sync_write+0xd1/0x110
Jan  5 12:46:42 testbox kernel: [653395.522503]  [<c03f1082>] ? rtc_update_irq+0x92/0xa0
Jan  5 12:46:42 testbox kernel: [653395.522503]  [<c0156a70>] ? autoremove_wake_function+0x0/0x50
Jan  5 12:46:42 testbox kernel: [653395.522503]  [<c02b22f0>] ? apparmor_file_permission+0x20/0x30
Jan  5 12:46:42 testbox kernel: [653395.522503]  [<c029049f>] ? security_file_permission+0xf/0x20
Jan  5 12:46:42 testbox kernel: [653395.522503]  [<c01c69d4>] ? rw_verify_area+0x54/0xd0
Jan  5 12:46:42 testbox kernel: [653395.522503]  [<c01c6ec8>] vfs_write+0x98/0x110
Jan  5 12:46:42 testbox kernel: [653395.522503]  [<c01c6760>] ? do_sync_write+0x0/0x110
Jan  5 12:46:42 testbox kernel: [653395.522503]  [<c01c6ffd>] sys_write+0x3d/0x70
Jan  5 12:46:42 testbox kernel: [653395.522503]  [<c0109eef>] sysenter_do_call+0x12/0x2f
Jan  5 12:46:42 testbox kernel: [653395.526521] hpet1: lost 3 rtc interrupts
Jan  5 12:47:16 testbox kernel: [653429.890004] BUG: soft lockup - CPU#0 stuck for 61s! [hal-acl-tool:4168]
Jan  5 12:47:16 testbox kernel: [653429.890005] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan  5 12:47:16 testbox kernel: [653429.891694]
Jan  5 12:47:16 testbox kernel: [653429.891694] Pid: 4168, comm: hal-acl-tool Tainted: G      D W  (2.6.28-19-server #66-Ubuntu) System Product Name
Jan  5 12:47:16 testbox kernel: [653429.891694] EIP: 0060:[<c012907f>] EFLAGS: 00000293 CPU: 0
Jan  5 12:47:16 testbox kernel: [653429.891694] EIP is at __ticket_spin_lock+0xf/0x20
Jan  5 12:47:16 testbox kernel: [653429.891694] EAX: f5d962b8 EBX: d6ba3b00 ECX: f5d96200 EDX: 00004f4d
Jan  5 12:47:16 testbox kernel: [653429.891694] ESI: d20d3000 EDI: f5d962b8 EBP: e0491c60 ESP: e0491c60
Jan  5 12:47:16 testbox kernel: [653429.891694]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan  5 12:47:16 testbox kernel: [653429.891694] CR0: 8005003b CR2: 09202568 CR3: 2a445000 CR4: 000006b0
Jan  5 12:47:16 testbox kernel: [653429.891694] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan  5 12:47:16 testbox kernel: [653429.891694] DR6: ffff0ff0 DR7: 00000400
Jan  5 12:47:16 testbox kernel: [653429.891694] Call Trace:
Jan  5 12:47:16 testbox kernel: [653429.891694]  [<c050da98>] _spin_lock+0x8/0x10
Jan  5 12:47:16 testbox kernel: [653429.891694]  [<c025345d>] __journal_try_to_free_buffer+0x4d/0xc0
Jan  5 12:47:16 testbox kernel: [653429.891694]  [<c0253532>] journal_try_to_free_buffers+0x62/0x100
Jan  5 12:47:16 testbox kernel: [653429.891694]  [<c021bda5>] ext3_releasepage+0x45/0x70
Jan  5 12:47:16 testbox kernel: [653429.891694]  [<c0196c8e>] try_to_release_page+0x2e/0x50
Jan  5 12:47:16 testbox kernel: [653429.891694]  [<c01a0346>] __invalidate_mapping_pages+0x136/0x150
Jan  5 12:47:16 testbox kernel: [653429.891694]  [<c01a0372>] invalidate_mapping_pages+0x12/0x20
Jan  5 12:47:16 testbox kernel: [653429.891694]  [<c01d98ba>] prune_icache+0x21a/0x260
Jan  5 12:47:16 testbox kernel: [653429.891694]  [<c01d993e>] shrink_icache_memory+0x3e/0x50
Jan  5 12:47:16 testbox kernel: [653429.891694]  [<c01a086f>] shrink_slab+0x11f/0x180
Jan  5 12:47:16 testbox kernel: [653429.891694]  [<c01a2995>] do_try_to_free_pages+0x155/0x2c0
Jan  5 12:47:16 testbox kernel: [653429.891694]  [<c0156a8b>] ? autoremove_wake_function+0x1b/0x50
Jan  5 12:47:16 testbox kernel: [653429.891694]  [<c01a2bec>] try_to_free_pages+0x6c/0x80
Jan  5 12:47:16 testbox kernel: [653429.891694]  [<c01a13a0>] ? isolate_pages_global+0x0/0x70
Jan  5 12:47:16 testbox kernel: [653429.891694]  [<c019bdbb>] __alloc_pages_internal+0x1cb/0x450
Jan  5 12:47:16 testbox kernel: [653429.891694]  [<c01c004b>] new_slab+0x7b/0x1e0
Jan  5 12:47:16 testbox kernel: [653429.891694]  [<c01c1165>] ? unfreeze_slab+0x75/0xb0
Jan  5 12:47:16 testbox kernel: [653429.891694]  [<c01c141b>] __slab_alloc+0x20b/0x260
Jan  5 12:47:16 testbox kernel: [653429.891694]  [<c01c1516>] kmem_cache_alloc+0xa6/0xb0
Jan  5 12:47:16 testbox kernel: [653429.891694]  [<c013ef4c>] ? dup_task_struct+0x1c/0x110
Jan  5 12:47:16 testbox kernel: [653429.891694]  [<c013ef4c>] dup_task_struct+0x1c/0x110
Jan  5 12:47:16 testbox kernel: [653429.891694]  [<c013fd7e>] copy_process+0x6e/0xcf0
Jan  5 12:47:16 testbox kernel: [653429.891694]  [<c01db240>] ? alloc_fd+0xe0/0x100
Jan  5 12:47:16 testbox kernel: [653429.891694]  [<c01c7bbc>] ? get_empty_filp+0x6c/0x170
Jan  5 12:47:16 testbox kernel: [653429.891694]  [<c050da98>] ? _spin_lock+0x8/0x10
Jan  5 12:47:16 testbox kernel: [653429.891694]  [<c0140a62>] do_fork+0x62/0x300
Jan  5 12:47:16 testbox kernel: [653429.891694]  [<c01cd6ef>] ? sys_pipe2+0x3f/0x60
Jan  5 12:47:16 testbox kernel: [653429.891694]  [<c01083df>] sys_clone+0x2f/0x40
Jan  5 12:47:16 testbox kernel: [653429.891694]  [<c0109eef>] sysenter_do_call+0x12/0x2f
Jan  5 12:47:48 testbox kernel: [653461.022504] BUG: soft lockup - CPU#3 stuck for 61s! [rsync:30532]
Jan  5 12:47:48 testbox kernel: [653461.022504] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan  5 12:47:48 testbox kernel: [653461.022504]
Jan  5 12:47:48 testbox kernel: [653461.022504] Pid: 30532, comm: rsync Tainted: G      D W  (2.6.28-19-server #66-Ubuntu) System Product Name
Jan  5 12:47:48 testbox kernel: [653461.022504] EIP: 0060:[<c0129085>] EFLAGS: 00000297 CPU: 3
Jan  5 12:47:48 testbox kernel: [653461.022504] EIP is at __ticket_spin_lock+0x15/0x20
Jan  5 12:47:48 testbox kernel: [653461.022504] EAX: f5d962b8 EBX: e0c8de00 ECX: 00000000 EDX: 00004e4d
Jan  5 12:47:48 testbox kernel: [653461.022504] ESI: f5d96200 EDI: f6d00450 EBP: db0f1cc8 ESP: db0f1cc8
Jan  5 12:47:48 testbox kernel: [653461.022504]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan  5 12:47:48 testbox kernel: [653461.022504] CR0: 8005003b CR2: 00000000 CR3: 1b0d9000 CR4: 000006b0
Jan  5 12:47:48 testbox kernel: [653461.022504] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan  5 12:47:48 testbox kernel: [653461.022504] DR6: ffff0ff0 DR7: 00000400
Jan  5 12:47:48 testbox kernel: [653461.022504] Call Trace:
Jan  5 12:47:48 testbox kernel: [653461.022504]  [<c050da98>] _spin_lock+0x8/0x10
Jan  5 12:47:48 testbox kernel: [653461.022504]  [<c025393e>] journal_dirty_data+0x5e/0x1f0
Jan  5 12:47:48 testbox kernel: [653461.022504]  [<c021b048>] ext3_journal_dirty_data+0x18/0x50
Jan  5 12:47:48 testbox kernel: [653461.022504]  [<c021b7ef>] walk_page_buffers+0x2f/0x80
Jan  5 12:47:48 testbox kernel: [653461.022504]  [<c021bbf3>] ext3_ordered_write_end+0x63/0x1b0
Jan  5 12:47:48 testbox kernel: [653461.022504]  [<c021b030>] ? ext3_journal_dirty_data+0x0/0x50
Jan  5 12:47:48 testbox kernel: [653461.022504]  [<c0195e6d>] ? iov_iter_copy_from_user_atomic+0x5d/0xc0
Jan  5 12:47:48 testbox kernel: [653461.022504]  [<c0195fdf>] generic_perform_write+0x10f/0x190
Jan  5 12:47:48 testbox kernel: [653461.022504]  [<c01970ad>] generic_file_buffered_write+0x5d/0x120
Jan  5 12:47:48 testbox kernel: [653461.022504]  [<c01986ab>] __generic_file_aio_write_nolock+0x19b/0x520
Jan  5 12:47:48 testbox kernel: [653461.022504]  [<c013949e>] ? resched_task+0x1e/0x70
Jan  5 12:47:48 testbox kernel: [653461.022504]  [<c0198b3e>] generic_file_aio_write+0x5e/0xd0
Jan  5 12:47:48 testbox kernel: [653461.022504]  [<c02194fd>] ext3_file_write+0x2d/0xc0
Jan  5 12:47:48 testbox kernel: [653461.022504]  [<c01c6831>] do_sync_write+0xd1/0x110
Jan  5 12:47:48 testbox kernel: [653461.022504]  [<c03f1082>] ? rtc_update_irq+0x92/0xa0
Jan  5 12:47:48 testbox kernel: [653461.022504]  [<c0156a70>] ? autoremove_wake_function+0x0/0x50
Jan  5 12:47:48 testbox kernel: [653461.022504]  [<c02b22f0>] ? apparmor_file_permission+0x20/0x30
Jan  5 12:47:48 testbox kernel: [653461.022504]  [<c029049f>] ? security_file_permission+0xf/0x20
Jan  5 12:47:48 testbox kernel: [653461.022504]  [<c01c69d4>] ? rw_verify_area+0x54/0xd0
Jan  5 12:47:48 testbox kernel: [653461.022504]  [<c01c6ec8>] vfs_write+0x98/0x110
Jan  5 12:47:48 testbox kernel: [653461.022504]  [<c01c6760>] ? do_sync_write+0x0/0x110
Jan  5 12:47:48 testbox kernel: [653461.022504]  [<c01c6ffd>] sys_write+0x3d/0x70
Jan  5 12:47:48 testbox kernel: [653461.022504]  [<c0109eef>] sysenter_do_call+0x12/0x2f
Jan  5 12:47:48 testbox kernel: [653461.025921] hpet1: lost 2 rtc interrupts
Jan  5 12:48:22 testbox kernel: [653495.390004] BUG: soft lockup - CPU#0 stuck for 61s! [hal-acl-tool:4168]
Jan  5 12:48:22 testbox kernel: [653495.390005] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan  5 12:48:22 testbox kernel: [653495.391695]
Jan  5 12:48:22 testbox kernel: [653495.391695] Pid: 4168, comm: hal-acl-tool Tainted: G      D W  (2.6.28-19-server #66-Ubuntu) System Product Name
Jan  5 12:48:22 testbox kernel: [653495.391695] EIP: 0060:[<c0129083>] EFLAGS: 00000293 CPU: 0
Jan  5 12:48:22 testbox kernel: [653495.391695] EIP is at __ticket_spin_lock+0x13/0x20
Jan  5 12:48:22 testbox kernel: [653495.391695] EAX: f5d962b8 EBX: d6ba3b00 ECX: f5d96200 EDX: 00004f4d
Jan  5 12:48:22 testbox kernel: [653495.391695] ESI: d20d3000 EDI: f5d962b8 EBP: e0491c60 ESP: e0491c60
Jan  5 12:48:22 testbox kernel: [653495.391695]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan  5 12:48:22 testbox kernel: [653495.391695] CR0: 8005003b CR2: 09202568 CR3: 2a445000 CR4: 000006b0
Jan  5 12:48:22 testbox kernel: [653495.391695] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan  5 12:48:22 testbox kernel: [653495.391695] DR6: ffff0ff0 DR7: 00000400
Jan  5 12:48:22 testbox kernel: [653495.391695] Call Trace:
Jan  5 12:48:22 testbox kernel: [653495.391695]  [<c050da98>] _spin_lock+0x8/0x10
Jan  5 12:48:22 testbox kernel: [653495.391695]  [<c025345d>] __journal_try_to_free_buffer+0x4d/0xc0
Jan  5 12:48:22 testbox kernel: [653495.391695]  [<c0253532>] journal_try_to_free_buffers+0x62/0x100
Jan  5 12:48:22 testbox kernel: [653495.391695]  [<c021bda5>] ext3_releasepage+0x45/0x70
Jan  5 12:48:22 testbox kernel: [653495.391695]  [<c0196c8e>] try_to_release_page+0x2e/0x50
Jan  5 12:48:22 testbox kernel: [653495.391695]  [<c01a0346>] __invalidate_mapping_pages+0x136/0x150
Jan  5 12:48:22 testbox kernel: [653495.391695]  [<c01a0372>] invalidate_mapping_pages+0x12/0x20
Jan  5 12:48:22 testbox kernel: [653495.391695]  [<c01d98ba>] prune_icache+0x21a/0x260
Jan  5 12:48:22 testbox kernel: [653495.391695]  [<c01d993e>] shrink_icache_memory+0x3e/0x50
Jan  5 12:48:22 testbox kernel: [653495.391695]  [<c01a086f>] shrink_slab+0x11f/0x180
Jan  5 12:48:22 testbox kernel: [653495.391695]  [<c01a2995>] do_try_to_free_pages+0x155/0x2c0
Jan  5 12:48:22 testbox kernel: [653495.391695]  [<c0156a8b>] ? autoremove_wake_function+0x1b/0x50
Jan  5 12:48:22 testbox kernel: [653495.391695]  [<c01a2bec>] try_to_free_pages+0x6c/0x80
Jan  5 12:48:22 testbox kernel: [653495.391695]  [<c01a13a0>] ? isolate_pages_global+0x0/0x70
Jan  5 12:48:22 testbox kernel: [653495.391695]  [<c019bdbb>] __alloc_pages_internal+0x1cb/0x450
Jan  5 12:48:22 testbox kernel: [653495.391695]  [<c01c004b>] new_slab+0x7b/0x1e0
Jan  5 12:48:22 testbox kernel: [653495.391695]  [<c01c1165>] ? unfreeze_slab+0x75/0xb0
Jan  5 12:48:22 testbox kernel: [653495.391695]  [<c01c141b>] __slab_alloc+0x20b/0x260
Jan  5 12:48:22 testbox kernel: [653495.391695]  [<c01c1516>] kmem_cache_alloc+0xa6/0xb0
Jan  5 12:48:22 testbox kernel: [653495.391695]  [<c013ef4c>] ? dup_task_struct+0x1c/0x110
Jan  5 12:48:22 testbox kernel: [653495.391695]  [<c013ef4c>] dup_task_struct+0x1c/0x110
Jan  5 12:48:22 testbox kernel: [653495.391695]  [<c013fd7e>] copy_process+0x6e/0xcf0
Jan  5 12:48:22 testbox kernel: [653495.391695]  [<c01db240>] ? alloc_fd+0xe0/0x100
Jan  5 12:48:22 testbox kernel: [653495.391695]  [<c01c7bbc>] ? get_empty_filp+0x6c/0x170
Jan  5 12:48:22 testbox kernel: [653495.391695]  [<c050da98>] ? _spin_lock+0x8/0x10
Jan  5 12:48:22 testbox kernel: [653495.391695]  [<c0140a62>] do_fork+0x62/0x300
Jan  5 12:48:22 testbox kernel: [653495.391695]  [<c01cd6ef>] ? sys_pipe2+0x3f/0x60
Jan  5 12:48:22 testbox kernel: [653495.391695]  [<c01083df>] sys_clone+0x2f/0x40
Jan  5 12:48:22 testbox kernel: [653495.391695]  [<c0109eef>] sysenter_do_call+0x12/0x2f
Jan  5 12:48:53 testbox kernel: [653526.522503] BUG: soft lockup - CPU#3 stuck for 61s! [rsync:30532]
Jan  5 12:48:53 testbox kernel: [653526.522504] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan  5 12:48:53 testbox kernel: [653526.522504]
Jan  5 12:48:53 testbox kernel: [653526.522504] Pid: 30532, comm: rsync Tainted: G      D W  (2.6.28-19-server #66-Ubuntu) System Product Name
Jan  5 12:48:53 testbox kernel: [653526.522504] EIP: 0060:[<c0129085>] EFLAGS: 00000297 CPU: 3
Jan  5 12:48:53 testbox kernel: [653526.522504] EIP is at __ticket_spin_lock+0x15/0x20
Jan  5 12:48:53 testbox kernel: [653526.522504] EAX: f5d962b8 EBX: e0c8de00 ECX: 00000000 EDX: 00004e4d
Jan  5 12:48:53 testbox kernel: [653526.522504] ESI: f5d96200 EDI: f6d00450 EBP: db0f1cc8 ESP: db0f1cc8
Jan  5 12:48:53 testbox kernel: [653526.522504]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan  5 12:48:53 testbox kernel: [653526.522504] CR0: 8005003b CR2: 00000000 CR3: 1b0d9000 CR4: 000006b0
Jan  5 12:48:53 testbox kernel: [653526.522504] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan  5 12:48:53 testbox kernel: [653526.522504] DR6: ffff0ff0 DR7: 00000400
Jan  5 12:48:53 testbox kernel: [653526.522504] Call Trace:
Jan  5 12:48:53 testbox kernel: [653526.522504]  [<c050da98>] _spin_lock+0x8/0x10
Jan  5 12:48:53 testbox kernel: [653526.522504]  [<c025393e>] journal_dirty_data+0x5e/0x1f0
Jan  5 12:48:53 testbox kernel: [653526.522504]  [<c021b048>] ext3_journal_dirty_data+0x18/0x50
Jan  5 12:48:53 testbox kernel: [653526.522504]  [<c021b7ef>] walk_page_buffers+0x2f/0x80
Jan  5 12:48:53 testbox kernel: [653526.522504]  [<c021bbf3>] ext3_ordered_write_end+0x63/0x1b0
Jan  5 12:48:53 testbox kernel: [653526.522504]  [<c021b030>] ? ext3_journal_dirty_data+0x0/0x50
Jan  5 12:48:53 testbox kernel: [653526.522504]  [<c0195e6d>] ? iov_iter_copy_from_user_atomic+0x5d/0xc0
Jan  5 12:48:53 testbox kernel: [653526.522504]  [<c0195fdf>] generic_perform_write+0x10f/0x190
Jan  5 12:48:53 testbox kernel: [653526.522504]  [<c01970ad>] generic_file_buffered_write+0x5d/0x120
Jan  5 12:48:53 testbox kernel: [653526.522504]  [<c01986ab>] __generic_file_aio_write_nolock+0x19b/0x520
Jan  5 12:48:53 testbox kernel: [653526.522504]  [<c013949e>] ? resched_task+0x1e/0x70
Jan  5 12:48:53 testbox kernel: [653526.522504]  [<c0198b3e>] generic_file_aio_write+0x5e/0xd0
Jan  5 12:48:53 testbox kernel: [653526.522504]  [<c02194fd>] ext3_file_write+0x2d/0xc0
Jan  5 12:48:53 testbox kernel: [653526.522504]  [<c01c6831>] do_sync_write+0xd1/0x110
Jan  5 12:48:53 testbox kernel: [653526.522504]  [<c03f1082>] ? rtc_update_irq+0x92/0xa0
Jan  5 12:48:53 testbox kernel: [653526.522504]  [<c0156a70>] ? autoremove_wake_function+0x0/0x50
Jan  5 12:48:53 testbox kernel: [653526.522504]  [<c02b22f0>] ? apparmor_file_permission+0x20/0x30
Jan  5 12:48:53 testbox kernel: [653526.522504]  [<c029049f>] ? security_file_permission+0xf/0x20
Jan  5 12:48:53 testbox kernel: [653526.522504]  [<c01c69d4>] ? rw_verify_area+0x54/0xd0
Jan  5 12:48:53 testbox kernel: [653526.522504]  [<c01c6ec8>] vfs_write+0x98/0x110
Jan  5 12:48:53 testbox kernel: [653526.522504]  [<c01c6760>] ? do_sync_write+0x0/0x110
Jan  5 12:48:53 testbox kernel: [653526.522504]  [<c01c6ffd>] sys_write+0x3d/0x70
Jan  5 12:48:53 testbox kernel: [653526.522504]  [<c0109eef>] sysenter_do_call+0x12/0x2f
Jan  5 12:48:53 testbox kernel: [653526.526518] hpet1: lost 3 rtc interrupts
Jan  5 12:49:27 testbox kernel: [653560.880003] BUG: soft lockup - CPU#0 stuck for 61s! [hal-acl-tool:4168]
Jan  5 12:49:27 testbox kernel: [653560.880005] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan  5 12:49:27 testbox kernel: [653560.881694]
Jan  5 12:49:27 testbox kernel: [653560.881694] Pid: 4168, comm: hal-acl-tool Tainted: G      D W  (2.6.28-19-server #66-Ubuntu) System Product Name
Jan  5 12:49:27 testbox kernel: [653560.881694] EIP: 0060:[<c012907f>] EFLAGS: 00000293 CPU: 0
Jan  5 12:49:27 testbox kernel: [653560.881694] EIP is at __ticket_spin_lock+0xf/0x20
Jan  5 12:49:27 testbox kernel: [653560.881694] EAX: f5d962b8 EBX: d6ba3b00 ECX: f5d96200 EDX: 00004f4d
Jan  5 12:49:27 testbox kernel: [653560.881694] ESI: d20d3000 EDI: f5d962b8 EBP: e0491c60 ESP: e0491c60
Jan  5 12:49:27 testbox kernel: [653560.881694]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan  5 12:49:27 testbox kernel: [653560.881694] CR0: 8005003b CR2: 09202568 CR3: 2a445000 CR4: 000006b0
Jan  5 12:49:27 testbox kernel: [653560.881694] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan  5 12:49:27 testbox kernel: [653560.881694] DR6: ffff0ff0 DR7: 00000400
Jan  5 12:49:27 testbox kernel: [653560.881694] Call Trace:
Jan  5 12:49:27 testbox kernel: [653560.881694]  [<c050da98>] _spin_lock+0x8/0x10
Jan  5 12:49:27 testbox kernel: [653560.881694]  [<c025345d>] __journal_try_to_free_buffer+0x4d/0xc0
Jan  5 12:49:27 testbox kernel: [653560.881694]  [<c0253532>] journal_try_to_free_buffers+0x62/0x100
Jan  5 12:49:27 testbox kernel: [653560.881694]  [<c021bda5>] ext3_releasepage+0x45/0x70
Jan  5 12:49:27 testbox kernel: [653560.881694]  [<c0196c8e>] try_to_release_page+0x2e/0x50
Jan  5 12:49:27 testbox kernel: [653560.881694]  [<c01a0346>] __invalidate_mapping_pages+0x136/0x150
Jan  5 12:49:27 testbox kernel: [653560.881694]  [<c01a0372>] invalidate_mapping_pages+0x12/0x20
Jan  5 12:49:27 testbox kernel: [653560.881694]  [<c01d98ba>] prune_icache+0x21a/0x260
Jan  5 12:49:27 testbox kernel: [653560.881694]  [<c01d993e>] shrink_icache_memory+0x3e/0x50
Jan  5 12:49:27 testbox kernel: [653560.881694]  [<c01a086f>] shrink_slab+0x11f/0x180
Jan  5 12:49:27 testbox kernel: [653560.881694]  [<c01a2995>] do_try_to_free_pages+0x155/0x2c0
Jan  5 12:49:27 testbox kernel: [653560.881694]  [<c0156a8b>] ? autoremove_wake_function+0x1b/0x50
Jan  5 12:49:27 testbox kernel: [653560.881694]  [<c01a2bec>] try_to_free_pages+0x6c/0x80
Jan  5 12:49:27 testbox kernel: [653560.881694]  [<c01a13a0>] ? isolate_pages_global+0x0/0x70
Jan  5 12:49:27 testbox kernel: [653560.881694]  [<c019bdbb>] __alloc_pages_internal+0x1cb/0x450
Jan  5 12:49:27 testbox kernel: [653560.881694]  [<c01c004b>] new_slab+0x7b/0x1e0
Jan  5 12:49:27 testbox kernel: [653560.881694]  [<c01c1165>] ? unfreeze_slab+0x75/0xb0
Jan  5 12:49:27 testbox kernel: [653560.881694]  [<c01c141b>] __slab_alloc+0x20b/0x260
Jan  5 12:49:27 testbox kernel: [653560.881694]  [<c01c1516>] kmem_cache_alloc+0xa6/0xb0
Jan  5 12:49:27 testbox kernel: [653560.881694]  [<c013ef4c>] ? dup_task_struct+0x1c/0x110
Jan  5 12:49:27 testbox kernel: [653560.881694]  [<c013ef4c>] dup_task_struct+0x1c/0x110
Jan  5 12:49:27 testbox kernel: [653560.881694]  [<c013fd7e>] copy_process+0x6e/0xcf0
Jan  5 12:49:27 testbox kernel: [653560.881694]  [<c01db240>] ? alloc_fd+0xe0/0x100
Jan  5 12:49:27 testbox kernel: [653560.881694]  [<c01c7bbc>] ? get_empty_filp+0x6c/0x170
Jan  5 12:49:27 testbox kernel: [653560.881694]  [<c050da98>] ? _spin_lock+0x8/0x10
Jan  5 12:49:27 testbox kernel: [653560.881694]  [<c0140a62>] do_fork+0x62/0x300
Jan  5 12:49:27 testbox kernel: [653560.881694]  [<c01cd6ef>] ? sys_pipe2+0x3f/0x60
Jan  5 12:49:27 testbox kernel: [653560.881694]  [<c01083df>] sys_clone+0x2f/0x40
Jan  5 12:49:27 testbox kernel: [653560.881694]  [<c0109eef>] sysenter_do_call+0x12/0x2f
Jan  5 12:49:59 testbox kernel: [653592.022503] BUG: soft lockup - CPU#3 stuck for 61s! [rsync:30532]
Jan  5 12:49:59 testbox kernel: [653592.022504] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan  5 12:49:59 testbox kernel: [653592.022504]
Jan  5 12:49:59 testbox kernel: [653592.022504] Pid: 30532, comm: rsync Tainted: G      D W  (2.6.28-19-server #66-Ubuntu) System Product Name
Jan  5 12:49:59 testbox kernel: [653592.022504] EIP: 0060:[<c0129085>] EFLAGS: 00000297 CPU: 3
Jan  5 12:49:59 testbox kernel: [653592.022504] EIP is at __ticket_spin_lock+0x15/0x20
Jan  5 12:49:59 testbox kernel: [653592.022504] EAX: f5d962b8 EBX: e0c8de00 ECX: 00000000 EDX: 00004e4d
Jan  5 12:49:59 testbox kernel: [653592.022504] ESI: f5d96200 EDI: f6d00450 EBP: db0f1cc8 ESP: db0f1cc8
Jan  5 12:49:59 testbox kernel: [653592.022504]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan  5 12:49:59 testbox kernel: [653592.022504] CR0: 8005003b CR2: 00000000 CR3: 1b0d9000 CR4: 000006b0
Jan  5 12:49:59 testbox kernel: [653592.022504] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan  5 12:49:59 testbox kernel: [653592.022504] DR6: ffff0ff0 DR7: 00000400
Jan  5 12:49:59 testbox kernel: [653592.022504] Call Trace:
Jan  5 12:49:59 testbox kernel: [653592.022504]  [<c050da98>] _spin_lock+0x8/0x10
Jan  5 12:49:59 testbox kernel: [653592.022504]  [<c025393e>] journal_dirty_data+0x5e/0x1f0
Jan  5 12:49:59 testbox kernel: [653592.022504]  [<c021b048>] ext3_journal_dirty_data+0x18/0x50
Jan  5 12:49:59 testbox kernel: [653592.022504]  [<c021b7ef>] walk_page_buffers+0x2f/0x80
Jan  5 12:49:59 testbox kernel: [653592.022504]  [<c021bbf3>] ext3_ordered_write_end+0x63/0x1b0
Jan  5 12:49:59 testbox kernel: [653592.022504]  [<c021b030>] ? ext3_journal_dirty_data+0x0/0x50
Jan  5 12:49:59 testbox kernel: [653592.022504]  [<c0195e6d>] ? iov_iter_copy_from_user_atomic+0x5d/0xc0
Jan  5 12:49:59 testbox kernel: [653592.022504]  [<c0195fdf>] generic_perform_write+0x10f/0x190
Jan  5 12:49:59 testbox kernel: [653592.022504]  [<c01970ad>] generic_file_buffered_write+0x5d/0x120
Jan  5 12:49:59 testbox kernel: [653592.022504]  [<c01986ab>] __generic_file_aio_write_nolock+0x19b/0x520
Jan  5 12:49:59 testbox kernel: [653592.022504]  [<c013949e>] ? resched_task+0x1e/0x70
Jan  5 12:49:59 testbox kernel: [653592.022504]  [<c0198b3e>] generic_file_aio_write+0x5e/0xd0
Jan  5 12:49:59 testbox kernel: [653592.022504]  [<c02194fd>] ext3_file_write+0x2d/0xc0
Jan  5 12:49:59 testbox kernel: [653592.022504]  [<c01c6831>] do_sync_write+0xd1/0x110
Jan  5 12:49:59 testbox kernel: [653592.022504]  [<c03f1082>] ? rtc_update_irq+0x92/0xa0
Jan  5 12:49:59 testbox kernel: [653592.022504]  [<c0156a70>] ? autoremove_wake_function+0x0/0x50
Jan  5 12:49:59 testbox kernel: [653592.022504]  [<c02b22f0>] ? apparmor_file_permission+0x20/0x30
Jan  5 12:49:59 testbox kernel: [653592.022504]  [<c029049f>] ? security_file_permission+0xf/0x20
Jan  5 12:49:59 testbox kernel: [653592.022504]  [<c01c69d4>] ? rw_verify_area+0x54/0xd0
Jan  5 12:49:59 testbox kernel: [653592.022504]  [<c01c6ec8>] vfs_write+0x98/0x110
Jan  5 12:49:59 testbox kernel: [653592.022504]  [<c01c6760>] ? do_sync_write+0x0/0x110
Jan  5 12:49:59 testbox kernel: [653592.022504]  [<c01c6ffd>] sys_write+0x3d/0x70
Jan  5 12:49:59 testbox kernel: [653592.022504]  [<c0109eef>] sysenter_do_call+0x12/0x2f
Jan  5 12:49:59 testbox kernel: [653592.025918] hpet1: lost 2 rtc interrupts
Jan  5 12:50:33 testbox kernel: [653626.380003] BUG: soft lockup - CPU#0 stuck for 61s! [hal-acl-tool:4168]
Jan  5 12:50:33 testbox kernel: [653626.380005] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan  5 12:50:33 testbox kernel: [653626.381684]
Jan  5 12:50:33 testbox kernel: [653626.381684] Pid: 4168, comm: hal-acl-tool Tainted: G      D W  (2.6.28-19-server #66-Ubuntu) System Product Name
Jan  5 12:50:33 testbox kernel: [653626.381684] EIP: 0060:[<c0129081>] EFLAGS: 00000293 CPU: 0
Jan  5 12:50:33 testbox kernel: [653626.381684] EIP is at __ticket_spin_lock+0x11/0x20
Jan  5 12:50:33 testbox kernel: [653626.381684] EAX: f5d962b8 EBX: d6ba3b00 ECX: f5d96200 EDX: 00004f4d
Jan  5 12:50:33 testbox kernel: [653626.381684] ESI: d20d3000 EDI: f5d962b8 EBP: e0491c60 ESP: e0491c60
Jan  5 12:50:33 testbox kernel: [653626.381684]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan  5 12:50:33 testbox kernel: [653626.381684] CR0: 8005003b CR2: 09202568 CR3: 2a445000 CR4: 000006b0
Jan  5 12:50:33 testbox kernel: [653626.381684] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan  5 12:50:33 testbox kernel: [653626.381684] DR6: ffff0ff0 DR7: 00000400
Jan  5 12:50:33 testbox kernel: [653626.381684] Call Trace:
Jan  5 12:50:33 testbox kernel: [653626.381684]  [<c050da98>] _spin_lock+0x8/0x10
Jan  5 12:50:33 testbox kernel: [653626.381684]  [<c025345d>] __journal_try_to_free_buffer+0x4d/0xc0
Jan  5 12:50:33 testbox kernel: [653626.381684]  [<c0253532>] journal_try_to_free_buffers+0x62/0x100
Jan  5 12:50:33 testbox kernel: [653626.381684]  [<c021bda5>] ext3_releasepage+0x45/0x70
Jan  5 12:50:33 testbox kernel: [653626.381684]  [<c0196c8e>] try_to_release_page+0x2e/0x50
Jan  5 12:50:33 testbox kernel: [653626.381684]  [<c01a0346>] __invalidate_mapping_pages+0x136/0x150
Jan  5 12:50:33 testbox kernel: [653626.381684]  [<c01a0372>] invalidate_mapping_pages+0x12/0x20
Jan  5 12:50:33 testbox kernel: [653626.381684]  [<c01d98ba>] prune_icache+0x21a/0x260
Jan  5 12:50:33 testbox kernel: [653626.381684]  [<c01d993e>] shrink_icache_memory+0x3e/0x50
Jan  5 12:50:33 testbox kernel: [653626.381684]  [<c01a086f>] shrink_slab+0x11f/0x180
Jan  5 12:50:33 testbox kernel: [653626.381684]  [<c01a2995>] do_try_to_free_pages+0x155/0x2c0
Jan  5 12:50:33 testbox kernel: [653626.381684]  [<c0156a8b>] ? autoremove_wake_function+0x1b/0x50
Jan  5 12:50:33 testbox kernel: [653626.381684]  [<c01a2bec>] try_to_free_pages+0x6c/0x80
Jan  5 12:50:33 testbox kernel: [653626.381684]  [<c01a13a0>] ? isolate_pages_global+0x0/0x70
Jan  5 12:50:33 testbox kernel: [653626.381684]  [<c019bdbb>] __alloc_pages_internal+0x1cb/0x450
Jan  5 12:50:33 testbox kernel: [653626.381684]  [<c01c004b>] new_slab+0x7b/0x1e0
Jan  5 12:50:33 testbox kernel: [653626.381684]  [<c01c1165>] ? unfreeze_slab+0x75/0xb0
Jan  5 12:50:33 testbox kernel: [653626.381684]  [<c01c141b>] __slab_alloc+0x20b/0x260
Jan  5 12:50:33 testbox kernel: [653626.381684]  [<c01c1516>] kmem_cache_alloc+0xa6/0xb0
Jan  5 12:50:33 testbox kernel: [653626.381684]  [<c013ef4c>] ? dup_task_struct+0x1c/0x110
Jan  5 12:50:33 testbox kernel: [653626.381684]  [<c013ef4c>] dup_task_struct+0x1c/0x110
Jan  5 12:50:33 testbox kernel: [653626.381684]  [<c013fd7e>] copy_process+0x6e/0xcf0
Jan  5 12:50:33 testbox kernel: [653626.381684]  [<c01db240>] ? alloc_fd+0xe0/0x100
Jan  5 12:50:33 testbox kernel: [653626.381684]  [<c01c7bbc>] ? get_empty_filp+0x6c/0x170
Jan  5 12:50:33 testbox kernel: [653626.381684]  [<c050da98>] ? _spin_lock+0x8/0x10
Jan  5 12:50:33 testbox kernel: [653626.381684]  [<c0140a62>] do_fork+0x62/0x300
Jan  5 12:50:33 testbox kernel: [653626.381684]  [<c01cd6ef>] ? sys_pipe2+0x3f/0x60
Jan  5 12:50:33 testbox kernel: [653626.381684]  [<c01083df>] sys_clone+0x2f/0x40
Jan  5 12:50:33 testbox kernel: [653626.381684]  [<c0109eef>] sysenter_do_call+0x12/0x2f
Jan  5 12:51:04 testbox kernel: [653657.522503] BUG: soft lockup - CPU#3 stuck for 61s! [rsync:30532]
Jan  5 12:51:04 testbox kernel: [653657.522504] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan  5 12:51:04 testbox kernel: [653657.522504]
Jan  5 12:51:04 testbox kernel: [653657.522504] Pid: 30532, comm: rsync Tainted: G      D W  (2.6.28-19-server #66-Ubuntu) System Product Name
Jan  5 12:51:04 testbox kernel: [653657.522504] EIP: 0060:[<c0129083>] EFLAGS: 00000297 CPU: 3
Jan  5 12:51:04 testbox kernel: [653657.522504] EIP is at __ticket_spin_lock+0x13/0x20
Jan  5 12:51:04 testbox kernel: [653657.522504] EAX: f5d962b8 EBX: e0c8de00 ECX: 00000000 EDX: 00004e4d
Jan  5 12:51:04 testbox kernel: [653657.522504] ESI: f5d96200 EDI: f6d00450 EBP: db0f1cc8 ESP: db0f1cc8
Jan  5 12:51:04 testbox kernel: [653657.522504]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan  5 12:51:04 testbox kernel: [653657.522504] CR0: 8005003b CR2: 00000000 CR3: 1b0d9000 CR4: 000006b0
Jan  5 12:51:04 testbox kernel: [653657.522504] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan  5 12:51:04 testbox kernel: [653657.522504] DR6: ffff0ff0 DR7: 00000400
Jan  5 12:51:04 testbox kernel: [653657.522504] Call Trace:
Jan  5 12:51:04 testbox kernel: [653657.522504]  [<c050da98>] _spin_lock+0x8/0x10
Jan  5 12:51:04 testbox kernel: [653657.522504]  [<c025393e>] journal_dirty_data+0x5e/0x1f0
Jan  5 12:51:04 testbox kernel: [653657.522504]  [<c021b048>] ext3_journal_dirty_data+0x18/0x50
Jan  5 12:51:04 testbox kernel: [653657.522504]  [<c021b7ef>] walk_page_buffers+0x2f/0x80
Jan  5 12:51:04 testbox kernel: [653657.522504]  [<c021bbf3>] ext3_ordered_write_end+0x63/0x1b0
Jan  5 12:51:04 testbox kernel: [653657.522504]  [<c021b030>] ? ext3_journal_dirty_data+0x0/0x50
Jan  5 12:51:04 testbox kernel: [653657.522504]  [<c0195e6d>] ? iov_iter_copy_from_user_atomic+0x5d/0xc0
Jan  5 12:51:04 testbox kernel: [653657.522504]  [<c0195fdf>] generic_perform_write+0x10f/0x190
Jan  5 12:51:04 testbox kernel: [653657.522504]  [<c01970ad>] generic_file_buffered_write+0x5d/0x120
Jan  5 12:51:04 testbox kernel: [653657.522504]  [<c01986ab>] __generic_file_aio_write_nolock+0x19b/0x520
Jan  5 12:51:04 testbox kernel: [653657.522504]  [<c013949e>] ? resched_task+0x1e/0x70
Jan  5 12:51:04 testbox kernel: [653657.522504]  [<c0198b3e>] generic_file_aio_write+0x5e/0xd0
Jan  5 12:51:04 testbox kernel: [653657.522504]  [<c02194fd>] ext3_file_write+0x2d/0xc0
Jan  5 12:51:04 testbox kernel: [653657.522504]  [<c01c6831>] do_sync_write+0xd1/0x110
Jan  5 12:51:04 testbox kernel: [653657.522504]  [<c03f1082>] ? rtc_update_irq+0x92/0xa0
Jan  5 12:51:04 testbox kernel: [653657.522504]  [<c0156a70>] ? autoremove_wake_function+0x0/0x50
Jan  5 12:51:04 testbox kernel: [653657.522504]  [<c02b22f0>] ? apparmor_file_permission+0x20/0x30
Jan  5 12:51:04 testbox kernel: [653657.522504]  [<c029049f>] ? security_file_permission+0xf/0x20
Jan  5 12:51:04 testbox kernel: [653657.522504]  [<c01c69d4>] ? rw_verify_area+0x54/0xd0
Jan  5 12:51:04 testbox kernel: [653657.522504]  [<c01c6ec8>] vfs_write+0x98/0x110
Jan  5 12:51:04 testbox kernel: [653657.522504]  [<c01c6760>] ? do_sync_write+0x0/0x110
Jan  5 12:51:04 testbox kernel: [653657.522504]  [<c01c6ffd>] sys_write+0x3d/0x70
Jan  5 12:51:04 testbox kernel: [653657.522504]  [<c0109eef>] sysenter_do_call+0x12/0x2f
Jan  5 12:51:04 testbox kernel: [653657.526512] hpet1: lost 3 rtc interrupts
Jan  5 12:51:38 testbox kernel: [653691.880004] BUG: soft lockup - CPU#0 stuck for 61s! [hal-acl-tool:4168]
Jan  5 12:51:38 testbox kernel: [653691.880005] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan  5 12:51:38 testbox kernel: [653691.880859]
Jan  5 12:51:38 testbox kernel: [653691.880859] Pid: 4168, comm: hal-acl-tool Tainted: G      D W  (2.6.28-19-server #66-Ubuntu) System Product Name
Jan  5 12:51:38 testbox kernel: [653691.880859] EIP: 0060:[<c0129085>] EFLAGS: 00000293 CPU: 0
Jan  5 12:51:38 testbox kernel: [653691.880859] EIP is at __ticket_spin_lock+0x15/0x20
Jan  5 12:51:38 testbox kernel: [653691.880859] EAX: f5d962b8 EBX: d6ba3b00 ECX: f5d96200 EDX: 00004f4d
Jan  5 12:51:38 testbox kernel: [653691.880859] ESI: d20d3000 EDI: f5d962b8 EBP: e0491c60 ESP: e0491c60
Jan  5 12:51:38 testbox kernel: [653691.880859]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan  5 12:51:38 testbox kernel: [653691.880859] CR0: 8005003b CR2: 09202568 CR3: 2a445000 CR4: 000006b0
Jan  5 12:51:38 testbox kernel: [653691.880859] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan  5 12:51:38 testbox kernel: [653691.880859] DR6: ffff0ff0 DR7: 00000400
Jan  5 12:51:38 testbox kernel: [653691.880859] Call Trace:
Jan  5 12:51:38 testbox kernel: [653691.880859]  [<c050da98>] _spin_lock+0x8/0x10
Jan  5 12:51:38 testbox kernel: [653691.880859]  [<c025345d>] __journal_try_to_free_buffer+0x4d/0xc0
Jan  5 12:51:38 testbox kernel: [653691.880859]  [<c0253532>] journal_try_to_free_buffers+0x62/0x100
Jan  5 12:51:38 testbox kernel: [653691.880859]  [<c021bda5>] ext3_releasepage+0x45/0x70
Jan  5 12:51:38 testbox kernel: [653691.880859]  [<c0196c8e>] try_to_release_page+0x2e/0x50
Jan  5 12:51:38 testbox kernel: [653691.880859]  [<c01a0346>] __invalidate_mapping_pages+0x136/0x150
Jan  5 12:51:38 testbox kernel: [653691.880859]  [<c01a0372>] invalidate_mapping_pages+0x12/0x20
Jan  5 12:51:38 testbox kernel: [653691.880859]  [<c01d98ba>] prune_icache+0x21a/0x260
Jan  5 12:51:38 testbox kernel: [653691.880859]  [<c01d993e>] shrink_icache_memory+0x3e/0x50
Jan  5 12:51:38 testbox kernel: [653691.880859]  [<c01a086f>] shrink_slab+0x11f/0x180
Jan  5 12:51:38 testbox kernel: [653691.880859]  [<c01a2995>] do_try_to_free_pages+0x155/0x2c0
Jan  5 12:51:38 testbox kernel: [653691.880859]  [<c0156a8b>] ? autoremove_wake_function+0x1b/0x50
Jan  5 12:51:38 testbox kernel: [653691.880859]  [<c01a2bec>] try_to_free_pages+0x6c/0x80
Jan  5 12:51:38 testbox kernel: [653691.880859]  [<c01a13a0>] ? isolate_pages_global+0x0/0x70
Jan  5 12:51:38 testbox kernel: [653691.880859]  [<c019bdbb>] __alloc_pages_internal+0x1cb/0x450
Jan  5 12:51:38 testbox kernel: [653691.880859]  [<c01c004b>] new_slab+0x7b/0x1e0
Jan  5 12:51:38 testbox kernel: [653691.880859]  [<c01c1165>] ? unfreeze_slab+0x75/0xb0
Jan  5 12:51:38 testbox kernel: [653691.880859]  [<c01c141b>] __slab_alloc+0x20b/0x260
Jan  5 12:51:38 testbox kernel: [653691.880859]  [<c01c1516>] kmem_cache_alloc+0xa6/0xb0
Jan  5 12:51:38 testbox kernel: [653691.880859]  [<c013ef4c>] ? dup_task_struct+0x1c/0x110
Jan  5 12:51:38 testbox kernel: [653691.880859]  [<c013ef4c>] dup_task_struct+0x1c/0x110
Jan  5 12:51:38 testbox kernel: [653691.880859]  [<c013fd7e>] copy_process+0x6e/0xcf0
Jan  5 12:51:38 testbox kernel: [653691.880859]  [<c01db240>] ? alloc_fd+0xe0/0x100
Jan  5 12:51:38 testbox kernel: [653691.880859]  [<c01c7bbc>] ? get_empty_filp+0x6c/0x170
Jan  5 12:51:38 testbox kernel: [653691.880859]  [<c050da98>] ? _spin_lock+0x8/0x10
Jan  5 12:51:38 testbox kernel: [653691.880859]  [<c0140a62>] do_fork+0x62/0x300
Jan  5 12:51:38 testbox kernel: [653691.880859]  [<c01cd6ef>] ? sys_pipe2+0x3f/0x60
Jan  5 12:51:38 testbox kernel: [653691.880859]  [<c01083df>] sys_clone+0x2f/0x40
Jan  5 12:51:38 testbox kernel: [653691.880859]  [<c0109eef>] sysenter_do_call+0x12/0x2f
Jan  5 12:52:10 testbox kernel: [653723.022504] BUG: soft lockup - CPU#3 stuck for 61s! [rsync:30532]
Jan  5 12:52:10 testbox kernel: [653723.022504] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan  5 12:52:10 testbox kernel: [653723.022504]
Jan  5 12:52:10 testbox kernel: [653723.022504] Pid: 30532, comm: rsync Tainted: G      D W  (2.6.28-19-server #66-Ubuntu) System Product Name
Jan  5 12:52:10 testbox kernel: [653723.022504] EIP: 0060:[<c0129083>] EFLAGS: 00000297 CPU: 3
Jan  5 12:52:10 testbox kernel: [653723.022504] EIP is at __ticket_spin_lock+0x13/0x20
Jan  5 12:52:10 testbox kernel: [653723.022504] EAX: f5d962b8 EBX: e0c8de00 ECX: 00000000 EDX: 00004e4d
Jan  5 12:52:10 testbox kernel: [653723.022504] ESI: f5d96200 EDI: f6d00450 EBP: db0f1cc8 ESP: db0f1cc8
Jan  5 12:52:10 testbox kernel: [653723.022504]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan  5 12:52:10 testbox kernel: [653723.022504] CR0: 8005003b CR2: 00000000 CR3: 1b0d9000 CR4: 000006b0
Jan  5 12:52:10 testbox kernel: [653723.022504] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan  5 12:52:10 testbox kernel: [653723.022504] DR6: ffff0ff0 DR7: 00000400
Jan  5 12:52:10 testbox kernel: [653723.022504] Call Trace:
Jan  5 12:52:10 testbox kernel: [653723.022504]  [<c050da98>] _spin_lock+0x8/0x10
Jan  5 12:52:10 testbox kernel: [653723.022504]  [<c025393e>] journal_dirty_data+0x5e/0x1f0
Jan  5 12:52:10 testbox kernel: [653723.022504]  [<c021b048>] ext3_journal_dirty_data+0x18/0x50
Jan  5 12:52:10 testbox kernel: [653723.022504]  [<c021b7ef>] walk_page_buffers+0x2f/0x80
Jan  5 12:52:10 testbox kernel: [653723.022504]  [<c021bbf3>] ext3_ordered_write_end+0x63/0x1b0
Jan  5 12:52:10 testbox kernel: [653723.022504]  [<c021b030>] ? ext3_journal_dirty_data+0x0/0x50
Jan  5 12:52:10 testbox kernel: [653723.022504]  [<c0195e6d>] ? iov_iter_copy_from_user_atomic+0x5d/0xc0
Jan  5 12:52:10 testbox kernel: [653723.022504]  [<c0195fdf>] generic_perform_write+0x10f/0x190
Jan  5 12:52:10 testbox kernel: [653723.022504]  [<c01970ad>] generic_file_buffered_write+0x5d/0x120
Jan  5 12:52:10 testbox kernel: [653723.022504]  [<c01986ab>] __generic_file_aio_write_nolock+0x19b/0x520
Jan  5 12:52:10 testbox kernel: [653723.022504]  [<c013949e>] ? resched_task+0x1e/0x70
Jan  5 12:52:10 testbox kernel: [653723.022504]  [<c0198b3e>] generic_file_aio_write+0x5e/0xd0
Jan  5 12:52:10 testbox kernel: [653723.022504]  [<c02194fd>] ext3_file_write+0x2d/0xc0
Jan  5 12:52:10 testbox kernel: [653723.022504]  [<c01c6831>] do_sync_write+0xd1/0x110
Jan  5 12:52:10 testbox kernel: [653723.022504]  [<c03f1082>] ? rtc_update_irq+0x92/0xa0
Jan  5 12:52:10 testbox kernel: [653723.022504]  [<c0156a70>] ? autoremove_wake_function+0x0/0x50
Jan  5 12:52:10 testbox kernel: [653723.022504]  [<c02b22f0>] ? apparmor_file_permission+0x20/0x30
Jan  5 12:52:10 testbox kernel: [653723.022504]  [<c029049f>] ? security_file_permission+0xf/0x20
Jan  5 12:52:10 testbox kernel: [653723.022504]  [<c01c69d4>] ? rw_verify_area+0x54/0xd0
Jan  5 12:52:10 testbox kernel: [653723.022504]  [<c01c6ec8>] vfs_write+0x98/0x110
Jan  5 12:52:10 testbox kernel: [653723.022504]  [<c01c6760>] ? do_sync_write+0x0/0x110
Jan  5 12:52:10 testbox kernel: [653723.022504]  [<c01c6ffd>] sys_write+0x3d/0x70
Jan  5 12:52:10 testbox kernel: [653723.022504]  [<c0109eef>] sysenter_do_call+0x12/0x2f
Jan  5 12:52:10 testbox kernel: [653723.025920] hpet1: lost 3 rtc interrupts
Jan  5 12:52:44 testbox kernel: [653757.380003] BUG: soft lockup - CPU#0 stuck for 61s! [hal-acl-tool:4168]
Jan  5 12:52:44 testbox kernel: [653757.380005] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan  5 12:52:44 testbox kernel: [653757.381694]
Jan  5 12:52:44 testbox kernel: [653757.381694] Pid: 4168, comm: hal-acl-tool Tainted: G      D W  (2.6.28-19-server #66-Ubuntu) System Product Name
Jan  5 12:52:44 testbox kernel: [653757.381694] EIP: 0060:[<c012907f>] EFLAGS: 00000293 CPU: 0
Jan  5 12:52:44 testbox kernel: [653757.381694] EIP is at __ticket_spin_lock+0xf/0x20
Jan  5 12:52:44 testbox kernel: [653757.381694] EAX: f5d962b8 EBX: d6ba3b00 ECX: f5d96200 EDX: 00004f4d
Jan  5 12:52:44 testbox kernel: [653757.381694] ESI: d20d3000 EDI: f5d962b8 EBP: e0491c60 ESP: e0491c60
Jan  5 12:52:44 testbox kernel: [653757.381694]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan  5 12:52:44 testbox kernel: [653757.381694] CR0: 8005003b CR2: 09202568 CR3: 2a445000 CR4: 000006b0
Jan  5 12:52:44 testbox kernel: [653757.381694] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan  5 12:52:44 testbox kernel: [653757.381694] DR6: ffff0ff0 DR7: 00000400
Jan  5 12:52:44 testbox kernel: [653757.381694] Call Trace:
Jan  5 12:52:44 testbox kernel: [653757.381694]  [<c050da98>] _spin_lock+0x8/0x10
Jan  5 12:52:44 testbox kernel: [653757.381694]  [<c025345d>] __journal_try_to_free_buffer+0x4d/0xc0
Jan  5 12:52:44 testbox kernel: [653757.381694]  [<c0253532>] journal_try_to_free_buffers+0x62/0x100
Jan  5 12:52:44 testbox kernel: [653757.381694]  [<c021bda5>] ext3_releasepage+0x45/0x70
Jan  5 12:52:44 testbox kernel: [653757.381694]  [<c0196c8e>] try_to_release_page+0x2e/0x50
Jan  5 12:52:44 testbox kernel: [653757.381694]  [<c01a0346>] __invalidate_mapping_pages+0x136/0x150
Jan  5 12:52:44 testbox kernel: [653757.381694]  [<c01a0372>] invalidate_mapping_pages+0x12/0x20
Jan  5 12:52:44 testbox kernel: [653757.381694]  [<c01d98ba>] prune_icache+0x21a/0x260
Jan  5 12:52:44 testbox kernel: [653757.381694]  [<c01d993e>] shrink_icache_memory+0x3e/0x50
Jan  5 12:52:44 testbox kernel: [653757.381694]  [<c01a086f>] shrink_slab+0x11f/0x180
Jan  5 12:52:44 testbox kernel: [653757.381694]  [<c01a2995>] do_try_to_free_pages+0x155/0x2c0
Jan  5 12:52:44 testbox kernel: [653757.381694]  [<c0156a8b>] ? autoremove_wake_function+0x1b/0x50
Jan  5 12:52:44 testbox kernel: [653757.381694]  [<c01a2bec>] try_to_free_pages+0x6c/0x80
Jan  5 12:52:44 testbox kernel: [653757.381694]  [<c01a13a0>] ? isolate_pages_global+0x0/0x70
Jan  5 12:52:44 testbox kernel: [653757.381694]  [<c019bdbb>] __alloc_pages_internal+0x1cb/0x450
Jan  5 12:52:44 testbox kernel: [653757.381694]  [<c01c004b>] new_slab+0x7b/0x1e0
Jan  5 12:52:44 testbox kernel: [653757.381694]  [<c01c1165>] ? unfreeze_slab+0x75/0xb0
Jan  5 12:52:44 testbox kernel: [653757.381694]  [<c01c141b>] __slab_alloc+0x20b/0x260
Jan  5 12:52:44 testbox kernel: [653757.381694]  [<c01c1516>] kmem_cache_alloc+0xa6/0xb0
Jan  5 12:52:44 testbox kernel: [653757.381694]  [<c013ef4c>] ? dup_task_struct+0x1c/0x110
Jan  5 12:52:44 testbox kernel: [653757.381694]  [<c013ef4c>] dup_task_struct+0x1c/0x110
Jan  5 12:52:44 testbox kernel: [653757.381694]  [<c013fd7e>] copy_process+0x6e/0xcf0
Jan  5 12:52:44 testbox kernel: [653757.381694]  [<c01db240>] ? alloc_fd+0xe0/0x100
Jan  5 12:52:44 testbox kernel: [653757.381694]  [<c01c7bbc>] ? get_empty_filp+0x6c/0x170
Jan  5 12:52:44 testbox kernel: [653757.381694]  [<c050da98>] ? _spin_lock+0x8/0x10
Jan  5 12:52:44 testbox kernel: [653757.381694]  [<c0140a62>] do_fork+0x62/0x300
Jan  5 12:52:44 testbox kernel: [653757.381694]  [<c01cd6ef>] ? sys_pipe2+0x3f/0x60
Jan  5 12:52:44 testbox kernel: [653757.381694]  [<c01083df>] sys_clone+0x2f/0x40
Jan  5 12:52:44 testbox kernel: [653757.381694]  [<c0109eef>] sysenter_do_call+0x12/0x2f
Jan  5 12:53:15 testbox kernel: [653788.512503] BUG: soft lockup - CPU#3 stuck for 61s! [rsync:30532]
Jan  5 12:53:15 testbox kernel: [653788.512504] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan  5 12:53:15 testbox kernel: [653788.512504]
Jan  5 12:53:15 testbox kernel: [653788.512504] Pid: 30532, comm: rsync Tainted: G      D W  (2.6.28-19-server #66-Ubuntu) System Product Name
Jan  5 12:53:15 testbox kernel: [653788.512504] EIP: 0060:[<c0129081>] EFLAGS: 00000297 CPU: 3
Jan  5 12:53:15 testbox kernel: [653788.512504] EIP is at __ticket_spin_lock+0x11/0x20
Jan  5 12:53:15 testbox kernel: [653788.512504] EAX: f5d962b8 EBX: e0c8de00 ECX: 00000000 EDX: 00004e4d
Jan  5 12:53:15 testbox kernel: [653788.512504] ESI: f5d96200 EDI: f6d00450 EBP: db0f1cc8 ESP: db0f1cc8
Jan  5 12:53:15 testbox kernel: [653788.512504]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan  5 12:53:15 testbox kernel: [653788.512504] CR0: 8005003b CR2: 00000000 CR3: 1b0d9000 CR4: 000006b0
Jan  5 12:53:15 testbox kernel: [653788.512504] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan  5 12:53:15 testbox kernel: [653788.512504] DR6: ffff0ff0 DR7: 00000400
Jan  5 12:53:15 testbox kernel: [653788.512504] Call Trace:
Jan  5 12:53:15 testbox kernel: [653788.512504]  [<c050da98>] _spin_lock+0x8/0x10
Jan  5 12:53:15 testbox kernel: [653788.512504]  [<c025393e>] journal_dirty_data+0x5e/0x1f0
Jan  5 12:53:15 testbox kernel: [653788.512504]  [<c021b048>] ext3_journal_dirty_data+0x18/0x50
Jan  5 12:53:15 testbox kernel: [653788.512504]  [<c021b7ef>] walk_page_buffers+0x2f/0x80
Jan  5 12:53:15 testbox kernel: [653788.512504]  [<c021bbf3>] ext3_ordered_write_end+0x63/0x1b0
Jan  5 12:53:15 testbox kernel: [653788.512504]  [<c021b030>] ? ext3_journal_dirty_data+0x0/0x50
Jan  5 12:53:15 testbox kernel: [653788.512504]  [<c0195e6d>] ? iov_iter_copy_from_user_atomic+0x5d/0xc0
Jan  5 12:53:15 testbox kernel: [653788.512504]  [<c0195fdf>] generic_perform_write+0x10f/0x190
Jan  5 12:53:15 testbox kernel: [653788.512504]  [<c01970ad>] generic_file_buffered_write+0x5d/0x120
Jan  5 12:53:15 testbox kernel: [653788.512504]  [<c01986ab>] __generic_file_aio_write_nolock+0x19b/0x520
Jan  5 12:53:15 testbox kernel: [653788.512504]  [<c013949e>] ? resched_task+0x1e/0x70
Jan  5 12:53:15 testbox kernel: [653788.512504]  [<c0198b3e>] generic_file_aio_write+0x5e/0xd0
Jan  5 12:53:15 testbox kernel: [653788.512504]  [<c02194fd>] ext3_file_write+0x2d/0xc0
Jan  5 12:53:15 testbox kernel: [653788.512504]  [<c01c6831>] do_sync_write+0xd1/0x110
Jan  5 12:53:15 testbox kernel: [653788.512504]  [<c03f1082>] ? rtc_update_irq+0x92/0xa0
Jan  5 12:53:15 testbox kernel: [653788.512504]  [<c0156a70>] ? autoremove_wake_function+0x0/0x50
Jan  5 12:53:15 testbox kernel: [653788.512504]  [<c02b22f0>] ? apparmor_file_permission+0x20/0x30
Jan  5 12:53:15 testbox kernel: [653788.512504]  [<c029049f>] ? security_file_permission+0xf/0x20
Jan  5 12:53:15 testbox kernel: [653788.512504]  [<c01c69d4>] ? rw_verify_area+0x54/0xd0
Jan  5 12:53:15 testbox kernel: [653788.512504]  [<c01c6ec8>] vfs_write+0x98/0x110
Jan  5 12:53:15 testbox kernel: [653788.512504]  [<c01c6760>] ? do_sync_write+0x0/0x110
Jan  5 12:53:15 testbox kernel: [653788.512504]  [<c01c6ffd>] sys_write+0x3d/0x70
Jan  5 12:53:15 testbox kernel: [653788.512504]  [<c0109eef>] sysenter_do_call+0x12/0x2f
Jan  5 12:53:15 testbox kernel: [653788.516515] hpet1: lost 3 rtc interrupts
Jan  5 12:53:49 testbox kernel: [653822.880003] BUG: soft lockup - CPU#0 stuck for 61s! [hal-acl-tool:4168]
Jan  5 12:53:49 testbox kernel: [653822.880005] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan  5 12:53:49 testbox kernel: [653822.881681]
Jan  5 12:53:49 testbox kernel: [653822.881681] Pid: 4168, comm: hal-acl-tool Tainted: G      D W  (2.6.28-19-server #66-Ubuntu) System Product Name
Jan  5 12:53:49 testbox kernel: [653822.881681] EIP: 0060:[<c012907f>] EFLAGS: 00000293 CPU: 0
Jan  5 12:53:49 testbox kernel: [653822.881681] EIP is at __ticket_spin_lock+0xf/0x20
Jan  5 12:53:49 testbox kernel: [653822.881681] EAX: f5d962b8 EBX: d6ba3b00 ECX: f5d96200 EDX: 00004f4d
Jan  5 12:53:49 testbox kernel: [653822.881681] ESI: d20d3000 EDI: f5d962b8 EBP: e0491c60 ESP: e0491c60
Jan  5 12:53:49 testbox kernel: [653822.881681]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan  5 12:53:49 testbox kernel: [653822.881681] CR0: 8005003b CR2: 09202568 CR3: 2a445000 CR4: 000006b0
Jan  5 12:53:49 testbox kernel: [653822.881681] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan  5 12:53:49 testbox kernel: [653822.881681] DR6: ffff0ff0 DR7: 00000400
Jan  5 12:53:49 testbox kernel: [653822.881681] Call Trace:
Jan  5 12:53:49 testbox kernel: [653822.881681]  [<c050da98>] _spin_lock+0x8/0x10
Jan  5 12:53:49 testbox kernel: [653822.881681]  [<c025345d>] __journal_try_to_free_buffer+0x4d/0xc0
Jan  5 12:53:49 testbox kernel: [653822.881681]  [<c0253532>] journal_try_to_free_buffers+0x62/0x100
Jan  5 12:53:49 testbox kernel: [653822.881681]  [<c021bda5>] ext3_releasepage+0x45/0x70
Jan  5 12:53:49 testbox kernel: [653822.881681]  [<c0196c8e>] try_to_release_page+0x2e/0x50
Jan  5 12:53:49 testbox kernel: [653822.881681]  [<c01a0346>] __invalidate_mapping_pages+0x136/0x150
Jan  5 12:53:49 testbox kernel: [653822.881681]  [<c01a0372>] invalidate_mapping_pages+0x12/0x20
Jan  5 12:53:49 testbox kernel: [653822.881681]  [<c01d98ba>] prune_icache+0x21a/0x260
Jan  5 12:53:49 testbox kernel: [653822.881681]  [<c01d993e>] shrink_icache_memory+0x3e/0x50
Jan  5 12:53:49 testbox kernel: [653822.881681]  [<c01a086f>] shrink_slab+0x11f/0x180
Jan  5 12:53:49 testbox kernel: [653822.881681]  [<c01a2995>] do_try_to_free_pages+0x155/0x2c0
Jan  5 12:53:49 testbox kernel: [653822.881681]  [<c0156a8b>] ? autoremove_wake_function+0x1b/0x50
Jan  5 12:53:49 testbox kernel: [653822.881681]  [<c01a2bec>] try_to_free_pages+0x6c/0x80
Jan  5 12:53:49 testbox kernel: [653822.881681]  [<c01a13a0>] ? isolate_pages_global+0x0/0x70
Jan  5 12:53:50 testbox kernel: [653822.881681]  [<c019bdbb>] __alloc_pages_internal+0x1cb/0x450
Jan  5 12:53:50 testbox kernel: [653822.881681]  [<c01c004b>] new_slab+0x7b/0x1e0
Jan  5 12:53:50 testbox kernel: [653822.881681]  [<c01c1165>] ? unfreeze_slab+0x75/0xb0
Jan  5 12:53:50 testbox kernel: [653822.881681]  [<c01c141b>] __slab_alloc+0x20b/0x260
Jan  5 12:53:50 testbox kernel: [653822.881681]  [<c01c1516>] kmem_cache_alloc+0xa6/0xb0
Jan  5 12:53:50 testbox kernel: [653822.881681]  [<c013ef4c>] ? dup_task_struct+0x1c/0x110
Jan  5 12:53:50 testbox kernel: [653822.881681]  [<c013ef4c>] dup_task_struct+0x1c/0x110
Jan  5 12:53:50 testbox kernel: [653822.881681]  [<c013fd7e>] copy_process+0x6e/0xcf0
Jan  5 12:53:50 testbox kernel: [653822.881681]  [<c01db240>] ? alloc_fd+0xe0/0x100
Jan  5 12:53:50 testbox kernel: [653822.881681]  [<c01c7bbc>] ? get_empty_filp+0x6c/0x170
Jan  5 12:53:50 testbox kernel: [653822.881681]  [<c050da98>] ? _spin_lock+0x8/0x10
Jan  5 12:53:50 testbox kernel: [653822.881681]  [<c0140a62>] do_fork+0x62/0x300
Jan  5 12:53:50 testbox kernel: [653822.881681]  [<c01cd6ef>] ? sys_pipe2+0x3f/0x60
Jan  5 12:53:50 testbox kernel: [653822.881681]  [<c01083df>] sys_clone+0x2f/0x40
Jan  5 12:53:50 testbox kernel: [653822.881681]  [<c0109eef>] sysenter_do_call+0x12/0x2f
Jan  5 12:54:21 testbox kernel: [653854.012503] BUG: soft lockup - CPU#3 stuck for 61s! [rsync:30532]
Jan  5 12:54:21 testbox kernel: [653854.012504] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan  5 12:54:21 testbox kernel: [653854.012504]
Jan  5 12:54:21 testbox kernel: [653854.012504] Pid: 30532, comm: rsync Tainted: G      D W  (2.6.28-19-server #66-Ubuntu) System Product Name
Jan  5 12:54:21 testbox kernel: [653854.012504] EIP: 0060:[<c0129085>] EFLAGS: 00000297 CPU: 3
Jan  5 12:54:21 testbox kernel: [653854.012504] EIP is at __ticket_spin_lock+0x15/0x20
Jan  5 12:54:21 testbox kernel: [653854.012504] EAX: f5d962b8 EBX: e0c8de00 ECX: 00000000 EDX: 00004e4d
Jan  5 12:54:21 testbox kernel: [653854.012504] ESI: f5d96200 EDI: f6d00450 EBP: db0f1cc8 ESP: db0f1cc8
Jan  5 12:54:21 testbox kernel: [653854.012504]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan  5 12:54:21 testbox kernel: [653854.012504] CR0: 8005003b CR2: 00000000 CR3: 1b0d9000 CR4: 000006b0
Jan  5 12:54:21 testbox kernel: [653854.012504] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan  5 12:54:21 testbox kernel: [653854.012504] DR6: ffff0ff0 DR7: 00000400
Jan  5 12:54:21 testbox kernel: [653854.012504] Call Trace:
Jan  5 12:54:21 testbox kernel: [653854.012504]  [<c050da98>] _spin_lock+0x8/0x10
Jan  5 12:54:21 testbox kernel: [653854.012504]  [<c025393e>] journal_dirty_data+0x5e/0x1f0
Jan  5 12:54:21 testbox kernel: [653854.012504]  [<c021b048>] ext3_journal_dirty_data+0x18/0x50
Jan  5 12:54:21 testbox kernel: [653854.012504]  [<c021b7ef>] walk_page_buffers+0x2f/0x80
Jan  5 12:54:21 testbox kernel: [653854.012504]  [<c021bbf3>] ext3_ordered_write_end+0x63/0x1b0
Jan  5 12:54:21 testbox kernel: [653854.012504]  [<c021b030>] ? ext3_journal_dirty_data+0x0/0x50
Jan  5 12:54:21 testbox kernel: [653854.012504]  [<c0195e6d>] ? iov_iter_copy_from_user_atomic+0x5d/0xc0
Jan  5 12:54:21 testbox kernel: [653854.012504]  [<c0195fdf>] generic_perform_write+0x10f/0x190
Jan  5 12:54:21 testbox kernel: [653854.012504]  [<c01970ad>] generic_file_buffered_write+0x5d/0x120
Jan  5 12:54:21 testbox kernel: [653854.012504]  [<c01986ab>] __generic_file_aio_write_nolock+0x19b/0x520
Jan  5 12:54:21 testbox kernel: [653854.012504]  [<c013949e>] ? resched_task+0x1e/0x70
Jan  5 12:54:21 testbox kernel: [653854.012504]  [<c0198b3e>] generic_file_aio_write+0x5e/0xd0
Jan  5 12:54:21 testbox kernel: [653854.012504]  [<c02194fd>] ext3_file_write+0x2d/0xc0
Jan  5 12:54:21 testbox kernel: [653854.012504]  [<c01c6831>] do_sync_write+0xd1/0x110
Jan  5 12:54:21 testbox kernel: [653854.012504]  [<c03f1082>] ? rtc_update_irq+0x92/0xa0
Jan  5 12:54:21 testbox kernel: [653854.012504]  [<c0156a70>] ? autoremove_wake_function+0x0/0x50
Jan  5 12:54:21 testbox kernel: [653854.012504]  [<c02b22f0>] ? apparmor_file_permission+0x20/0x30
Jan  5 12:54:21 testbox kernel: [653854.012504]  [<c029049f>] ? security_file_permission+0xf/0x20
Jan  5 12:54:21 testbox kernel: [653854.012504]  [<c01c69d4>] ? rw_verify_area+0x54/0xd0
Jan  5 12:54:21 testbox kernel: [653854.012504]  [<c01c6ec8>] vfs_write+0x98/0x110
Jan  5 12:54:21 testbox kernel: [653854.012504]  [<c01c6760>] ? do_sync_write+0x0/0x110
Jan  5 12:54:21 testbox kernel: [653854.012504]  [<c01c6ffd>] sys_write+0x3d/0x70
Jan  5 12:54:21 testbox kernel: [653854.012504]  [<c0109eef>] sysenter_do_call+0x12/0x2f
Jan  5 12:54:21 testbox kernel: [653854.015940] hpet1: lost 2 rtc interrupts
Jan  5 12:54:55 testbox kernel: [653888.370004] BUG: soft lockup - CPU#0 stuck for 61s! [hal-acl-tool:4168]
Jan  5 12:54:55 testbox kernel: [653888.370005] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan  5 12:54:55 testbox kernel: [653888.371683]
Jan  5 12:54:55 testbox kernel: [653888.371683] Pid: 4168, comm: hal-acl-tool Tainted: G      D W  (2.6.28-19-server #66-Ubuntu) System Product Name
Jan  5 12:54:55 testbox kernel: [653888.371683] EIP: 0060:[<c012907f>] EFLAGS: 00000293 CPU: 0
Jan  5 12:54:55 testbox kernel: [653888.371683] EIP is at __ticket_spin_lock+0xf/0x20
Jan  5 12:54:55 testbox kernel: [653888.371683] EAX: f5d962b8 EBX: d6ba3b00 ECX: f5d96200 EDX: 00004f4d
Jan  5 12:54:55 testbox kernel: [653888.371683] ESI: d20d3000 EDI: f5d962b8 EBP: e0491c60 ESP: e0491c60
Jan  5 12:54:55 testbox kernel: [653888.371683]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan  5 12:54:55 testbox kernel: [653888.371683] CR0: 8005003b CR2: 09202568 CR3: 2a445000 CR4: 000006b0
Jan  5 12:54:55 testbox kernel: [653888.371683] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan  5 12:54:55 testbox kernel: [653888.371683] DR6: ffff0ff0 DR7: 00000400
Jan  5 12:54:55 testbox kernel: [653888.371683] Call Trace:
Jan  5 12:54:55 testbox kernel: [653888.371683]  [<c050da98>] _spin_lock+0x8/0x10
Jan  5 12:54:55 testbox kernel: [653888.371683]  [<c025345d>] __journal_try_to_free_buffer+0x4d/0xc0
Jan  5 12:54:55 testbox kernel: [653888.371683]  [<c0253532>] journal_try_to_free_buffers+0x62/0x100
Jan  5 12:54:55 testbox kernel: [653888.371683]  [<c021bda5>] ext3_releasepage+0x45/0x70
Jan  5 12:54:55 testbox kernel: [653888.371683]  [<c0196c8e>] try_to_release_page+0x2e/0x50
Jan  5 12:54:55 testbox kernel: [653888.371683]  [<c01a0346>] __invalidate_mapping_pages+0x136/0x150
Jan  5 12:54:55 testbox kernel: [653888.371683]  [<c01a0372>] invalidate_mapping_pages+0x12/0x20
Jan  5 12:54:55 testbox kernel: [653888.371683]  [<c01d98ba>] prune_icache+0x21a/0x260
Jan  5 12:54:55 testbox kernel: [653888.371683]  [<c01d993e>] shrink_icache_memory+0x3e/0x50
Jan  5 12:54:55 testbox kernel: [653888.371683]  [<c01a086f>] shrink_slab+0x11f/0x180
Jan  5 12:54:55 testbox kernel: [653888.371683]  [<c01a2995>] do_try_to_free_pages+0x155/0x2c0
Jan  5 12:54:55 testbox kernel: [653888.371683]  [<c0156a8b>] ? autoremove_wake_function+0x1b/0x50
Jan  5 12:54:55 testbox kernel: [653888.371683]  [<c01a2bec>] try_to_free_pages+0x6c/0x80
Jan  5 12:54:55 testbox kernel: [653888.371683]  [<c01a13a0>] ? isolate_pages_global+0x0/0x70
Jan  5 12:54:55 testbox kernel: [653888.371683]  [<c019bdbb>] __alloc_pages_internal+0x1cb/0x450
Jan  5 12:54:55 testbox kernel: [653888.371683]  [<c01c004b>] new_slab+0x7b/0x1e0
Jan  5 12:54:55 testbox kernel: [653888.371683]  [<c01c1165>] ? unfreeze_slab+0x75/0xb0
Jan  5 12:54:55 testbox kernel: [653888.371683]  [<c01c141b>] __slab_alloc+0x20b/0x260
Jan  5 12:54:55 testbox kernel: [653888.371683]  [<c01c1516>] kmem_cache_alloc+0xa6/0xb0
Jan  5 12:54:55 testbox kernel: [653888.371683]  [<c013ef4c>] ? dup_task_struct+0x1c/0x110
Jan  5 12:54:55 testbox kernel: [653888.371683]  [<c013ef4c>] dup_task_struct+0x1c/0x110
Jan  5 12:54:55 testbox kernel: [653888.371683]  [<c013fd7e>] copy_process+0x6e/0xcf0
Jan  5 12:54:55 testbox kernel: [653888.371683]  [<c01db240>] ? alloc_fd+0xe0/0x100
Jan  5 12:54:55 testbox kernel: [653888.371683]  [<c01c7bbc>] ? get_empty_filp+0x6c/0x170
Jan  5 12:54:55 testbox kernel: [653888.371683]  [<c050da98>] ? _spin_lock+0x8/0x10
Jan  5 12:54:55 testbox kernel: [653888.371683]  [<c0140a62>] do_fork+0x62/0x300
Jan  5 12:54:55 testbox kernel: [653888.371683]  [<c01cd6ef>] ? sys_pipe2+0x3f/0x60
Jan  5 12:54:55 testbox kernel: [653888.371683]  [<c01083df>] sys_clone+0x2f/0x40
Jan  5 12:54:55 testbox kernel: [653888.371683]  [<c0109eef>] sysenter_do_call+0x12/0x2f
 


Tags:

ubuntu, crashcpu, kernel, mb, raid, testbox, lockup, cpu, acl, modules, linked, vmnet, vmci, vmmon, binfmt_misc, drbd, output, input_polldev, ocfs, _stackglue, _dlmfs, _dlm, _nodemanager, configfs, temp, hwmon_vid, lp, snd_hda_intel, snd_pcm_oss, snd_mixer_oss, snd_pcm, snd_seq_dummy, snd_seq_oss, snd_seq_midi, snd_rawmidi, snd_seq_midi_event, snd_seq, snd_timer, snd_seq_device, psmouse, ppdev, snd, soundcore, serio_raw, pcspkr, c_piix, parport_pc, parport, shpchp, snd_page_alloc, usb_storage, mii, async_xor, async_memcpy, async_tx, xor, multipath, linear, fbcon, tileblit, font, bitblit, softcursor, pid, comm, tainted, server, eip, eflags, __ticket_spin_lock, eax, ebx, ecx, edx, esi, edi, ebp, esp, ds, es, fs, gs, ss, cr, ffff, ff, _spin_lock, __journal_try_to_free_buffer, xc, journal_try_to_free_buffers, bda, ext, _releasepage, try_to_release_page, __invalidate_mapping_pages, invalidate_mapping_pages, prune_icache, shrink_icache_memory, shrink_slab, do_try_to_free_pages, autoremove_wake_function, bec, try_to_free_pages, isolate_pages_global, bdbb, __alloc_pages_internal, cb, new_slab, unfreeze_slab, xb, __slab_alloc, kmem_cache_alloc, xa, ef, dup_task_struct, fd, copy_process, xcf, db, alloc_fd, xe, bbc, get_empty_filp, do_fork, sys_pipe, df, sys_clone, eef, sysenter_do_call, rsync, cc, journal_dirty_data, _journal_dirty_data, walk_page_buffers, bbf, _ordered_write_end, iov_iter_copy_from_user_atomic, fdf, generic_perform_write, generic_file_buffered_write, ab, __generic_file_aio_write_nolock, resched_task, generic_file_aio_write, xd, _file_write, do_sync_write, rtc_update_irq, apparmor_file_permission, security_file_permission, xf, rw_verify_area, ec, vfs_write, ffd, sys_write, hpet, rtc, interrupts,

Latest Articles

  • Cisco Unified Communication Manager (CUCM) - How To Add Phones
  • pptp / pptpd not working in DD-WRT iptables / router
  • systemd-journald high memory usage solution
  • How to Install FreePBX in Linux Debian Ubuntu Mint Guide
  • How To Install Cisco's CUCM (Cisco Unified Communication Manager) 12 Guide
  • Linux Ubuntu Redhat How To Extract Images from PDF
  • Linux and Windows Dual Boot Issue NIC Won't work After Booting Windows
  • Cisco CME How To Enable ACD hunt groups
  • How to install gns3 on Linux Ubuntu Mint
  • How to convert audio for Asterisk .wav format
  • Using Cisco CME Router with Asterisk as a dial-peer
  • Cisco CME How To Configure SIP Trunk VOIP
  • Virtualbox host Only Network Error Failed to save host network interface parameter - Cannot change gateway IP of host only network
  • Cisco CME and C7200 Router Testing and Learning Environment on Ubuntu 20+ Setup Tutorial Guide
  • Abusive IP ranges blacklist
  • How to Install Any OS on a Physical Drive from Windows Using VMware Workstation (Linux, Windows, BSD)
  • CDN Cloudflare how to set and preserve the real IP of the client without modifying application code on Apache
  • CentOS 7 fix Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=container error was 14: curl#6 -
  • Ubuntu Debian How To Install Recommended Packages Automatically
  • How to set Linux Ubuntu Redhat Debian Command Line http https socks proxy for yum apt