Ubuntu 9.04 Crash

CPU/Kernel/MB/RAID problem?

Jan 5 12:45:05 testbox kernel: [653298.890004] BUG: soft lockup - CPU#0 stuck for 61s! [hal-acl-tool:4168]
Jan 5 12:45:05 testbox kernel: [653298.890005] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:45:05 testbox kernel: [653298.891687]
Jan 5 12:45:05 testbox kernel: [653298.891687] Pid: 4168, comm: hal-acl-tool Tainted: G      D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:45:05 testbox kernel: [653298.891687] EIP: 0060:[<c0129085>] EFLAGS: 00000293 CPU: 0
Jan 5 12:45:05 testbox kernel: [653298.891687] EIP is at __ticket_spin_lock+0x15/0x20
Jan 5 12:45:05 testbox kernel: [653298.891687] EAX: f5d962b8 EBX: d6ba3b00 ECX: f5d96200 EDX: 00004f4d
Jan 5 12:45:05 testbox kernel: [653298.891687] ESI: d20d3000 EDI: f5d962b8 EBP: e0491c60 ESP: e0491c60
Jan 5 12:45:05 testbox kernel: [653298.891687] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:45:05 testbox kernel: [653298.891687] CR0: 8005003b CR2: 09202568 CR3: 2a445000 CR4: 000006b0
Jan 5 12:45:05 testbox kernel: [653298.891687] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:45:05 testbox kernel: [653298.891687] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:45:05 testbox kernel: [653298.891687] Call Trace:
Jan 5 12:45:05 testbox kernel: [653298.891687] [<c050da98>] _spin_lock+0x8/0x10
Jan 5 12:45:05 testbox kernel: [653298.891687] [<c025345d>] __journal_try_to_free_buffer+0x4d/0xc0
Jan 5 12:45:05 testbox kernel: [653298.891687] [<c0253532>] journal_try_to_free_buffers+0x62/0x100
Jan 5 12:45:05 testbox kernel: [653298.891687] [<c021bda5>] ext3_releasepage+0x45/0x70
Jan 5 12:45:05 testbox kernel: [653298.891687] [<c0196c8e>] try_to_release_page+0x2e/0x50
Jan 5 12:45:05 testbox kernel: [653298.891687] [<c01a0346>] __invalidate_mapping_pages+0x136/0x150
Jan 5 12:45:05 testbox kernel: [653298.891687] [<c01a0372>] invalidate_mapping_pages+0x12/0x20
Jan 5 12:45:05 testbox kernel: [653298.891687] [<c01d98ba>] prune_icache+0x21a/0x260
Jan 5 12:45:05 testbox kernel: [653298.891687] [<c01d993e>] shrink_icache_memory+0x3e/0x50
Jan 5 12:45:05 testbox kernel: [653298.891687] [<c01a086f>] shrink_slab+0x11f/0x180
Jan 5 12:45:05 testbox kernel: [653298.891687] [<c01a2995>] do_try_to_free_pages+0x155/0x2c0
Jan 5 12:45:05 testbox kernel: [653298.891687] [<c0156a8b>] ? autoremove_wake_function+0x1b/0x50
Jan 5 12:45:05 testbox kernel: [653298.891687] [<c01a2bec>] try_to_free_pages+0x6c/0x80
Jan 5 12:45:05 testbox kernel: [653298.891687] [<c01a13a0>] ? isolate_pages_global+0x0/0x70
Jan 5 12:45:05 testbox kernel: [653298.891687] [<c019bdbb>] __alloc_pages_internal+0x1cb/0x450
Jan 5 12:45:05 testbox kernel: [653298.891687] [<c01c004b>] new_slab+0x7b/0x1e0
Jan 5 12:45:05 testbox kernel: [653298.891687] [<c01c1165>] ? unfreeze_slab+0x75/0xb0
Jan 5 12:45:05 testbox kernel: [653298.891687] [<c01c141b>] __slab_alloc+0x20b/0x260
Jan 5 12:45:05 testbox kernel: [653298.891687] [<c01c1516>] kmem_cache_alloc+0xa6/0xb0
Jan 5 12:45:05 testbox kernel: [653298.891687] [<c013ef4c>] ? dup_task_struct+0x1c/0x110
Jan 5 12:45:05 testbox kernel: [653298.891687] [<c013ef4c>] dup_task_struct+0x1c/0x110
Jan 5 12:45:05 testbox kernel: [653298.891687] [<c013fd7e>] copy_process+0x6e/0xcf0
Jan 5 12:45:05 testbox kernel: [653298.891687] [<c01db240>] ? alloc_fd+0xe0/0x100
Jan 5 12:45:05 testbox kernel: [653298.891687] [<c01c7bbc>] ? get_empty_filp+0x6c/0x170
Jan 5 12:45:05 testbox kernel: [653298.891687] [<c050da98>] ? _spin_lock+0x8/0x10
Jan 5 12:45:05 testbox kernel: [653298.891687] [<c0140a62>] do_fork+0x62/0x300
Jan 5 12:45:05 testbox kernel: [653298.891687] [<c01cd6ef>] ? sys_pipe2+0x3f/0x60
Jan 5 12:45:05 testbox kernel: [653298.891687] [<c01083df>] sys_clone+0x2f/0x40
Jan 5 12:45:05 testbox kernel: [653298.891687] [<c0109eef>] sysenter_do_call+0x12/0x2f
Jan 5 12:45:37 testbox kernel: [653330.032503] BUG: soft lockup - CPU#3 stuck for 61s! [rsync:30532]
Jan 5 12:45:37 testbox kernel: [653330.032503] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:45:37 testbox kernel: [653330.032503]
Jan 5 12:45:37 testbox kernel: [653330.032503] Pid: 30532, comm: rsync Tainted: G      D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:45:37 testbox kernel: [653330.032503] EIP: 0060:[<c0129083>] EFLAGS: 00000297 CPU: 3
Jan 5 12:45:37 testbox kernel: [653330.032503] EIP is at __ticket_spin_lock+0x13/0x20
Jan 5 12:45:37 testbox kernel: [653330.032503] EAX: f5d962b8 EBX: e0c8de00 ECX: 00000000 EDX: 00004e4d
Jan 5 12:45:37 testbox kernel: [653330.032503] ESI: f5d96200 EDI: f6d00450 EBP: db0f1cc8 ESP: db0f1cc8
Jan 5 12:45:37 testbox kernel: [653330.032503] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:45:37 testbox kernel: [653330.032503] CR0: 8005003b CR2: 00000000 CR3: 1b0d9000 CR4: 000006b0
Jan 5 12:45:37 testbox kernel: [653330.032503] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:45:37 testbox kernel: [653330.032503] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:45:37 testbox kernel: [653330.032503] Call Trace:
Jan 5 12:45:37 testbox kernel: [653330.032503] [<c050da98>] _spin_lock+0x8/0x10
Jan 5 12:45:37 testbox kernel: [653330.032503] [<c025393e>] journal_dirty_data+0x5e/0x1f0
Jan 5 12:45:37 testbox kernel: [653330.032503] [<c021b048>] ext3_journal_dirty_data+0x18/0x50
Jan 5 12:45:37 testbox kernel: [653330.032503] [<c021b7ef>] walk_page_buffers+0x2f/0x80
Jan 5 12:45:37 testbox kernel: [653330.032503] [<c021bbf3>] ext3_ordered_write_end+0x63/0x1b0
Jan 5 12:45:37 testbox kernel: [653330.032503] [<c021b030>] ? ext3_journal_dirty_data+0x0/0x50
Jan 5 12:45:37 testbox kernel: [653330.032503] [<c0195e6d>] ? iov_iter_copy_from_user_atomic+0x5d/0xc0
Jan 5 12:45:37 testbox kernel: [653330.032503] [<c0195fdf>] generic_perform_write+0x10f/0x190
Jan 5 12:45:37 testbox kernel: [653330.032503] [<c01970ad>] generic_file_buffered_write+0x5d/0x120
Jan 5 12:45:37 testbox kernel: [653330.032503] [<c01986ab>] __generic_file_aio_write_nolock+0x19b/0x520
Jan 5 12:45:37 testbox kernel: [653330.032503] [<c013949e>] ? resched_task+0x1e/0x70
Jan 5 12:45:37 testbox kernel: [653330.032503] [<c0198b3e>] generic_file_aio_write+0x5e/0xd0
Jan 5 12:45:37 testbox kernel: [653330.032503] [<c02194fd>] ext3_file_write+0x2d/0xc0
Jan 5 12:45:37 testbox kernel: [653330.032503] [<c01c6831>] do_sync_write+0xd1/0x110
Jan 5 12:45:37 testbox kernel: [653330.032503] [<c03f1082>] ? rtc_update_irq+0x92/0xa0
Jan 5 12:45:37 testbox kernel: [653330.032503] [<c0156a70>] ? autoremove_wake_function+0x0/0x50
Jan 5 12:45:37 testbox kernel: [653330.032503] [<c02b22f0>] ? apparmor_file_permission+0x20/0x30
Jan 5 12:45:37 testbox kernel: [653330.032503] [<c029049f>] ? security_file_permission+0xf/0x20
Jan 5 12:45:37 testbox kernel: [653330.032503] [<c01c69d4>] ? rw_verify_area+0x54/0xd0
Jan 5 12:45:37 testbox kernel: [653330.032503] [<c01c6ec8>] vfs_write+0x98/0x110
Jan 5 12:45:37 testbox kernel: [653330.032503] [<c01c6760>] ? do_sync_write+0x0/0x110
Jan 5 12:45:37 testbox kernel: [653330.032503] [<c01c6ffd>] sys_write+0x3d/0x70
Jan 5 12:45:37 testbox kernel: [653330.032503] [<c0109eef>] sysenter_do_call+0x12/0x2f
Jan 5 12:45:37 testbox kernel: [653330.035919] hpet1: lost 3 rtc interrupts
Jan 5 12:46:11 testbox kernel: [653364.390003] BUG: soft lockup - CPU#0 stuck for 61s! [hal-acl-tool:4168]
Jan 5 12:46:11 testbox kernel: [653364.390005] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:46:11 testbox kernel: [653364.391694]
Jan 5 12:46:11 testbox kernel: [653364.391694] Pid: 4168, comm: hal-acl-tool Tainted: G      D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:46:11 testbox kernel: [653364.391694] EIP: 0060:[<c0129083>] EFLAGS: 00000293 CPU: 0
Jan 5 12:46:11 testbox kernel: [653364.391694] EIP is at __ticket_spin_lock+0x13/0x20
Jan 5 12:46:11 testbox kernel: [653364.391694] EAX: f5d962b8 EBX: d6ba3b00 ECX: f5d96200 EDX: 00004f4d
Jan 5 12:46:11 testbox kernel: [653364.391694] ESI: d20d3000 EDI: f5d962b8 EBP: e0491c60 ESP: e0491c60
Jan 5 12:46:11 testbox kernel: [653364.391694] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:46:11 testbox kernel: [653364.391694] CR0: 8005003b CR2: 09202568 CR3: 2a445000 CR4: 000006b0
Jan 5 12:46:11 testbox kernel: [653364.391694] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:46:11 testbox kernel: [653364.391694] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:46:11 testbox kernel: [653364.391694] Call Trace:
Jan 5 12:46:11 testbox kernel: [653364.391694] [<c050da98>] _spin_lock+0x8/0x10
Jan 5 12:46:11 testbox kernel: [653364.391694] [<c025345d>] __journal_try_to_free_buffer+0x4d/0xc0
Jan 5 12:46:11 testbox kernel: [653364.391694] [<c0253532>] journal_try_to_free_buffers+0x62/0x100
Jan 5 12:46:11 testbox kernel: [653364.391694] [<c021bda5>] ext3_releasepage+0x45/0x70
Jan 5 12:46:11 testbox kernel: [653364.391694] [<c0196c8e>] try_to_release_page+0x2e/0x50
Jan 5 12:46:11 testbox kernel: [653364.391694] [<c01a0346>] __invalidate_mapping_pages+0x136/0x150
Jan 5 12:46:11 testbox kernel: [653364.391694] [<c01a0372>] invalidate_mapping_pages+0x12/0x20
Jan 5 12:46:11 testbox kernel: [653364.391694] [<c01d98ba>] prune_icache+0x21a/0x260
Jan 5 12:46:11 testbox kernel: [653364.391694] [<c01d993e>] shrink_icache_memory+0x3e/0x50
Jan 5 12:46:11 testbox kernel: [653364.391694] [<c01a086f>] shrink_slab+0x11f/0x180
Jan 5 12:46:11 testbox kernel: [653364.391694] [<c01a2995>] do_try_to_free_pages+0x155/0x2c0
Jan 5 12:46:11 testbox kernel: [653364.391694] [<c0156a8b>] ? autoremove_wake_function+0x1b/0x50
Jan 5 12:46:11 testbox kernel: [653364.391694] [<c01a2bec>] try_to_free_pages+0x6c/0x80
Jan 5 12:46:11 testbox kernel: [653364.391694] [<c01a13a0>] ? isolate_pages_global+0x0/0x70
Jan 5 12:46:11 testbox kernel: [653364.391694] [<c019bdbb>] __alloc_pages_internal+0x1cb/0x450
Jan 5 12:46:11 testbox kernel: [653364.391694] [<c01c004b>] new_slab+0x7b/0x1e0
Jan 5 12:46:11 testbox kernel: [653364.391694] [<c01c1165>] ? unfreeze_slab+0x75/0xb0
Jan 5 12:46:11 testbox kernel: [653364.391694] [<c01c141b>] __slab_alloc+0x20b/0x260
Jan 5 12:46:11 testbox kernel: [653364.391694] [<c01c1516>] kmem_cache_alloc+0xa6/0xb0
Jan 5 12:46:11 testbox kernel: [653364.391694] [<c013ef4c>] ? dup_task_struct+0x1c/0x110
Jan 5 12:46:11 testbox kernel: [653364.391694] [<c013ef4c>] dup_task_struct+0x1c/0x110
Jan 5 12:46:11 testbox kernel: [653364.391694] [<c013fd7e>] copy_process+0x6e/0xcf0
Jan 5 12:46:11 testbox kernel: [653364.391694] [<c01db240>] ? alloc_fd+0xe0/0x100
Jan 5 12:46:11 testbox kernel: [653364.391694] [<c01c7bbc>] ? get_empty_filp+0x6c/0x170
Jan 5 12:46:11 testbox kernel: [653364.391694] [<c050da98>] ? _spin_lock+0x8/0x10
Jan 5 12:46:11 testbox kernel: [653364.391694] [<c0140a62>] do_fork+0x62/0x300
Jan 5 12:46:11 testbox kernel: [653364.391694] [<c01cd6ef>] ? sys_pipe2+0x3f/0x60
Jan 5 12:46:11 testbox kernel: [653364.391694] [<c01083df>] sys_clone+0x2f/0x40
Jan 5 12:46:11 testbox kernel: [653364.391694] [<c0109eef>] sysenter_do_call+0x12/0x2f
Jan 5 12:46:42 testbox kernel: [653395.522503] BUG: soft lockup - CPU#3 stuck for 61s! [rsync:30532]
Jan 5 12:46:42 testbox kernel: [653395.522503] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:46:42 testbox kernel: [653395.522503]
Jan 5 12:46:42 testbox kernel: [653395.522503] Pid: 30532, comm: rsync Tainted: G      D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:46:42 testbox kernel: [653395.522503] EIP: 0060:[<c0129085>] EFLAGS: 00000297 CPU: 3
Jan 5 12:46:42 testbox kernel: [653395.522503] EIP is at __ticket_spin_lock+0x15/0x20
Jan 5 12:46:42 testbox kernel: [653395.522503] EAX: f5d962b8 EBX: e0c8de00 ECX: 00000000 EDX: 00004e4d
Jan 5 12:46:42 testbox kernel: [653395.522503] ESI: f5d96200 EDI: f6d00450 EBP: db0f1cc8 ESP: db0f1cc8
Jan 5 12:46:42 testbox kernel: [653395.522503] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:46:42 testbox kernel: [653395.522503] CR0: 8005003b CR2: 00000000 CR3: 1b0d9000 CR4: 000006b0
Jan 5 12:46:42 testbox kernel: [653395.522503] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:46:42 testbox kernel: [653395.522503] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:46:42 testbox kernel: [653395.522503] Call Trace:
Jan 5 12:46:42 testbox kernel: [653395.522503] [<c050da98>] _spin_lock+0x8/0x10
Jan 5 12:46:42 testbox kernel: [653395.522503] [<c025393e>] journal_dirty_data+0x5e/0x1f0
Jan 5 12:46:42 testbox kernel: [653395.522503] [<c021b048>] ext3_journal_dirty_data+0x18/0x50
Jan 5 12:46:42 testbox kernel: [653395.522503] [<c021b7ef>] walk_page_buffers+0x2f/0x80
Jan 5 12:46:42 testbox kernel: [653395.522503] [<c021bbf3>] ext3_ordered_write_end+0x63/0x1b0
Jan 5 12:46:42 testbox kernel: [653395.522503] [<c021b030>] ? ext3_journal_dirty_data+0x0/0x50
Jan 5 12:46:42 testbox kernel: [653395.522503] [<c0195e6d>] ? iov_iter_copy_from_user_atomic+0x5d/0xc0
Jan 5 12:46:42 testbox kernel: [653395.522503] [<c0195fdf>] generic_perform_write+0x10f/0x190
Jan 5 12:46:42 testbox kernel: [653395.522503] [<c01970ad>] generic_file_buffered_write+0x5d/0x120
Jan 5 12:46:42 testbox kernel: [653395.522503] [<c01986ab>] __generic_file_aio_write_nolock+0x19b/0x520
Jan 5 12:46:42 testbox kernel: [653395.522503] [<c013949e>] ? resched_task+0x1e/0x70
Jan 5 12:46:42 testbox kernel: [653395.522503] [<c0198b3e>] generic_file_aio_write+0x5e/0xd0
Jan 5 12:46:42 testbox kernel: [653395.522503] [<c02194fd>] ext3_file_write+0x2d/0xc0
Jan 5 12:46:42 testbox kernel: [653395.522503] [<c01c6831>] do_sync_write+0xd1/0x110
Jan 5 12:46:42 testbox kernel: [653395.522503] [<c03f1082>] ? rtc_update_irq+0x92/0xa0
Jan 5 12:46:42 testbox kernel: [653395.522503] [<c0156a70>] ? autoremove_wake_function+0x0/0x50
Jan 5 12:46:42 testbox kernel: [653395.522503] [<c02b22f0>] ? apparmor_file_permission+0x20/0x30
Jan 5 12:46:42 testbox kernel: [653395.522503] [<c029049f>] ? security_file_permission+0xf/0x20
Jan 5 12:46:42 testbox kernel: [653395.522503] [<c01c69d4>] ? rw_verify_area+0x54/0xd0
Jan 5 12:46:42 testbox kernel: [653395.522503] [<c01c6ec8>] vfs_write+0x98/0x110
Jan 5 12:46:42 testbox kernel: [653395.522503] [<c01c6760>] ? do_sync_write+0x0/0x110
Jan 5 12:46:42 testbox kernel: [653395.522503] [<c01c6ffd>] sys_write+0x3d/0x70
Jan 5 12:46:42 testbox kernel: [653395.522503] [<c0109eef>] sysenter_do_call+0x12/0x2f
Jan 5 12:46:42 testbox kernel: [653395.526521] hpet1: lost 3 rtc interrupts
Jan 5 12:47:16 testbox kernel: [653429.890004] BUG: soft lockup - CPU#0 stuck for 61s! [hal-acl-tool:4168]
Jan 5 12:47:16 testbox kernel: [653429.890005] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:47:16 testbox kernel: [653429.891694]
Jan 5 12:47:16 testbox kernel: [653429.891694] Pid: 4168, comm: hal-acl-tool Tainted: G      D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:47:16 testbox kernel: [653429.891694] EIP: 0060:[<c012907f>] EFLAGS: 00000293 CPU: 0
Jan 5 12:47:16 testbox kernel: [653429.891694] EIP is at __ticket_spin_lock+0xf/0x20
Jan 5 12:47:16 testbox kernel: [653429.891694] EAX: f5d962b8 EBX: d6ba3b00 ECX: f5d96200 EDX: 00004f4d
Jan 5 12:47:16 testbox kernel: [653429.891694] ESI: d20d3000 EDI: f5d962b8 EBP: e0491c60 ESP: e0491c60
Jan 5 12:47:16 testbox kernel: [653429.891694] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:47:16 testbox kernel: [653429.891694] CR0: 8005003b CR2: 09202568 CR3: 2a445000 CR4: 000006b0
Jan 5 12:47:16 testbox kernel: [653429.891694] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:47:16 testbox kernel: [653429.891694] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:47:16 testbox kernel: [653429.891694] Call Trace:
Jan 5 12:47:16 testbox kernel: [653429.891694] [<c050da98>] _spin_lock+0x8/0x10
Jan 5 12:47:16 testbox kernel: [653429.891694] [<c025345d>] __journal_try_to_free_buffer+0x4d/0xc0
Jan 5 12:47:16 testbox kernel: [653429.891694] [<c0253532>] journal_try_to_free_buffers+0x62/0x100
Jan 5 12:47:16 testbox kernel: [653429.891694] [<c021bda5>] ext3_releasepage+0x45/0x70
Jan 5 12:47:16 testbox kernel: [653429.891694] [<c0196c8e>] try_to_release_page+0x2e/0x50
Jan 5 12:47:16 testbox kernel: [653429.891694] [<c01a0346>] __invalidate_mapping_pages+0x136/0x150
Jan 5 12:47:16 testbox kernel: [653429.891694] [<c01a0372>] invalidate_mapping_pages+0x12/0x20
Jan 5 12:47:16 testbox kernel: [653429.891694] [<c01d98ba>] prune_icache+0x21a/0x260
Jan 5 12:47:16 testbox kernel: [653429.891694] [<c01d993e>] shrink_icache_memory+0x3e/0x50
Jan 5 12:47:16 testbox kernel: [653429.891694] [<c01a086f>] shrink_slab+0x11f/0x180
Jan 5 12:47:16 testbox kernel: [653429.891694] [<c01a2995>] do_try_to_free_pages+0x155/0x2c0
Jan 5 12:47:16 testbox kernel: [653429.891694] [<c0156a8b>] ? autoremove_wake_function+0x1b/0x50
Jan 5 12:47:16 testbox kernel: [653429.891694] [<c01a2bec>] try_to_free_pages+0x6c/0x80
Jan 5 12:47:16 testbox kernel: [653429.891694] [<c01a13a0>] ? isolate_pages_global+0x0/0x70
Jan 5 12:47:16 testbox kernel: [653429.891694] [<c019bdbb>] __alloc_pages_internal+0x1cb/0x450
Jan 5 12:47:16 testbox kernel: [653429.891694] [<c01c004b>] new_slab+0x7b/0x1e0
Jan 5 12:47:16 testbox kernel: [653429.891694] [<c01c1165>] ? unfreeze_slab+0x75/0xb0
Jan 5 12:47:16 testbox kernel: [653429.891694] [<c01c141b>] __slab_alloc+0x20b/0x260
Jan 5 12:47:16 testbox kernel: [653429.891694] [<c01c1516>] kmem_cache_alloc+0xa6/0xb0
Jan 5 12:47:16 testbox kernel: [653429.891694] [<c013ef4c>] ? dup_task_struct+0x1c/0x110
Jan 5 12:47:16 testbox kernel: [653429.891694] [<c013ef4c>] dup_task_struct+0x1c/0x110
Jan 5 12:47:16 testbox kernel: [653429.891694] [<c013fd7e>] copy_process+0x6e/0xcf0
Jan 5 12:47:16 testbox kernel: [653429.891694] [<c01db240>] ? alloc_fd+0xe0/0x100
Jan 5 12:47:16 testbox kernel: [653429.891694] [<c01c7bbc>] ? get_empty_filp+0x6c/0x170
Jan 5 12:47:16 testbox kernel: [653429.891694] [<c050da98>] ? _spin_lock+0x8/0x10
Jan 5 12:47:16 testbox kernel: [653429.891694] [<c0140a62>] do_fork+0x62/0x300
Jan 5 12:47:16 testbox kernel: [653429.891694] [<c01cd6ef>] ? sys_pipe2+0x3f/0x60
Jan 5 12:47:16 testbox kernel: [653429.891694] [<c01083df>] sys_clone+0x2f/0x40
Jan 5 12:47:16 testbox kernel: [653429.891694] [<c0109eef>] sysenter_do_call+0x12/0x2f
Jan 5 12:47:48 testbox kernel: [653461.022504] BUG: soft lockup - CPU#3 stuck for 61s! [rsync:30532]
Jan 5 12:47:48 testbox kernel: [653461.022504] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:47:48 testbox kernel: [653461.022504]
Jan 5 12:47:48 testbox kernel: [653461.022504] Pid: 30532, comm: rsync Tainted: G      D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:47:48 testbox kernel: [653461.022504] EIP: 0060:[<c0129085>] EFLAGS: 00000297 CPU: 3
Jan 5 12:47:48 testbox kernel: [653461.022504] EIP is at __ticket_spin_lock+0x15/0x20
Jan 5 12:47:48 testbox kernel: [653461.022504] EAX: f5d962b8 EBX: e0c8de00 ECX: 00000000 EDX: 00004e4d
Jan 5 12:47:48 testbox kernel: [653461.022504] ESI: f5d96200 EDI: f6d00450 EBP: db0f1cc8 ESP: db0f1cc8
Jan 5 12:47:48 testbox kernel: [653461.022504] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:47:48 testbox kernel: [653461.022504] CR0: 8005003b CR2: 00000000 CR3: 1b0d9000 CR4: 000006b0
Jan 5 12:47:48 testbox kernel: [653461.022504] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:47:48 testbox kernel: [653461.022504] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:47:48 testbox kernel: [653461.022504] Call Trace:
Jan 5 12:47:48 testbox kernel: [653461.022504] [<c050da98>] _spin_lock+0x8/0x10
Jan 5 12:47:48 testbox kernel: [653461.022504] [<c025393e>] journal_dirty_data+0x5e/0x1f0
Jan 5 12:47:48 testbox kernel: [653461.022504] [<c021b048>] ext3_journal_dirty_data+0x18/0x50
Jan 5 12:47:48 testbox kernel: [653461.022504] [<c021b7ef>] walk_page_buffers+0x2f/0x80
Jan 5 12:47:48 testbox kernel: [653461.022504] [<c021bbf3>] ext3_ordered_write_end+0x63/0x1b0
Jan 5 12:47:48 testbox kernel: [653461.022504] [<c021b030>] ? ext3_journal_dirty_data+0x0/0x50
Jan 5 12:47:48 testbox kernel: [653461.022504] [<c0195e6d>] ? iov_iter_copy_from_user_atomic+0x5d/0xc0
Jan 5 12:47:48 testbox kernel: [653461.022504] [<c0195fdf>] generic_perform_write+0x10f/0x190
Jan 5 12:47:48 testbox kernel: [653461.022504] [<c01970ad>] generic_file_buffered_write+0x5d/0x120
Jan 5 12:47:48 testbox kernel: [653461.022504] [<c01986ab>] __generic_file_aio_write_nolock+0x19b/0x520
Jan 5 12:47:48 testbox kernel: [653461.022504] [<c013949e>] ? resched_task+0x1e/0x70
Jan 5 12:47:48 testbox kernel: [653461.022504] [<c0198b3e>] generic_file_aio_write+0x5e/0xd0
Jan 5 12:47:48 testbox kernel: [653461.022504] [<c02194fd>] ext3_file_write+0x2d/0xc0
Jan 5 12:47:48 testbox kernel: [653461.022504] [<c01c6831>] do_sync_write+0xd1/0x110
Jan 5 12:47:48 testbox kernel: [653461.022504] [<c03f1082>] ? rtc_update_irq+0x92/0xa0
Jan 5 12:47:48 testbox kernel: [653461.022504] [<c0156a70>] ? autoremove_wake_function+0x0/0x50
Jan 5 12:47:48 testbox kernel: [653461.022504] [<c02b22f0>] ? apparmor_file_permission+0x20/0x30
Jan 5 12:47:48 testbox kernel: [653461.022504] [<c029049f>] ? security_file_permission+0xf/0x20
Jan 5 12:47:48 testbox kernel: [653461.022504] [<c01c69d4>] ? rw_verify_area+0x54/0xd0
Jan 5 12:47:48 testbox kernel: [653461.022504] [<c01c6ec8>] vfs_write+0x98/0x110
Jan 5 12:47:48 testbox kernel: [653461.022504] [<c01c6760>] ? do_sync_write+0x0/0x110
Jan 5 12:47:48 testbox kernel: [653461.022504] [<c01c6ffd>] sys_write+0x3d/0x70
Jan 5 12:47:48 testbox kernel: [653461.022504] [<c0109eef>] sysenter_do_call+0x12/0x2f
Jan 5 12:47:48 testbox kernel: [653461.025921] hpet1: lost 2 rtc interrupts
Jan 5 12:48:22 testbox kernel: [653495.390004] BUG: soft lockup - CPU#0 stuck for 61s! [hal-acl-tool:4168]
Jan 5 12:48:22 testbox kernel: [653495.390005] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:48:22 testbox kernel: [653495.391695]
Jan 5 12:48:22 testbox kernel: [653495.391695] Pid: 4168, comm: hal-acl-tool Tainted: G      D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:48:22 testbox kernel: [653495.391695] EIP: 0060:[<c0129083>] EFLAGS: 00000293 CPU: 0
Jan 5 12:48:22 testbox kernel: [653495.391695] EIP is at __ticket_spin_lock+0x13/0x20
Jan 5 12:48:22 testbox kernel: [653495.391695] EAX: f5d962b8 EBX: d6ba3b00 ECX: f5d96200 EDX: 00004f4d
Jan 5 12:48:22 testbox kernel: [653495.391695] ESI: d20d3000 EDI: f5d962b8 EBP: e0491c60 ESP: e0491c60
Jan 5 12:48:22 testbox kernel: [653495.391695] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:48:22 testbox kernel: [653495.391695] CR0: 8005003b CR2: 09202568 CR3: 2a445000 CR4: 000006b0
Jan 5 12:48:22 testbox kernel: [653495.391695] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:48:22 testbox kernel: [653495.391695] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:48:22 testbox kernel: [653495.391695] Call Trace:
Jan 5 12:48:22 testbox kernel: [653495.391695] [<c050da98>] _spin_lock+0x8/0x10
Jan 5 12:48:22 testbox kernel: [653495.391695] [<c025345d>] __journal_try_to_free_buffer+0x4d/0xc0
Jan 5 12:48:22 testbox kernel: [653495.391695] [<c0253532>] journal_try_to_free_buffers+0x62/0x100
Jan 5 12:48:22 testbox kernel: [653495.391695] [<c021bda5>] ext3_releasepage+0x45/0x70
Jan 5 12:48:22 testbox kernel: [653495.391695] [<c0196c8e>] try_to_release_page+0x2e/0x50
Jan 5 12:48:22 testbox kernel: [653495.391695] [<c01a0346>] __invalidate_mapping_pages+0x136/0x150
Jan 5 12:48:22 testbox kernel: [653495.391695] [<c01a0372>] invalidate_mapping_pages+0x12/0x20
Jan 5 12:48:22 testbox kernel: [653495.391695] [<c01d98ba>] prune_icache+0x21a/0x260
Jan 5 12:48:22 testbox kernel: [653495.391695] [<c01d993e>] shrink_icache_memory+0x3e/0x50
Jan 5 12:48:22 testbox kernel: [653495.391695] [<c01a086f>] shrink_slab+0x11f/0x180
Jan 5 12:48:22 testbox kernel: [653495.391695] [<c01a2995>] do_try_to_free_pages+0x155/0x2c0
Jan 5 12:48:22 testbox kernel: [653495.391695] [<c0156a8b>] ? autoremove_wake_function+0x1b/0x50
Jan 5 12:48:22 testbox kernel: [653495.391695] [<c01a2bec>] try_to_free_pages+0x6c/0x80
Jan 5 12:48:22 testbox kernel: [653495.391695] [<c01a13a0>] ? isolate_pages_global+0x0/0x70
Jan 5 12:48:22 testbox kernel: [653495.391695] [<c019bdbb>] __alloc_pages_internal+0x1cb/0x450
Jan 5 12:48:22 testbox kernel: [653495.391695] [<c01c004b>] new_slab+0x7b/0x1e0
Jan 5 12:48:22 testbox kernel: [653495.391695] [<c01c1165>] ? unfreeze_slab+0x75/0xb0
Jan 5 12:48:22 testbox kernel: [653495.391695] [<c01c141b>] __slab_alloc+0x20b/0x260
Jan 5 12:48:22 testbox kernel: [653495.391695] [<c01c1516>] kmem_cache_alloc+0xa6/0xb0
Jan 5 12:48:22 testbox kernel: [653495.391695] [<c013ef4c>] ? dup_task_struct+0x1c/0x110
Jan 5 12:48:22 testbox kernel: [653495.391695] [<c013ef4c>] dup_task_struct+0x1c/0x110
Jan 5 12:48:22 testbox kernel: [653495.391695] [<c013fd7e>] copy_process+0x6e/0xcf0
Jan 5 12:48:22 testbox kernel: [653495.391695] [<c01db240>] ? alloc_fd+0xe0/0x100
Jan 5 12:48:22 testbox kernel: [653495.391695] [<c01c7bbc>] ? get_empty_filp+0x6c/0x170
Jan 5 12:48:22 testbox kernel: [653495.391695] [<c050da98>] ? _spin_lock+0x8/0x10
Jan 5 12:48:22 testbox kernel: [653495.391695] [<c0140a62>] do_fork+0x62/0x300
Jan 5 12:48:22 testbox kernel: [653495.391695] [<c01cd6ef>] ? sys_pipe2+0x3f/0x60
Jan 5 12:48:22 testbox kernel: [653495.391695] [<c01083df>] sys_clone+0x2f/0x40
Jan 5 12:48:22 testbox kernel: [653495.391695] [<c0109eef>] sysenter_do_call+0x12/0x2f
Jan 5 12:48:53 testbox kernel: [653526.522503] BUG: soft lockup - CPU#3 stuck for 61s! [rsync:30532]
Jan 5 12:48:53 testbox kernel: [653526.522504] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:48:53 testbox kernel: [653526.522504]
Jan 5 12:48:53 testbox kernel: [653526.522504] Pid: 30532, comm: rsync Tainted: G      D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:48:53 testbox kernel: [653526.522504] EIP: 0060:[<c0129085>] EFLAGS: 00000297 CPU: 3
Jan 5 12:48:53 testbox kernel: [653526.522504] EIP is at __ticket_spin_lock+0x15/0x20
Jan 5 12:48:53 testbox kernel: [653526.522504] EAX: f5d962b8 EBX: e0c8de00 ECX: 00000000 EDX: 00004e4d
Jan 5 12:48:53 testbox kernel: [653526.522504] ESI: f5d96200 EDI: f6d00450 EBP: db0f1cc8 ESP: db0f1cc8
Jan 5 12:48:53 testbox kernel: [653526.522504] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:48:53 testbox kernel: [653526.522504] CR0: 8005003b CR2: 00000000 CR3: 1b0d9000 CR4: 000006b0
Jan 5 12:48:53 testbox kernel: [653526.522504] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:48:53 testbox kernel: [653526.522504] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:48:53 testbox kernel: [653526.522504] Call Trace:
Jan 5 12:48:53 testbox kernel: [653526.522504] [<c050da98>] _spin_lock+0x8/0x10
Jan 5 12:48:53 testbox kernel: [653526.522504] [<c025393e>] journal_dirty_data+0x5e/0x1f0
Jan 5 12:48:53 testbox kernel: [653526.522504] [<c021b048>] ext3_journal_dirty_data+0x18/0x50
Jan 5 12:48:53 testbox kernel: [653526.522504] [<c021b7ef>] walk_page_buffers+0x2f/0x80
Jan 5 12:48:53 testbox kernel: [653526.522504] [<c021bbf3>] ext3_ordered_write_end+0x63/0x1b0
Jan 5 12:48:53 testbox kernel: [653526.522504] [<c021b030>] ? ext3_journal_dirty_data+0x0/0x50
Jan 5 12:48:53 testbox kernel: [653526.522504] [<c0195e6d>] ? iov_iter_copy_from_user_atomic+0x5d/0xc0
Jan 5 12:48:53 testbox kernel: [653526.522504] [<c0195fdf>] generic_perform_write+0x10f/0x190
Jan 5 12:48:53 testbox kernel: [653526.522504] [<c01970ad>] generic_file_buffered_write+0x5d/0x120
Jan 5 12:48:53 testbox kernel: [653526.522504] [<c01986ab>] __generic_file_aio_write_nolock+0x19b/0x520
Jan 5 12:48:53 testbox kernel: [653526.522504] [<c013949e>] ? resched_task+0x1e/0x70
Jan 5 12:48:53 testbox kernel: [653526.522504] [<c0198b3e>] generic_file_aio_write+0x5e/0xd0
Jan 5 12:48:53 testbox kernel: [653526.522504] [<c02194fd>] ext3_file_write+0x2d/0xc0
Jan 5 12:48:53 testbox kernel: [653526.522504] [<c01c6831>] do_sync_write+0xd1/0x110
Jan 5 12:48:53 testbox kernel: [653526.522504] [<c03f1082>] ? rtc_update_irq+0x92/0xa0
Jan 5 12:48:53 testbox kernel: [653526.522504] [<c0156a70>] ? autoremove_wake_function+0x0/0x50
Jan 5 12:48:53 testbox kernel: [653526.522504] [<c02b22f0>] ? apparmor_file_permission+0x20/0x30
Jan 5 12:48:53 testbox kernel: [653526.522504] [<c029049f>] ? security_file_permission+0xf/0x20
Jan 5 12:48:53 testbox kernel: [653526.522504] [<c01c69d4>] ? rw_verify_area+0x54/0xd0
Jan 5 12:48:53 testbox kernel: [653526.522504] [<c01c6ec8>] vfs_write+0x98/0x110
Jan 5 12:48:53 testbox kernel: [653526.522504] [<c01c6760>] ? do_sync_write+0x0/0x110
Jan 5 12:48:53 testbox kernel: [653526.522504] [<c01c6ffd>] sys_write+0x3d/0x70
Jan 5 12:48:53 testbox kernel: [653526.522504] [<c0109eef>] sysenter_do_call+0x12/0x2f
Jan 5 12:48:53 testbox kernel: [653526.526518] hpet1: lost 3 rtc interrupts
Jan 5 12:49:27 testbox kernel: [653560.880003] BUG: soft lockup - CPU#0 stuck for 61s! [hal-acl-tool:4168]
Jan 5 12:49:27 testbox kernel: [653560.880005] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:49:27 testbox kernel: [653560.881694]
Jan 5 12:49:27 testbox kernel: [653560.881694] Pid: 4168, comm: hal-acl-tool Tainted: G      D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:49:27 testbox kernel: [653560.881694] EIP: 0060:[<c012907f>] EFLAGS: 00000293 CPU: 0
Jan 5 12:49:27 testbox kernel: [653560.881694] EIP is at __ticket_spin_lock+0xf/0x20
Jan 5 12:49:27 testbox kernel: [653560.881694] EAX: f5d962b8 EBX: d6ba3b00 ECX: f5d96200 EDX: 00004f4d
Jan 5 12:49:27 testbox kernel: [653560.881694] ESI: d20d3000 EDI: f5d962b8 EBP: e0491c60 ESP: e0491c60
Jan 5 12:49:27 testbox kernel: [653560.881694] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:49:27 testbox kernel: [653560.881694] CR0: 8005003b CR2: 09202568 CR3: 2a445000 CR4: 000006b0
Jan 5 12:49:27 testbox kernel: [653560.881694] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:49:27 testbox kernel: [653560.881694] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:49:27 testbox kernel: [653560.881694] Call Trace:
Jan 5 12:49:27 testbox kernel: [653560.881694] [<c050da98>] _spin_lock+0x8/0x10
Jan 5 12:49:27 testbox kernel: [653560.881694] [<c025345d>] __journal_try_to_free_buffer+0x4d/0xc0
Jan 5 12:49:27 testbox kernel: [653560.881694] [<c0253532>] journal_try_to_free_buffers+0x62/0x100
Jan 5 12:49:27 testbox kernel: [653560.881694] [<c021bda5>] ext3_releasepage+0x45/0x70
Jan 5 12:49:27 testbox kernel: [653560.881694] [<c0196c8e>] try_to_release_page+0x2e/0x50
Jan 5 12:49:27 testbox kernel: [653560.881694] [<c01a0346>] __invalidate_mapping_pages+0x136/0x150
Jan 5 12:49:27 testbox kernel: [653560.881694] [<c01a0372>] invalidate_mapping_pages+0x12/0x20
Jan 5 12:49:27 testbox kernel: [653560.881694] [<c01d98ba>] prune_icache+0x21a/0x260
Jan 5 12:49:27 testbox kernel: [653560.881694] [<c01d993e>] shrink_icache_memory+0x3e/0x50
Jan 5 12:49:27 testbox kernel: [653560.881694] [<c01a086f>] shrink_slab+0x11f/0x180
Jan 5 12:49:27 testbox kernel: [653560.881694] [<c01a2995>] do_try_to_free_pages+0x155/0x2c0
Jan 5 12:49:27 testbox kernel: [653560.881694] [<c0156a8b>] ? autoremove_wake_function+0x1b/0x50
Jan 5 12:49:27 testbox kernel: [653560.881694] [<c01a2bec>] try_to_free_pages+0x6c/0x80
Jan 5 12:49:27 testbox kernel: [653560.881694] [<c01a13a0>] ? isolate_pages_global+0x0/0x70
Jan 5 12:49:27 testbox kernel: [653560.881694] [<c019bdbb>] __alloc_pages_internal+0x1cb/0x450
Jan 5 12:49:27 testbox kernel: [653560.881694] [<c01c004b>] new_slab+0x7b/0x1e0
Jan 5 12:49:27 testbox kernel: [653560.881694] [<c01c1165>] ? unfreeze_slab+0x75/0xb0
Jan 5 12:49:27 testbox kernel: [653560.881694] [<c01c141b>] __slab_alloc+0x20b/0x260
Jan 5 12:49:27 testbox kernel: [653560.881694] [<c01c1516>] kmem_cache_alloc+0xa6/0xb0
Jan 5 12:49:27 testbox kernel: [653560.881694] [<c013ef4c>] ? dup_task_struct+0x1c/0x110
Jan 5 12:49:27 testbox kernel: [653560.881694] [<c013ef4c>] dup_task_struct+0x1c/0x110
Jan 5 12:49:27 testbox kernel: [653560.881694] [<c013fd7e>] copy_process+0x6e/0xcf0
Jan 5 12:49:27 testbox kernel: [653560.881694] [<c01db240>] ? alloc_fd+0xe0/0x100
Jan 5 12:49:27 testbox kernel: [653560.881694] [<c01c7bbc>] ? get_empty_filp+0x6c/0x170
Jan 5 12:49:27 testbox kernel: [653560.881694] [<c050da98>] ? _spin_lock+0x8/0x10
Jan 5 12:49:27 testbox kernel: [653560.881694] [<c0140a62>] do_fork+0x62/0x300
Jan 5 12:49:27 testbox kernel: [653560.881694] [<c01cd6ef>] ? sys_pipe2+0x3f/0x60
Jan 5 12:49:27 testbox kernel: [653560.881694] [<c01083df>] sys_clone+0x2f/0x40
Jan 5 12:49:27 testbox kernel: [653560.881694] [<c0109eef>] sysenter_do_call+0x12/0x2f
Jan 5 12:49:59 testbox kernel: [653592.022503] BUG: soft lockup - CPU#3 stuck for 61s! [rsync:30532]
Jan 5 12:49:59 testbox kernel: [653592.022504] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:49:59 testbox kernel: [653592.022504]
Jan 5 12:49:59 testbox kernel: [653592.022504] Pid: 30532, comm: rsync Tainted: G      D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:49:59 testbox kernel: [653592.022504] EIP: 0060:[<c0129085>] EFLAGS: 00000297 CPU: 3
Jan 5 12:49:59 testbox kernel: [653592.022504] EIP is at __ticket_spin_lock+0x15/0x20
Jan 5 12:49:59 testbox kernel: [653592.022504] EAX: f5d962b8 EBX: e0c8de00 ECX: 00000000 EDX: 00004e4d
Jan 5 12:49:59 testbox kernel: [653592.022504] ESI: f5d96200 EDI: f6d00450 EBP: db0f1cc8 ESP: db0f1cc8
Jan 5 12:49:59 testbox kernel: [653592.022504] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:49:59 testbox kernel: [653592.022504] CR0: 8005003b CR2: 00000000 CR3: 1b0d9000 CR4: 000006b0
Jan 5 12:49:59 testbox kernel: [653592.022504] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:49:59 testbox kernel: [653592.022504] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:49:59 testbox kernel: [653592.022504] Call Trace:
Jan 5 12:49:59 testbox kernel: [653592.022504] [<c050da98>] _spin_lock+0x8/0x10
Jan 5 12:49:59 testbox kernel: [653592.022504] [<c025393e>] journal_dirty_data+0x5e/0x1f0
Jan 5 12:49:59 testbox kernel: [653592.022504] [<c021b048>] ext3_journal_dirty_data+0x18/0x50
Jan 5 12:49:59 testbox kernel: [653592.022504] [<c021b7ef>] walk_page_buffers+0x2f/0x80
Jan 5 12:49:59 testbox kernel: [653592.022504] [<c021bbf3>] ext3_ordered_write_end+0x63/0x1b0
Jan 5 12:49:59 testbox kernel: [653592.022504] [<c021b030>] ? ext3_journal_dirty_data+0x0/0x50
Jan 5 12:49:59 testbox kernel: [653592.022504] [<c0195e6d>] ? iov_iter_copy_from_user_atomic+0x5d/0xc0
Jan 5 12:49:59 testbox kernel: [653592.022504] [<c0195fdf>] generic_perform_write+0x10f/0x190
Jan 5 12:49:59 testbox kernel: [653592.022504] [<c01970ad>] generic_file_buffered_write+0x5d/0x120
Jan 5 12:49:59 testbox kernel: [653592.022504] [<c01986ab>] __generic_file_aio_write_nolock+0x19b/0x520
Jan 5 12:49:59 testbox kernel: [653592.022504] [<c013949e>] ? resched_task+0x1e/0x70
Jan 5 12:49:59 testbox kernel: [653592.022504] [<c0198b3e>] generic_file_aio_write+0x5e/0xd0
Jan 5 12:49:59 testbox kernel: [653592.022504] [<c02194fd>] ext3_file_write+0x2d/0xc0
Jan 5 12:49:59 testbox kernel: [653592.022504] [<c01c6831>] do_sync_write+0xd1/0x110
Jan 5 12:49:59 testbox kernel: [653592.022504] [<c03f1082>] ? rtc_update_irq+0x92/0xa0
Jan 5 12:49:59 testbox kernel: [653592.022504] [<c0156a70>] ? autoremove_wake_function+0x0/0x50
Jan 5 12:49:59 testbox kernel: [653592.022504] [<c02b22f0>] ? apparmor_file_permission+0x20/0x30
Jan 5 12:49:59 testbox kernel: [653592.022504] [<c029049f>] ? security_file_permission+0xf/0x20
Jan 5 12:49:59 testbox kernel: [653592.022504] [<c01c69d4>] ? rw_verify_area+0x54/0xd0
Jan 5 12:49:59 testbox kernel: [653592.022504] [<c01c6ec8>] vfs_write+0x98/0x110
Jan 5 12:49:59 testbox kernel: [653592.022504] [<c01c6760>] ? do_sync_write+0x0/0x110
Jan 5 12:49:59 testbox kernel: [653592.022504] [<c01c6ffd>] sys_write+0x3d/0x70
Jan 5 12:49:59 testbox kernel: [653592.022504] [<c0109eef>] sysenter_do_call+0x12/0x2f
Jan 5 12:49:59 testbox kernel: [653592.025918] hpet1: lost 2 rtc interrupts
Jan 5 12:50:33 testbox kernel: [653626.380003] BUG: soft lockup - CPU#0 stuck for 61s! [hal-acl-tool:4168]
Jan 5 12:50:33 testbox kernel: [653626.380005] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:50:33 testbox kernel: [653626.381684]
Jan 5 12:50:33 testbox kernel: [653626.381684] Pid: 4168, comm: hal-acl-tool Tainted: G      D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:50:33 testbox kernel: [653626.381684] EIP: 0060:[<c0129081>] EFLAGS: 00000293 CPU: 0
Jan 5 12:50:33 testbox kernel: [653626.381684] EIP is at __ticket_spin_lock+0x11/0x20
Jan 5 12:50:33 testbox kernel: [653626.381684] EAX: f5d962b8 EBX: d6ba3b00 ECX: f5d96200 EDX: 00004f4d
Jan 5 12:50:33 testbox kernel: [653626.381684] ESI: d20d3000 EDI: f5d962b8 EBP: e0491c60 ESP: e0491c60
Jan 5 12:50:33 testbox kernel: [653626.381684] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:50:33 testbox kernel: [653626.381684] CR0: 8005003b CR2: 09202568 CR3: 2a445000 CR4: 000006b0
Jan 5 12:50:33 testbox kernel: [653626.381684] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:50:33 testbox kernel: [653626.381684] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:50:33 testbox kernel: [653626.381684] Call Trace:
Jan 5 12:50:33 testbox kernel: [653626.381684] [<c050da98>] _spin_lock+0x8/0x10
Jan 5 12:50:33 testbox kernel: [653626.381684] [<c025345d>] __journal_try_to_free_buffer+0x4d/0xc0
Jan 5 12:50:33 testbox kernel: [653626.381684] [<c0253532>] journal_try_to_free_buffers+0x62/0x100
Jan 5 12:50:33 testbox kernel: [653626.381684] [<c021bda5>] ext3_releasepage+0x45/0x70
Jan 5 12:50:33 testbox kernel: [653626.381684] [<c0196c8e>] try_to_release_page+0x2e/0x50
Jan 5 12:50:33 testbox kernel: [653626.381684] [<c01a0346>] __invalidate_mapping_pages+0x136/0x150
Jan 5 12:50:33 testbox kernel: [653626.381684] [<c01a0372>] invalidate_mapping_pages+0x12/0x20
Jan 5 12:50:33 testbox kernel: [653626.381684] [<c01d98ba>] prune_icache+0x21a/0x260
Jan 5 12:50:33 testbox kernel: [653626.381684] [<c01d993e>] shrink_icache_memory+0x3e/0x50
Jan 5 12:50:33 testbox kernel: [653626.381684] [<c01a086f>] shrink_slab+0x11f/0x180
Jan 5 12:50:33 testbox kernel: [653626.381684] [<c01a2995>] do_try_to_free_pages+0x155/0x2c0
Jan 5 12:50:33 testbox kernel: [653626.381684] [<c0156a8b>] ? autoremove_wake_function+0x1b/0x50
Jan 5 12:50:33 testbox kernel: [653626.381684] [<c01a2bec>] try_to_free_pages+0x6c/0x80
Jan 5 12:50:33 testbox kernel: [653626.381684] [<c01a13a0>] ? isolate_pages_global+0x0/0x70
Jan 5 12:50:33 testbox kernel: [653626.381684] [<c019bdbb>] __alloc_pages_internal+0x1cb/0x450
Jan 5 12:50:33 testbox kernel: [653626.381684] [<c01c004b>] new_slab+0x7b/0x1e0
Jan 5 12:50:33 testbox kernel: [653626.381684] [<c01c1165>] ? unfreeze_slab+0x75/0xb0
Jan 5 12:50:33 testbox kernel: [653626.381684] [<c01c141b>] __slab_alloc+0x20b/0x260
Jan 5 12:50:33 testbox kernel: [653626.381684] [<c01c1516>] kmem_cache_alloc+0xa6/0xb0
Jan 5 12:50:33 testbox kernel: [653626.381684] [<c013ef4c>] ? dup_task_struct+0x1c/0x110
Jan 5 12:50:33 testbox kernel: [653626.381684] [<c013ef4c>] dup_task_struct+0x1c/0x110
Jan 5 12:50:33 testbox kernel: [653626.381684] [<c013fd7e>] copy_process+0x6e/0xcf0
Jan 5 12:50:33 testbox kernel: [653626.381684] [<c01db240>] ? alloc_fd+0xe0/0x100
Jan 5 12:50:33 testbox kernel: [653626.381684] [<c01c7bbc>] ? get_empty_filp+0x6c/0x170
Jan 5 12:50:33 testbox kernel: [653626.381684] [<c050da98>] ? _spin_lock+0x8/0x10
Jan 5 12:50:33 testbox kernel: [653626.381684] [<c0140a62>] do_fork+0x62/0x300
Jan 5 12:50:33 testbox kernel: [653626.381684] [<c01cd6ef>] ? sys_pipe2+0x3f/0x60
Jan 5 12:50:33 testbox kernel: [653626.381684] [<c01083df>] sys_clone+0x2f/0x40
Jan 5 12:50:33 testbox kernel: [653626.381684] [<c0109eef>] sysenter_do_call+0x12/0x2f
Jan 5 12:51:04 testbox kernel: [653657.522503] BUG: soft lockup - CPU#3 stuck for 61s! [rsync:30532]
Jan 5 12:51:04 testbox kernel: [653657.522504] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:51:04 testbox kernel: [653657.522504]
Jan 5 12:51:04 testbox kernel: [653657.522504] Pid: 30532, comm: rsync Tainted: G      D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:51:04 testbox kernel: [653657.522504] EIP: 0060:[<c0129083>] EFLAGS: 00000297 CPU: 3
Jan 5 12:51:04 testbox kernel: [653657.522504] EIP is at __ticket_spin_lock+0x13/0x20
Jan 5 12:51:04 testbox kernel: [653657.522504] EAX: f5d962b8 EBX: e0c8de00 ECX: 00000000 EDX: 00004e4d
Jan 5 12:51:04 testbox kernel: [653657.522504] ESI: f5d96200 EDI: f6d00450 EBP: db0f1cc8 ESP: db0f1cc8
Jan 5 12:51:04 testbox kernel: [653657.522504] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:51:04 testbox kernel: [653657.522504] CR0: 8005003b CR2: 00000000 CR3: 1b0d9000 CR4: 000006b0
Jan 5 12:51:04 testbox kernel: [653657.522504] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:51:04 testbox kernel: [653657.522504] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:51:04 testbox kernel: [653657.522504] Call Trace:
Jan 5 12:51:04 testbox kernel: [653657.522504] [<c050da98>] _spin_lock+0x8/0x10
Jan 5 12:51:04 testbox kernel: [653657.522504] [<c025393e>] journal_dirty_data+0x5e/0x1f0
Jan 5 12:51:04 testbox kernel: [653657.522504] [<c021b048>] ext3_journal_dirty_data+0x18/0x50
Jan 5 12:51:04 testbox kernel: [653657.522504] [<c021b7ef>] walk_page_buffers+0x2f/0x80
Jan 5 12:51:04 testbox kernel: [653657.522504] [<c021bbf3>] ext3_ordered_write_end+0x63/0x1b0
Jan 5 12:51:04 testbox kernel: [653657.522504] [<c021b030>] ? ext3_journal_dirty_data+0x0/0x50
Jan 5 12:51:04 testbox kernel: [653657.522504] [<c0195e6d>] ? iov_iter_copy_from_user_atomic+0x5d/0xc0
Jan 5 12:51:04 testbox kernel: [653657.522504] [<c0195fdf>] generic_perform_write+0x10f/0x190
Jan 5 12:51:04 testbox kernel: [653657.522504] [<c01970ad>] generic_file_buffered_write+0x5d/0x120
Jan 5 12:51:04 testbox kernel: [653657.522504] [<c01986ab>] __generic_file_aio_write_nolock+0x19b/0x520
Jan 5 12:51:04 testbox kernel: [653657.522504] [<c013949e>] ? resched_task+0x1e/0x70
Jan 5 12:51:04 testbox kernel: [653657.522504] [<c0198b3e>] generic_file_aio_write+0x5e/0xd0
Jan 5 12:51:04 testbox kernel: [653657.522504] [<c02194fd>] ext3_file_write+0x2d/0xc0
Jan 5 12:51:04 testbox kernel: [653657.522504] [<c01c6831>] do_sync_write+0xd1/0x110
Jan 5 12:51:04 testbox kernel: [653657.522504] [<c03f1082>] ? rtc_update_irq+0x92/0xa0
Jan 5 12:51:04 testbox kernel: [653657.522504] [<c0156a70>] ? autoremove_wake_function+0x0/0x50
Jan 5 12:51:04 testbox kernel: [653657.522504] [<c02b22f0>] ? apparmor_file_permission+0x20/0x30
Jan 5 12:51:04 testbox kernel: [653657.522504] [<c029049f>] ? security_file_permission+0xf/0x20
Jan 5 12:51:04 testbox kernel: [653657.522504] [<c01c69d4>] ? rw_verify_area+0x54/0xd0
Jan 5 12:51:04 testbox kernel: [653657.522504] [<c01c6ec8>] vfs_write+0x98/0x110
Jan 5 12:51:04 testbox kernel: [653657.522504] [<c01c6760>] ? do_sync_write+0x0/0x110
Jan 5 12:51:04 testbox kernel: [653657.522504] [<c01c6ffd>] sys_write+0x3d/0x70
Jan 5 12:51:04 testbox kernel: [653657.522504] [<c0109eef>] sysenter_do_call+0x12/0x2f
Jan 5 12:51:04 testbox kernel: [653657.526512] hpet1: lost 3 rtc interrupts
Jan 5 12:51:38 testbox kernel: [653691.880004] BUG: soft lockup - CPU#0 stuck for 61s! [hal-acl-tool:4168]
Jan 5 12:51:38 testbox kernel: [653691.880005] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:51:38 testbox kernel: [653691.880859]
Jan 5 12:51:38 testbox kernel: [653691.880859] Pid: 4168, comm: hal-acl-tool Tainted: G      D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:51:38 testbox kernel: [653691.880859] EIP: 0060:[<c0129085>] EFLAGS: 00000293 CPU: 0
Jan 5 12:51:38 testbox kernel: [653691.880859] EIP is at __ticket_spin_lock+0x15/0x20
Jan 5 12:51:38 testbox kernel: [653691.880859] EAX: f5d962b8 EBX: d6ba3b00 ECX: f5d96200 EDX: 00004f4d
Jan 5 12:51:38 testbox kernel: [653691.880859] ESI: d20d3000 EDI: f5d962b8 EBP: e0491c60 ESP: e0491c60
Jan 5 12:51:38 testbox kernel: [653691.880859] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:51:38 testbox kernel: [653691.880859] CR0: 8005003b CR2: 09202568 CR3: 2a445000 CR4: 000006b0
Jan 5 12:51:38 testbox kernel: [653691.880859] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:51:38 testbox kernel: [653691.880859] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:51:38 testbox kernel: [653691.880859] Call Trace:
Jan 5 12:51:38 testbox kernel: [653691.880859] [<c050da98>] _spin_lock+0x8/0x10
Jan 5 12:51:38 testbox kernel: [653691.880859] [<c025345d>] __journal_try_to_free_buffer+0x4d/0xc0
Jan 5 12:51:38 testbox kernel: [653691.880859] [<c0253532>] journal_try_to_free_buffers+0x62/0x100
Jan 5 12:51:38 testbox kernel: [653691.880859] [<c021bda5>] ext3_releasepage+0x45/0x70
Jan 5 12:51:38 testbox kernel: [653691.880859] [<c0196c8e>] try_to_release_page+0x2e/0x50
Jan 5 12:51:38 testbox kernel: [653691.880859] [<c01a0346>] __invalidate_mapping_pages+0x136/0x150
Jan 5 12:51:38 testbox kernel: [653691.880859] [<c01a0372>] invalidate_mapping_pages+0x12/0x20
Jan 5 12:51:38 testbox kernel: [653691.880859] [<c01d98ba>] prune_icache+0x21a/0x260
Jan 5 12:51:38 testbox kernel: [653691.880859] [<c01d993e>] shrink_icache_memory+0x3e/0x50
Jan 5 12:51:38 testbox kernel: [653691.880859] [<c01a086f>] shrink_slab+0x11f/0x180
Jan 5 12:51:38 testbox kernel: [653691.880859] [<c01a2995>] do_try_to_free_pages+0x155/0x2c0
Jan 5 12:51:38 testbox kernel: [653691.880859] [<c0156a8b>] ? autoremove_wake_function+0x1b/0x50
Jan 5 12:51:38 testbox kernel: [653691.880859] [<c01a2bec>] try_to_free_pages+0x6c/0x80
Jan 5 12:51:38 testbox kernel: [653691.880859] [<c01a13a0>] ? isolate_pages_global+0x0/0x70
Jan 5 12:51:38 testbox kernel: [653691.880859] [<c019bdbb>] __alloc_pages_internal+0x1cb/0x450
Jan 5 12:51:38 testbox kernel: [653691.880859] [<c01c004b>] new_slab+0x7b/0x1e0
Jan 5 12:51:38 testbox kernel: [653691.880859] [<c01c1165>] ? unfreeze_slab+0x75/0xb0
Jan 5 12:51:38 testbox kernel: [653691.880859] [<c01c141b>] __slab_alloc+0x20b/0x260
Jan 5 12:51:38 testbox kernel: [653691.880859] [<c01c1516>] kmem_cache_alloc+0xa6/0xb0
Jan 5 12:51:38 testbox kernel: [653691.880859] [<c013ef4c>] ? dup_task_struct+0x1c/0x110
Jan 5 12:51:38 testbox kernel: [653691.880859] [<c013ef4c>] dup_task_struct+0x1c/0x110
Jan 5 12:51:38 testbox kernel: [653691.880859] [<c013fd7e>] copy_process+0x6e/0xcf0
Jan 5 12:51:38 testbox kernel: [653691.880859] [<c01db240>] ? alloc_fd+0xe0/0x100
Jan 5 12:51:38 testbox kernel: [653691.880859] [<c01c7bbc>] ? get_empty_filp+0x6c/0x170
Jan 5 12:51:38 testbox kernel: [653691.880859] [<c050da98>] ? _spin_lock+0x8/0x10
Jan 5 12:51:38 testbox kernel: [653691.880859] [<c0140a62>] do_fork+0x62/0x300
Jan 5 12:51:38 testbox kernel: [653691.880859] [<c01cd6ef>] ? sys_pipe2+0x3f/0x60
Jan 5 12:51:38 testbox kernel: [653691.880859] [<c01083df>] sys_clone+0x2f/0x40
Jan 5 12:51:38 testbox kernel: [653691.880859] [<c0109eef>] sysenter_do_call+0x12/0x2f
Jan 5 12:52:10 testbox kernel: [653723.022504] BUG: soft lockup - CPU#3 stuck for 61s! [rsync:30532]
Jan 5 12:52:10 testbox kernel: [653723.022504] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:52:10 testbox kernel: [653723.022504]
Jan 5 12:52:10 testbox kernel: [653723.022504] Pid: 30532, comm: rsync Tainted: G      D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:52:10 testbox kernel: [653723.022504] EIP: 0060:[<c0129083>] EFLAGS: 00000297 CPU: 3
Jan 5 12:52:10 testbox kernel: [653723.022504] EIP is at __ticket_spin_lock+0x13/0x20
Jan 5 12:52:10 testbox kernel: [653723.022504] EAX: f5d962b8 EBX: e0c8de00 ECX: 00000000 EDX: 00004e4d
Jan 5 12:52:10 testbox kernel: [653723.022504] ESI: f5d96200 EDI: f6d00450 EBP: db0f1cc8 ESP: db0f1cc8
Jan 5 12:52:10 testbox kernel: [653723.022504] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:52:10 testbox kernel: [653723.022504] CR0: 8005003b CR2: 00000000 CR3: 1b0d9000 CR4: 000006b0
Jan 5 12:52:10 testbox kernel: [653723.022504] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:52:10 testbox kernel: [653723.022504] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:52:10 testbox kernel: [653723.022504] Call Trace:
Jan 5 12:52:10 testbox kernel: [653723.022504] [<c050da98>] _spin_lock+0x8/0x10
Jan 5 12:52:10 testbox kernel: [653723.022504] [<c025393e>] journal_dirty_data+0x5e/0x1f0
Jan 5 12:52:10 testbox kernel: [653723.022504] [<c021b048>] ext3_journal_dirty_data+0x18/0x50
Jan 5 12:52:10 testbox kernel: [653723.022504] [<c021b7ef>] walk_page_buffers+0x2f/0x80
Jan 5 12:52:10 testbox kernel: [653723.022504] [<c021bbf3>] ext3_ordered_write_end+0x63/0x1b0
Jan 5 12:52:10 testbox kernel: [653723.022504] [<c021b030>] ? ext3_journal_dirty_data+0x0/0x50
Jan 5 12:52:10 testbox kernel: [653723.022504] [<c0195e6d>] ? iov_iter_copy_from_user_atomic+0x5d/0xc0
Jan 5 12:52:10 testbox kernel: [653723.022504] [<c0195fdf>] generic_perform_write+0x10f/0x190
Jan 5 12:52:10 testbox kernel: [653723.022504] [<c01970ad>] generic_file_buffered_write+0x5d/0x120
Jan 5 12:52:10 testbox kernel: [653723.022504] [<c01986ab>] __generic_file_aio_write_nolock+0x19b/0x520
Jan 5 12:52:10 testbox kernel: [653723.022504] [<c013949e>] ? resched_task+0x1e/0x70
Jan 5 12:52:10 testbox kernel: [653723.022504] [<c0198b3e>] generic_file_aio_write+0x5e/0xd0
Jan 5 12:52:10 testbox kernel: [653723.022504] [<c02194fd>] ext3_file_write+0x2d/0xc0
Jan 5 12:52:10 testbox kernel: [653723.022504] [<c01c6831>] do_sync_write+0xd1/0x110
Jan 5 12:52:10 testbox kernel: [653723.022504] [<c03f1082>] ? rtc_update_irq+0x92/0xa0
Jan 5 12:52:10 testbox kernel: [653723.022504] [<c0156a70>] ? autoremove_wake_function+0x0/0x50
Jan 5 12:52:10 testbox kernel: [653723.022504] [<c02b22f0>] ? apparmor_file_permission+0x20/0x30
Jan 5 12:52:10 testbox kernel: [653723.022504] [<c029049f>] ? security_file_permission+0xf/0x20
Jan 5 12:52:10 testbox kernel: [653723.022504] [<c01c69d4>] ? rw_verify_area+0x54/0xd0
Jan 5 12:52:10 testbox kernel: [653723.022504] [<c01c6ec8>] vfs_write+0x98/0x110
Jan 5 12:52:10 testbox kernel: [653723.022504] [<c01c6760>] ? do_sync_write+0x0/0x110
Jan 5 12:52:10 testbox kernel: [653723.022504] [<c01c6ffd>] sys_write+0x3d/0x70
Jan 5 12:52:10 testbox kernel: [653723.022504] [<c0109eef>] sysenter_do_call+0x12/0x2f
Jan 5 12:52:10 testbox kernel: [653723.025920] hpet1: lost 3 rtc interrupts
Jan 5 12:52:44 testbox kernel: [653757.380003] BUG: soft lockup - CPU#0 stuck for 61s! [hal-acl-tool:4168]
Jan 5 12:52:44 testbox kernel: [653757.380005] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:52:44 testbox kernel: [653757.381694]
Jan 5 12:52:44 testbox kernel: [653757.381694] Pid: 4168, comm: hal-acl-tool Tainted: G      D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:52:44 testbox kernel: [653757.381694] EIP: 0060:[<c012907f>] EFLAGS: 00000293 CPU: 0
Jan 5 12:52:44 testbox kernel: [653757.381694] EIP is at __ticket_spin_lock+0xf/0x20
Jan 5 12:52:44 testbox kernel: [653757.381694] EAX: f5d962b8 EBX: d6ba3b00 ECX: f5d96200 EDX: 00004f4d
Jan 5 12:52:44 testbox kernel: [653757.381694] ESI: d20d3000 EDI: f5d962b8 EBP: e0491c60 ESP: e0491c60
Jan 5 12:52:44 testbox kernel: [653757.381694] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:52:44 testbox kernel: [653757.381694] CR0: 8005003b CR2: 09202568 CR3: 2a445000 CR4: 000006b0
Jan 5 12:52:44 testbox kernel: [653757.381694] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:52:44 testbox kernel: [653757.381694] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:52:44 testbox kernel: [653757.381694] Call Trace:
Jan 5 12:52:44 testbox kernel: [653757.381694] [<c050da98>] _spin_lock+0x8/0x10
Jan 5 12:52:44 testbox kernel: [653757.381694] [<c025345d>] __journal_try_to_free_buffer+0x4d/0xc0
Jan 5 12:52:44 testbox kernel: [653757.381694] [<c0253532>] journal_try_to_free_buffers+0x62/0x100
Jan 5 12:52:44 testbox kernel: [653757.381694] [<c021bda5>] ext3_releasepage+0x45/0x70
Jan 5 12:52:44 testbox kernel: [653757.381694] [<c0196c8e>] try_to_release_page+0x2e/0x50
Jan 5 12:52:44 testbox kernel: [653757.381694] [<c01a0346>] __invalidate_mapping_pages+0x136/0x150
Jan 5 12:52:44 testbox kernel: [653757.381694] [<c01a0372>] invalidate_mapping_pages+0x12/0x20
Jan 5 12:52:44 testbox kernel: [653757.381694] [<c01d98ba>] prune_icache+0x21a/0x260
Jan 5 12:52:44 testbox kernel: [653757.381694] [<c01d993e>] shrink_icache_memory+0x3e/0x50
Jan 5 12:52:44 testbox kernel: [653757.381694] [<c01a086f>] shrink_slab+0x11f/0x180
Jan 5 12:52:44 testbox kernel: [653757.381694] [<c01a2995>] do_try_to_free_pages+0x155/0x2c0
Jan 5 12:52:44 testbox kernel: [653757.381694] [<c0156a8b>] ? autoremove_wake_function+0x1b/0x50
Jan 5 12:52:44 testbox kernel: [653757.381694] [<c01a2bec>] try_to_free_pages+0x6c/0x80
Jan 5 12:52:44 testbox kernel: [653757.381694] [<c01a13a0>] ? isolate_pages_global+0x0/0x70
Jan 5 12:52:44 testbox kernel: [653757.381694] [<c019bdbb>] __alloc_pages_internal+0x1cb/0x450
Jan 5 12:52:44 testbox kernel: [653757.381694] [<c01c004b>] new_slab+0x7b/0x1e0
Jan 5 12:52:44 testbox kernel: [653757.381694] [<c01c1165>] ? unfreeze_slab+0x75/0xb0
Jan 5 12:52:44 testbox kernel: [653757.381694] [<c01c141b>] __slab_alloc+0x20b/0x260
Jan 5 12:52:44 testbox kernel: [653757.381694] [<c01c1516>] kmem_cache_alloc+0xa6/0xb0
Jan 5 12:52:44 testbox kernel: [653757.381694] [<c013ef4c>] ? dup_task_struct+0x1c/0x110
Jan 5 12:52:44 testbox kernel: [653757.381694] [<c013ef4c>] dup_task_struct+0x1c/0x110
Jan 5 12:52:44 testbox kernel: [653757.381694] [<c013fd7e>] copy_process+0x6e/0xcf0
Jan 5 12:52:44 testbox kernel: [653757.381694] [<c01db240>] ? alloc_fd+0xe0/0x100
Jan 5 12:52:44 testbox kernel: [653757.381694] [<c01c7bbc>] ? get_empty_filp+0x6c/0x170
Jan 5 12:52:44 testbox kernel: [653757.381694] [<c050da98>] ? _spin_lock+0x8/0x10
Jan 5 12:52:44 testbox kernel: [653757.381694] [<c0140a62>] do_fork+0x62/0x300
Jan 5 12:52:44 testbox kernel: [653757.381694] [<c01cd6ef>] ? sys_pipe2+0x3f/0x60
Jan 5 12:52:44 testbox kernel: [653757.381694] [<c01083df>] sys_clone+0x2f/0x40
Jan 5 12:52:44 testbox kernel: [653757.381694] [<c0109eef>] sysenter_do_call+0x12/0x2f
Jan 5 12:53:15 testbox kernel: [653788.512503] BUG: soft lockup - CPU#3 stuck for 61s! [rsync:30532]
Jan 5 12:53:15 testbox kernel: [653788.512504] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:53:15 testbox kernel: [653788.512504]
Jan 5 12:53:15 testbox kernel: [653788.512504] Pid: 30532, comm: rsync Tainted: G      D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:53:15 testbox kernel: [653788.512504] EIP: 0060:[<c0129081>] EFLAGS: 00000297 CPU: 3
Jan 5 12:53:15 testbox kernel: [653788.512504] EIP is at __ticket_spin_lock+0x11/0x20
Jan 5 12:53:15 testbox kernel: [653788.512504] EAX: f5d962b8 EBX: e0c8de00 ECX: 00000000 EDX: 00004e4d
Jan 5 12:53:15 testbox kernel: [653788.512504] ESI: f5d96200 EDI: f6d00450 EBP: db0f1cc8 ESP: db0f1cc8
Jan 5 12:53:15 testbox kernel: [653788.512504] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:53:15 testbox kernel: [653788.512504] CR0: 8005003b CR2: 00000000 CR3: 1b0d9000 CR4: 000006b0
Jan 5 12:53:15 testbox kernel: [653788.512504] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:53:15 testbox kernel: [653788.512504] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:53:15 testbox kernel: [653788.512504] Call Trace:
Jan 5 12:53:15 testbox kernel: [653788.512504] [<c050da98>] _spin_lock+0x8/0x10
Jan 5 12:53:15 testbox kernel: [653788.512504] [<c025393e>] journal_dirty_data+0x5e/0x1f0
Jan 5 12:53:15 testbox kernel: [653788.512504] [<c021b048>] ext3_journal_dirty_data+0x18/0x50
Jan 5 12:53:15 testbox kernel: [653788.512504] [<c021b7ef>] walk_page_buffers+0x2f/0x80
Jan 5 12:53:15 testbox kernel: [653788.512504] [<c021bbf3>] ext3_ordered_write_end+0x63/0x1b0
Jan 5 12:53:15 testbox kernel: [653788.512504] [<c021b030>] ? ext3_journal_dirty_data+0x0/0x50
Jan 5 12:53:15 testbox kernel: [653788.512504] [<c0195e6d>] ? iov_iter_copy_from_user_atomic+0x5d/0xc0
Jan 5 12:53:15 testbox kernel: [653788.512504] [<c0195fdf>] generic_perform_write+0x10f/0x190
Jan 5 12:53:15 testbox kernel: [653788.512504] [<c01970ad>] generic_file_buffered_write+0x5d/0x120
Jan 5 12:53:15 testbox kernel: [653788.512504] [<c01986ab>] __generic_file_aio_write_nolock+0x19b/0x520
Jan 5 12:53:15 testbox kernel: [653788.512504] [<c013949e>] ? resched_task+0x1e/0x70
Jan 5 12:53:15 testbox kernel: [653788.512504] [<c0198b3e>] generic_file_aio_write+0x5e/0xd0
Jan 5 12:53:15 testbox kernel: [653788.512504] [<c02194fd>] ext3_file_write+0x2d/0xc0
Jan 5 12:53:15 testbox kernel: [653788.512504] [<c01c6831>] do_sync_write+0xd1/0x110
Jan 5 12:53:15 testbox kernel: [653788.512504] [<c03f1082>] ? rtc_update_irq+0x92/0xa0
Jan 5 12:53:15 testbox kernel: [653788.512504] [<c0156a70>] ? autoremove_wake_function+0x0/0x50
Jan 5 12:53:15 testbox kernel: [653788.512504] [<c02b22f0>] ? apparmor_file_permission+0x20/0x30
Jan 5 12:53:15 testbox kernel: [653788.512504] [<c029049f>] ? security_file_permission+0xf/0x20
Jan 5 12:53:15 testbox kernel: [653788.512504] [<c01c69d4>] ? rw_verify_area+0x54/0xd0
Jan 5 12:53:15 testbox kernel: [653788.512504] [<c01c6ec8>] vfs_write+0x98/0x110
Jan 5 12:53:15 testbox kernel: [653788.512504] [<c01c6760>] ? do_sync_write+0x0/0x110
Jan 5 12:53:15 testbox kernel: [653788.512504] [<c01c6ffd>] sys_write+0x3d/0x70
Jan 5 12:53:15 testbox kernel: [653788.512504] [<c0109eef>] sysenter_do_call+0x12/0x2f
Jan 5 12:53:15 testbox kernel: [653788.516515] hpet1: lost 3 rtc interrupts
Jan 5 12:53:49 testbox kernel: [653822.880003] BUG: soft lockup - CPU#0 stuck for 61s! [hal-acl-tool:4168]
Jan 5 12:53:49 testbox kernel: [653822.880005] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:53:49 testbox kernel: [653822.881681]
Jan 5 12:53:49 testbox kernel: [653822.881681] Pid: 4168, comm: hal-acl-tool Tainted: G      D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:53:49 testbox kernel: [653822.881681] EIP: 0060:[<c012907f>] EFLAGS: 00000293 CPU: 0
Jan 5 12:53:49 testbox kernel: [653822.881681] EIP is at __ticket_spin_lock+0xf/0x20
Jan 5 12:53:49 testbox kernel: [653822.881681] EAX: f5d962b8 EBX: d6ba3b00 ECX: f5d96200 EDX: 00004f4d
Jan 5 12:53:49 testbox kernel: [653822.881681] ESI: d20d3000 EDI: f5d962b8 EBP: e0491c60 ESP: e0491c60
Jan 5 12:53:49 testbox kernel: [653822.881681] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:53:49 testbox kernel: [653822.881681] CR0: 8005003b CR2: 09202568 CR3: 2a445000 CR4: 000006b0
Jan 5 12:53:49 testbox kernel: [653822.881681] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:53:49 testbox kernel: [653822.881681] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:53:49 testbox kernel: [653822.881681] Call Trace:
Jan 5 12:53:49 testbox kernel: [653822.881681] [<c050da98>] _spin_lock+0x8/0x10
Jan 5 12:53:49 testbox kernel: [653822.881681] [<c025345d>] __journal_try_to_free_buffer+0x4d/0xc0
Jan 5 12:53:49 testbox kernel: [653822.881681] [<c0253532>] journal_try_to_free_buffers+0x62/0x100
Jan 5 12:53:49 testbox kernel: [653822.881681] [<c021bda5>] ext3_releasepage+0x45/0x70
Jan 5 12:53:49 testbox kernel: [653822.881681] [<c0196c8e>] try_to_release_page+0x2e/0x50
Jan 5 12:53:49 testbox kernel: [653822.881681] [<c01a0346>] __invalidate_mapping_pages+0x136/0x150
Jan 5 12:53:49 testbox kernel: [653822.881681] [<c01a0372>] invalidate_mapping_pages+0x12/0x20
Jan 5 12:53:49 testbox kernel: [653822.881681] [<c01d98ba>] prune_icache+0x21a/0x260
Jan 5 12:53:49 testbox kernel: [653822.881681] [<c01d993e>] shrink_icache_memory+0x3e/0x50
Jan 5 12:53:49 testbox kernel: [653822.881681] [<c01a086f>] shrink_slab+0x11f/0x180
Jan 5 12:53:49 testbox kernel: [653822.881681] [<c01a2995>] do_try_to_free_pages+0x155/0x2c0
Jan 5 12:53:49 testbox kernel: [653822.881681] [<c0156a8b>] ? autoremove_wake_function+0x1b/0x50
Jan 5 12:53:49 testbox kernel: [653822.881681] [<c01a2bec>] try_to_free_pages+0x6c/0x80
Jan 5 12:53:49 testbox kernel: [653822.881681] [<c01a13a0>] ? isolate_pages_global+0x0/0x70
Jan 5 12:53:50 testbox kernel: [653822.881681] [<c019bdbb>] __alloc_pages_internal+0x1cb/0x450
Jan 5 12:53:50 testbox kernel: [653822.881681] [<c01c004b>] new_slab+0x7b/0x1e0
Jan 5 12:53:50 testbox kernel: [653822.881681] [<c01c1165>] ? unfreeze_slab+0x75/0xb0
Jan 5 12:53:50 testbox kernel: [653822.881681] [<c01c141b>] __slab_alloc+0x20b/0x260
Jan 5 12:53:50 testbox kernel: [653822.881681] [<c01c1516>] kmem_cache_alloc+0xa6/0xb0
Jan 5 12:53:50 testbox kernel: [653822.881681] [<c013ef4c>] ? dup_task_struct+0x1c/0x110
Jan 5 12:53:50 testbox kernel: [653822.881681] [<c013ef4c>] dup_task_struct+0x1c/0x110
Jan 5 12:53:50 testbox kernel: [653822.881681] [<c013fd7e>] copy_process+0x6e/0xcf0
Jan 5 12:53:50 testbox kernel: [653822.881681] [<c01db240>] ? alloc_fd+0xe0/0x100
Jan 5 12:53:50 testbox kernel: [653822.881681] [<c01c7bbc>] ? get_empty_filp+0x6c/0x170
Jan 5 12:53:50 testbox kernel: [653822.881681] [<c050da98>] ? _spin_lock+0x8/0x10
Jan 5 12:53:50 testbox kernel: [653822.881681] [<c0140a62>] do_fork+0x62/0x300
Jan 5 12:53:50 testbox kernel: [653822.881681] [<c01cd6ef>] ? sys_pipe2+0x3f/0x60
Jan 5 12:53:50 testbox kernel: [653822.881681] [<c01083df>] sys_clone+0x2f/0x40
Jan 5 12:53:50 testbox kernel: [653822.881681] [<c0109eef>] sysenter_do_call+0x12/0x2f
Jan 5 12:54:21 testbox kernel: [653854.012503] BUG: soft lockup - CPU#3 stuck for 61s! [rsync:30532]
Jan 5 12:54:21 testbox kernel: [653854.012504] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:54:21 testbox kernel: [653854.012504]
Jan 5 12:54:21 testbox kernel: [653854.012504] Pid: 30532, comm: rsync Tainted: G      D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:54:21 testbox kernel: [653854.012504] EIP: 0060:[<c0129085>] EFLAGS: 00000297 CPU: 3
Jan 5 12:54:21 testbox kernel: [653854.012504] EIP is at __ticket_spin_lock+0x15/0x20
Jan 5 12:54:21 testbox kernel: [653854.012504] EAX: f5d962b8 EBX: e0c8de00 ECX: 00000000 EDX: 00004e4d
Jan 5 12:54:21 testbox kernel: [653854.012504] ESI: f5d96200 EDI: f6d00450 EBP: db0f1cc8 ESP: db0f1cc8
Jan 5 12:54:21 testbox kernel: [653854.012504] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:54:21 testbox kernel: [653854.012504] CR0: 8005003b CR2: 00000000 CR3: 1b0d9000 CR4: 000006b0
Jan 5 12:54:21 testbox kernel: [653854.012504] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:54:21 testbox kernel: [653854.012504] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:54:21 testbox kernel: [653854.012504] Call Trace:
Jan 5 12:54:21 testbox kernel: [653854.012504] [<c050da98>] _spin_lock+0x8/0x10
Jan 5 12:54:21 testbox kernel: [653854.012504] [<c025393e>] journal_dirty_data+0x5e/0x1f0
Jan 5 12:54:21 testbox kernel: [653854.012504] [<c021b048>] ext3_journal_dirty_data+0x18/0x50
Jan 5 12:54:21 testbox kernel: [653854.012504] [<c021b7ef>] walk_page_buffers+0x2f/0x80
Jan 5 12:54:21 testbox kernel: [653854.012504] [<c021bbf3>] ext3_ordered_write_end+0x63/0x1b0
Jan 5 12:54:21 testbox kernel: [653854.012504] [<c021b030>] ? ext3_journal_dirty_data+0x0/0x50
Jan 5 12:54:21 testbox kernel: [653854.012504] [<c0195e6d>] ? iov_iter_copy_from_user_atomic+0x5d/0xc0
Jan 5 12:54:21 testbox kernel: [653854.012504] [<c0195fdf>] generic_perform_write+0x10f/0x190
Jan 5 12:54:21 testbox kernel: [653854.012504] [<c01970ad>] generic_file_buffered_write+0x5d/0x120
Jan 5 12:54:21 testbox kernel: [653854.012504] [<c01986ab>] __generic_file_aio_write_nolock+0x19b/0x520
Jan 5 12:54:21 testbox kernel: [653854.012504] [<c013949e>] ? resched_task+0x1e/0x70
Jan 5 12:54:21 testbox kernel: [653854.012504] [<c0198b3e>] generic_file_aio_write+0x5e/0xd0
Jan 5 12:54:21 testbox kernel: [653854.012504] [<c02194fd>] ext3_file_write+0x2d/0xc0
Jan 5 12:54:21 testbox kernel: [653854.012504] [<c01c6831>] do_sync_write+0xd1/0x110
Jan 5 12:54:21 testbox kernel: [653854.012504] [<c03f1082>] ? rtc_update_irq+0x92/0xa0
Jan 5 12:54:21 testbox kernel: [653854.012504] [<c0156a70>] ? autoremove_wake_function+0x0/0x50
Jan 5 12:54:21 testbox kernel: [653854.012504] [<c02b22f0>] ? apparmor_file_permission+0x20/0x30
Jan 5 12:54:21 testbox kernel: [653854.012504] [<c029049f>] ? security_file_permission+0xf/0x20
Jan 5 12:54:21 testbox kernel: [653854.012504] [<c01c69d4>] ? rw_verify_area+0x54/0xd0
Jan 5 12:54:21 testbox kernel: [653854.012504] [<c01c6ec8>] vfs_write+0x98/0x110
Jan 5 12:54:21 testbox kernel: [653854.012504] [<c01c6760>] ? do_sync_write+0x0/0x110
Jan 5 12:54:21 testbox kernel: [653854.012504] [<c01c6ffd>] sys_write+0x3d/0x70
Jan 5 12:54:21 testbox kernel: [653854.012504] [<c0109eef>] sysenter_do_call+0x12/0x2f
Jan 5 12:54:21 testbox kernel: [653854.015940] hpet1: lost 2 rtc interrupts
Jan 5 12:54:55 testbox kernel: [653888.370004] BUG: soft lockup - CPU#0 stuck for 61s! [hal-acl-tool:4168]
Jan 5 12:54:55 testbox kernel: [653888.370005] Modules linked in: vmnet vmci vmmon binfmt_misc drbd video output input_polldev ocfs2_stackglue ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs k8temp hwmon_vid lp snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse ppdev snd soundcore serio_raw pcspkr i2c_piix4 parport_pc parport shpchp snd_page_alloc usb_storage r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor
Jan 5 12:54:55 testbox kernel: [653888.371683]
Jan 5 12:54:55 testbox kernel: [653888.371683] Pid: 4168, comm: hal-acl-tool Tainted: G      D W (2.6.28-19-server #66-Ubuntu) System Product Name
Jan 5 12:54:55 testbox kernel: [653888.371683] EIP: 0060:[<c012907f>] EFLAGS: 00000293 CPU: 0
Jan 5 12:54:55 testbox kernel: [653888.371683] EIP is at __ticket_spin_lock+0xf/0x20
Jan 5 12:54:55 testbox kernel: [653888.371683] EAX: f5d962b8 EBX: d6ba3b00 ECX: f5d96200 EDX: 00004f4d
Jan 5 12:54:55 testbox kernel: [653888.371683] ESI: d20d3000 EDI: f5d962b8 EBP: e0491c60 ESP: e0491c60
Jan 5 12:54:55 testbox kernel: [653888.371683] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jan 5 12:54:55 testbox kernel: [653888.371683] CR0: 8005003b CR2: 09202568 CR3: 2a445000 CR4: 000006b0
Jan 5 12:54:55 testbox kernel: [653888.371683] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Jan 5 12:54:55 testbox kernel: [653888.371683] DR6: ffff0ff0 DR7: 00000400
Jan 5 12:54:55 testbox kernel: [653888.371683] Call Trace:
Jan 5 12:54:55 testbox kernel: [653888.371683] [<c050da98>] _spin_lock+0x8/0x10
Jan 5 12:54:55 testbox kernel: [653888.371683] [<c025345d>] __journal_try_to_free_buffer+0x4d/0xc0
Jan 5 12:54:55 testbox kernel: [653888.371683] [<c0253532>] journal_try_to_free_buffers+0x62/0x100
Jan 5 12:54:55 testbox kernel: [653888.371683] [<c021bda5>] ext3_releasepage+0x45/0x70
Jan 5 12:54:55 testbox kernel: [653888.371683] [<c0196c8e>] try_to_release_page+0x2e/0x50
Jan 5 12:54:55 testbox kernel: [653888.371683] [<c01a0346>] __invalidate_mapping_pages+0x136/0x150
Jan 5 12:54:55 testbox kernel: [653888.371683] [<c01a0372>] invalidate_mapping_pages+0x12/0x20
Jan 5 12:54:55 testbox kernel: [653888.371683] [<c01d98ba>] prune_icache+0x21a/0x260
Jan 5 12:54:55 testbox kernel: [653888.371683] [<c01d993e>] shrink_icache_memory+0x3e/0x50
Jan 5 12:54:55 testbox kernel: [653888.371683] [<c01a086f>] shrink_slab+0x11f/0x180
Jan 5 12:54:55 testbox kernel: [653888.371683] [<c01a2995>] do_try_to_free_pages+0x155/0x2c0
Jan 5 12:54:55 testbox kernel: [653888.371683] [<c0156a8b>] ? autoremove_wake_function+0x1b/0x50
Jan 5 12:54:55 testbox kernel: [653888.371683] [<c01a2bec>] try_to_free_pages+0x6c/0x80
Jan 5 12:54:55 testbox kernel: [653888.371683] [<c01a13a0>] ? isolate_pages_global+0x0/0x70
Jan 5 12:54:55 testbox kernel: [653888.371683] [<c019bdbb>] __alloc_pages_internal+0x1cb/0x450
Jan 5 12:54:55 testbox kernel: [653888.371683] [<c01c004b>] new_slab+0x7b/0x1e0
Jan 5 12:54:55 testbox kernel: [653888.371683] [<c01c1165>] ? unfreeze_slab+0x75/0xb0
Jan 5 12:54:55 testbox kernel: [653888.371683] [<c01c141b>] __slab_alloc+0x20b/0x260
Jan 5 12:54:55 testbox kernel: [653888.371683] [<c01c1516>] kmem_cache_alloc+0xa6/0xb0
Jan 5 12:54:55 testbox kernel: [653888.371683] [<c013ef4c>] ? dup_task_struct+0x1c/0x110
Jan 5 12:54:55 testbox kernel: [653888.371683] [<c013ef4c>] dup_task_struct+0x1c/0x110
Jan 5 12:54:55 testbox kernel: [653888.371683] [<c013fd7e>] copy_process+0x6e/0xcf0
Jan 5 12:54:55 testbox kernel: [653888.371683] [<c01db240>] ? alloc_fd+0xe0/0x100
Jan 5 12:54:55 testbox kernel: [653888.371683] [<c01c7bbc>] ? get_empty_filp+0x6c/0x170
Jan 5 12:54:55 testbox kernel: [653888.371683] [<c050da98>] ? _spin_lock+0x8/0x10
Jan 5 12:54:55 testbox kernel: [653888.371683] [<c0140a62>] do_fork+0x62/0x300
Jan 5 12:54:55 testbox kernel: [653888.371683] [<c01cd6ef>] ? sys_pipe2+0x3f/0x60
Jan 5 12:54:55 testbox kernel: [653888.371683] [<c01083df>] sys_clone+0x2f/0x40
Jan 5 12:54:55 testbox kernel: [653888.371683] [<c0109eef>] sysenter_do_call+0x12/0x2f