Dell Server Workstation iDRAC Dead after Firmware Update Solution R720, R320, R730
This is certainly a poor design, as you can read many seasoned admins who have updated their iDRAC only to have it killed. One possible cause is not by doing all the incremental updates, doing updates from an old iDRAC to one many revisions newer is a sure way to kill things, but even then there is no guarantee based on the amount of failures.
List of threads of people's dead iDRAC's:
https://www.reddit.com/r/homelab/comments/a49b4y/r720_idrac_help/?rdt=40792
https://www.reddit.com/r/homelab/comments/960wx8/dell_r720_idrac_recovery/
https://community.spiceworks.com/t/alert-idrac-not-responding-rebooting/776991
The first solution is don't mass update, as we've seen entire datacenters or racks full of Dell machines with dead iDRAC due to mass update attempts. It is likely not practical to do this fix on a wide scale unless you can migrate the applications to another facility or rack as this requires the servers to be pulled off the rack or at least pulled by the rails and to open the case of course.
Try This First (only if you have lights on the iDRAC):
This solution is for if your iDRAC is totally dead, no flashing lights, dead NIC etc...
You will first need a USB to TTL adapter like this Amazon affil link.
Another successful recovery: http://blog.ignoranthack.me/?p=86
You will see Dell themselves advising to replace the motherboard, but in many cases by connecting to the UART below with a USB to TTL cable, you can tell the iDRAC to reflash from the SDCARD image.
Based off this example of a Dell R720, we can see the UART pins which are labelled by this github user.
This area is at the back right of the server (if looking from the server from the front) and will require you to remove the riser cage to access this area.
Step 1.) Connect to USB to TTL adapter.
Be sure to use the correct pins below (remember the TX side of the TTL goes to the RX side of the UART and the the RX side of the TTL goes to the TX side of the UART). If in doubt, use a multimeter to make sure you aren't connecting to the wrong pins/getting voltage where you shouldn't be.
Note About Format of SD Card
I have seen this occur on some servers, I am not sure how it was done like this but the SD Card came formatted as the root drive, without any partition. It seems that U-Boot recovery DOES NOT like this, or the fact that it is a 16GB Card even though it is from Dell.
UTIL RECOVER:Transport:sd TargetMMC:EMMC File:firmimg.d7
UTIL RECOVER:ERROR:Could not locate 'firmimg.d7' in FAT or EXT2 file system.
If you get the errors above and you are sure the file exists, then it is worth doing the following.
1.) Backup all files from the SD Card.
2.) Completely wipe the card.
3.) Format a single partition as 2GB as ext2 (ext2 not ext3 or ext4 or it won't work as the environment does not have the kernel modules to load anything but FAT and EXT2).
Step 2.) Recover
Sometimes you may need to go through 2 rounds of bootup procedure by hitting all the annoying keys to get the server booted normally for this to work.
Connect using minicom or screen and run these commands:
screen /dev/ttyUSB0 115200
The connection speed is crucial, otherwise you will see nothing or garbage characters.
If all goes well you should see like below.
Make sure you have the file firmimg.d7 or whatever the name one is from the iDRAC firmware on the SD card that inserts at the front of the server.
util recover -emmc -from sd -f firmimg.d7 -noreset -clear
Some versions will not have this -from_sd option
iDRAC8=> util recover -emmc -from_sd -f firmimg.d6 -noreset -clear
util recover:mcgetopt:ERROR: option "-from_sd" was not found.
Then once the above completes do this:
util reset
Short iDRAC DBG interrupt/reboot pins
If you see a shell prompt like below (SH7757), you will need to short the DBG pins in the image from earlier.
Use a paper clip to short the two pins shown above as the interrupt/reboot pins in the diagram above.
After shorting you should see U-Boot again:
U-Boot 2009.08 (Mar 10 2012 - 02:52:09) Avocent (0.0.3) EVB, Build: jenkins-12G_iDRAC_Releases-402
CPU: SH-4A
BOARD: R0P7757LC00xxRL (C0 step) board
BOOT: Secure, HRK generated
DRAM: 240MB
(240MB of 256MB total DRAM is available on U-Boot)
ENV: Using primary env area.
In: serial
Out: serial
Err: serial
PCIe: Bridge loaded with 0x18000 bytes
WDT2: Booted Lower Vector, 'uboot1'
sh_mmcif: 0, sh-sdhi: 1
Net: sh_eth.0, sh_g_eth.0
INFO: 00:002 Start-up -to- util_idrac_main()
INFO: 00:004 U-Boot 2009.08 (Mar 10 2012 - 02:52:09) Avocent (0.0.3) EVB
INFO: 00:007 U-Boot checkin date(03-02-2012) Version(1.0.175)
INFO: 00:006 iDRAC PPID <NULL>
INFO: 00:003 SPI NOR init 4096 KiB N25Q032 bus=0 cs=0, speed=1000000, mode=3
INFO: 00:007 SH-4A Product: Major Ver=0x31 Minor Ver=0x13 C3 Little endian
Family=0x10 Major Ver=0x30 Minor Ver=0x0b
PASS: 00:016 Dedicated monolithic mgmt NIC disabled
INFO: 00:128 BCM54610 OUI=0x3fffff Model=0x3f Revision=0x0f PhyAddr=1
INFO: 00:362 SD CARD: Device: sh-sdhi Manufacturer ID: 41 OEM: 3432
Name: DELL1 Tran Speed: 25000000 Rd Block Len: 512
SD version 2.0 High Capacity: Yes Capacity: -843055104
INFO: 00:059 EMMC: Device: sh_mmcif Manufacturer ID: 90 OEM: 14a
Name: HYNIX Tran Speed: 25000000 Rd Block Len: 512
MMC version 4.0 High Capacity: Yes Capacity: 0
INFO: 00:019 CPLD: Major Ver=0x1 Minor Ver=0x0 Maint Ver=0x3
Planar: Type=0x02 Rev=0x8 Rework=0x8 Scratch/PathRetry=0x00
PASS: 00:013 Coin cell detected good, AD=0x397 low water=0x2c1
PASS: 00:008 PCIe C0 Ver=0.13 MCTP en, CRC=0x19592849 @0x8efbf3a8 cnt=0x18000
INFO: 00:007 Init PCIe mailbox(PCIe 0xFFEE0150=0x40010000)
INFO: 00:006 mode=normal
INFO: 00:002 reset_cause=ac
PASS: 00:005 Booted Lower Vector, 'uboot1' wdt2cnt=0
INFO: 00:005 wdt0cnt=0
PASS: 00:003 Clear CH1/CH2, clear 4K shared memory@0xffcaa000 on AC power up
PASS: 00:007 SMR0 no sermux env, default 0xd4
INFO: 00:004 GRACR=0x3c HISEL=0x00 SIRQCR5_D=0x03 SIRQCR6_D=0x01 LADMSK0=0xff2
MRSTCR0=0xfedffe7f MRSTCR1=0xfff3ff0f MRSTCR2=0x7f80feff
BARMAP=0x1 BCR=0x85000000 NCER=0x01fc NCMCR=0x0006 NCCSR=0x0303
PASS: 00:021 etherc0=78:45:C4:F6:88:68
getherc0=78:45:C4:F6:88:69
INFO: 00:010 Fan logic for monolithic planar type 2
fan1 - def 0000 1fff 3d fan2 - def 0000 1fff 3d
fan3 - def 0000 1fff 3d fan4 - def 0000 1fff 3d
fan5 - def 0000 1fff 3d fan6 - def 0000 1fff 3d
fan7 - def 0000 1fff 3d fan8 - def 0000 1fff 3d
INFO: 00:076 Env and backup CRC'ed ok
*** no text signature found ***
INFO: 00:536 Sync eMMC/SPI NOR/Alternate u-boot images
PASS: 00:258 Current u-boot1 1.0.175 verified with 'ubootN'
Trailer Struct - Missing start token, exp=0xc0de1111 rec=0x0
U-boot2 in sync with u-boot1 1.0.175
PASS: 03:607 Verify OS Images N: CRC32 OK: Kernel=0x4d21804a Rootfs=0x40007db7
PASS: 00:008 Boot device=emmc Boot partition1/N
Boot Path Retry:P1/N=0 P5/N-1=0
INFO: 00:000 05:198
WDT2: Disabled
Initialization complete, proceed with boot
*** no text signature found ***
No. Device
---------------------
1 EVB SD/MMC
4 FLASH
Current Device ==> 1
OS_WDT0_EN:180 seconds, Re-load ILRAM:
Boot Path Retry:P1/N=0 P5/N-1=0
Copying kernel image ... OK
## Booting kernel from Legacy Image at 82000000 ...
Image Name: sh-linux
Image Type: SuperH Linux Kernel Image (gzip compressed)
Data Size: 2535377 Bytes = 2.4 MB
Load Address: 80001000
Entry Point: 80002000
Verifying Checksum ... OK
Uncompressing Kernel Image ... OK
WDT2: Disable in do_bootm_linux()
sh_eth_init
Monolithic/DRB
init started: BusyBox v1.18.4 (2012-03-10 03:35:27 UTC)
starting pid 415, tty '': '/etc/init.d/rcS'
waitfor: pid 415 ...
setting net.ipv6.conf.default.accept_dad based on kernel bootup option
enable normal accept DAD for ipv6
Sat Jan 1 00:00:01 UTC 2000 To execute command __source /etc/sysapps_script/I_1000_mountfs.sh__
Sat Jan 1 00:00:03 UTC 2000 To execute command __source /etc/sysapps_script/I_1001_trigwdt.sh__
<========= Start H/W watchdog first... =========>
Sat Jan 1 00:00:03 UTC 2000 To execute command __source /etc/sysapps_script/I_1100_nfs.sh__
Sat Jan 1 00:00:03 UTC 2000 To execute command __source /etc/sysapps_script/I_1105_ps.sh__
XXXXXX /dev/mmcblk0p11
/dev/mmcblk0p11: recovering journal
fsck.ext3: Attempt to read block from filesystem resulted in short read while trying to re-open /dev/mmcblk0p11
e2fsck: io manager magic bad!
e2fsck 1.41.14 (22-Dec-2010)
fsck.ext3: Attempt to read block from filesystem resulted in short read while trying to open /dev/mmcblk0p11
Could this be a zero-length partition?
mount: mounting /dev/mmcblk0p11 on /flash/data0 failed: Invalid argument
mounting private storage fail, formatting partition
/etc/init.d/rcS: /etc/sysapps_script/I_1105_ps.sh: line 29: mkfs.ext3: Input/output error
/etc/init.d/rcS: /etc/sysapps_script/I_1105_ps.sh: line 29: tune2fs: Input/output error
mount: mounting /dev/mmcblk0p11 on /flash/data0 failed: Invalid argument
Sat Jan 1 00:00:04 UTC 2000 To execute command __source /etc/sysapps_script/I_1106_clear_ps.sh__
Sat Jan 1 00:00:04 UTC 2000 To execute command __source /etc/sysapps_script/I_1150_SYS_Drv.sh__
insmod: can't insert '/lib/modules/aess_eventhandlerdrv.ko': Input/output error
insmod: can't insert '/lib/modules/aess_kcsdrv.ko': Input/output error
insmod: can't insert '/lib/modules/aess_biospostdrv.ko': Input/output error
insmod: can't insert '/lib/modules/aess_gpiodrv.ko': Input/output error
insmod: can't insert '/lib/modules/aess_sgpiodrv.ko': Input/output error
insmod: can't insert '/lib/modules/aess_dynairqdrv.ko': Input/output error
insmod: can't insert '/lib/modules/aess_i2c_hwctrldrv.ko': Input/output error
insmod: can't insert '/lib/modules/aess_i2cdrv.ko': Input/output error
insmod: can't insert '/lib/modules/aess_spi1drv.ko': Input/output error
insmod: can't insert '/lib/modules/aess_pecisensordrv.ko': Input/output error
insmod: can't insert '/lib/modules/cryptodev.ko': Input/output error
insmod: read error from '/lib/modules/sh_tsip.ko': Input/output error
insmod: can't insert '/lib/modules/sh_tsip.ko': Input/output error
insmod: can't insert '/lib/modules/aess_fansensordrv.ko': Input/output error
insmod: can't insert '/lib/modules/aess_pwmdrv.ko': Input/output error
insmod: can't insert '/lib/modules/aess_timerdrv.ko': Input/output error
insmod: can't insert '/lib/modules/dell_cplddrv.ko': Input/output error
insmod: can't insert '/lib/modules/aess_rspidrv.ko': Input/output error
insmod: can't insert '/lib/modules/dell_fpdrv.ko': Input/output error
insmod: can't insert '/lib/modules/aess_adcsensordrv.ko': Input/output error
insmod: can't insert '/lib/modules/VKCSDriver.ko': Input/output error
insmod: can't insert '/lib/modules/aess_memdrv.ko': Input/output error
insmod: read error from '/lib/modules/sh_pbi.ko': Input/output error
insmod: can't insert '/lib/modules/sh_pbi.ko': Input/output error
Sat Jan 1 00:00:04 UTC 2000 To execute command __source /etc/sysapps_script/I_1170_mknode.sh__
Sat Jan 1 00:00:05 UTC 2000 To execute command __/etc/sysapps_script/config_lib_init.sh__
mounting Platform Data 1 partition
mount: mounting /dev/mmcblk0p9 on /tmp/pd0/ failed: Input/output error
/etc/sysapps_script/config_lib_init.sh: line 28: /etc/default/ipmi/getsysid: not found
mkdir: can't create directory '/flash/data0/config/': Read-only file system
/etc/sysapps_script/config_lib_init.sh: cd: line 38: can't cd to /flash/data0/config/
ln: ./lmcfg.txt: Read-only file system
ln: ./platcfggrp.txt: Read-only file system
ln: ./platcfgfld.txt: Read-only file system
ln: ./cfgfld.txt: Read-only file system
ln: ./cfggrp.txt: Read-only file system
ln: ./gencfggrp.txt: Read-only file system
ln: ./gencfgfld.txt: Read-only file system
ln: ./altdefaults.txt: Read-only file system
cp: can't stat '/tmp/pd0/network_config/Orca/iDRACnet.default': No such file or directory
cp: can't stat '/tmp/pd0/network_config/Orca/NICSelection.sh': No such file or directory
Sat Jan 1 00:00:06 UTC 2000 To execute command __source /etc/sysapps_script/I_1160_OSINET.sh__
Sat Jan 1 00:00:06 UTC 2000 To execute command __source /etc/sysapps_script/I_1200_etc_defaults.sh__
cp: can't create '/flash/data0/etc/hosts': Read-only file system
mkdir: can't create directory '/flash/data0/sysconfig': Read-only file system
mkdir: can't create directory '/flash/data0/ipsec': Read-only file system
/etc/sysapps_script/network_dev_init.sh: line 34: /tmp/pd0/ipmi/getsysid: not found
cp: can't create '/flash/data0/etc/hosts': Read-only file system
Sat Jan 1 00:00:06 UTC 2000 To execute command __source /etc/sysapps_script/I_1210_sysklog_syslogd.sh__
mkdir: can't create directory '/var/log/raclogd/': No such file or directory
touch: /var/log/raclogd/raclog: No such file or directory
chmod: /var/log/raclogd/raclog: No such file or directory
/etc/init.d/rcS: /etc/sysapps_script/I_1210_sysklog_syslogd.sh: line 16: /etc/sysapps_script/syslog.sh: Input/output error
Sat Jan 1 00:00:07 UTC 2000 To execute command __source /etc/sysapps_script/I_trab_asdf.sh__
insmod: read error from '/lib/modules/trab.ko': Input/output error
insmod: can't insert '/lib/modules/trab.ko': Input/output error
Sat Jan 1 00:00:07 UTC 2000 To execute command __source /etc/sysapps_script/I_usb_drv.sh__
NOTICE: Not full features configuration - kbdmouse not operational
insmod: can't insert '/lib/modules/g_r8a66597_udc.ko': Input/output error
failed load of g_r8a66597_udc
Sat Jan 1 00:00:07 UTC 2000 To execute command __source /etc/sysapps_script/I_video_drv.sh__
insmod: can't insert '/lib/modules/aess_video.ko': Input/output error
waitfor: pid 415, wpid 415
Please press Enter to activate this console. ^[[5~^[[5~
starting pid 550, tty '': '-/bin/sh'
Other useful recon output of how things work
iDRAC8=> util recover
UTIL RECOVER:Transport:tftp TargetMMC:EMMC File:firmimg.d7
UTIL RECOVER:Loading file "firmimg.d7" from TFTP server IP "192.168.0.100"
reset PHY of sh_g_eth.0
sh_eth: phy reset timeout
sh_eth: phy config timeout
UTIL RECOVER:ERROR:TFTP load.
UTIL FAIL
iDRAC8 commands
iDRAC8=> util help
Available "util" subcommands:
evc [-en
evc [-rise] [-fall] [-monitor]
fan [-set
fan [-init]
flash [-sd | -spi | -emmc]
[-b
iDRAC8=>
iDRAC8=> util recover -emmc -from sd -f firmimg.d6 -noreset -clear
util recover:mcgetopt:ERROR: option "-clear" was not found.
UTIL FAIL
iDRAC8=>
iDRAC8=> util recover -emmc -from sd -f firmimg.d6 -noreset
UTIL RECOVER:Transport:sd TargetMMC:EMMC File:firmimg.d6
UTIL RECOVER:ERROR:Could not locate 'firmimg.d6' in FAT or EXT2 file system.
UTIL FAIL
iDRAC8=> util recover -emmc -from sd -f firmimg.d7 -noreset
UTIL RECOVER:Transport:sd TargetMMC:EMMC File:firmimg.d7
UTIL RECOVER:ERROR:Could not locate 'firmimg.d7' in FAT or EXT2 file system.
U-Boot 2009.08 (Jun 11 2018 - 05:56:29) Avocent (0.0.3) EVB
CPU: SH-4A
BOARD: R0P7757LC00xxRL (Cn step) board
BOOT: Secure, HRK generated
DRAM: 240MB
(240MB of 256MB total DRAM is available on U-Boot)
ENV: Using primary env area.
In: serial
Out: serial
Err: serial
WDT2: Booted Lower Vector, 'uboot1'
sh_mmcif: 0, sh-sdhi: 1
Net: sh_eth.0, sh_g_eth.0
INFO: 00:002 Start-up -to- util_idrac_main()
INFO: 00:005 U-Boot 2009.08 (Jun 11 2018 - 05:56:29) Avocent (0.0.3) EVB
INFO: 00:006 U-Boot checkin date(08-11-2016) Version(1.1.32)
INFO: 00:006 iDRAC PPID
INFO: 00:003 SPI NOR init 4096 KiB N25Q032 bus=0 cs=0, speed=1000000, mode=3
INFO: 00:007 SH-4A Product: Major Ver=0x31 Minor Ver=0x13 C3 Little endian
Family=0x10 Major Ver=0x30 Minor Ver=0x0b
INFO: 00:140 BCM54610 OUI=0x00d897 Model=0x26 Revision=0x0a PhyAddr=1
INFO: 00:006 Dedicated monolithic mgmt NIC disabled
INFO: 00:340 SD CARD: Device: sh-sdhi Manufacturer ID: 41 OEM: 3432
Name: DELL1 Tran Speed: 25000000 Rd Blk Len: 512
SD ver 2.0 HC: Yes cap(<2gb): 3451912192 cap(>2gb): 0
ext_rev: 0 rst_n: 0 ext[96]: 0
INFO: 00:063 EMMC: Device: sh_mmcif Manufacturer ID: 90 OEM: 14a
Name: HYNIX Tran Speed: 25000000 Rd Blk Len: 512
MMC ver 4.0 HC: Yes cap(<2gb): 0 cap(>2gb): 7569408
ext_rev: 5 rst_n: 0 ext[96]: 0
INFO: 00:023 CPLD: Major Ver=0x1 Minor Ver=0x0 Maint Ver=0x3
Planar: Type=0x02 Rev=0x8 Rework=0x8 Scratch/PathRetry=0x0f
PASS: 00:014 Coin cell detected good, AD=0x397 low water=0x2c1
PASS: 00:008 PCIe SH7757_C0 Ver=0.15 MCTP en, CRC=0x8e9b6875 @0x8efbd914 cnt=0x18000
INFO: 00:008 Init PCIe mailbox(PCIe 0xFFEE0150=0x40010000)
INFO: 00:006 mode=normal
INFO: 00:002 reset_cause=recover
PASS: 00:005 Booted Lower Vector, 'uboot1' wdt2cnt=0
INFO: 00:006 wdt0cnt=0 cpld_recover_cnt=0 ddr_recover_cnt=0
PASS: 00:006 SMR0 no sermux env, default 0xd4
INFO: 00:004 GRACR=0x3c HISEL=0x00 SIRQCR5_D=0x03 SIRQCR6_D=0x01 LADMSK0=0xff2
MRSTCR0=0xfedffe7f MRSTCR1=0xfff3ff0f MRSTCR2=0x6f80feff
BARMAP=0x1 BCR=0x85000000 NCER=0x01fc NCMCR=0x0006 NCCSR=0x0303
PASS: 00:021 etherc0=
getherc0=
INFO: 00:008 Fan logic is not modified on Non-AC power up
INFO: 00:052 Env and backup CRC'ed ok
*** no text signature found ***
INFO: 00:696 Sync eMMC/SPI NOR/Alternate u-boot images
PASS: 00:077 Current u-boot1 1.1.32 verified with 'ubootN1'
Trailer Struct - Missing start token, exp=0xc0de1111 rec=0x0
FAIL: 06:813 Verify OS Images N-1: Rootfs crc exp=0xe8caa151 rec=0xcfb9c0a7
blk_start=0x48003 blk_size=0x301f0 ENV bcnt=0x603e000
FAIL: 00:013 Boot device=emmc Boot partition5/N-1
Boot Path Retry:P1/N=3 P5/N-1=3 Recover
INFO: 00:000 15:265
Hit any key to stop autoboot: 0
WDT2: Disable in abortboot()
OSWDT: Disable in abortboot()
CPLD_BMCRDY: Enable BMC_MIN_RDY in abortboot(). Prevent BIOS reset.
NOTE: After stopping u-boot in this development mode. You may need to
warm/cold reset the server when booting iDRAC manually as BIOS
may have already viewed iDRAC as unresponsive.
RECOVER:Max retries occured for both N/N-1 paths, OR forced recover.
Full Recovery Output
Sometimes it does not go well especially if you didn't know your EMMC is bad where it stores the firmware.
Not having the -noreset and -clear seems to have created this situation, with the waitpid but at that point the iDRAC lights and NIC were flashing. I then hard powered down and had to use the DBG pins to get back into U-Boot.
iDRAC8=> util recover -emmc -from sd
UTIL RECOVER:Transport:sd TargetMMC:EMMC File:firmimg.d7
Loading file "firmimg.d7" from mmc device 1:1 (xxb1)
54542562 bytes read
UTIL RECOVER:SD load passed from EXT2 fs.
UTIL RECOVER:Transport time [sec:mil]: 46:594
Clear OS images in partition/s.
Clear kernelN, rootfsN, ubootN
'EMMC' blk size=[0x200][512] Erase to 0xffffffff
Mem buf size=[0x06000000][100663296] Total bytes=[0x08000000][134217728]
blocks[0x1:0x40001][1:262145]
Blocks [0x30001:0x40000] Buf[0x88000000]..fill buffer..erase/write
Completed loop 1, Elapsed time in hr:min:sec:mil 14:122
Clear rootfsN1
'EMMC' blk size=[0x200][512] Erase to 0xffffffff
Mem buf size=[0x06000000][100663296] Total bytes=[0x06F80000][116916224]
blocks[0x48003:0x7fc03][294915:523267]
Blocks [0x78003:0x7fc02] Buf[0x88000000]..fill buffer..erase/write
Completed loop 1, Elapsed time in hr:min:sec:mil 09:950
Clear ubootN1
'EMMC' blk size=[0x200][512] Erase to 0xffffffff
Mem buf size=[0x06000000][100663296] Total bytes=[0x00080000][524288]
blocks[0x7fc04:0x80004][523268:524292]
Blocks [0x7fc04:0x80003] Buf[0x88000000]..fill buffer..erase/write
Completed loop 1, Elapsed time in hr:min:sec:mil 00:133
*** no text signature found ***
*** Updating Partition 1 ***
Checking image header CRC ... OK
Checking platform env ID..... OK
Checking kernel image CRC ... OK
Checking rootfs image CRC ... OK
Checking u-boot image CRC ... OK
Skipping u-boot update ...... NO
Checking Platform image CRC . OK
Checking processor support .. OK
Copying kernel image ........ OK
Copying rootfs .............. OK
Copying u-boot1 to flash..... OK
Copying u-boot2 to flash..... OK
Copying u-boot to MMC........ OK
Copying platform image ...... OK
*** Updating Partition 5 ***
Checking image header CRC ... OK
Checking platform env ID..... OK
Checking kernel image CRC ... OK
Checking rootfs image CRC ... OK
Checking u-boot image CRC ... OK
Skipping u-boot update ...... NO
Checking Platform image CRC . OK
Checking processor support .. OK
Copying kernel image ........ OK
Copying rootfs .............. OK
Copying u-boot1 to flash..... OK
Copying u-boot2 to flash..... OK
Copying u-boot to MMC........ OK
Copying platform image ...... OK
Erasing SPI flash at 0x100000...Writing to SPI flash...done
Erasing SPI flash at 0x110000...Writing to SPI flash...done
UTIL RECOVER:Program time [sec:mil]:01:14:394
UTIL RECOVER:Total update time [sec:mil]:02:00:992
Re-load 0x1000Kbytes from u-boot offset 0x0 into ILRAM at 0x0
Force Address Error to reset iDRAC
U-Boot 2009.08 (Mar 10 2012 - 02:52:09) Avocent (0.0.3) EVB, Build: jenkins-12G_iDRAC_Releases-402
CPU: SH-4A
BOARD: R0P7757LC00xxRL (C0 step) board
BOOT: Secure, HRK generated
DRAM: 240MB
(240MB of 256MB total DRAM is available on U-Boot)
ENV: Using primary env area.
In: serial
Out: serial
Err: serial
WDT2: Booted Lower Vector, 'uboot1'
sh_mmcif: 0, sh-sdhi: 1
Net: sh_eth.0, sh_g_eth.0
INFO: 00:002 Start-up -to- util_idrac_main()
INFO: 00:005 U-Boot 2009.08 (Mar 10 2012 - 02:52:09) Avocent (0.0.3) EVB
INFO: 00:006 U-Boot checkin date(03-02-2012) Version(1.0.175)
INFO: 00:006 iDRAC PPID
INFO: 00:003 SPI NOR init 4096 KiB N25Q032 bus=0 cs=0, speed=1000000, mode=3
INFO: 00:007 SH-4A Product: Major Ver=0x31 Minor Ver=0x13 C3 Little endian
Family=0x10 Major Ver=0x30 Minor Ver=0x0b
PASS: 00:016 Dedicated monolithic mgmt NIC disabled
INFO: 00:128 BCM54610 OUI=0x3fffff Model=0x3f Revision=0x0f PhyAddr=1
INFO: 00:365 SD CARD: Device: sh-sdhi Manufacturer ID: 41 OEM: 3432
Name: DELL1 Tran Speed: 25000000 Rd Block Len: 512
SD version 2.0 High Capacity: Yes Capacity: -843055104
INFO: 00:058 EMMC: Device: sh_mmcif Manufacturer ID: 90 OEM: 14a
Name: HYNIX Tran Speed: 25000000 Rd Block Len: 512
MMC version 4.0 High Capacity: Yes Capacity: 0
INFO: 00:019 CPLD: Major Ver=0x1 Minor Ver=0x0 Maint Ver=0x3
Planar: Type=0x02 Rev=0x8 Rework=0x0 Scratch/PathRetry=0x00
PASS: 00:014 Coin cell detected good, AD=0x397 low water=0x2c1
PASS: 00:007 PCIe C0 Ver=0.13 MCTP en, CRC=0x19592849 @0x8efbf3a8 cnt=0x18000
INFO: 00:008 Init PCIe mailbox(PCIe 0xFFEE0150=0x40010000)
INFO: 00:005 mode=normal
INFO: 00:002 reset_cause=board
PASS: 00:005 Booted Lower Vector, 'uboot1' wdt2cnt=0
INFO: 00:005 wdt0cnt=0
PASS: 00:003 SMR0 no sermux env, default 0xd4
INFO: 00:004 GRACR=0x3c HISEL=0x00 SIRQCR5_D=0x03 SIRQCR6_D=0x01 LADMSK0=0xff2
MRSTCR0=0xfedffe7f MRSTCR1=0xfff3ff0f MRSTCR2=0x7f80feff
BARMAP=0x1 BCR=0x85000000 NCER=0x01fc NCMCR=0x0006 NCCSR=0x0303
PASS: 00:021 etherc0=78:45:C4:F6:88:68
getherc0=78:45:C4:F6:88:69
INFO: 00:007 Fan logic is not modified on Non-AC power up
INFO: 00:053 Env and backup CRC'ed ok
Erasing SPI flash at 0x100000...Writing to SPI flash...done
Erasing SPI flash at 0x110000...Writing to SPI flash...done
*** no text signature found ***
INFO: 02:537 Sync eMMC/SPI NOR/Alternate u-boot images
PASS: 00:076 Current u-boot1 1.0.175 verified with 'ubootN'
Trailer Struct - Missing start token, exp=0xc0de1111 rec=0x0
PASS: 03:612 Verify OS Images N: CRC32 OK: Kernel=0x4d21804a Rootfs=0x40007db7
PASS: 00:008 Boot device=emmc Boot partition1/N
Boot Path Retry:P1/N=0 P5/N-1=0
INFO: 00:000 06:990
WDT2: Disabled
Initialization complete, proceed with boot
*** no text signature found ***
No. Device
---------------------
1 EVB SD/MMC
4 FLASH
Current Device ==> 1
OS_WDT0_EN:180 seconds, Re-load ILRAM:
Boot Path Retry:P1/N=0 P5/N-1=0
Copying kernel image ... OK
## Booting kernel from Legacy Image at 82000000 ...
Image Name: sh-linux
Image Type: SuperH Linux Kernel Image (gzip compressed)
Data Size: 2535377 Bytes = 2.4 MB
Load Address: 80001000
Entry Point: 80002000
Verifying Checksum ... OK
Uncompressing Kernel Image ... OK
WDT2: Disable in do_bootm_linux()
sh_eth_init
Monolithic/DRB
init started: BusyBox v1.18.4 (2012-03-10 03:35:27 UTC)
starting pid 415, tty '': '/etc/init.d/rcS'
waitfor: pid 415 ...
setting net.ipv6.conf.default.accept_dad based on kernel bootup option
enable normal accept DAD for ipv6
Sat Jan 1 00:00:01 UTC 2000 To execute command __source /etc/sysapps_script/I_1000_mountfs.sh__
Sat Jan 1 00:00:03 UTC 2000 To execute command __source /etc/sysapps_script/I_1001_trigwdt.sh__
<========= Start H/W watchdog first... =========>
Sat Jan 1 00:00:03 UTC 2000 To execute command __source /etc/sysapps_script/I_1100_nfs.sh__
Sat Jan 1 00:00:03 UTC 2000 To execute command __source /etc/sysapps_script/I_1105_ps.sh__
XXXXXX /dev/mmcblk0p11
/dev/mmcblk0p11: recovering journal
/dev/mmcblk0p11: Clearing orphaned inode 304 (uid=0, gid=0, mode=0100644, size=156)
/dev/mmcblk0p11: clean, 660/2048 files, 2831/4096 blocks (check in 5 mounts)
e2fsck 1.41.14 (22-Dec-2010)
/dev/mmcblk0p11: clean, 660/2048 files, 2831/4096 blocks (check in 5 mounts)
Sat Jan 1 00:00:04 UTC 2000 To execute command __source /etc/sysapps_script/I_1106_clear_ps.sh__
Sat Jan 1 00:00:04 UTC 2000 To execute command __source /etc/sysapps_script/I_1150_SYS_Drv.sh__
Sat Jan 1 00:00:07 UTC 2000 To execute command __source /etc/sysapps_script/I_1170_mknode.sh__
Sat Jan 1 00:00:08 UTC 2000 To execute command __/etc/sysapps_script/config_lib_init.sh__
mounting Platform Data 1 partition
Sat Jan 1 00:00:10 UTC 2000 To execute command __source /etc/sysapps_script/I_1160_OSINET.sh__
Sat Jan 1 00:00:10 UTC 2000 To execute command __source /etc/sysapps_script/I_1200_etc_defaults.sh__
Sat Jan 1 00:00:10 UTC 2000 To execute command __source /etc/sysapps_script/I_1210_sysklog_syslogd.sh__
starting syslogd
/etc/sysapps_script/syslog.sh: /sbin/aim not running
waitfor: pid 415, wpid 705
syslogd: /mmc1/idraclogs: Read-only file system
Jan 1 00:00:11 (none) syslogd 1.4.2: restart.
Jan 1 00:00:12 (none) kernel: klogd 1.4.2, log source = /proc/kmsg started.
Jan 1 00:00:12 (none) kernel: Cannot find map file.
Jan 1 00:00:12 (none) kernel: No module symbols loaded - kernel modules not enabled.
Jan 1 00:00:12 (none) kernel: cannot find any symbols, turning off symbol lookups
Jan 1 00:00:12 (none) kernel: Linux version 2.6.30 (jenkins@vmbrd-dh-r5-253.us.dell.com) (gcc version 4.3.4 (GCC) ) #1 Sat Mar 10 02:59:38 UTC 2012
Jan 1 00:00:12 (none) kernel: Boot params:
Jan 1 00:00:12 (none) kernel: ... MOUNT_ROOT_RDONLY - 00000000
Jan 1 00:00:12 (none) kernel: ... RAMDISK_FLAGS - 00000000
Jan 1 00:00:12 (none) kernel: ... ORIG_ROOT_DEV - 00000000
Jan 1 00:00:12 (none) kernel: ... LOADER_TYPE - 00000000
Jan 1 00:00:12 (none) kernel: ... INITRD_START - 00000000
Jan 1 00:00:12 (none) kernel: ... INITRD_SIZE - 00000000
Jan 1 00:00:12 (none) kernel: NMI Buffer at 0x83000000
Jan 1 00:00:12 (none) kernel: Booting machvec: SH7757LCR
Sat Jan 1 00:00:12 UTC 2000 To execute command __source /etc/sysapps_script/I_trab_asdf.sh__
Sat Jan 1 00:00:12 UTC 2000 To execute command __source /etc/sysapps_script/I_usb_drv.sh__
Jan 1 00:00:12 (none) kernel: Renesas R0P7757LC0012RL support.
Jan 1 00:00:12 (none) kernel: Node 0: start_pfn = 0x40000, low = 0x4ea00
Jan 1 00:00:12 (none) kernel: Zone PFN ranges:
Jan 1 00:00:12 (none) kernel: Normal 0x00040000 -> 0x0004ea00
Jan 1 00:00:12 (none) kernel: Movable zone start PFN for each node
Jan 1 00:00:12 (none) kernel: early_node_map[1] activIRQ 50/r8a66597_udc: IRQF_DISABLED is not guaranteed on shared IRQs
e PFN ranges
JIRQ 57/r8a66597_udc: IRQF_DISABLED is not guaranteed on shared IRQs
an 1 00:00:12 (none) kernel: 0: 0x00040000 -> 0x0004ea00
Jan 1 00:00:12 (none) kernel: On node 0 totalpages: 59904
Jan 1 00:00:12 (none) kernel: free_area_init_node: node 0, pgdat 8048f3ac, node_mem_map 804f3000
Jan 1 00:00:12 (none) kernel: Normal zone: 468 pages used for memmap
Jan 1 00:00:12 (none) kernel: Normal zone: 0 pages reserved
Jan 1 00:00:12 (none) kernel: Normal zone: 59436 pages, LIFO batch:15
Jan 1 00:00:12 (none) kernel: Built 1 zonelists in Zone order, mobility grouping on. Total pages: 59436
Jan 1 00:00:12 (none) kernel: Kernel command line: root=/dev/mmcblk0p2 rootwait rw rootfstype=squashfs mem=239616k console=ttyS2,115200 mac1=78:45:C4:F6:88:68 mac2=78:45:C4:F6:88:69 mode=normal reset_cause=board nmi_buf=0x83000000 quiet
Jan 1 00:00:12 (none) kernel: TRAB buffer at 8eb00000 allocated
Jan 1 00:00:12 (none) kernel: NR_IRQS:384
Jan 1 00:00:12 (none) kernel: PID hash table entries: 1024 (order: 10, 4096 bytes)
Jan 1 00:00:12 (none) kernel: Using tmu for system timer
Jan 1 00:00:12 (none) kernel: Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
Jan 1 00:00:12 (none) kernel: Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
Jan 1 00:00:12 (none) kernel: PVR=10300b00 CVR=73440410 PRR=00003113
Jan 1 00:00:12 (none) kernel: I-cache : n_ways=4 n_sets=256 way_incr=8192
Jan 1 00:00:12 (none) kernel: I-cache : entry_mask=0x00001fe0 alias_mask=0x00001000 n_aliases=2
Jan 1 00:00:12 (none) kernel: D-cache : n_ways=4 n_sets=256 way_incr=8192
Jan 1 00:00:12 (none) kernel: D-cache : entry_mask=0x00001fe0 alias_mask=0x00001000 n_ahid_init: Desc: Keyboard/Mouse Function
liases=2
Jan hid_init: Name: g_kbdmouse
1 00:00:12 (nonehid_init: Version: 20110426
) kernel: Memoryhid_bind: Using Major Number 233
: 232192k/239616hid_bind: using r8a66597_udc, Keyboard ep1 Mouse ep2
k available (347hid_bind: Mouse REL ep3
0k kernel code, 1200k data, 156k init)
Jan 1 00:00:12 (none) kernel: Calibrating delay loop... 230.19 BogoMIPS (lpj=1150976)
Jan 1 00:00:12 (none) kernel: Security Framework initialized
Jan 1 00:00:12 (none) kernel: Mount-cache hash table entries: 512
Jan 1 00:00:12 (none) kernel: CPU: SH7757
Jan 1 00:00:12 (none) kernel: net_namespace: 980 bytes
Jan 1 00:00:12 (none) kernel: PCI: Using configuration type 1
Jan 1 00:00:12 (none) kernel: NET: Registered protocol family 16
Jan 1 00:00:12 (none) kernel: sh pinmux: sh7757_pfc handling gpio 0 -> 504
Jan 1 00:00:12 (none) kernel: bio: create slab at 0
Jan 1 00:00:12 (none) kernel: usbcore: registered new interface driver usbfs
Jan 1 00:00:12 (none) kernel: usbcore: registered new interface driver hub
Jan 1 00:00:12 (none) kernel: usbcore: registered new device driver usb
Jan 1 00:00:12 (none) kernel: DMA: Registering DMA API.
Jan 1 00:00:12 (none) kernel: DMA: Registering sh_dmac handler (24 channels).
Jan 1 00:00:12 (none) kernel: Autoconfig PCI channel 0x80488900
Jan 1 00:00:12 (none) kernel: Scanning bus 00, I/O 0xfed00000:0xff100000, Mem 0x04000000:0x08000000
Jan 1 00:00:12 (none) kernel: NET: Registered protocol family 2
Jan 1 00:00:12 (none) kernel: IP route cache hash table entries: 2048 (order: 1, 8192 bytes)
Jan 1 00:00:12 (none) kernel: TCP established hash table entries: 8192 (order: 4, 65536 bytes)
Jan 1 00:00:12 (none) kernel: TCP bind hash table entries: 8192 (order: 3, 32768 bytes)
Jan 1 00:00:12 (none) kernel: TCP: Hash tables configured (established 8192 bind 8192)
Jan 1 00:00:12 (none) kernel: TCP reno registered
Jan 1 00:00:12 (none) kernel: NET: Registered protocol family 1
Jan 1 00:00:12 (none) kernel: Switched to high resolution mode on CPU 0
Jan 1 00:00:12 (none) kernel: enable PCIe shared memory area
Jan 1 00:00:12 (none) kernel: audit: initializing netlink socket (disabled)
Jan 1 00:00:12 (none) kernel: type=2000 audit(946684800.310:1): initialized
Jan 1 00:00:12 (none) kernel: squashfs: version 4.0 (2009/01/31) Phillip Lougher
Jan 1 00:00:12 (none) kernel: JFFS2 version 2.2. (NAND) (SUMMARY) © 2001-2006 Red Hat, Inc.
Jan 1 00:00:12 (none) kernel: fuse init (API version 7.11)
Jan 1 00:00:12 (none) kernel: msgmni has been set to 453
Jan 1 00:00:12 (none) kernel: alg: No test for cipher_null (cipher_null-generic)
Jan 1 00:00:12 (none) kernel: alg: No test for ecb(cipher_null) (ecb-cipher_null)
Jan 1 00:00:12 (none) kernel: alg: No test for digest_null (digest_null-generic)
Jan 1 00:00:12 (none) kernel: alg: No test for compress_null (compress_null-generic)
Jan 1 00:00:12 (none) kernel: alg: No test for stdrng (krng)
Jan 1 00:00:12 (none) kernel: io scheduler noop registered
Jan 1 00:00:12 (none) kernel: io scheduler anticipatory registered
Jan 1 00:00:12 (none) kernel: io scheduler deadline registered (default)
Jan 1 00:00:12 (none) kernel: io scheduler cfq registered
Jan 1 00:00:12 (none) kernel: Serial: 8250/16550 driver, 2 ports, IRQ sharing disabled
Jan 1 00:00:12 (none) kernel: SuperH SCI(F) driver initialized
Jan 1 00:00:12 (none) kernel: sh-sci: ttyS0 at MMIO 0xfe4b0000 (irq = 40) is a scif
Jan 1 00:00:12 (none) kernel: sh-sci: ttyS1 at MMIO 0xfe4c0000 (irq = 76) is a scif
Jan 1 00:00:12 (none) kernel: sh-sci: ttyS2 at MMIO 0xfe4d0000 (irq = 104) is a scif
Jan 1 00:00:12 (none) kernel: console [ttyS2] enabled
Jan 1 00:00:12 (none) kernel: sh-sci: ttyS3 at MMIO 0xfe4c0000 (irq = 76) is a scif
Jan 1 00:00:12 (none) kernel: sh-sci: ttyS4 at MMIO 0xfe4c0000 (irq = 76) is a scif
Jan 1 00:00:12 (none) kernel: sh-sci: ttyS5 at MMIO 0xfe4c0000 (irq = 76) is a scif
Jan 1 00:00:12 (none) kernel: sh-sci: ttyS6 at MMIO 0xfe4c0000 (irq = 76) is a scif
Jan 1 00:00:12 (none) kernel: brd: module loaded
Jan 1 00:00:12 (none) kernel: loop: module loaded
Jan 1 00:00:12 (none) kernel: nbd: registered device at major 43
Jan 1 00:00:12 (none) kernel: sh_eth_init
Jan 1 00:00:12 (none) kernel:
Jan 1 00:00:12 (none) kernel: Monolithic/DRB
Jan 1 00:00:12 (none) kernel: sh_mii: probed
Jan 1 00:00:12 (none) kernel: Base address at 0xfee00000, 00:00:00:00:00:00, IRQ 315.
Jan 1 00:00:12 (none) kernel: sh_mii: probed
Jan 1 00:00:12 (none) kernel: Base address at 0xfef00800, 00:00:00:00:00:00, IRQ 84.
Jan 1 00:00:12 (none) kernel: sh_spi sh_spi.0: registered master spi0
Jan 1 00:00:12 (none) kernel: m25p80 spi0.1: n25Q032a (4096 Kbytes)
Jan 1 00:00:12 (none) kernel: Creating 9 MTD partitions on "m25p80":
Jan 1 00:00:12 (none) kernel: 0x000000000000-0x000000080000 : "u-boot1"
Jan 1 00:00:12 (none) kernel: 0x000000080000-0x000000100000 : "u-boot2"
Jan 1 00:00:12 (none) kernel: 0x000000100000-0x000000110000 : "env1"
Jan 1 00:00:12 (none) kernel: 0x000000110000-0x000000120000 : "env2"
Jan 1 00:00:12 (none) kernel: 0x000000120000-0x000000130000 : "fru"
Jan 1 00:00:12 (none) kernel: 0x000000130000-0x000000140000 : "res1"
Jan 1 00:00:12 (none) kernel: 0x000000140000-0x0000001c0000 : "tracebuf"
Jan 1 00:00:12 (none) kernel: 0x0000001c0000-0x000000340000 : "lcl"
Jan 1 00:00:12 (none) kernel: 0x000000340000-0x000000400000 : "res2"
Jan 1 00:00:12 (none) kernel: sh_spi sh_spi.0: registered child spi0.1
Jan 1 00:00:12 (none) kernel: spi_sh: version 2009-06-10
Jan 1 00:00:12 (none) kernel: mknod /dev/aess_wdtdrv c 10 130
Jan 1 00:00:12 (none) kernel: aess_wdtdrv: initialized. (nowayout=0)
Jan 1 00:00:12 (none) kernel: device-mapper: ioctl: 4.14.0-ioctl (2008-04-23) initialised: dm-devel@redhat.com
Jan 1 00:00:12 (none) kernel: sh_mmcif: driver version 2010-03-17
Jan 1 00:00:12 (none) kernel: sh-sdhi: power control pin initiated
Jan 1 00:00:12 (none) kernel: mmc1: power off - 8f
Jan 1 00:00:12 (none) kernel: heartbeat: version 0.1.1 loaded
Jan 1 00:00:12 (none) kernel: sermux sermux: version 2010-05-26
Jan 1 00:00:12 (none) kernel: nf_conntrack version 0.5.0 (3744 buckets, 14976 max)
Jan 1 00:00:12 (none) kernel: IPv4 over IPv4 tunneling driver
Jan 1 00:00:12 (none) kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
Jan 1 00:00:12 (none) kernel: TCP cubic registered
Jan 1 00:00:12 (none) kernel: NET: Registered protocol family 10
Jan 1 00:00:12 (none) kernel: lo: Disabled Privacy Extensions
Jan 1 00:00:12 (none) kernel: tunl0: Disabled Privacy Extensions
Jan 1 00:00:12 (none) kernel: ip6_tables: (C) 2000-2006 Netfilter Core Team
Jan 1 00:00:12 (none) kernel: IPv6 over IPv4 tunneling driver
Jan 1 00:00:12 (none) kernel: sit0: Disabled Privacy Extensions
Jan 1 00:00:12 (none) kernel: NET: Registered protocol family 17
Jan 1 00:00:12 (none) kernel: RPC: Registered udp transport module.
Jan 1 00:00:12 (none) kernel: RPC: Registered tcp transport module.
Jan 1 00:00:12 (none) kernel: 802.1Q VLAN Support v1.8 Ben Greear
Tags:
dell, server, workstation, idrac, firmware, update, seasoned, admins, updated, incremental, updates, revisions, newer, failures, threads, https, www, reddit, homelab, comments, _idrac_help, rdt, wx, dell_r, _idrac_recovery, en, conversations, rack, servers, responding, poweredge, xd, hardware, iidrac, initialization, ccf, ca, rebooting, spiceworks, df, ve, datacenters, racks, attempts, migrate, applications, requires, rails, kbdoc, firmimg, flashing, nic, etc, usb, ttl, adapter, amazon, affil, http, blog, ignoranthack, advising, motherboard, connecting, uart, reflash, sdcard, pins, labelled, github, user, riser, tx, rx, multimeter, aren, voltage, shouldn, bootup, booted, minicom, commands, inserts, util, emmc, from_sd, noreset, completes, reset,