I have a 3CR860-95 and I have successfully extracted the file system from the firmware image provided by 3Com. The firmware image is not like the images for other Linux based routers. In particular, the firmware image file does not directly contain a kernel image. It only contains a cramfs image with a 16 byte header prepended to it. I had to strip that header off for cramfsck to recognize the image. I used the reverse endianness patch for cramfsck.
Interestingly, a few of the fields in the cramfs image's superblock had been zeroed out. Possibly an attempt by the 3Com people to prevent reverse engineering? I had to modify cramfsck to omit the consistency checks it normally does on those fields. When I was finally able to extract the file system from the image, I discovered what appeared to be a minimal root file system, with bin, dev, lib, mnt, proc, and sbin. Most of the directories were empty, except for ld-2.2.5.so and libc-2.2.5.so in the lib directory and init, upgrade, and upgrade_flash in the sbin directory. These programs must relate to the process of writing the new system into flash on the device.
My best guess is that upon receiving a new firmware file by HTTP, the router stores the firmware in RAM and reboots. Then rather than the kernel setting root to the normal flash device, it sets root to the cramfs image in RAM and boots to there. The init program in that image automatically invokes upgrade and/or upgrade_flash.
There are two files in the root of the upgrade file system, kernel_fs.bin and rootfs.mips. I have not been able to ascertain the format of kernel_fs.bin, but I would assume that it contains the kernel image to be written to flash. The rootfs.mips is yet another cramfs image, but this one begins 1024 bytes into the file. Stripping those 1024 bytes off, cramfsck is able to extract the file system from this image, though it caused a segfault on my machine, so I am not sure whether it got all the way through or not. It did extract a fairly complete-looking file system. The web site hosted by the router is found in /home/product and the web server resides in /home/boa. There is a minimal set of system tools present in /sbin including ash, cat, chmod, chroot, cp, dmesg, grep, gzip, ifconfig, insmod, mkdir, mount, nice, rmdir, sync, syslogd, umount, and a number of others. I will post the complete OfficeConnectSbinListing
My next step is to get mkcramfs working to repackage the file systems and see if I come up with nearly identical (except for the zeroed fields and prepended headers) cramfs images. If so, then I will try modifying some of the web pages, building a new firmware image, and upgrading the device with it. If this goes well, then I may attempt to reset the root password manually and add a telnet or ssh daemon, compiled with the MIPS cross-compiler toolchain provided by 3Com. If I can get a root terminal over Ethernet to my router, then I will be in business.
I got into some serious trouble with my router a few days ago. I created a new firmware image using all knowledge I have gathered thus far, but it was incorrect. By my best estimation, I created an image with a valid kernel but an invalid root file system. The router would start to boot, but the LED would then just go on blinking forever. Since the kernel started correctly, the watchdog was satisfied and the failsafe firmware wouldn't kick in. I thought I was S.O.L. with nothing more than a very expensive 4-port switch on my hands. Just this morning, I had the router opened up and by a stroke of luck, I got it to load its failsafe firmware. While booting the router, I shorted a couple of the data pins on one of the flash chips to ground, thus likely providing corrupt data to the CPU, so the kernel couldn't start and the watchdog was not satisfied, and the system booted the failsafe firmware. I have now flashed the stock 3Com firmware back into the router and all appears well.
My efforts have not been totally unsuccessful, though. I did succeed in repackaging the "outer" cramfs image containing the flash programs, the kernel image, and the root file system image. It had a different length and checksum than the 3Com image, but it flashed into the device just fine. The problem I encoutered was when I tried to repackage the root file system image inside the upgrade image. Obviously, I didn't get the format right. Back to the drawing board.
Note: you can also load the failsafe firmware by holding in the reset button when powering on the router.
If anyone else wants to take a crack at figuring this thing out, I'm posting an archive of the root filesystem from 3CR860-95 firmware version 1.04 at http://www.mattwhitlock.com/3CR860-0104H-rootfs.tar.bz2