Today, mrmt, ishikawa, kmuto, Ar-, nnn went to Kashiwa NOC where some Debian JP servers and JLA servers reside. They upgraded their servers to sarge and I, from my home remotely, upgraded Debian JP servers, JLA servers and my server to sarge and kernel with help of them.

misato.debian.or.jp: this is old server. backup necessary files/data and recalled.

plat.debian.or.jp (debian archive mirror): this leaned in the rack (maybe due to the earthquake at Kanto area recently). Shutdowned and relocated. IDE controller was somehow broken, so other old server is used as replacement. While replacing, /dev/md1 was broken. /dev/md1 is partition for debian archive mirrors. I tried to fix it. At first, I fix partition type of hdb1 (Linux -> Linux auto raid detect) and recreate md.

 # mdadm --create /dev/md1 --level=linear --raid-devices=2 /dev/hda1 /dev/hdb1

Then, I could fsck on /dev/md1. However, many inodes were broken, so it seems a lot of time to fix it (note that still fscking snapshot.debian.net old archive disks (1.4TB) for 40 days...) All contents in the partition was just mirrored contents, so I recreate the ext3 filesystem and mirror from upstreams. It would take 1 day or so to recover full mirrors.

arashi.debian.or.jp (Debian JP web/mail server): After upgraded sarge, kmuto upgrade the kernel to new version. He rebooted but no response. He finally found that VGA card was broken. He replaced VGA card from recalled server (misato.debian.or.jp) and it worked. I noticed upgraded dancer-ircd couldn't communicate with older version of it. (claimed failure to match password). So, I upgrade all of our servers in our IRC network.

minori.linux.or.jp (JLA backend server) and norika.linux.or.jp: I upgraded them to sarge. Except dancer-ircd problem as above, the problem (reported by takei) was that customized cvs pserver line that I put in inetd.conf was disabled by cvs package. I re-enabled it.

These works took 5 hours, and still mirroring.