On Oct 11th 14:35 JST, www.jp.debian.org (arashi.debian.or.jp) happened to be down without any notice. I'm afraid it would be hardware trouble or kernel trouble, since we got such trouble in yesterday too. Yesterday, mrmt went to the NOC and booted it up again, but this time nobody could go to the NOC, so we couldn't fix it soon.

For the time being, we've setup www.debian.or.jp on the new machine. Once arashi.debian.or.jp boot up again, we'll move the services to the new machine.

mrmt went to NOC, and found the server had died due to kernel panic. He resets the machine and it runs back again without any problem. Thanks.

On Sept 13th, 10:00 JST, www.jp.debian.org happened to fall silent, several servers had not responded and we couldn't login via ssh any more. We suspect this is because OOM or disk errors. It also seemed that named also be dead, so that ssh connection won't be established (probably, killed because tcpwrapper recognized the connection from UNKNOWN hosts).

% slogin -v arashi.debian.or.jp
OpenSSH_3.8.1p1 Debian-8.sarge.4, OpenSSL 0.9.7g 11 Apr 2005
debug1: Reading configuration data /home/ukai/.ssh/config
debug1: Applying options for *.debian.or.jp
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Connecting to arashi.debian.or.jp [210.157.158.37] port 22.
debug1: Connection established.
debug1: identity file /home/ukai/.ssh/identity type 0
debug1: identity file /home/ukai/.ssh/id_rsa type 1
debug1: identity file /home/ukai/.ssh/id_dsa type 2
ssh_exchange_identification: Connection closed by remote host
zsh: exit 255   slogin -v arashi.debian.or.jp

We're scheduling those who could get to the NOC.

I've setup debian-secure-testing mirror on http://ftp2.jp.debian.org/debian-secure-testing/ and rsync ftp2.jp.debian.org::debian-secure-testing.

debian-secure-testing would be archived in snapshot.debian.net.

Finally, all mirror has been recovered, so ftp.debian.or.jp is available now.

Today, mrmt, ishikawa, kmuto, Ar-, nnn went to Kashiwa NOC where some Debian JP servers and JLA servers reside. They upgraded their servers to sarge and I, from my home remotely, upgraded Debian JP servers, JLA servers and my server to sarge and kernel with help of them.

misato.debian.or.jp: this is old server. backup necessary files/data and recalled.

plat.debian.or.jp (debian archive mirror): this leaned in the rack (maybe due to the earthquake at Kanto area recently). Shutdowned and relocated. IDE controller was somehow broken, so other old server is used as replacement. While replacing, /dev/md1 was broken. /dev/md1 is partition for debian archive mirrors. I tried to fix it. At first, I fix partition type of hdb1 (Linux -> Linux auto raid detect) and recreate md.

 # mdadm --create /dev/md1 --level=linear --raid-devices=2 /dev/hda1 /dev/hdb1

Then, I could fsck on /dev/md1. However, many inodes were broken, so it seems a lot of time to fix it (note that still fscking snapshot.debian.net old archive disks (1.4TB) for 40 days...) All contents in the partition was just mirrored contents, so I recreate the ext3 filesystem and mirror from upstreams. It would take 1 day or so to recover full mirrors.

arashi.debian.or.jp (Debian JP web/mail server): After upgraded sarge, kmuto upgrade the kernel to new version. He rebooted but no response. He finally found that VGA card was broken. He replaced VGA card from recalled server (misato.debian.or.jp) and it worked. I noticed upgraded dancer-ircd couldn't communicate with older version of it. (claimed failure to match password). So, I upgrade all of our servers in our IRC network.

minori.linux.or.jp (JLA backend server) and norika.linux.or.jp: I upgraded them to sarge. Except dancer-ircd problem as above, the problem (reported by takei) was that customized cvs pserver line that I put in inetd.conf was disabled by cvs package. I re-enabled it.

These works took 5 hours, and still mirroring.

I've noticed that ftp2.jp.debian.org didn't have debian-archive. Maybe, I forgot to mirror debian-archive when the filesystem for mirrors was regenerated.

I start mirroring debian-archive from hanzubon.jp that would be nearest debian-archive site.