Server dying at inconvenient times ..

Posted by: lordvan 4 years, 10 months ago

(0 comments)

So our older server - which still had a few network shares on - decided to die .. on Saturday evening around 22:00 .. yay .. -- did I mention that server was also still the internet gateway for .. well everything? -- + DHCP server was on that (and DNS, but that is mirrored anyway so whatever)

Fortunately I had another small server already lined up and (mostly) prepared to take over the routing - and while I was at it i made it the DHCP server too for now.

So apart from the fact that the silly router we got from our ISP likes to not accept a different MAC address with the same IP as before immediately I had to wait a bit for that to work, but then just set up DHCP and the internet was up & running again .

Now about figuring out what is wrong with the server .. switiching it back on the bios gives cirtical warnings about voltages .. wonderful.. I managed to get it to boot to a root shell once more - which allowed me to copy /etc (which included the up2date DHCP config so yay!)

Since it seems to be either the power supply or maybe the - by now at least 15 years old - mainboard I decided *not* to dwell on that much longer that evening, and after internet and a few other basics were back online I called it a night.

Next morning 07:30 I'm back in the server room, opening up the server, but not seeing any easy way to use a normal (non-server) power supply to easily test things out.. after some consideration plan B: swap (some of) the hard disks over to the new server (which fortunately has 3 HDDs in, but has 8 slots as I went for the bigger one -- which was definitely good considering now!.

So i eject the old hdd from the hot-swap and try to remove the HDD .. to find out I have to go into the workshop because some joker decided to use Torx screws for that and not the usual cross/phillips .. after that out of the way the 2 hdds were in the new server and found.. but INACTIVE ARRAY was shown .. took me a while to figure out how to deal with it -- I purposely did only put one of each of the mirrored disks in the new server as a precaution.

So anyway the steps to get them working:

1. run mdadm -E --scan
2. add the arrays to /etc/mdadm.conf
3. then - because the array is incomplete need to stop, then re-assemble (with --run) like this: mdadm --stop /dev/md121 ;  mdadm --assemble --run /dev/md121 /dev/sdd2

After that it started up again and in mdstat the inactive changed to active:

md121 : active raid1 sdd2[1]
      505920 blocks [2/1] [_U]

Fortunately the LVM on that server had different names than on the newer server so no conflicts there, so I could just mount the devices.

But instead of mounting the Data shares on that server I directly passed it with virtio to the KVM/QEMU instance with my Univention Coroporate Server fileserver - as described here in my old blog post - just add a 2nd -disk and reboot it: Running UCS (Univention Corporate Server) Core on Gentoo with kvm + using an LVM volume

• ← Google+ going away today - Japanese Learners Community,..
• UCS (Univention Corporate Server) Nagios NRPE Command ... not defined →