Yet more reasons I hate control panels.
I had to move a site last night from one server, hades to another server, marion, and both servers are running Insipid, which has a backup and restore feature. So the plan was to backup the site from hades, then restore it on marion.
It took several attempts on my part to get this process to work. One has to realize that when Insipid says the operation was “sucessful” all that was “sucessful” was your request for the operation—the operation itself is a separate process that will notify you of the actual sucess (or failure thereof) via email.
Hate hate hate.
Now, one aspect of this site is that it has its own IP (Internet Protocol) address. And lo', once I got the site on marion, marion was listening in on the IP address.
Good.
But hades was still listening in on the site as well.
Bad.
Not wanting to actually remove the site from hades until I know this is working, I then decided to manually remove the IP address (not knowing how one even approaches this using Insipid).
>
```
ip addr del XXXXXXXXXXXXXX/24 dev eth0
```
Try to view the site, and the request is going to hades.
Okay, the switch it's on is probably still sending traffic to hades. Clear the ARP (Address Resolution Protocol) cache on the switch.
Try to view the site, and the request is going to hades.
Check hades and see that it really wants to hold onto that IP address. Use both ip and ifconfig to nuke the IP from hades.
Try to view the site, and the request is going to hades.
Clear the ARP cache on the switch.
Try to view the site, and the request is going to hades.
Okay, shut down the port that hades is plugged into on the switch, clear the ARP cache.
Try to view the site, and the request is now going to marion.
Good.
Now, that was late last night (between 3:00 and 4:00 am during which I stupidly answered the phone and took a tech support call around 3:30 am dealing with an email issue—sigh).
This morning, requests for said site are now going to hades.
Hate hate hate hate hate.
Okay, [DELETED-nuke-DELETED] delete the site from hades, make sure it doesn't have the IP address, shutdown the port it's plugged into on the switch, clear the ARP cache on the switch and okay, requests are now going to marion.
I even double check to make absolutely sure that no other sites are on this IP address. There aren't.
Hate hate hate hate hate.
I know why I hate control panels—I don't feel in control. And when something breaks, I have no idea how to fix it. Oh, I typically know what's wrong, and how one could fix it, if one weren't running a control panel. How to fix it within the context of the control panel? That, I don't know (oh, I suppose one could dive into the internals of the control panel but a) that kind of defeats the purpose of a control panel, which supposedly makes Unix administration easy and b) we use three or four different control panels, which all work differently, which means we need to become experts in using all these control panels which again, kind of defeats the purpose of a control panel. Either that, or I'm bitter that all my experience in administrating a Linux system is no longer applicable and that I have to relearn all this crap four new times, just to administrate a Linux system).
Hate hate hate hate hate.
Subject: No subject
Posted-by: Sean (Staff)
Date: 06-14-2006 3:25am EDT
> Moved site to marion. It”s disabled on hades and hopefully, hades won't try to reassert the IP address.
“Response to trouble ticket last night after moving the site”
I fully expect that in a few hours, I'll have to revisit this situation again.
hades took control again. This time, we found a process, ntpd (Network Time Protocol—which keeps the clocks on all the servers in sync) had explicitely bound to each IP address on hades, as well as Apache [1] apparently still configured for the site in question (then what the XXXXXXX XXXX good is Insipid if it doesn't restart Apache?).
Okay, maybe now hades won't take over the address.
It's happened twice since the last update. Short of rebooting hades the next time it happens, I can't think of what else might be causing it to respond to an IP address it's no longer programmed to respond to.
Found the script buried in /etc that had the IP address and nuked it. That seems to have taken care of the problem.
For now.