Building redundancy into DB architecture on Amazon

[carmstrong]

Using RightScale, we have a master/slave database architecture. However, on more than one occasion, our master DB has gone down. Amazon just seems to shrug and say "the underlying host became unresponsive". We have to manually promote the slave to master, launch a new slave, and let it synch. We then kill the old server. For the duration of this process, our site is unavailable.

That sort of thing really isn't acceptable to us... is there an alternative to maximize redundancy? How do other people deal with this sort of thing?

[Chris Fordham - RightScale]

Check and ensure you are using the latest ServerTemplate and MCI.
If using for example a CentOS 5.4 MCI, this is aged, discontinued by CentOS and also doesn't use a CentOS kernel (known issues).

[carmstrong]

Thanks Chris. We are indeed using a CentOS 5.4 MCI, as was built by our RightScale integration engineer last summer. Should we swap this image out for a CentOS 5.6 on our own, or should we contact RightScale support? I'd imagine there could be issues in doing this without significant testing.

Additionally, is there a mailing list we can subscribe to so we can receive important updates like this?

[Chris Fordham - RightScale]

Originally Posted by carmstrong

Thanks Chris. We are indeed using a CentOS 5.4 MCI, as was built by our RightScale integration engineer last summer. Should we swap this image out for a CentOS 5.6 on our own, or should we contact RightScale support? I'd imagine there could be issues in doing this without significant testing.

This is a significant change, utilizing your consultative support in your support plan is recommended. In any case, dev/test in a separate deployment with the new configuration is a good practice. Depending on the ServerTemplates you are using, upgrading those and in turn the images may be just as easy.

Originally Posted by carmstrong

Additionally, is there a mailing list we can subscribe to so we can receive important updates like this?

There isn't yet, though we'd love to hear your feedback for one on http://feedback.rightscale.com/

[carmstrong]

Chris, our database servers are using this image:

RightImage_CentOS_5.4_x64_v5.6 - 11H1

Can you confirm that this image is affected?

[Chris Fordham - RightScale]

Originally Posted by carmstrong

Chris, our database servers are using this image:

RightImage_CentOS_5.4_x64_v5.6 - 11H1

Can you confirm that this image is affected?

Yes, any CentOS 5.4 image (or image that uses an Amazon/fc8 AKI).