{"id":176,"date":"2012-01-30T18:18:39","date_gmt":"2012-01-30T17:18:39","guid":{"rendered":"http:\/\/www.simweb.ch\/blog\/?p=176"},"modified":"2012-02-19T15:10:29","modified_gmt":"2012-02-19T14:10:29","slug":"taking-gear-out-off-service","status":"publish","type":"post","link":"https:\/\/www.simweb.ch\/blog\/2012\/01\/taking-gear-out-off-service\/","title":{"rendered":"Taking gear out off service"},"content":{"rendered":"

This is a story about good practise and \"don'ts\" in sysadmin work,\u00a0 inspired by Bryan's (@bdha<\/a>) blog entry about the\u00a0 first law of systems administration<\/a>. Recently I\u00a0 got some used gear for lab\u00a0 - the story of this gear, a set of FC switches - remembered me of a lesson about what can go wrong and why I was once told to do things a certain way when administering systems in production. - I was also bitten by similar but never that hurting experiences thouch<\/p>\n

Expect the unexpected (say hello to Murphy)
\n<\/strong><\/p>\n

Once in the ol' days my current boss (and former sysadmin) told us: \"If you migrate a service and power off the old server or whatever important gear: \"Do NOT disassemble, throw\/give away the old thing too soon. One never knows if you forgot a tiny detail and you might be happy to have the old box back running in short period.\"<\/em><\/p>\n

Today we have virtualization, conf management like and revision control who can help you track things for config changes, migrations. Nonetheless it can be very dangerous.<\/p>\n

What he also told me was: \"Plan for the unexpected, you really never know what's going to happen in a migration if the system is complex. Also plan your time and resources and don't squeeze too much into that time window.\"<\/em> (you might need a little bit of sleep though?)<\/a><\/p>\n

Reality...<\/strong><\/p>\n

Now came day X where the customer (me) wanted to get his used gear. I was kindly asked fo an additional week due to their migration, fine. When I finally went there I had to realize that they were actually in the process of migrating the SAN traffic to the remaining fabric switches. And guess what? - Boom that's when the \"unexpected situation\"\u00a0 happened even with highly available virtualization and clustering and a multipathed, multi-controller FC SAN. I had to come a back in a couple of hours for the time they fixed their production environment. I was only grumpy becase I had to wait additional time, but they were quite exhausted after this exercise...<\/p>\n

Conclusion<\/strong><\/p>\n

Doing massive changes on your critical production environment\u00a0 without any time reserve is not sane. I may sometimes try to rush into changes but not this way. Now I have had the chance to experience such a situation as an outsider. Lots of Sysadmin wisdom gets outdated quickly, but some doesn't. - This rule seems to be part of this wisdom. Stick to it, you not only hurt yourself by not following it.<\/p>\n","protected":false},"excerpt":{"rendered":"

This is a story about good practise and “don’ts” in sysadmin work,\u00a0 inspired by Bryan’s (@bdha) blog entry about the\u00a0 first law of systems administration. Recently I\u00a0 got some used gear for lab\u00a0 – the story of this gear, a set of FC switches – remembered me of a lesson about what can go wrong … more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_crdt_document":"","footnotes":""},"categories":[1],"tags":[],"class_list":["post-176","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/www.simweb.ch\/blog\/wp-json\/wp\/v2\/posts\/176","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.simweb.ch\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.simweb.ch\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.simweb.ch\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.simweb.ch\/blog\/wp-json\/wp\/v2\/comments?post=176"}],"version-history":[{"count":0,"href":"https:\/\/www.simweb.ch\/blog\/wp-json\/wp\/v2\/posts\/176\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.simweb.ch\/blog\/wp-json\/wp\/v2\/media?parent=176"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.simweb.ch\/blog\/wp-json\/wp\/v2\/categories?post=176"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.simweb.ch\/blog\/wp-json\/wp\/v2\/tags?post=176"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}