The Vintage Hardware Stack: a new life for old hardware

Remember my Shameless call for hardware, some time ago? In that post I explained in short my idea of a pile of old hardware that could serve as an unique playground for young administrators. Well, I’ve made some big progress with that. Last june, my employer thought it was a good idea to put an interview online in which I explain the Vintage Hardware Stack initiative. Good read, if you’re good at Dutch. But if you’re not, I’ve taken the liberty to translate the Dutch post so you can all read it. A tad long, but nevertheless still quite a project. Have fun reading!

Thanks to OGD that I could translate the post and place it on my own blog. Oh, and we’re always open for donations!

THE VINTAGE HARDWARE STACK: A NEW PURPOSE FOR OLD HARDWARE

The VHS, also known as the Vintage Hardware Stack, is a playground for young engineers in which they can play, test and learn whatever they want. Everything in a safe and controlled environment, not connected to any customer infrastructure whatsoever. Very valuable for for OGD and its customers indeed.

The VHS idea was developed by lead engineer Robbert Erents, back in 2012. The concept is to collect old hardware for educational purposes; He offers customers to pick up their decommissioned equipment in exchange for a delicious pie. Or two.

Within two years, Robbert collected a fairly decent stock of servers, switches, server racks and storage and placed it in the OGD warehouse. More and more engineers ask him for airtime in The Stack, or advice how to address specific storage or virtualization related issues. We asked him a couple of questions about it.

Things are running quite smoothly with the VHS ‘ey?

“Absolutely! We receive donations quite regularly, management gave us our own room in the new OGD building, more and more engineers participate and the number of success stories continues to grow. It’s even more special when you realize that it all started out with just a couple of old servers, some years ago at my home. Hooking them up, installing an OS, creating a domain, firing up a DHCP server and a file share. Even then I saw way more possibilities with that old stuff. How about gathering old devices that don’t perform that well, but still have the same set of possibilities as newer equiptment? You could tear them apart, rebuild everything and do it all over again! Unfortunately, I didn’t have the budget, time and vision at that time to develop the whole concept.”

And then, OGD ict-diensten organized Technival (Technival, a technical summer event for all amployees of OGD, content in Dutch) for the first time. Robbert and a couple of colleagues came up with the concept of a “Virarium” as an activity.

A Virarium?

“Yes, a Virarium. One has an aquarium with fish, others have lizards in a terrarium tank. And we wanted a Virarium. We wanted to place a number of virtual machines in a highly isolated virtual environment and infect them with viruses. Then design a highly interactive dashboard on which the spread of the viruses is visualized. On this dashboard we could monitor the health and infections of the VMs. Why? Just because we can.”

How did you start that?

“A decent Virarium need hardware to run on. Eventually I found a fullsize rack with a HP blade center, an EVA SAN and some network switches. I picked the stuff up with a small truck and placed it in the OGD warehouse. And then I got the momentum, I had a really nice set of devices to start with. Not just for Virarium, but for a whole new initiative.”

To start with what: the VHS and the Hardware Wednesdays?

“Yes, exactly! With these devices, I got a place of my own in the warehouse. Separate power lines, storage space and uplinks. Finally I could do something with it. The devices present in the VHS are used and often not really high-performance, like the blade center mentioned above. But, they offer most (if not all) of the functionalities and capabilities used in newer stuff. This makes them the ideal toys for our most famous OGD Tech-playground: The Hardware Wednesdays.”

Yep, I saw the Stack. But, now there’s way more than just one rack?

“True, true. We’ve moved on! We are collecting old customer hardware at a regular basis. Thanks to these donations the VHS stock stays at an acceptable level. Insert new stuff, get rid of really old devices. Yes, we sometimes dispose old hardware too. We’re not hoarders. And, when incorporating new equipment and throwing away the old, all data carriers and configurations are wiped. No customer data will be present on VHS devices. Primarily because it’s part of the agreement with the donating customer, but the data is in our way too. We have complete runbooks and automated processes to wipe any device clean.”

What do you do on those Hardware Wednesdays, exactly?

“The whole idea of a Hardware Wednesday is click and play with devices present in the VHS. So everybody grabs his or her favorite stuff, plugs it in and starts to configure it according to their ideas. I always have some maintenance jobs ready, but own initiative is way more important. They can recreate customer issues, test designs or just have a go at a basic implementation. The whole concept is “Get Experienced”. Young engineers have an opportunity to have a go at (components of) an infrastructure. Or how to break it. Pull plugs, cripple a configuration and see if and how the infrastructure will be impacted. Learn while doing without impacting customers, it doesn’t get any better than that.”

Sounds like fun!

“Oh, it really is! But it’s not just the techside that’s important. Young enigineers also learn to regard and execute their work as a project: plan, create deliverables like documentation and explain what they are doing, and implement your plan. And they do a lot! Even cleaning up their own mess like keyboards, power cords, chairs and empty beer bottles afterwards. Oh, and don’t forget the audio set that’s used for good music. We like to keep the place clean.”

Is the VHS just used for these Hardware Wednesdays?

“Oh no, the Stack has way more use cases than just a Hardware Wednesday. It can be used for Proof of Concepts, for example. Or think of an engineer that wants to test his design, or implement a new technology that doesn’t master yet. Then he can use the Stack to test the product and his design around it. Or redundancy tests: pull a powerplug and see if your environment still operates as designed. If not, then review your design and start over. No harm done. Or preconfig a greenfield setup to transport it to the customer site later on? Go ahead.”

And what do your customers think about all this?

“They’re extremely happy with this. They are aware of the enormous advantages it has, not just a pile of useless old hardware. A great example that I like to tell is of one of a valuable customer that runs his production environment at the same hardware we have in the Stack. That customer wanted to perform a firmware upgrade on those components, but was a bit hesitant. What will the outage be, is there a rollback scenario, what expertise is needed? With the help of the VHS, 4 engineers armed with USB sticks performed an upgrade on our equipment and documented every step. Within two days they had a complete runbook with impact analysis, actions, screenshots and management summaries to convince their manager that the upgrade was assigned as low risk as could possibly be prepared. Now that’s great.

What do you think the future of the Vintage Hardware Stack looks like?

There is still a lot to be done. Register everything in the Stack, develop pods of devices for specific use cases, create manuals to reste devices to factory default etc. We got our own space in our new building, so we can create a semi-permanent setup. Until the next donation ..