Next hurdle, how do I manage the infrastructure and categorise it
- Core – always on, candidate for reserved instances
- DevInfra – Development Infrastructure – almost always on, 20 hours a day minimum.
- TestInfra – Testing Infrastructure, on for about 16 hours a day
- DemandOnly – Demand instances only, manual startup, always shut down every day if running
Firstly I did not want to have to worry about checking startup, shutdown, backups, snapshots etc… all day, I needed a way to set up a machine with the right software that enables me to schedule automation jobs, keep a history, work with Windows Operating Systems and the AWS .NET SDK, oh yeah and I didn’t want to pay for it either.
Because it’s all built on Windows. If you’re not using PowerShell to build up and configure your Windows machines you’re doing it wrong.
I know the product well (always good to stick with known, knowns) and it really is a great tool with good online support. It enables scheduling of jobs that can run virtually anything, it can build software, chain jobs together in pipelines and it has a ton of plugins too. Sure it’s a Java tool but only an idiot would assume you can only look for answers in the Microsoft world.
After a month of solid scripting and testing I had created enough PowerShell scripts and functions that enabled me to do the following with the instances in my VPC all controlled through Jenkins using the metadata tags
- Startup and Shutdown of instances
- Core on all the time, DevInfra on 20 hours per day, TestInfra on 16 hours a day
- Snapshots – Core and DevInfra snapshots are created every day
- S3 Database Backups – All my database full backups that ran every night were copied to S3
- Redundancy – New snapshots created were also copied over to the US West Region every night
- Environment rebuilds – Cloud Formation scripts ran every night to rebuild the test environments so we had a totally clean machine to deploy to daily
- AWS Cleanup - I created jobs to clean up S3 and instance snapshots once they got older than a couple of weeks
The best part about this solution was that if we added new instances we just tagged them appropriately and then all the maintenance of the startup, shutdown, snapshotting took care of itself. Matter of fact I stopped looking at it after a couple of months as it all ran like clockwork.
Using AWS for your dev and testing is an absolute win over the more traditional methods (physical servers - arrgh!, VMWare - Hosts forever running out of capacity).
Sure it will take time and investment in your DevOps staff to plan for and use it appropriately (show me a new infrastructure technology that doesn't need it), but the payoff in being able to completely automate your environments, increase/decrease resources as you need, scale up instances (such as your build servers when they are starting to run hard) and the lower TCO is impossible to ignore. Best of all once you have done it once, and done it properly, you can reuse a lot of what you created for other projects and clients.