Ever since we started TestingBot (almost 2 years ago!) we've been running TestingBot on Amazon AWS (EC2 + S3 + other services).
These last few months however, we've been moving everything from Amazon to our own private cloud.
Originally Amazon AWS seemed like a good fit for us: easy to setup, manage and maintain. We'd scale up and down, depending on the number of tests our customers were running. As it turned out, AWS has its disadvantages:
- Noisy neighbours: sometimes instances would behave much slower than usual, because other people on the same hypervisor were using all the hypervisor's resources.
- Expensive: AWS is expensive, as soon as we start an instance, we're billed for the entire hour, even if we only need to run a 2 minute test on it.
In July we started looking into running our own private cloud on a bunch of dedicated servers. Originally we planned to use VMWare's vSphere and vCenter, but after testing VMWare for 2 weeks we concluded it would not satisfy our needs:
- Expensive: complicated/expensive licensing + expensive support
- Black box: whenever something went wrong it was hard to troubleshoot since we can't look at the code. VMWare does have good documentation though.
- Complicated API: we needed an API that would help us automate launching/destroying VMs. VMWare's APIs are complicated to test and use.
After we ditched VMWare, we decided to look into an open-source solution: KVM + QEMU.
This turned out to be an instant success: easy to install, setup and use. Together with libvirt we quickly had a proof of concept system where we could easily launch and destroy virtual machines.
Now we're running our own cloud; as soon as a customer wants to run a test we spin up a VM in less than 10 seconds, run the test and destroy the VM after the test has finished. This way we guarantee pristine VMs, fast tests and a secure environment.
Together with these changes, we've changed some more things on TestingBot:
- Updated our OSX VMs to OSX Mavericks
- Added IE11 to our grid
- Created a "prerun" capability to download and run any program you specify before running your test so you can customise the VM to your liking.
