In this article, we'd like to talk about how we at TestingBot make sure that new code that goes into production is tested thoroughly.
As a cloud testing company, we deal with testing on a daily basis. For us to provide a good service, it's essential that we use the service ourselves.
For this reason, everything at TestingBot is being tested by TestingBot itself (*inception*).
For our website we've got a batch of Selenium tests which test various functionality on several browsers and platforms.
Here's what our tests are verifying:
- Does the page render correctly on each browser?
- Do the buttons work, do they trigger the correct action?
- Does the pricing page work, does upgrading your subscription function correctly?
- Can we create a new account, can we log in?
- Does the member dashboard work, can we see and edit tests
- and a lot more test cases...
We've configured our test runner to be able to run on several environments via a configuration setting:
- local: tests run on our own computer via TestingBot Tunnel
- staging: tests run on our own staging environment
- production: tests run on the live environment (TestingBot.com)
For each environment, we use a separate database, to make sure we do not mix data between environments.
Running tests against code that's about to go into production.
Repository Setup
We make sure that our production webserver syncs code from our master branch.
We've also toggled a setting in our repository that we can only push new code in our development branch, not in our master branch.
The only one with a access to push to master is our Jenkins CI.
We've also added a GIT hook to our repository that will trigger a Jenkins CI Job as soon as we push new code into the repository.
This Jenkins CI job does the following:
- Sync the development branch on our staging server, so that the staging webserver is serving the latest code, including the code we just pushed.
-
Run a lint checker on the new code, to make sure we don't introduce syntax errors in our code.
The reason we do this first is because it's very fast to do this, and because it doesn't make sense to first run longer tests when the code has been broken anyway. - Run all tests against this staging webserver. Since we use a separate database, the production data is not affected.
- As soon as one of the steps above fail, the Job fails. If all steps succeed, we continue to the last step: Jenkins will merge the newly pushed code into the master branch.
When we deploy now, the new code will be included in production.
The process outlined above makes sure that we always know that new changes that go into production pass all tests.
Deploying to production
Once Jenkins informs us that all tests pass (we use a Slack hook), we can deploy the new code.
Once the code is deployed, we will run the same tests again, this time we set the environment variable to production.
If all goes well, these tests should all pass, since they already passed in our staging environment.
Improvements
Since we're an Automated Testing company, we'd like to improve this process in the future.
When the staging tests pass, we can configure our Jenkins to automatically deploy the new code from the master branch. At this point, we've basically reached the Continuous Deployment level.
How TestingBot can help you
Interested in using this approach for your product? At TestingBot, we provide over 4600 browser combinations to run tests against.
Once you've created some Selenium tests and connected the tests with our grid, you can start testing your code on various platforms, browsers and devices.
