** This post is a small diversion from the main subject of this blog.
I am occasionally publishing technology posts related to my profession. **
In this series of posts I would like to share my experience from a process of selecting technology to use as part of my organization’s devops process.
My organization is a Microsoft based development house.
We use TFS 2015 as our source control and CI platform and we are starting to use the release management feature as well.
In order to achieve a complete process we were looking into tests automation framework for sanity tests of our websites and web services after their automatic deployment.
I reviewed these technologies:
- Powershell script
- web performance tests (webtest files which are part of web and load performance projects in visual studio)
Before I begin reviewing the options I would like to share my opinion on unit tests.
Unit tests should have been a natural answer to this sort of quest I was in, but I think of it more like an holy grail.
To my opinion, quality unit tests require experience and knowledge of relevant methodologies.
Without knowing how to write good unit tests, the basic measurements will be code coverage and time invested in writing and maintenance.
The task of training, building knowledge and eventually manage and monitor the quality were not worth the price
Since we have a deployed environment that should be working, the system can be tested at a much higher level than unit, sub system or system tests
Powershell provides an ability to test web services that expose soap methods so I will be able to not only verify sanity of the service, I will also be able to test a process flow and verify with assertions that it is valid.
The simplicity had me going fast forward – I didn’t have to build special projects and take care of compilation and build tasks.
Powershell has an out of the box support as a step in the build or release processes in TFS so I managed to integrate it in our flows quickly.
In terms of reporting it sent errors to the build and release reports and was able to stop them in case of errors.
Another aspect is usage.
Writing powershell script is pretty similar to writing C# code and visual studio provides an IDE to write powershell scripts.
So in terms of methodology developers were able to write the tests as part of their development tasks and apply them to the CI/ CD processes by simply adding the files to a folder in the source control which later on will be scanned during build and all contents will be executed.
I had issues related to the way services were exposed – only soap messages are supported, so services exposed using net.pipe or net.tcp were not discovered and could not be tested.
Fortunately for us, since we are using an ESB layer all services were exposed correctly and were available for tests.
Another issue is related to testing our websites which is the usual client layer.
I didn’t get to test this option since I moved on to the next candidate but, to my opinion, if web services were exposed using a nice proxy object, calling the web site required hard coding of all post and get strings sent to the server.
I don’t know of any ability to record or create a script to call the web site automatically so testing web sites could be a pretty daunting task to achieve.
To quickly summarize the powershell option:
- Quickly connect to simple and complex web services
- Easy adoption by .net developers
- Easy integration to TFS build and release processes
- Only testing of web services layer
- No support for protocols other than SOAP
- Requires writing of code (coding will be irrelevant with the other candidates)
In the coming weeks I will add my summary of the other 2 options.