This post is a part of my degree project “How can we adjust to Working Agile with Cloud-based Microservices in CI/CD” written during the last course of a software test engineer education. All information from the thesis is not included and some text is changed. The original document is available as a pdf Link to pdf here This is part 2 of 5. First part can be found here: Link to first part
In this post I will go thru some methodology how and why I collected the data presented in these project posts. It felt needed to add this to clarify why some topics are mentioned and why some subjects might seem a bit selective.
Data Collection
To find the best approach for what to include in this research and which areas that needed to be a part of the analysis to make it reliable, I first contacted several different Test Engineers within different fields of testing. I wanted to form a better understanding in if they come across this topic and how they experience it affecting their work. It was also to get a feel of the level of knowledge if there was a high or low level of knowledge. These questions and conversations were only used to get a better understand of which tests that are focused on and to use that in my decision how to select and limit the research topics. I found it important to have this information from the perspective of test engineers, even more so with my background as a system developer. To not fall into the track of development and coding and down-prioritize the importance of the testing role. The answers were collected and used as a guiding point and not as a part of the actual analysis.
Different methods were used to collect data for the final analysis:
Discussions and questions
After the first contact with the test engineers, I decided to continue the same approach in the analysis. For importance of empiric data and to reliability in the analysis the discussions was made with the questions. The discussions were as mentioned above made in written form for the reason of time zone differences and time limit of the project. But this was a
conscious choice to get an international perspective and see if the work strategies could vary depending upon location. The topic varies in the conversations depending upon the interviewed persons background and experience. For this reason, I found it important to let the person answer from their perspective but to use a semi-structured method in the follow-up questions. To keep this method in all conversations made it possible to keep the data comparable and dependable. I used a professional method that I have experience in from former education and used for several of years in earlier professions.
Online Research
After the first discussions and my experience in system development I researched the different areas of cloud-based, microservices, test planning and test strategies. At first, I made an activity diagram of an application, to have a visual overview of what parts we have included in an enterprise sized application. This way I could go thru the different sections and research the common test approaches done within that area. This gave me a more clear and specific way to narrow down the test strategy. Here it was of importance to use reliable sources, I tried to avoid personal blogposts and information, to keep the data fact-based and not opinion-based. Some physical books were also used, these was selected for the reason of their focus on specific topics within my research area.
Experiential Testing
To form a better understanding of best practices and to see the reach of different testing methods I built a small application, the application itself is not functional only built to get the tests to run. This was only made for the purpose of trying the methods and in some cases see the impact in performance. This does not include larger testing as end-to-end testing, since I found that it would be too time-consuming to develop the size of application to make the result relevant. During the time of the research, I also had access to different software and services that I analyzed and, in some cases, tried different test scenarios. This also gave me the possibility to access their documents and other educational material.
Ethics while research
The first conversations and questions asked on this topic was asked without this research in mind. The discussion started with my own interest in this topic, the questions asked was out of curiosity what other opinions was on the topic. It was this that lead up to the decision of this degree project, to deepen my own knowledge. For that reason, the people I contacted was not aware of the later purpose of our earlier conversations. All people involved was contacted again and asked permission to use their data in this research. And the purpose of the follow up conversations was clearly explained and once again asked permission to be used in the study. It was significant important that the participants felt comfortable to discuss freely therefore all was given the option to be anonymous and to specify if there were any specifics that they didn´t want me to use. Only a few wanted to be anonymous, and most was positive to share their names and profession. When starting to analyze the data I made the decision to remove all names and only where it could be relevant for clarification mention their area of profession. This conclusion was made upon both respect of privacy and readability. Also as mentioned above, the information evaluation of the research material was a high priority.
Analysis method
The get an overview of the data from the discussions I followed Boyatzis explanation of structured thematic analysis (Boyatzis, 2009). First, I used the research questions to identify various categories, these categories were later studied from the perspective of the main questions. This thematic analysis method gave themes with relevant and reliable data that could be used for further analytics. It was taken into consideration that the method had weak parts and can be viewed as it has been directed towards certain results based upon the interviewed and interviewers’ subjective perspectives. This made it of significant importance to use the data based upon the theme of the questions. Also, the further research of the themes identified was made with careful consideration of information evaluation and keep it neutrally relevant to the themes.
Interviews with Test Engineers
For the interviews there was 14 different people answering the questions. The responses varied in both in topic direction and length. As mentioned in the Ethics while research section some conversations was started on the topic before the research started, some of the subjects discussed is outside of this research and will therefore not be a part of the results.
Experiential Research
Here various categories within testing are researched, some categories were researched only with document and fact analysis others included hands-on testing. The methods and software used was selected upon varied reasons that will be clarified in their sections that follows. Not all experiential research will be presented in detail or with visual content.
In the next post we will see all themes identified based upon the answers from respondents
Links to previous posts in this series can be found here:
Link to part 1
Link to the next post can be found here:
Link to part 3