Part 1 of 2
This text includes different areas of non-functional testing and a specific video service product is used as a focus object. I won´t go in to details about the research done on this service but I have gone thru their documentation both from developer backlogs, technical manuals and policies to gain a better understanding of the product. You can read more about the video service here: Link to video service
This first part includes the requirements:
- Explain the differences between functional and non-functional testing. Why are both needed to be done on a product like video service.
- Explain performance and stress testing. Explain why they need to be used at the video service.
- Explain how monitoring can be done when we do Performance and Stress tests. Explain why it is needed for the video service.
- What is security testing. Explain why it is needed for the video service
- Which are the most common tools to use for non-functional testing. When we test the video service, which could be correct to use, and why.
Differences between functional and non-functional testing
Explain the differences between functional and non-functional testing. Why are both needed to be done on a product like video service.
Differences
Simple explained we could say that functional testing is what our product do, the actions and non-functional testing is how it does it, the performance.
Functional testing is to check that our video service is up running, functions like log in form, buttons, menus, video player pass our requirements. We look at what works or not, is the UI looking correct, can we click thru everything. Focus is not about performance. It meets the business requirements, test cases pass, functional testing is done.
Some of the testing types we use for functional testing:
- Globalization
- Integration
- Interoperability
- Unit
- User Acceptance
- Regression
- Smoke
Non-functional testing is about behavior, how the product is performing. When we know that the functional tests have been done, we start analyzing the product. Is the log in form usable, does it meet accessibility expectations, does it load within expected time. In the functional test we cleared that the log in form looks correct now we check so that tab order behaves correct, that a screen reader can be used for the functions. Is the performance of video service meeting requirements at any level of users logged in at the same time. Is the user’s data and privacy protected.
Some of the testing types we use for non-functional testing:
- Compliance
- Disaster Recovering
- Load
- Performance
- Portability
- Scalability
- Stress
- Usability
Why is both needed
Functional testing is needed on the video service to check that it meets all business requirements. The video service has different areas of functions and to make sure all areas of the product connect and behaves like said we need to evaluate the functions. An example is the Digital meeting service:
This service is available on both web and mobile.
We have the log in form for users which is connected to an electronic identification service. For clients there is two identification options, for the meeting organizer there is three.
When logged in the client should come to a waiting area before they are let into the meeting. This feature is optional, the meeting organizer makes that selection before the meeting. The meeting displays both a video, sound, and a chat function.
In the topic of functional testing we need to validate the video service actions.
First is the log in, the forms UI needs to be tested, that the form fields are linked to correct functions, buttons can be clicked. That it connects with third party integration. Does an authorization and authentication check to give correct system access. If the meeting organizer has selected the waiting room option, this is another are to check the functions. And on the meeting organizers side the feature of letting the client in to the meeting is another function to be tested.
When the users have access to the video meeting, the video, sound, and chat functions needs to meet the requirements. UI might be different depending upon device used, if we have different code base for these UI´s functional testing needs to be done on both.
When these functional tests are done and the test cases pass, we still don´t know if the video service meets expectations.
Let’s go back to the first step again, login form. Non-functional test areas here could be Usability, can we fill in the form with a keyboard only, does the screen reader work correctly. Do we have more than one e-identification options for the users who needs a simplified version.
Security, is the link between login and third-party integration safe.
What will happen if a high amount of people is using the feature at the same time.
The waiting room area performance is another area, we know it works after testing the functions, but let´s assume the expectation for it is to shift to the video meeting within 2 seconds after the meeting organizer clicks to let the client enter. Here we need to test the performance.
In the video meeting non-functional tests needs to include performance for video streaming, network compatibility (like what happens if the internet connection is slow), is the sound and chat secure. Is the information storage protected.
What is Performance and Stress testing
Explain performance and stress testing. Explain why they need to be used at the video service.
Performance testing is a testing process where we evaluate the response time on specific product elements or/and the overall system. How the system performs it´s tasks, including response time, Load time, error rate, CPU utilization, latency, data query execution and so on.
To see the system behavior when it is under a heavy workload, we use Stress testing. Stress testing gives the data of system limits, behavior when reaching limits. This can include: many users at the same time, traffic is redirected from another service, like one server goes down and traffic suddenly must go thru another and double the workload.
For the video services product multi-party call performance where the users can connect to the service using different third-party video call services. With many users, video, different platforms connected, and similar features offered in this service it is many areas performance testing needs to cover. The login time for each user, network latency, server request, user volumes. All of it needs be verified to meet the expectations of, speed, robustness, reliability, scalability.
Stress tests is needed to evaluate the behavior of the system facing user peak activity. If there are more users than normal in one multi-party call and the system as whole, if there is a peak of calls going at the same time. This means more stress on both the login feature and on functions included in the call service. The test should give us the answer what happens when it goes over the peak and fails and how easily it recovers.
Monitoring Performance and Stress tests
Explain how monitoring can be done when we do Performance and Stress tests. Explain why it is needed for the video service.
When the product is tested and released into production it still needs to be observed, without interference. Monitoring is done to ensure that the software is performing as expected and if there is any spontaneous behavior in the system. The process includes to identify, measure, and evaluate the performance to discover and resolve issues.
The stress tests have given us the metrics of what the system can manage and how gracefully it recovers from like user peak activity, CPU levels during this event. Even when we have this data, we need to monitor the system when a similar scenario appears. Here we can use a monitor tool that gives us an alert if the CPU level goes above expected level when there is a certain number of users active.
Video service offers a video infrastructure service, this service is for organizations within specialized healthcare. They can all connect with other users in the network, it includes a personalized phone number, third-party video gatekeeper. It can also be used for international connections. With this system it means that lots of various products needs to be connected, the user activity is unpredictable. To watch the system performance on all levels is important, both to have data of user activity, connection performance, availability and all other features. An example of reason to monitor stress tests could be similar to the ongoing pandemic, many healthcare organizations and hospitals have limited resources, they need to connect with many other specialists to organize surgeries, treatments and so on. This could mean a user activity peak well over what earlier been evaluated in the stress, spike tests. To monitor the system and have logged metrics gives us data to analyze, see if there are areas not reaching expectations.
Security Testing
What is security testing. Explain why it is needed for the video service
Security testing is about evaluate if there is any weaknesses, loopholes, or risks of breaches in the system. When doing security testing we look at the requirements, system infrastructure and frameworks. There are several types of security testing, some examples:
- Ethical hacking
- Penetration testing
- Posture assessment
- Risk assessment
- Security auditing
- Security scanning
One technique that is used to analyze the areas and develop test strategies for security risks is Threat modeling. Here the first step is to understand how the system works, including functions, connectivity, processes. Secondly the first system analyze is defined in sections and ranked according to importance. Then all sections are explored for weaknesses and threats. First here the mitigation strategies are created for each of the threats.
The video service is used for healthcare organizations, there is an expectation of confidentiality, high security, and protected data. Let´s use the example of the video infrastructure service again. Within this system, healthcare organizations data is stored, users log in with their e-identification, videocalls, chats and client information are shared. There are third-party applications connected. This means several areas that can have potential vulnerabilities to be targets of threats. The service needs to secure enough to keep sensitive information protected.
Testing tools for Non-Functional testing
Which are the most common tools to use for non-functional testing. When we test the video service, which could be correct to use, and why.
LoadRunner – Is measuring system performance behavior under load. Can test end to end system performance, measuring system behavior and generate scripts by recording them.
JMeter – Open-source tool. Can analyze and measure system performance. It is used for performance, functional and load, HTTP testing.
Grafana – Open-source tool. Used for interactive visualization, analysis and monitoring of data.
Wireshark – Open-source tool. A network protocol analyzer to capture and analyze data in real-time.
Burp Suite – A web penetration testing framework. Used for web security and popular for penetration testing.
Lighthouse-ci – A suite of free tools to use during Continuous Integration. Can be used for several testing purposes like accessibility, performance.
Axe DevTools – Accessibility testing. Can be used for pages and component-level testing.
NVDA – Non-Visual Desktop Access. Open-source screen reader.
For doing non-functional tests with the testing tools above I would say these could be beneficial to use:
- Lighthouse-ci. If the system is cloud based, I would say this one. For monitoring the system during CI process this one can cover performance and accessibility monitoring. It includes, graphs, metrics, long time storage of data, compare reports and more.
- Burp Suite. As mentioned above the video service is a system that holds sensitive data and need to be secure. To use Burp Suite for penetration tests on all services and connections to third-party applications, databases, servers would find vulnerabilities in these areas.
- NVDA. The video service needs to be accessible for screen readers. NVDA is open-source and one of the most common to use.
- Axe DevTools. If we assume that Lighthouse already is running in CI, this one might not be necessary. Even if manually accessibility testing needs to be done too. But Axe DevTools comes in two versions, either a browser extension or a CI tool. The browser extension can be used for accessibility tests for the functions on video service. Video service is both used by professionals and public clients. To make sure it is accessible the extension can be used for all categories of disabilities with a measurement level of WCAG 2.1 standard.