USING CHATGPT? All software, including ChatGPT, requires testing before being deployed to production.
What’s ChatGPT and why AI technologies look threatening to engineers? ChatGPT, an acronym for Chat Generative Pre-trained Transformer, is a chatbot developed
by OpenAI. It was released in beta version to
the public on November 30, 2022. At first glance, it may seem like an exceptionally intelligent automation tool capable of solving numerous problems. However, upon closer examination it may not be so. We explore the challenges and why ChatGPT is far from being a magical solution. To begin with, various sources have highlighted limitations, and users who have experimented with different versions of ChatGPT have noticed them too.
- ChatGPT has limited knowledge of events that have
taken place after September 2021 – so if you ask it to tell a story from the recent past or analyze specific events or data from this period, this may be an issue;
- ChatGPT sometimes writes plausible-sounding but
incorrect or nonsensical answers – ChatGPT occasionally generates responses that sound plausible but are incorrect or nonsensical. This behavior is common among large language models and is referred to as hallucination. Hallucination, also known as confabulation or delusion, refers to a confident response by an AI that does not appear to be justified
by its training data. It means that sometimes users may
receive fabricated data, but not the actual facts. There are a lot of examples, such as when ChatGPT was asked for proof that dinosaurs built a civilization, ChatGPT claimed there were fossil remains of dinosaur tools and stated,”Some species of dinosaurs even developed primitive forms of art, such as engravings on stones.” ChatGPT endeavors to decline prompts that might violate its content policy. However, presently, there remain certain vulnerabilities that users have managed to exploit, potentially leading to the generation of misleading or false information as per their intentions. This vulnerability persists, and engineers are diligently seeking the most effective solution to address this issue.
- Not all languages are supported by ChatGPT yet, the
service works best with the English language for now. However, as ChatGPT’s popularity continues to soar, several successful features have been implemented to enhance its utility: Basic Service: This foundational offering allows users to engage in dialogues and receive answers to their queries. Powered by advanced machine learning algorithms, this highly capable chatbot processes immeasurable amounts of data to generate responses. It comprehends both spoken and written human language, enabling it to grasp incoming information and provide relevant responses. ChatGPT Plus Premium Service: Users opting for the premium service gain access to the latest versions and updates, along with perks such as uninterrupted service during peak periods, priority access to new features, and faster response times. In July 2023, OpenAI expanded the offering by making its proprietary Code Interpreter plugin available to all ChatGPT Plus subscribers. This Interpreter offers a wide variety of functionalities, including data analysis, instant data formatting, personalized data scientist services, creative problemsolving, musical preference analysis, video editing, and seamless file upload/download with image extraction. Mobile App Launch: In May 2023, OpenAI launched the ChatGPT mobile app for iOS users, extending its accessibility. An Android version of the app has been rolled out to select countries initially, with plans for further expansion in the near future. ChatGPT can:
- Answer questions
- Solve math equations
- Translate between languages
- Debug and fix code
- Write a story/poem
- Classify things
Sounds scary from an engineering perspective, doesn’t it? Let’s try to understand if our fear is justified. Is it truly feasible for AI to completely replace the entire manual QA process and a significant portion of automation QA Lately, there has been increasing chatter about ChatGPT and its potential to displace various engineering roles, particularly QA Engineers. However, it’s essential to refrain from being too discouraging about AI. It appears unlikely that ChatGPT can entirely supplant all testing efforts or significantly reduce the need for manual and automation QA work. The notion that emerging technologies and a new generation of skilled workers will render current jobs obsolete is far from new. As an example, the term “Luddite” traces its origins back to a movement of 19th-century textile workers in England
who opposed textile machinery, fearing it would devalue their highly specialized craftsmanship. In the realm of software testing, similar concerns arose over a decade ago when marketers touted test execution automation as a “silver bullet” promising improved testing efficiency. It was believed that this automation would significantly reduce testing budgets and the human effort required to accomplish project goals. Upon analyzing the aforementioned limitations, it becomes evident that the responses generated by ChatGPT for user queries may not always be 100% efficient. Therefore, it is imperative we review the provided results, make necessary adjustments, and update queries as required. Furthermore, it is important to acknowledge that ChatGPT may not comprehensively cover all topics and queries due to its inherent limitations. In essence, human intervention is essential to pose the right questions to ChatGPT and to review its responses. The process of proofreading demands knowledgeable individuals, and rather than replacing productivity, it serves to enhance it. It can certainly enhance current manual procedures and assist Software Development Engineers in Test (SDETs) and Automation Quality Analysts (AQAs) in achieving more. However, it’s crucial to understand that automation can never replace human involvement; it can only streamline and expedite our tasks, making them more efficient. The positive outcome of automation is that it accelerates individual processes for Quality Assurance (QA) teams and various components of the Software Development Life Cycle (SDLC), including design and development. These individual enhancements ultimately result in swifter software delivery, allowing for changes of greater functional complexity and logical intricacy. Consequently, this leaves less time for comprehensive testing, which is where new tools could potentially play a significant role. How can you effectively harness the power of ChatGPT in your testing processes? Both manual and automated testers stand to gain significant advantages from utilizing such robust tools, provided they are willing to invest time and effort in learning how to leverage them effectively. It is strongly recommended for test leads and test managers to allocate some time for piloting and experimenting with
ChatGPT. The benefits include process enhancement and the improved quality and efficiency of software Here are some evident ways to make the most of ChatGPT for your testing needs: Test Documentation Generation: ChatGPT can be employed to create initial drafts of various test-related documents, including Test Strategies, Test Policies, Test Scenarios, and more. By asking precise and targeted questions, you can obtain more meaningful answers, reducing the need for extensive post-generation revisions before finalizing your documents. Documentation Validation: Testers can enlist ChatGPT’s help in reviewing and enhancing provided documentation, such as user manuals, guides, and help documents. ChatGPT can assist in aligning the documentation with the software’s features and functionalities, ensuring accuracy and clarity. Complex Scenario Simulation: ChatGPT can be tasked with simulating intricate scenarios that might be challenging to replicate manually. These scenarios may involve conditions that are not easily triggered, and ChatGPT can provide valuable feedback based on these Diverse Testing Types: ChatGPT can also be used for various testing types, including usability, load, and performance testing. By emulating desired user interactions, it can provide feedback on how the software performs under different conditions, aiding in identifying potential issues. Automated Scenario Generation: Automation Quality Analysts (AQAs) can leverage ChatGPT to request the coding of specific scenarios using their preferred programming language. This automation streamlines scenario generation, improving efficiency and accuracy in testing processes. It’s essential to reiterate that ChatGPT is a valuable tool for testers across various roles. However, it’s crucial to use it in conjunction with traditional testing methods and human judgment. Any content generated by AI should undergo a thorough review and refinement before being considered a final version for your future deliverables. Furthermore, it’s important to recognize that ChatGPT cannot entirely replace the necessity for specialized testing techniques and methods. This includes tasks such as creating automated testing frameworks from scratch or conducting security assessments, which require expertise and the use of highly specialized tools handled
by experts in the field.
It’s important not to take the use of ChatGPT too lightly, as every tool has its limitations. The future across various domains appears to carry certain risks. If people aren’t replaced by AI, they may heavily rely on ChatGPT’s feedback, potentially leading to issues or a reliance on AI for precision, as seen in the example of a judge in Pakistan using ChatGPT to decide a legal matter concerning a 13-year-old accused. However, this doesn’t necessarily apply to engineering. In engineering, we may encounter individuals who become complacent and rely solely on tools, while others may use ChatGPT to enhance their productivity and deliver their best work. In summary, it might seem that ChatGPT could tempt engineers to take shortcuts and spend less time on their work than initially estimated. But let’s pause here and consider a crucial point: any attempts at cheating or cutting corners will eventually come to light, and ChatGPT itself could play a role in that. There are already tools available to detect AI-generated text, and we can expect more in the future. Therefore, the optimal way to leverage this tool is to streamline routine and mundane tasks, creating more room for professional growth and the art of testing.
Author: Ramella Basenko is a Lead QA Engineer
& Engineering Manager at AgileEngine, has 10 years of experience in the QA area. Her daily work mainly focuses on process improvements and project transformations as well as team management and career growth of QA professionals within the company. ISTQB Full Advanced level certificate holder. Has degrees in German philology, Business and Administration. Speaker at conferences and webinars in the field of software quality and certification of QA specialists.
TMAP®: Quality Engineering for SAP certification Is an advanced certification specifically crafted for individuals involved in the testing, acceptance, and implementation of SAP solutions. It equips professionals with the essential knowledge and expertise required to champion quality and excellence in their SAP projects. TMAP: Quality Engineering for SAP Email: exam@isqi.org Email: tmap@isqi.org