AI with Robot Framework

As seasoned DevOps and test automation professionals, it’s no surprise we found ourselves once more at Robocon, which is a yearly conference about Robot Framework, the acceptance testing framework where you can write tests in natural language.

In this blog post, we highlight one topic that’s of particular interest to developers at the moment: Artificial intelligence (AI).

In the dynamic realm of Generative AI, models like ChatGPT are igniting curiosity about their potential use in various activities in test automation. David Fogl, an experienced developer and test automation specialist passionate about advancing the use of Robot Framework, highlighted some key possibilities.

David’s pros and cons of Generative AI in Robot Framework

Test data generation:

David delved into how Generative AI automates the creation of diverse test data and cautioned against over-reliance on AI-generated data that may overlook critical real-world scenarios.

Edge case identification:

He explored AI's potential in uncovering obscure edge cases, juxtaposed with its limitations in understanding the contextual relevance.

Dynamic XPath generation with AI:

AI's capability in generating adaptive XPaths was examined along with scenarios where AI-generated XPaths may lack reliability or efficiency.

AI integration via listener API:

David showcased the ease of integrating AI with Robot Framework while addressing potential complexities and troubleshooting challenges introduced by the integration.

API test scenarios generation:

He showed how Generative AI aids in formulating comprehensive and robust API test scenarios, simplifying the QA process for API testing.

Automating SQL test cases:

The potential of Generative AI in writing SQL automation tests was covered, enhancing the efficiency and accuracy of database testing.

See David’s full talk, where he delves into the points above.

Our thoughts on Generative AI in Robot Framework

Inspired by David’s talk, we wanted to cover the following possible use cases for AI in the future of test automation.

Creating a Robot script from the manual test case

We want to repeat David’s point in the talk: AI cannot replace human ingenuity, and it's unlikely to do so in the foreseeable future. It's essential to have the capability to craft Robot test cases from manual test cases firsthand. Otherwise, how can you ensure that the AI-generated script effectively tests the intended scenarios?

We’ve seen instances of human-crafted but superficial tests that pass without actually assessing the functionality of the application under test. There’s nothing to suggest that AI won’t do the same. Thus, a human needs to be in the driving seat and use AI as an assistant. Perhaps requiring both positive and negative tests from AI could help, thus making it more aware of its functionality and limits.

David brought up the fact that to be more effective, AI requires specific training with a dedicated model tailored to its intended function. Considering the simplicity of creating Robot test cases, investing time in developing and validating a functional model might exceed the time required to manually craft scripts. However, an open-source initiative might be the best course of action to encourage collaboration from like-minded people keen on contributing to the training of the AI model.

Once trained, machine learning models seamlessly integrate into the Robot Framework ecosystem, enhancing capabilities. These models aid in predictive test analysis, offering insights into potential failures or bottlenecks before they occur. They facilitate intelligent test prioritization, allowing your teams to focus on critical areas while optimizing testing efforts.

Visualizations 

Our own Aleksi Simell also did a talk at Robocon, focusing on test analysis with the help of visualizations. Humans are bad at reading raw input, so we require visualizations to help us make the correct analysis.

Inspired by David’s talk, we also think machine learning could, in the future, help in this process by analyzing the raw data for us while still providing the required visualizations to understand it. We are still left waiting for practical tooling around this idea, however.

Test strategy

Another inspiration taken from David’s talk is how AI can help with test strategy. Real-world applications showcase machine learning's prowess in identifying patterns, anomalies, and correlations within test data, leading to more effective testing strategies. However, challenges such as model interpretability, data quality, and algorithm selection must be carefully navigated to realize the full potential of integrating machine learning into Robot Framework.

Securing test data and infrastructure

David’s talk also birthed the idea of utilizing AI in the domain of security. Securing test data and infrastructure is a concern in the world of test automation. Traditional methods often fall short in protecting sensitive data and infrastructure assets from evolving threats. That's where AI-based solutions could step in, offering proactive measures to reduce risks and strengthen defenses. Techniques like behavioral analytics, anomaly detection, and adaptive authentication provide a strong shield against unauthorized access and malicious activities.

AI-enhanced encryption techniques will become (we think) crucial for maintaining the privacy of test data so that sensitive information remains confidential throughout the testing process.

AI-driven access control mechanisms bolster security by enforcing strict policies to prevent unauthorized access and data breaches. Continuous monitoring and threat detection, powered by AI algorithms, allow quick responses to security incidents, minimizing potential damage and downtime.

Implementing AI-based security solutions calls for a comprehensive approach. By integrating AI-driven compliance checks into your test processes, you ensure adherence to industry regulations and standards.

As the testing landscape evolves, AI-based solutions become increasingly indispensable for safeguarding test data and infrastructure integrity.

That’s a wrap for Robocon!

Once again, Robocon proved a hotspot for DevOps and test automation zealots, and while we only mentioned two of the talks, there was much to be taken away.

We look forward to the next Robocon and hope to see you there!