The pitfalls of running Robot Framework with Safari

When I started writing UI tests, my comfort-zone environment to work with was Chrome on Linux. I usually use this setup and I also had a good implementation of Selenium webdriver for it. 

But, as a QA Engineer, you need to think and behave as a user. In my new assignment, our platform was being accessed mostly by Google Chrome and Safari. The CTO came up to me and asked me about using Safari for our test automation. 

A quick look at the global statistics of browser usage shows that Safari is number two with a sizeable share of the market, which is why it’s worth trying out Safari for your test cases too. 

Robot Framework is my open source test automation framework of choice, no bias!

Top tips for getting started with Safari on macOS Mojave

The beginning was not that straightforward. Safari has some limitations for Selenium tests. One of them was that only one browser could be used. According to Apple, this “accurately reflects what a user can do in a macOS windowing environment”. 

Another hurdle was that you can’t upload files by standard Selenium commands, due to Glass Pane protection. The last one caused an inconsistency with how to select pop-up windows. 

In Chrome, you have built-in functions to select the latest opened pop-up, which also allows you to select the main window. In Safari, you can only select by URL and title, which creates forks in the code.

In my case, I was able to fix the one browser issue easily, because most of my tests rely on single browser interaction. The upload file was made by a second Chrome browser, which was a really ugly hack and the window, well, it just forked the code for specific actions in Safari. 

One aspect that was very disappointing about Safari is how it needs more robustness when dealing with the UI. Comparing to the test case I wrote for Chrome, I had to be much more careful when interacting with the elements on the screen, asserting more about element visibility and also position when writing the test cases, since Safari does not wait until a page is fully loaded.

This is how you select a window in Chrome

This is how you select a window in Safari

How to configure and run Safari tests on Azure

Here I’ll describe the resources I used to make Robot Framework test automation work on Safari, including some suggestions for improvements for the powers that be.

I was fascinated by Azure DevOps platform and how it’s easy to set up a pipeline, from builds to testing and deployment. The UI is really comfortable which makes it easy to visualize the workflows. 

Before trying this setup (macOS with Safari), I had previously set up my Chrome successfully on Linux tests using Microsoft hosted agents, which was quite a nice experience. 

I basically created an Agent Job, with Shell++ steps to set up and run Robot tests. I published them with Publish Test Results, which was really disappointing from the features perspective. It enabled me to publish xUnit reports, but no possibility to upload the HTML reports of Robot Framework. 

To make this happen, you had to use Copy Files task, and store the logs in a zip file. I found this a lot clunkier on the Robot Framework front when compared to the Jenkins plugin for Robot Framework.

From the macOS agent perspective, Azure offers almost no option other than run some manual script for debugging purposes. As the macOS agent comes with an enabled firewall, my tests were stuck waiting for confirmation for incoming traffic. 

To take a screenshot, Azure offers screenshot and video recording functions only if you are using MSTest framework. For this case, I used a paste and upload tool. But real-time access or screenshot by demand would be nice. 

How Azure can improve their Robot Framework experience

Coming from a Jenkins guy, mostly it was the publisher plugin that I missed the most. Robot Framework generates really good reports for QAs and it would be nice to have a tool that uses that. 

Conclusion

Working with Safari when writing automated tests with Robot Framework is clearly a little bit different than other browsers.

Even with some crucial limitations, you can use Safari to cover most of the end-to-end test situations and make sure that your users can experience a stable user interface. When it comes to compatibility, most of the code can be shared with Google Chrome tests, so don’t expect the need to rewrite most of your test cases.

Safari will only expect the more robust code since the Webdriver in Safari is not mature as ChromeDriver. Azure has very good support for Linux and macOS environments in its Azure DevOps Pipelines, and Robot Framework is easily set up in both.

In short, Azure, Robot Framework (RF 3.2.1, SeleniumLibrary 3.3.1) and Safari 12 can run smoothly together.