Selenium – technical

Blog: BELLATRIX Test Automation Framework for C# and JAVA

Wed, 16 Nov 2022 00:00:00 +0000

Over the last decade, a large ecosystem of Open Source projects has sprouted up around Selenium. Selenium is often used for automating web applications for testing purposes, but it does not include a testing framework. Nowadays, Selenium Ecosystem initiatives try to give popularity to popular open-source test automation frameworks maintained by people outside of the core Selenium maintainers. One of these frameworks is BELLATRIX, invented by Anton Angelov. It has two versions - C# and Java. A testing framework is an abstraction in which common code provides generic functionality (which can be selectively overridden) for testing different aspects of our applications- UI, API, security, performance, and many others.

BELLATRIX Test Automation Framework

The first version of BELLATRIX appeared on 26 December 2017. It was available only for C# initially, but written on the new back then .NET Core, allowing the framework to be used on all major operating systems (cross-platform). One huge advantage of BELLATRIX is its cross-technology readiness. It allows you to write tests for different technologies such as Web, Mobile, Desktop, and API. In BELLATRIX, we strive for the API for all modules to be as identical as possible.

The usage is simple. We suggest cloning BELLATRIX as a GIT sub-module. Then, any customizations, tests, and project-specific plug-ins should be placed in a project outside the BELLATRIX cloned repository. This way, you can quickly update to the latest version.

BELLATRIX official website, download and releases info

BELLATRIX official C# GitHub Page

BELLATRIX official Java GitHub Page

BELLATRIX C# Documentation

BELLATRIX Java Documentation

Let’s investigate how easy it is to create your first test with BELLATRIX in 15 minutes. The sample will showcase how to create a very basic test login into a website:

Open the BellatrixTestFramework.sln

Under the starthere folder, find the project you prefer: web, mobile, desktop, API

Open the BellatrixLoginTest.cs file. There you will find a sample test automating the login.

[TestClass]
public class LoginTestsMSTest : MSTest.WebTest
{
 public override void TestInit()
 {
 App.Navigation.Navigate("http://demos.bellatrix.solutions/my-account/");
 }

 [TestMethod]
 public void SuccessfullyLoginToMyAccount()
 {
 var userNameField = App.Components.CreateById<TextField>("username");
 var passwordField = App.Components.CreateById<Password>("password");
 var loginButton = App.Components.CreateByXpath<Button>("//button[@name='login']");

 userNameField.SetText("info@yourverybusywebsite.com");
 passwordField.SetPassword("yourverysecretp4ssw0rd$");
 loginButton.Click();

 var myAccountContentDiv = App.Components.CreateByClass<Div>("woocommerce-MyAccount-content");
 myAccountContentDiv.ValidateInnerTextContains("Hello John");

 var logoutLink = App.Components.CreateByInnerTextContaining<Anchor>("Log out");

 logoutLink.ValidateIsVisible();
 logoutLink.Click();
 }
}

All available services are available through the main App class. The Components property provides various Create methods for finding elements. They are generic, so you need to mention the type of the searched element. We have different elements because, for each of them, BELLATRIX offers various additional methods and assertions on top of native WebDriver methods. The sample code uses MSTest as the default test framework, but by changing the attributes, it will also work out of the box for NUnit. Of course, you need to change the base class namespace too.

Why BELLATRIX?

Let’s quickly list some of the essential things the framework brings to the table:

Multiple Test Environments Configuration

Every aspect of the framework can be controlled via a rich JSON configuration designed to work for many test environments. Web Project Configuration.

Customization

One of the hardest things to develop is to allow these generic frameworks to be extendable and customizable. Knowing how essential customization is, we utilize different ways to achieve it. The major one is about writing your own plug-ins.

Test Reliability

One of the biggest problems in test automation is handling timeouts and performing actions on elements that may not be on the page right now. BELLATRIX hides the complexity of searching and waiting for elements. Furthermore, when you perform an action or assertion against an element, we guarantee that once returned, it will be present.

A significant part of your tests are the assertions - checking whether some conditions are met. To handle such scenarios, we created elements Validate methods. They internally handle the whole complexity of waiting for some condition to happen.

updateCart.ValidateIsDisabled();
totalSpan.ValidateInnerTextIs("120.00€", timeout: 30, sleepInterval: 2);
messageAlert.ValidateIsNotVisible();

Complex Controls

BELLATRIX provides API that makes handling HTML tables and grids much easier HTML tables and grids.

Here is an example for asserting grid cells:

TestGrid.ForEachCell(cell => cell.AssertFontSize("14px"));
TestGrid.GetCell("Firstname", 1).ValidateInnerTextIs("Mary");
TestGrid.GetCell(0, 1).ValidateInnerTextIs("John");
TestGrid.GetCell<Employee>(cell => cell.PersonalEmail, 1).ValidateInnerTextIs("mary@hotmail.com");

There is much more complex stuff that you can do with both components so check the official documentation.

Integrations

Seamlessly integrate the framework with your existing tools and processes. Execute tests in the clouds, distribute and publish test results in reporting solutions. Maybe the most significant differentiators of the framework are its many integrations with popular tools such as Jira/qTest/Allure/ReportPortal and clouds such as AWS, Microsoft Azure, + many more. All these integrations leverage BELLATRIX plug-in architecture. For example, we have plug-ins for automatically generating/updating test cases based on our automated tests in qTest and AzureDevops or similarly creating automatic bug reports with steps to reproduce in Jira or Azure.

Dynamic Test Cases

Dynamic test cases are a unique feature in BELLATRIX, where the framework automatically generates test cases in a popular test case management system based on your automated tests. It will populate the title, description, and other necessary properties automatically. Moreover, it will generate human-readable steps and expected results. The most significant benefit is that it will keep up to date with your auto-generated test cases over time, no matter what you change in your tests. It is an excellent functionality which allows the non-technical people of your company to see what your tests are doing.

AI Validation of PDFs and Image

Azure Computer Vision is a service that can be used to extract printed and handwritten text from images and documents with mixed languages and writing styles. In contrast, Azure Form Recognizer is an AI-powered document extraction service that understands your document. You are not limited to PDFs only. You can use the same feature for extracting text from complex images. BELLATRIX comes with similar functionality based on the AWS cloud.

Email Testing

BELLATRIX offers a few utilities for email testing. There are a few scenarios where we need such integration. The first one is related to creating unique email inboxes and using them to submit various online forms. Later, we can read the emails via the services and check the content of the emails. It might be enough to verify the content via regular C#, or in some cases, we might need to interact with the email content in the browser.

There are tons of other integrations that we use on a daily basis in many big enterprise projects, such as cloud secrets management for securely storing credentials and other secrets. All BELLATRIX Integrations

Using BELLATRIX in your projects might save months/years of effort even if you have the required programming knowledge. This way, you can quickly focus on writing automated tests for your project. Usually, there are 4-6 major releases each year, including all bug fixes and new features. Check our release notes history. The framework is fully covered with over 4000 automated tests and offers rich documentation. We make sure to merge the new feature branches only when we are sure that everything is working. If some issue emerges after a major release, it is quickly fixed within a few days.

For feature requests or bug reports, you can submit them to our GitHub repositories. If you believe that the functionality you developed can be added to the CORE framework or you have a bug fix, please submit a PR so we can discuss it and potentially merge it. For anything else, you can reach us via our contact us form.

Blog: Observability in Selenium 4

Mon, 26 Apr 2021 00:00:00 +0000

Diagnosing problems when the Selenium server isn’t working has never been easy. With Selenium 4, we have integrated OpenTelemetry to help you troubleshoot issues, optimize performance, and provide visibility into the system. By making the Selenium server observable, we are putting more power into your hands.

Need for Observability

Selenium server enables distributed testing. Instead of running the browsers for tests locally, the tests use a remote browser driver that points to a server. The server makes “smart” decisions to run tests on different remote servers. Every such remote server is capable of hosting different types of browsers and browser versions.

Selenium 3 uses Hub as the coordinator. It receives new session requests and directs them to the appropriate Node. The Node is the remote end where the browser itself runs.

Selenium 4 extends this to provide users with a way to set up a full-blown distributed system. Essentially, Selenium 4 splits the Hub into different components with additional enhancements. However, the Node’s role remains the same. To ensure the backward compatibility and simplicity of setting up Grid, standalone and hub-node mode are also available in the new Selenium server.

Now picture running a full-blown distributed Selenium Grid with hundreds of Nodes with different browsers and browser versions. Testing at scale in such an infrastructure will involve large volumes of requests. How can one keep track of these requests? Enter Selenium Observability!

How Does Selenium Provide Observability?

Selenium uses OpenTelemetry to instrument tracing and event logs. Tracing keeps track of a request’s lifecycle. As a request moves through the distributed system, the trace of the request will contain all the information of its crucial operations performed along the way. Such operations are known as spans. A span can record timed logs called events, which ideally encapsulate the current state of the system. These are event logs.

By default, the Selenium server enables tracing. Selenium server can run in different modes:

Tracing and event logs are available for all the modes. The simplest way to see traces is in the form of console logs. By default, the Selenium server prints logs at the INFO level and above. To pass a logging level of choice while running a Selenium Grid use the log-level flag. Setting the log-level to FINE will display traces and event logs as console logs.

For a large scale system, consuming traces as logs might not be efficient. Visualizing and querying traces will quickly help troubleshoot a request failure easily. Jaeger seamlessly integrates with OpenTelemetry to provide a rich experience of querying, visualizing and collecting request traces.

Run java -jar selenium-server-<selenium-version>.jar info tracing.

It provides detailed and updated instructions to set up Jaeger with the Selenium server. Visualizing the traces makes it easy to interpret a trace and understand request flow. Now Grid users can effortlessly trace a request, drill down into an error or query them to observe what is happening.

Refer to Selenium Observability for details.

Full-Stack Tracing

The Java client binding supports tracing. Full-stack tracing allows tracing a request from the client to the server and back. Though the client is a single component, the client trace will contain spans that help build the request. It is easier to locate a client-side problem and fix the test if needed.

We already saw how the server supports tracing and how to leverage it. For the client-side, add the Opentelemetry dependency to your project setup and add the necessary system properties to export the traces to Jaeger. Refer to RemoteWebdriver client for detailed instructions.

Leveraging Traces and Event Logs

The key to fixing a problem lies in knowing the error. For a known error situation, the event logs have you covered. It provides detailed error information and stack traces. Query the collected traces for a time range by the error code to identify the frequency of the error. It will help determine if the error is transient or not and accordingly take action.

Observe the request latency to identify potential bottlenecks. Each trace will also contain time taken by each span. The issue could be in the underlying infrastructure setup, the network latency, or the code itself. Irrespective of the cause, it is easier to identify the problem area.

Observability is slowly becoming a must-have property of a system. Selenium is now observable! Go ahead try it out. Let us know if you find anything that requires improvement on our end.

Blog: Browser Testing and Tools WG Meeting @ TPAC 2020

Thu, 29 Oct 2020 00:00:00 +0000

It’s that time of the year where working groups from the W3C meet up to discuss the various standards that are being worked on.

Within the Browser Testing and Tools Working Group, there are 2 different standards.

We have:

The first is what is commonly supported by the Selenium Project and has support from Apple, Mozilla, Microsoft, and Google in their browsers. It is also supported by various Selenium in the cloud providers like Sauce Labs and BrowserStack.

As the world has moved on we have felt the need to add new APIs and move Selenium to be more event-driven. This is where we are learning, and collaborating, with projects like Puppeteer to make sure that we can improve the Browser Automation space. This is where the WebDriver-Bidi Specification comes in. It has broad support from the browser vendors so you can use official browsers and not be limited by the JavaScript sandbox. Some of the newer frameworks can’t guarantee that.

If you’re curious about we discussed this week, feel free to read it up on the W3 Wiki.

This post was originally shared on David Burns’ Blog

Blog: How to delete your master branch

Thu, 27 Aug 2020 00:00:00 +0000

At the Selenium project we practice trunk based development, in which trunk is the usual name of the default git branch of the repository. However, when the project was moved to GitHub, the repository followed the traditional use of master as a name for the default git branch.

With the intention of making the Selenium project an even more inclusive place where everyone is welcome, a decision was made to use trunk as the default git branch and, after the switch, delete the master branch. This change created a few challenges. This blog post will point out a few things you should watch out if you want to make the same change in your GitHub repository.

Broken Links

It is common to link specific parts of the code in the documentation, and that link normally contains the branch name. Double-check your documentation for links pointing to files living on the master branch, as they could end up as broken links after the change.

Mentions to the branch name

Similarly, the branch name gets mentioned in different parts of the repository, such as code comments, contribution instructions, issue and pull request templates, and the readme. Don’t forget to check these places.

GitHub repository badges

We all like to show off how our GitHub repository is doing by adding as many badges as we can. In many cases, those badges report a build status that depends on the branch where the build was executed. Make sure your Travis/CircleCI/GitHub Actions badge is pointing to the new branch.

Continuous Integration setups

A good practice in open source is to have a continuous integration setup to run the builds, execute tests and potentially do automated releases. Nowadays, the CI configuration is done in files (e.g. .travis.yml for Travis), and one important piece of that configuration is the name of the branch where the build should be executed. Similar to the previous point, double check that your new branch name is property configured in your CI integration.

Build scripts

At the Selenium project, we have a few custom build scripts that get executed through the continuous integration setup. An example is the script that generates the docs for the Java, Ruby and Python bindings. This script needs to know the code’s branch name to generate the docs. If you have scripts with similar purposes, check them after changing the branch name.

Open pull requests

All of the tasks above could probably be achieved with a text editor and a massive, but careful, “search and replace” across all the files in the repository. In short, the process we followed to move to the new branch trunk was:

Create a new branch called trunk, based on the master branch.
Do all the items described in the previous points.
Commit and push those changes.
Delete the master branch.

Nevertheless, one thing happened that we did not expect: after deleting the master branch, all the open pull requests got closed. This made sense, since they were all targeting the master branch. Therefore, before deleting your master branch, double check and, if needed, edit the open pull requests so they target the new branch.

It goes without saying that the name of your new branch can be any name that works well for your context and environment. For example, the repository that has the contents of the Selenium website uses now dev as the branch with the website source files and publish as the branch with the generated static website that gets published.

These are our lessons learned during the process of deleting the master branch in all the major repositories under the SeleniumHQ GitHub organization. I hope they are helpful if you decide to move from master as a name for your default branch.

This was originally posted at https://opensource.saucelabs.com/blog/how_to_delete_your_master_branch/

Blog: Moving to Trunk

Wed, 01 Jul 2020 00:00:00 +0000

Since the project started we have been following trunk based development. This was a very natural fit when we were using SVN over a decade ago on Google Code.

As Google Code shut down we moved to GitHub and the git model of doing things. We moved there mostly due to the gravity that GitHub had created in Open Source projects.

This meant that we followed the standard use of master as our trunk to work off. Now that GitHub, and services that use GitHub, have improved support for non-master branches as default we are moving our default branch to trunk. It describes how we, as a project, work and is a more inclusive term.

If you have pull requests based on master we will see about moving that over to the trunk branch ourselves. If we can’t, we may ask you to help with the rebasing.

We, as a project, want to make our community inclusive and this is just one step in making sure we are. Other steps we taking are improving our Code of Conduct and Community Guidelines.

Join us on Slack or IRC if you wish to discuss this further.

Blog: Source Control

Mon, 14 Jan 2013 00:00:00 +0000

This short technical note is to announce that the Selenium project is now using git on Google Code in place of subversion.

The move has been a long time in the making, and it’s largely thanks to the efforts of Kristian Rosenvold that we’ve been able to do the migration and retain the project history. The project owes him a huge thank you! We’re in the process of migrating the last bits and pieces (none of which are user facing), so there may be some last minute turbulence as we settle everything down.

Although the canonical source will be on Google Code, we’re working on setting up a github mirror. We’ll announce the location of that once it’s set up.

Blog: Going Atomic: How

Sun, 05 Sep 2010 00:00:00 +0000

This is the second of my technical posts. Again, if you’re interested in the internal workings of Selenium 2, then please skip straight to something else. If you’re interested in how and why we made some of the technical decisions on the project, keep reading….

We left our intrepid heroes in a tight spot: they’d decided to write a shared library of code, to be used by the various webdriver implementations and selenium core, but the requirements for doing this seemed to be at odds with it actually happening.

Fortunately, at about the same time we started down this path, Google Open Sourced the Closure compiler. This is a Javascript compiler that takes as input a set of Javascript files, and which outputs Javascript. It can be configured to either pass the code through untouched into a single file, or it can compile a script aggressively, removing unused code-paths and minifying the output as much as possible. The Closure compiler is used on a lot of Google products, so we know that it’s reliable and consistent.

In order to get the best out of the Closure compiler, we’re writing the atoms using the Closure library. This isn’t as well known as some of the other JS libraries out there, but it’s solid, well tested and is being actively developed. It also features an easy-to-use extension of JsUnit, which makes writing tests a far simpler task than might otherwise be the case, and it has an easy to use mechanism for modularizing code.

So, given that we could compile a single Javascript function (and it’s dependencies) into a minified fragment of JS, we were all set, right? Not quite.

The problem is that the atoms are being extracted from two frameworks that have a different way of viewing the world. As an example, Selenium 1’s “getAttribute” method only returns the value of a particular attribute, whereas WebDriver’s “getAttribute” method will return the value of either a property or an attribute (because sometimes it’s hard to remember whether something is an attribute or a property of an element)

As with all problems in computer science, an extra level of indirection is used to solve this issue.

We’re busy implementing the expected behaviour of both WebDriver’s and Selenium’s API on top of the atoms.

There is, of course, the obvious question about how we get this carefully compressed JS into a driver. One option would be to include the raw Javascript as files in each language binding, and pull them in as required. That’s possible, but it would make each language binding bloated, and would introduce a lot of duplication. The alternative is to push the atoms as far into the driver as possible, and this is what we do. As part of the build process for webdriver, we take the compressed JS and convert it into a form that can be consumed by a particular driver. For example, for the IE driver, we convert them into constants in a C header file. These constants can then be referred to by the driver and converted back into a script than be executed via the same mechanism that is used by “executeScript”.

What do we gain from this seemingly baroque approach? Other than the ability to share the same code between drivers? Many things. The cost of maintenance drops dramatically as we can fix a bug in one place and have that fix be picked up by every driver. Because we’re working in pure JS and just querying the DOM, we can run the unit tests in a browser whilst we’re developing the code. This leads to a very tight feedback cycle. It also makes it easier for developers not familiar with the code to take a look at how we do things, and send us patches (always appreciated!) Finally, we can ensure a consistency of result.

Right, any questions?

Blog: Going Atomic: Why?

Mon, 16 Aug 2010 00:00:00 +0000

This is the first in a series of technical posts by me about the internals of Selenium WebDriver. If you’re not interested in technical nitty-gritty, then feel free to step away now.

Still here? Excellent.

Let’s take a step back to just before the Selenium and WebDriver projects merged. There were, very obviously, two separate codebases. Looking closer and with a slightly different perspective, there were more than this. We used the test suites for webdriver to define the behaviour for multiple, largely independent, driver codebases. The IE driver was written in C, the HtmlUnit driver in Java and the Firefox driver is largely Javascript, and so on.

This means that there was a lot of “congruent code”: code that performed the same function but was implemented in a different way. The natural result of this was there was the possibility for behaviour to diverge between drivers. Worse, it meant that when a bug was found, we had to check it in every browser, and it wasn’t certain that an individual could actually fix the code. After all, not everyone is comfortable writing in all the languages we use on the project, or is au fait with all the technologies. For an Open Source project like Selenium, this is a major problem: we rely on a relatively small core of key developers backed up with a far larger team of individuals submitting small changes and fixes. Anything that makes it harder for us to function effectively as a development community is a Bad Thing.

So, we wanted a way off the island; a mechanism that would make it easy to share code between the various drivers and selenium core, that allowed us to fix a bug in one place only and have that fix ripple out to every driver that made use of this mechanism. More importantly, it had to be easy to use, and for someone not familiar with a raft of languages and technologies to quickly get started with.

What would this mechanism look like? Well, there’s a few things that feed into this, but the most important one, was that a majority of the code we’d think of merging was querying the state of the browser (“find an element”, “get the value of this attribute”) and, as Jason Huggins would point out to me at the drop of a hat, the natural language for querying the state of a browser is Javascript. One of the nice things with Javascript is that it’s possible to get a nice, fast development cycle going in it. Just modify a test, save and then hit “refresh” in the browser. That’s kind of attractive. Better still, there are a lot of developers familiar with Javascript.

So, we decided to use Javascript.

Because this shared code was to be composed of the smallest useful fragments of functionality required for browser automation we decided to refer to them as “Browser Automation Atoms”, or “atoms” for short. Rather than write them from scratch, the easiest thing to do was to extract them from the existing code — this is stuff that’s been battle-tested, so we know it’s robust.

There was one very obvious fly in the ointment: not every driver is written in Javascript. Although we have a mechanism available in every browser for executing JS, it’s wildly inefficient to dump an enormous lump of code on to the JS engine of the browser whenever you want to query the DOM. After all, most of the code would not be needed, and not all JS engines have been created equal. Some are blazingly fast. Others, not so much.

It would also be nice to break the code up into manageably-sized modules, rather than being in a single, monolithic file, which implies some clever “module loading” capability. Except this code isn’t always going to be executing inside an environment where writing “script” tags to load additional scripts is possible. You can’t do that in the guts of a firefox extension, though you can load files other ways. However we tie modules together will need to cope with that.

Ah! These opposing requirements: small modules containing the functions we want to use, no extraneous code, and for everything to be in a single file in order to minimize the pain of loading additional modules. That doesn’t sound like a very compatible list. How we resolved those differences is the topic of my next post….

Blog: How to use Selenium 2 with Maven

Fri, 30 Jul 2010 00:00:00 +0000

There are several ways to use Selenium 2:

If you don’t have Selenium 1.x legacy code, you might want to directly use on of the new WebDriver implemenations like ChromeDriver, HtmlUnitDriver, FirefoxDriver, or InternetExplorerDriver which provide a nice, small and easy to learn API.
If you have Selenium 1.x legacy code, you can still use the well known DefaultSelenium class or the new WebDriverBackedSelenium, which extends DefaultSelenium but uses one of the WebDriver implementations internally.

Whichever option you prefer, if you have want to use Maven, all you need to do is add the following dependency to your pom.xml:

<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium</artifactId>
<version>2.0a5</version>
</dependency>

If you know, that you will only use a certain WebDriver implementation, e.g. the FirefoxDriver, you don’t need to depend on the selenium artifact (which has dependencies to all WebDriver implementations as well as the support classes). Instead you can add the dependency to just the artifact you need, e.g.

<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-firefox-driver</artifactId>
<version>2.0a5</version>
</dependency>

When using a WebDriver implementation, there is no need to start a Selenium server – the browser will be directly started and remote controlled.

But if you are using DefaultSelenium (or the RemoteWebDriver implementation), you still need to start a Selenium server.

The best way is to download the standalone Selenium server jar and just use it.

Furthermore you can also embed the Selenium server into your own project, if you add the following dependency to your pom.xml:

<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-server</artifactId>
<version>2.0a5</version>
</dependency>

Now you can create a SeleniumServer instance yourself and start it.

Be aware, that the selenium-server artifact has a dependency to the servlet-api-2.5 artifact, which you need to exclude, if your project will be run in a web application container.

Well, I hope that covers everything you need to know on how to use Selenium 2 with Maven.

Michael