Selenium – grid

Blog: Scaling a Kubernetes Selenium Grid with KEDA

Fri, 19 Aug 2022 00:00:00 +0000

The Issue

If you have any experience with Selenium Grid and Kubernetes you will probably run into an issue with scaling. Kubernetes (K8S) works wonders for scaling up and down applications based on their CPU and Memory usage, but it is not so straightforward when it comes down to applications like Selenium Grid.

The issue is described quite well in this blog post. But in short, the Horizontal Pod AutoScaler (HPA) that is built into Kubernetes checks (by default) for resource consumption to determine if a deployment needs to be scaled up or down. This becomes an issue for Selenium Grid for a couple reasons:

The browser pods use a variable amount of resources depending on the demand of the current test. This means that all your browser pods may be in use but there isn’t enough CPU usage for the HPA to decide that a scale-up is needed, leaving tests waiting in the queue unnecessarily.
When Kubernetes decides to scale down a deployment it does so (for the most part) at random. You could have 10 tests running on 20 pods and need to scale down. More than likely at least one of the pods asked to terminate will still have a test running, resulting in connection failures.

How KEDA Helps

KEDA is a free and open-source Kubernetes event-driven autoscaling solution that extends the feature set of K8S’ HPA. This is done via plugins written by the community that feed KEDA’s metrics server with the information it needs to scale specific deployments up and down.

Specifically for Selenium Grid, we have a plugin that will tie into our grid to get the information it needs. Example of the used trigger:

triggers:
 - type: selenium-grid
 metadata:
 url: 'http://selenium-grid-url-or-ip:4444/graphql'
 browserName: 'chrome'

All of this gets saved as a Scaled-Object like so:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
 name: selenium-chrome-scaledobject
 namespace: <namespace of your browser pods>
 labels:
 deploymentName: selenium-chrome-node-deployment
spec:
 minReplicaCount: 0
 maxReplicaCount: 80
 scaleTargetRef:
 name: selenium-chrome-node-deployment
 triggers:
 - type: selenium-grid
 metadata:
 url: 'https://selenium-grid-url-or-ip:4444/graphql'
 browserName: 'chrome'

As an added bonus KEDA allows us to scale our deployments down to 0 when there is no use, something the normal out-of-the-box HPA does not allow.

Check more details at the documentation for Scaled-Object in KEDA.

A full example of how to implement this is further down in the article but KEDA solves one of our two issues. Now we can properly scale up and down based on the actual load on the Selenium Grid. Unfortunately scaling down still results in the likely possibility that a pod is still running a test and is told to terminate before it can finish.

Using PreStop and Drain

To combat this we are going to use a combination of K8s PreStop and Selenium Grid’s Drain functionality.

PreStop allows us to set a command or chain of commands that is run to completion before the container is told to stop.
Drain tells the selenium browser pod to finish its current test and then shut down.

Together these look like so in our browser pod yaml:

spec:
 template:
 spec:
 terminationGracePeriodSeconds: 3600
 ...
 ...
 containers:
 lifecycle:
 preStop:
 exec:
 command: ["/bin/sh", "-c", "curl --request POST 'localhost:5555/se/grid/node/drain' --header 'X-REGISTRATION-SECRET;'; tail --pid=$(pgrep -f '[n]ode --bind-host false --config /opt/selenium/config.toml') -f /dev/null; sleep 30s"]

Breaking this down

terminationGracePeriodSeconds is set to however long you wish to give your pods to gracefully terminate before being forced. In this case I give the pods 60 minutes to finish their test when asked to terminate. If you are also scaling your cluster nodes as a part of this you may need to increase the termination grace period for your cluster nodes as well.
When the pod is told to stop, the PreStop command is ran first.
We curl the localhost of our pod to tell it to drain. The pod will no longer accept new session requests and will finish its current test. More information on this can be found in the Selenium Grid documentation.
We then tail the internal node process that will continue to run until the node has been drained.
After this we give the pod 30 seconds to finish anything else before giving the full termination command.

And with that our application can now safely scale down our selenium browser deployments!

From Start to Finish

Install KEDA

You need to use version 2.8.0 or later, you can find the latest version number at the Selenium Grid Scaler docs.
kubectl apply -f https://github.com/kedacore/keda/releases/download/<Version_Number_Here>/keda-<Version_Number_Here>.yaml

Create and apply your scaled object(s)

As described earlier your scaled object will look like so:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
 name: selenium-chrome-scaledobject
 namespace: <namespace of your browser pods>
 labels:
 deploymentName: selenium-chrome-node-deployment
spec:
 minReplicaCount: 0
 maxReplicaCount: 80
 scaleTargetRef:
 name: selenium-chrome-node-deployment
 triggers:
 - type: selenium-grid
 metadata:
 url: 'https://selenium-grid-url-or-ip:4444/graphql'
 browserName: 'chrome'

You will need one of these for every browser you wish to scale.

Things to edit:

namespace should be the namespace that your selenium browser pods exist in
deploymentName are the name of your browser deployment
name (within spec) is also the name of your browser deployment
url is the url of your selenium grid
browserName is the name of the browser you are using
minReplicaCount and maxReplicaCount are the min and maximum pod count you want to have

If you plan to scale with Edge you will need at least version 2.8.0 of KEDA and will need to include sessionBrowserName as well in the trigger metadata:

triggers:
 - type: selenium-grid
 metadata:
 url: 'https://selenium-grid-url-or-ip:4444/graphql'
 browserName: 'MicrosoftEdge'
 sessionBrowserName: 'msedge'

This is due to a name change between the Edge sessions in the queue and the active sessions and is addressed through this pull request.

Once you have that ready just save it as a yaml file and apply with:

kubectl apply -f ./<scaled-object-file-name>.yaml --namespace=<browser_namespace>

Add PreStop commands to your browser pods

Set your terminationGracePeriodSeconds of your deployment to whatever the maximum time you wish to give the pods in order to terminate gracefully. Again you may need to also increase the grace period for your nodepool as well which will vary depending on your K8s provider.
Add the PreStop command to the container lifecycle spec:

spec:
 template:
 spec:
 terminationGracePeriodSeconds: 3600
 ...
 ...
 containers:
 lifecycle:
 preStop:
 exec:
 command: ["/bin/sh", "-c", "curl --request POST 'localhost:5555/se/grid/node/drain' --header 'X-REGISTRATION-SECRET;'; tail --pid=$(pgrep -f '[n]ode --bind-host false --config /opt/selenium/config.toml') -f /dev/null; sleep 30s"]

That is it, your Selenium Grid pods should now scale up and down properly without any lost sessions!

Blog: Removing Legacy Protocol Support

Fri, 20 May 2022 00:00:00 +0000

The Selenium team prides itself on how seriously it takes backwards compatibility. A lot of care has gone into the Java bindings, especially, to ensure very few breaking changes over the years. There is Selenium code written 15 years ago that can still run with Selenium 4 libraries! Providing this exceptional amount of support comes with a large maintenance burden, though, and we need to be able to properly meet the needs of the vast majority of our userbase.

TL/DR:

Support for the legacy JSON Wire Protocol will be removed from Java Selenium 4.9 (other languages have already removed this support)
Protocol conversions will stop in Selenium 4.9 Grid
You can ensure your sessions are W3C compliant by using Browser Options classes instead of the deprecated Desired Capabilities classes (and avoid using “set capability” methods directly)
If you rely on the current protocol conversion functionality, and it works for you, you can continue to use it with Selenium Grid 4.8

By far the biggest challenge in the past seven years of Selenium development has been transitioning the underlying implementation from the legacy JSON Wire Protocol to the new standardized W3C WebDriver Protocol. Because the WebDriver specification was being actively developed at the time, Selenium 3 supported both protocols simultaneously. We implemented “handshake” code to determine which protocol to use. When starting a session, Selenium would package the provided capabilities in both legacy and W3C formats, send them to the driver/server, and use whichever protocol was returned. This worked well for the most part. For many users, no changes were necessary to get W3C compliant sessions from new browsers (Firefox 47+ and Chrome 75+).

For the Java bindings, it was decided to take this approach one step further. Instead of just sending along what was provided in both formats, the code converted capabilities from the legacy protocol to the W3C protocol on behalf of the user. The Selenium Grid makes use of this code, so, regardless of which client language sends legacy capabilities to it, the Grid translates it to the W3C protocol for communication with the driver and then re-translates the results back to the legacy protocol. Because the code must make some assumptions and guesses for this to work, there are a lot of frustrating edge cases.

For Selenium 4.0, the Ruby, JavaScript, and .NET bindings, each removed the handshake code, so no legacy commands are used. Due to some issues that couldn’t be resolved before freezing the 3.x code, Python is waiting until Selenium 4.9 to remove its handshake code. The Selenium team intended to continue to support both protocols in the Grid and Java bindings throughout the 4.x releases, but running test suites written for Selenium 2 on the Selenium 4 Grid resulted in a larger than expected number of failures. Half a dozen bugs were discovered, some with a large effect.

The code in question is sufficiently complex that fixing the bugs without causing new problems would be a significant challenge. As such, we’ve decided to start simplifying the code base again by explicitly not supporting capabilities and commands from the legacy protocol in any of the Selenium codebase. If you are using the latest version of Java Selenium, everything that will be removed has already been marked deprecated, so double-check your usage to ensure there aren’t any surprises.

If you’re interested in the nitty-gritty details of some of the issues we found exploring this problem, you can look at the discussion in Selenium Issue #10374.

For more information on ensuring compatibility with the Grid, please follow our Selenium 4 Upgrade Guide

Blog: Observability in Selenium 4

Mon, 26 Apr 2021 00:00:00 +0000

Diagnosing problems when the Selenium server isn’t working has never been easy. With Selenium 4, we have integrated OpenTelemetry to help you troubleshoot issues, optimize performance, and provide visibility into the system. By making the Selenium server observable, we are putting more power into your hands.

Need for Observability

Selenium server enables distributed testing. Instead of running the browsers for tests locally, the tests use a remote browser driver that points to a server. The server makes “smart” decisions to run tests on different remote servers. Every such remote server is capable of hosting different types of browsers and browser versions.

Selenium 3 uses Hub as the coordinator. It receives new session requests and directs them to the appropriate Node. The Node is the remote end where the browser itself runs.

Selenium 4 extends this to provide users with a way to set up a full-blown distributed system. Essentially, Selenium 4 splits the Hub into different components with additional enhancements. However, the Node’s role remains the same. To ensure the backward compatibility and simplicity of setting up Grid, standalone and hub-node mode are also available in the new Selenium server.

Now picture running a full-blown distributed Selenium Grid with hundreds of Nodes with different browsers and browser versions. Testing at scale in such an infrastructure will involve large volumes of requests. How can one keep track of these requests? Enter Selenium Observability!

How Does Selenium Provide Observability?

Selenium uses OpenTelemetry to instrument tracing and event logs. Tracing keeps track of a request’s lifecycle. As a request moves through the distributed system, the trace of the request will contain all the information of its crucial operations performed along the way. Such operations are known as spans. A span can record timed logs called events, which ideally encapsulate the current state of the system. These are event logs.

By default, the Selenium server enables tracing. Selenium server can run in different modes:

Tracing and event logs are available for all the modes. The simplest way to see traces is in the form of console logs. By default, the Selenium server prints logs at the INFO level and above. To pass a logging level of choice while running a Selenium Grid use the log-level flag. Setting the log-level to FINE will display traces and event logs as console logs.

For a large scale system, consuming traces as logs might not be efficient. Visualizing and querying traces will quickly help troubleshoot a request failure easily. Jaeger seamlessly integrates with OpenTelemetry to provide a rich experience of querying, visualizing and collecting request traces.

Run java -jar selenium-server-<selenium-version>.jar info tracing.

It provides detailed and updated instructions to set up Jaeger with the Selenium server. Visualizing the traces makes it easy to interpret a trace and understand request flow. Now Grid users can effortlessly trace a request, drill down into an error or query them to observe what is happening.

Refer to Selenium Observability for details.

Full-Stack Tracing

The Java client binding supports tracing. Full-stack tracing allows tracing a request from the client to the server and back. Though the client is a single component, the client trace will contain spans that help build the request. It is easier to locate a client-side problem and fix the test if needed.

We already saw how the server supports tracing and how to leverage it. For the client-side, add the Opentelemetry dependency to your project setup and add the necessary system properties to export the traces to Jaeger. Refer to RemoteWebdriver client for detailed instructions.

Leveraging Traces and Event Logs

The key to fixing a problem lies in knowing the error. For a known error situation, the event logs have you covered. It provides detailed error information and stack traces. Query the collected traces for a time range by the error code to identify the frequency of the error. It will help determine if the error is transient or not and accordingly take action.

Observe the request latency to identify potential bottlenecks. Each trace will also contain time taken by each span. The issue could be in the underlying infrastructure setup, the network latency, or the code itself. Irrespective of the cause, it is easier to identify the problem area.

Observability is slowly becoming a must-have property of a system. Selenium is now observable! Go ahead try it out. Let us know if you find anything that requires improvement on our end.

Blog: Selenium Grid 1.0.8 Released

Thu, 10 Jun 2010 00:00:00 +0000

We’re pleased to announce the release of Selenium Grid 1.0.8. This release fixes multithreaded issues with the Selenium Grid hub that appeared randomly under heavy load. If you’ve ever seen a log message about HttpClient being accessed by multiple threads, you definitely want to upgrade. Even if you haven’t, this release is highly recommended for all.

You can download it now or view the changelog. The list of changes is:

Multi-threaded issue with access to HttpClient has been resolved, fixing random crashes on a heavily loaded grids

Many thanks go out to Chris Gulley for identifying and fixing the problem. As always, patches and bug reports are appreciated:

I mentioned in the release notes for 1.0.7 that barring any major issues, 1.1 would be the next release. This fix was major enough to warrant an intermediate release. We are still planning to put out 1.1 as the next feature release.

If you have any questions about Selenium Grid, please use either the user or the developer list, as is most appropriate for the nature of your question. Ongoing discussion about grid development should take place on the developer list.

Blog: Selenium Grid 1.0.7 Released

Tue, 11 May 2010 00:00:00 +0000

Following up on the Selenium Grid 1.0.6 release, which was mostly a bug fix release, we’re pleased to announce the release of Selenium Grid 1.0.7. This is another bug fix release that focuses primarily on how Selenium Grid handles browser sessions.

You can download it now or view the changelog. The list of changes are:

Hub now instructs remote controls to shutdown a session if the hub is going to remove it due to being idle for too long
Added ability to specify timeout period when waiting for remote controls to become available. Now the client can timeout gracefully and decide what it would like to do next
Handle common case of session ID not being found, either because it timed out or was already closed; rather than raising a NullPointerException we now raise a NoSuchSessionException
Truncate long command response bodies to 128 characters
Retry remote control checks up to three times before deciding that a remote control is unreliable or unresponsive. This prevents aggressive unregistration of hosts that are busy or suffer from network issues

As always, patches and bug reports are appreciated:

Barring any critical issues, the next release of Selenium Grid will be 1.1 and will feature an enhanced Web view as well as a status API. After that, we will be moving from GitHub to Google Code to live alongside the other Selenium projects.

Blog: The Future of Selenium Grid

Tue, 27 Apr 2010 00:00:00 +0000

As you likely know by now, after years of stewardship Philippe has stepped down as maintainer of Selenium Grid and has named me the new maintainer. In this post, I hope to address the most immediate questions about what this means for the future of Selenium Grid.

Background

I realize many of you may not know who I am. I’ve been using Selenium in some capacity for close to five years now. I love working on open source software and do so primarily through my membership in the Apache Software Foundation and through my GitHub account. I hang out on the #selenium channel with the handle “nirvdrum” and have a personal page that includes some articles I’ve written and other things I’m involved with. Perhaps more importantly, I also rely on Selenium Grid for my latest start-up so I’m very keen on seeing the project mature and evolve.

Project Resources

The new canonical project repository on GitHub is the nirvdrum fork. However, within in the next month or so the project will be migrated from git and GitHub to subversion and the Selenium project on Google Code. As much as I personally prefer GitHub to Google Code, it makes the most sense from a community perspective for all Selenium projects to be hosted in the same location. The move will be made after Selenium Grid stabilizes a bit more. The 1.0.6 release was the first step in that process, but there may be one or two more minor patch releases. After the move, I’ll try to maintain a synchronized mirror of the code on GitHub so those that prefer to develop patches with git can continue to do so.

The canonical issue tracker is the Selenium issue tracker on Google Code. We will not be using either JIRA or GitHub Issues. I’ll try to migrate what I can over to the new tracker, but if I miss anything, please re-open the issue over there.

Project Evolution

Selenium Grid currently only works with Selenium 1.x, and consequently the 1.x compatibility interface in Selenium 2. There is currently no support for WebDriver. However, I fully intend to steer the product in that direction, while maintaining support for existing Selenium Grid installations (i.e., based on Selenium 1.x).

Since the project is moving to Google Code, all of the existing Selenium committers will have the ability to commit to the codebase. This should make the project’s evolution a much more collaborative effort. I’ll kickstart the process and likely lead the development effort, but the role of maintainer will be diminished (in a positive way) by allowing the entire core team equal access to the project.

Others are welcome to chime as well. Selenium Grid development discussion will be taking place on the Selenium developers list. Please continue to use the Selenium users list for usage and support inquiries.

Blog: Selenium Grid 1.0.6 Released

Tue, 13 Apr 2010 00:00:00 +0000

Following up on the Selenium Grid 1.0.5 release, which added self-healing features to ease grid management, we’re pleased to announce the release of Selenium Grid 1.0.6. This is a bug fix release for 1.0.5 with some minor new additions.

You can download it now or view the changelog. The summary of changes are:

Fixed some concurrent modification issues that may have resulted in NullPointerExceptions
Fixed issue with shutting down workers with Rakefile
Added PID file support for hub and worker started via Rakefile, making it easier to use a process monitor such as monit

In case you missed the announcement, Philippe is no longer maintaining Selenium Grid. We thank him for his years of effort and helping the project get to where it is today. This is my first release as the new Selenium Grid maintainer. I will be getting out a new post shortly that lays out the future for the project [UPDATE: The promised post is live]. In the meanwhile, please note the new location of several key resources: