How to Run Selenium Regression Tests With GitHub Actions

1. Overview

Regression tests ensure builds are ready for release by providing critical feedback. As the codebase evolves with new requirements, bug fixes, and enhancements, running these tests frequently is essential to maintaining software quality.

It’s important to run regression tests in the CI/CD pipeline, as they execute automatically when we commit code to the remote repository, delivering quick feedback.

In this tutorial, we’ll learn how to run Selenium regression tests with GitHub Actions.

2. What Are GitHub Actions?

GitHub Actions automate the build, test, and deployment workflows that build and test every pull request to the remote repository, thus enabling faster feedback.

The feedback helps us in fixing the issues before we move to production. GitHub Actions workflows can run on Windows, macOS, and Linux virtual machines or self-hosted runners in the data center or cloud infrastructure.

We can also leverage GitHub Actions with various tools and frameworks to run regression tests. For example, GitHub Actions can help perform automated browser testing with Selenium, making regression tests more efficient.

3. Running Selenium Regression Tests With GitHub Actions

Before we talk about running the Selenium regression tests on GitHub Actions, let’s discuss the project setup, configuration, test scenario, and implementation.

3.1. Setup and Configuration

Let’s add the dependencies of Selenium WebDriver and dependencies of TestNG in the pom.xml file of the Maven project:

<dependency>
    <groupId>org.seleniumhq.selenium</groupId>
    <artifactId>selenium-java</artifactId>
    <version>4.27.0</version>
</dependency>
<dependency>
    <groupId>org.testng</groupId>
    <artifactId>testng</artifactId>
    <version>7.10.2</version>
    <scope>test</scope>
</dependency>

We also need to add the maven-surefire-plugin as it helps us in executing the testng.xml file when we run the Maven command mvn clean install using GitHub Actions workflow:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>3.2.5</version>
    <executions>
        <execution>
            <goals>
                <goal>test</goal>
            </goals>
        </execution>
    </executions>
    <configuration>
        <useSystemClassLoader>false</useSystemClassLoader>
        <properties>
            <property>
                <name>usedefaultlisteners</name>
                <value>false</value>
            </property>
        </properties>
        <suiteXmlFiles>
            <suiteXmlFile>testng.xml</suiteXmlFile>
        </suiteXmlFiles>
        <argLine>${argLine}</argLine>
    </configuration>
</plugin>

We’ll run Selenium regression tests using GitHub Actions on the cloud using platforms like LambdaTest.

LambdaTest is an AI-powered test execution platform that lets us perform automation testing across 3000+ real browsers, browser versions, and operating systems.

Let’s create a BaseTest.java to add all the Selenium and LambdaTest configuration details:

public class BaseTest {
    protected RemoteWebDriver driver;
    // Other code and methods...
}

We’ll be adding the setup() method in the BaseTest class that configures Selenium and LambdaTest before any test runs:

@BeforeTest
public void setup() {
    String userName = System.getenv("LT_USERNAME") == null 
      ? "LT_USERNAME" 
      : System.getenv("LT_USERNAME");
    String accessKey = System.getenv("LT_ACCESS_KEY") == null 
      ? "LT_ACCESS_KEY" 
      : System.getenv("LT_ACCESS_KEY");
    String gridUrl = "@hub.lambdatest.com/wd/hub";
    try {
        this.driver = new RemoteWebDriver(
          new URL("http://" + userName + ":" + accessKey + gridUrl),
          getChromeOptions()
        );
    } catch (MalformedURLException e) {
        LOG.error("Could not start the remote session on LambdaTest cloud grid");
    }
    this.driver.manage()
      .timeouts()
      .implicitlyWait(Duration.ofSeconds(5));
}

The LambdaTest capabilities related to the platform, browser version, etc., will be fetched from the getChromeOptions() method created in the same BaseTest class. To generate automation capabilities, we can use the LambdaTest Automation Capabilities Generator:

public ChromeOptions getChromeOptions() {
    var browserOptions = new ChromeOptions();
    browserOptions.setPlatformName("Windows 10");
    browserOptions.setBrowserVersion("latest");
    HashMap<String, Object> ltOptions = new HashMap<>();
    ltOptions.put("project", "LambdaTest e-commerce website automation");
    ltOptions.put("build", "LambdaTest e-commerceV1.0.0");
    ltOptions.put("name", "Homepage search product test");
    ltOptions.put("w3c", true);
    ltOptions.put("plugin", "java-testNG");
    browserOptions.setCapability("LT:Options", ltOptions);
    return browserOptions;
}

After all the tests are executed, the tearDown() method updates the test status on the LambdaTest cloud grid and closes the driver session:

@AfterTest
public void tearDown() {
    this.driver.executeScript("lambda-status=" + this.status);
    this.driver.quit();
}

We’ll automate the following test scenario using Selenium with the GitHub Actions on LambdaTest cloud grid on the latest Chrome browser.

3.2. Test Scenario

Let’s use the following test scenario to demonstrate Selenium regression testing with GitHub Actions:

1. Navigate to the LambdaTest eCommerce Playground website.
2. Search for a product on the homepage.
3. Check that the product search page returns the correct product details.

3.3. Test Implementation

Let’s create a LambdaTestEcommerceTests class to implement the test scenario by writing automated test scripts. This class extends the BaseTest class to inherit its fields and methods and reuses it in the test class:

public class LambdaTestEcommerceTests extends BaseTest {
    @Test
    public void whenUserSearchesForAProduct_thenSearchResultsShouldBeDisplayed() {
        String productName = "iPhone";
        driver.get("https://ecommerce-playground.lambdatest.io/");
        HomePage homePage = new HomePage(driver);
        SearchResultPage searchResultPage = homePage.searchProduct(productName);
        searchResultPage.verifySearchResultPageHeader(productName);
        this.status = "passed";
    }
}

There are two page object classes used in the test. The first one is the HomePage class, which has all the page objects and actions related to the homepage of the web application:

public class HomePage {
    private RemoteWebDriver driver;
    public HomePage(RemoteWebDriver driver) {
        this.driver = driver;
    }
    public SearchResultPage searchProduct(String productName) {
        WebElement searchBox = driver.findElement(By.name("search"));
        searchBox.sendKeys(productName);
        WebElement searchBtn = driver.findElement(By.cssSelector("button.type-text"));
        searchBtn.click();
        return new SearchResultPage(driver);
    }
}

The searchProduct() method accepts the productName as a method parameter and searches for the product with the name provided. After the successful search of the product, the search results page appears; hence, this method returns a new instance of the SearchResultPage class:

public class SearchResultPage {
    private RemoteWebDriver driver;
    public SearchResultPage(RemoteWebDriver driver) {
        this.driver = driver;
    }
    public void verifySearchResultPageHeader(String productName) {
        String pageHeader = driver.findElement(By.tagName("h1"))
          .getText();
        assertEquals(pageHeader, "Search - " + productName);
    }
}

The verifySearchResultPageHeader() method checks if the header of the search page has the text of the product that was searched for.

3.4. Test Execution

Let’s create a testng.xml file for executing Selenium regression tests on the LambdaTest cloud platform:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Selenium GitHub Actions demo">
  <test name="Searching a product from the HomePage">
      <classes>
          <class name="com.baeldung.LambdaTestEcommerceTests">
              <methods>
                  <include name="whenUserSearchesForAProduct_thenSearchResultsShouldBeDisplayed"/>
              </methods>
          </class>
      </classes>
  </test>
</suite>

3.5. Set up the GitHub Actions

The following are a few prerequisites to setup and configure GitHub Actions:

• GitHub account
• GitHub repository

Once we have the GitHub account and a repository created, we can move to the Actions tab and configure the Java with Maven Action workflow:

3.6. Create the GitHub Actions Workflow

GitHub automatically generates the workflow file for us. We can choose to continue with the same workflow provided by GitHub, or we can add our own content.

As we need to run the Selenium regression tests on the LambdaTest cloud platform, we’ll be updating the content of the workflow as per our requirements:

name: Java CI with Maven
on:
  push:
    branches:
      - "main"
      - "test-*"
  pull_request:
    branches:
      - "main"
      - "test-*"
jobs:
  build:
    runs-on: ubuntu-latest
    env:
      LT_USERNAME: ${{ secrets.LT_USERNAME }}
      LT_ACCESS_KEY: ${{ secrets.LT_ACCESS_KEY }}
    steps:
      - uses: actions/checkout@v4
      - name: Set up JDK 17
        uses: actions/setup-java@v4
        with:
          java-version: '17'
          distribution: 'temurin'
          cache: maven
      - name: Build and Test with Maven
        run: mvn clean install

We need to add the LambdaTest Username and Access Key as secrets for running the tests on the LambdaTest cloud grid. Additionally, we’ll be updating the Maven command to mvn clean install, as it downloads all the dependencies, builds the project, and runs the test in one go.

The Secrets value can be added to the repository by navigating to the GitHub repository Settings > Secrets and Variables > Actions:

On the Actions Secrets and Variables page, we click on New repository secret and add the secret variables and their values.

3.7. Trigger the GitHub Actions Workflow

We can trigger the GitHub Actions workflow using the on keyword in the workflow file for both single as well as multiple events.

Let’s follow these steps to trigger the workflow when pushing code to any branch or raising a pull request:

1. We add the workflow file to the GitHub repository. We’ll name our branch maven_github_actions to match the on push branches /on pull_request branches name to maven_*.

2. We make code changes in the maven_github_actions branch. We then push the code on the branch and raise a pull request to the main branch. After successfully pushing the code, the workflow triggers automatically, and we can check its status on the Actions tab of the GitHub repository:

Clicking on the workflow run lets us view detailed information about the workflow job.

After the workflow run is complete, we can view the Selenium regression test results on the LambdaTest Web Automation dashboard:

The build jobs actually run on the user-chosen configuration machine. The name of this machine is set in the workflow flow inside the jobs block using the runs-on keyword.

In our case, we ran the GitHub Actions workflow using the ubuntu-latest machine.

4. Conclusion

GitHub Actions helps in setting up automated pipelines and running various workflows, including regression tests. It triggers workflows on multiple events, such as pushing code to the repository or merging code into the main branch through a pull request. These workflow runs deliver speedy feedback on the build, enabling us to take corrective actions before releasing the application to production.

The source code used in this tutorial is available over on GitHub.