QA Automation Pipeline with CI/CD – Cucumber + Xray (Jira) + Selenium (Ruby) + Jenkins

With the popularity of Continuous Integration and Continuous Delivery practices, automated testing is gaining an increased slice of the overall testing specification. This is no surprise since automated tests provide great benefits over manual testing. It makes the testing process faster, systematic (thus avoiding errors), and reduces the effort associated with the execution and validation.

In this article, we would like to introduce to you an automation test framework developed by MeU focusing on two things: the test automation framework and continuous integration – This QA pipeline has helped us shorten delivery cycles and bring higher quality software and also has the ability to track QA/Feedback from testing directly in Agile boards, so all team members (tester, developer, PO, …) have visibility over the issues’ progress from a quality perspective immediately. In this framework, you can automate your tests using XRay (Jira), Cucumber, Selenium (Ruby), and Jenkins together for the project which written in Rails.

Let see the role of each tool below

• JIRA is used for tracking requirements and bugs.
• Xray for JIRA is a complete Test Management tool for JIRA. It is a full-featured add-on for JIRA that does not require any other software in order to run. In Xray, Automated Tests are executed by an external tool, which controls the test executions and the comparison of actual outcomes and the predicted ones. As the testing tool, we will use Cucumber because it allows you to execute plain-text functional descriptions as automated tests. You can find more information about how automated tests are created in Xray, as well as more information about Cucumber at below, if you want to know more detail, you can find it in Xray documentation and Cucumber’s website, respectively.
• Cucumber is a framework for behavior-driven development (BDD), where the test specification is done in a natural language, while the implementation of each specification’s phrase (i.e. steps) is done in code (e.g. Java, Ruby or some other language).
• Selenium – a framework for UI automation tests and web browser interaction,
• Jenkins – it is open source, allows you to run any kind of tests, and integrates with almost anything you can imagine. It has one of the biggest open source communities, and they bring Jenkins to new levels year after year.

Here is an overview of our model.

The BA (Business Analyst) will give the requirement to the test manager. The test manager will create new cucumber tests then create test execution on Jira. Testers will download cucumber files to the automation project then they will develop tests and commit the code to source control like Github or SVN,… After that, they will create and configure jobs for these tests on Jenkins to execute tests. According to our model,  after execution, the report will be imported back to Xray (Jira) – a test management tool. And all tests are running in headless browsers (PhantomJS (old), Chrome or Firefox headless mode,…). In addition, the test not only runs on localhost it also runs a cloud node (such as Saucelabs).

Let’s see how Cucumber works in a simple workflow

In this workflow, Tests are created and managed in Jira, thus Jira will be the master for the Cucumber scenarios.

1. Specify Cucumber tests using natural language, in Jira.
2. Export Cucumber features from Jira to the CI environment, using the Xray REST API.
3. Implement tests in code and commit them to the source code versioning system.
4. Execute tests in the CI environment.
5. Report results to Xray, using the Xray REST API.

That’s how it works.

Check out our video below to see How to Create a Cucumber Test with Xray

In our demo, we were just exporting Cucumber features manually. You can use the Xray for JIRA add-on for Jenkins it can help to simplify the process of continuous integration. With the setup of some Xray specific tasks, you can easily export Cucumber features, report automated testing results for many different automated testing frameworks and associate them with new or existing Tests, Test Executions, Test Plans, Test Environments, and more.

Check out our video below to see How to run Test Execution and Importing Test Results in JIRA

If you love our model and want to know more about the demo, please contact us via contact form.

But to understand more about this model, know how we select these tools for our solution, what are the advantages of tools, you need to have knowledge about technical, tools as well as frameworks which are used in our automation framework besides the Ruby language.

Let’s get started with Behavior Driven Development (BDD).

So what is BDD? We will clearly learn through the below picture

According to the above example, we find out the communication in project development is very important in project success. Communication is not just about describing your ideas to others, but you also have to get feedback to make sure that a person understands your idea. This is also the reason why the agile software development team has learned how to work in small steps, thus increasing the communication and feedback of stakeholders. Like the example above, developers have spent the whole time building a lot of code on the wrong idea that they think is the right idea.

BDD was created in order to focus on developing business values through a combination of information technology experts and business people, customers who are interested in or have the idea of building the project in the direction Information Technology by providing a common language – a structured language used by all members to connect all group activities with software (or with stakeholders). This reduces the division of the gap between business and technology. BDD helps teams focus their efforts on identifying, understanding and building valuable features that are important to businesses and ensuring that these features are well designed and implemented. BDD tools can help turn these requests into automated tests that help guide developers, verify features and record what the application does. BDD is not a method of software development in its own right. BDD combines, builds and enhances ideas from many of these methodologies.

BDD is developed based Test-Driven Development (TDD) and agile development method. TDD reverses the process of building a normal project and testing it later. In TDD, test scripts will be written first, then how to build project code to successfully pass the previously written scripts. Agile is a method of flexible software development through each stage with the aim of matching the changing of the project, you can also understand that agile method is the way to make products come to life as soon as possible. BDD is the perfect combination of TDD and Agile. Instead of customers, business stakeholders transfer requests to the development team or BAs without the opportunity to respond, BDD ensures that business partners respond to each other so that software designers can respond to actual demand.

Why? Why we should use BDD for our project?

According to the traditional way of working under the workflow above, we can see that there are a lot of lost, misunderstood, or possibly missing information that is transmitted, misunderstood, or have a lot of release modules for guests, that are not in line with the needs or the original documentation requirements.

In a BDD group, business analysts, developers, and testers collaborate to understand and identify requirements together, They use a common language that allows an easy, less path, less more vague from end-user requests that can be used, automated testing. These tests determine how software should work and they guide developers to work on software that focuses on really important features for Business.

Compared to the traditional scenario, this scenario uses many conversations, thus reducing the amount of information lost. In this way, a lot of ambiguity in translating the customer’s initial requirements into code, reports, and documents is eliminated.

From the above two examples, we can answer the question: why do we use BDD?

• First of all, BDD is to match the required of customers or business stakeholders about building the application: Documents written in natural, common language that all participants can understand. When reading a document, customers can understand whether software developers have met their requirements or have some changes.
• Live documents: Why is it a live document? Because common documents are written and shared with stakeholders, but with BDD, they are read and shared and stored on the computer at any time to tell you how exactly they are. In fact, instead of documents that are written once and then gradually outdated, it will become a creature that reflects the true state of the project.

Therefore, when using BDD, the software is developed towards behavior that always pays attention to creating products in accordance with customer needs, thereby improving product quality, creating useful products.

What are the benefits of BDD?

Reduced waste

• BDD tool focuses on developing and exploring applications, features that bring value to businesses, avoiding creating features of inappropriate features. Once building a feature that is not suitable for the project, this effort will be wasted. Similarly, the group that writes the feature according to the needs but not the requirements of the business, the group works together to make that feature, thus wasting more and more.
• BDD helps us avoid this kind of wasteful effort by helping groups focus on features that match the business goals of the business.
• BDD also reduces wasteful effort by allowing faster, more, and more useful feedback for users. This helps teams make changes sooner rather than later

Reduced costs

• The effect of reducing waste is reducing prices. By building features that fit the needs and requirements of the business, without wasting the effort of small features, businesses can save a large amount of money in the process of providing a product to end users by the business. and by improving the application coding process, you can save the cost of fixing errors and costs associated with the error.

Easier and safer changes

• BDD makes it easy to change as well as expand features in applications, systems, etc. Live documents are created from performance specifications using the terms that parties familiar related. Therefore, it helps stakeholders understand the process of working.
• BDD creates a comprehensive set of automated unit and acceptance tests, which reduces the risk of regression due to any new changes to the application.

Faster releases

• These comprehensive automated tests also significantly accelerate the release cycle. Testers do not need to perform manual testing of previously tested test cases, instead, we can use automated tests to start point and spend time on exploratory testing.

The disadvantage of BDD

• BDD requires high business engagement and collaboration: BDD is based on conversation and feedback, conversations will motivate the team to create a good product when understanding the requirements of the business and how they provide business values based on that request. If the stakeholders do not cooperate or 1 of the 2 parties does not participate in the conversation or they wait for the complete product giving feedback, that the effectiveness of BDD is not promoted.
• Poorly written tests can lead to higher test-maintenance costs
• Requires a certain skill and many groups start using BDD to see this is a significant challenge, especially for complex web applications.

Who will write BDD?

BDD promotes collaboration between project members as well as stakeholders. Therefore, all of these people will build the BDD file to provide the most accurate, general view of the project requirements.

How to write BDD?

BDD is written in plain text language called Gherkin.

This is the BDD writing structure, we will learn basic keywords to know the meaning and use them at Gherkin Reference

What is Cucumber?

In order to gain a better advantage in software testing in particular and in software development in general, today people take a step towards the Behavioral Driven Development project.

Behavioral Driven Development for us to create a good version for developers as well as potential customers. So initially the project, developers together and think about which test scripts will be adapted to successfully call this software/application. In this way, they give a set of scripts. experiment. All of these test scripts are in simple English language, so it also serves the purpose of the document. That is the gherkins language introduced above.

Cucumber is one such open source tool, which supports behavior driven development.

How Cucumber Works

This structure, from features down to automation library, is illustrated in the figure.

Originally, the cucumber was created as a tool on the command line by the ruby community. Since then it has been translated into many different development environments like Java, .. allowing everyone to benefit from it. When you run the cucumber tests, it will read your specifications from simple text files called features. Each scenario (including the steps of a feature written in gherkins format) is a list for cucumber to perform. In order to test Cucumber understand these features, we must follow some Gherkins rules. The features that we have provided for cucumber definition are mapped by programming code to perform any action described by this step. In the test suite, the step definition itself may be just One or two lines of authorized code for the support code library, dedicated to your application domain, know how to perform common tasks. on the system. Sometimes it may involve the use of automation libraries, such as Selenium browser automation library, to interact with the system itself.

Why Cucumber?

What makes Cucumber special compared to other support tools is that it is designed to ensure acceptance tests can be easily read and written by anyone in the project team, and the value of test tests as a tool for communication and collaboration between stakeholders. The readability of the cucumber tests helps attract the attention of the stakeholders in the project, enabling them to understand more, explore as well as their understand requirements in the project.

Due to the application and adherence to Gherkins rules – a common language, easy to use by stakeholders, it is one of the least technical tools used in project development.

Besides, Cucumber is also supported to develop in other languages such as Perl, PHP, Python, Net etc. and also used with other tools such as Selenium, Watir, and Capybara etc.

And Cucumber is completely free, it is an open source tool, used by computer programmers for testing other software.

CI/CD

Have you ever heard about CI/CD? They are common terms which are used in the software field. Today, in this article, we would like introducing to you an overview of CI/CD, so let’s get started with the question. What does CI mean?

CI – Continuous Integration is a software development practice where members of a team integrate their work frequently, usually, each person integrates at least daily – leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly – Martin Fowler

For example, developers have to push their code into the source code repository to improve their software project every day, multiple times in a day, so integrating happens continuously. For every time they commit, the CI system will build and test the software automatically. Then if there is any error, CI will send an email to the whole team, it’s easier to find where the error is and start to fix it immediately. Besides that CI also supports unit test, so the unit test will be run by the CI to make sure that the changed codes don’t interrupt the software. If somewhere fails, we will know and fix to make all test case pass as soon as possible. That’s how CI works.

How about CD? Does it stand for Continuous Delivery or Continuous Deployment?

First of all, we’ll find out what is Continuous Delivery?

Continuous Delivery is a set of techniques and tools to deploy the source code which was built and tested to the staging environment by automated. Continuous Delivery allows developers to do automated testing in the staging environment. After being tested the code is deployed to the production environment.

Same as Continuous Delivery, Continuous Deployment does anything it does, so what is different between them? how to distinguish them?

The answer is that the way it deploys from the staging environment to the production environment.

Continuous Delivery deploys to the production environment is triggered manually while the process in Continuous Deployment is fully automated. there’s no human intervention at all.

The difference between Continuous Delivery and Continuous Deployment is shown in the below picture.

Therefore, the CD is interpreted as continuous distribution or continuous deployment depending on how it deploys to the production environment.

Why we have to use CI/CD? What benefits it has?

• Reduces risk: because the code is tested and deployed frequently, an error will be detected sooner. It helps you reduce risks in your project. The error will be fixed immediately.
• Minimize repetitive processes
• Cost: Why? Because time is money, if you take too much time on finding and fixing the error, you may lose your money, so CI/CD can help you deal with that problem.
• Improved customer satisfaction: Fast release, risks are under control, no errors. Trust has been established.

Challenges

But everything has two sides, this is a reality. Besides benefits, here are a few common challenges of CI/CD.

• Technical challenges: To build a CI/CD platform, we have to use many different tools and technologies.
• Time: We need time to establish the system.
• People: To build this system, it requires people who have in-depth knowledge about CI/CD, how to use the related tools, technologies …

Tools

• Code Repositories: SVN, Mercurial, Git, Visual Source Safe…
• Continuous Build Systems: Jenkins, Bamboo, Cruise, Hudson, …
• Test Unit Frameworks: JUnit, Cucumber, CppUnit …
• Test report website: Mantis, TestLink, Redmine…
• Artifact Repositories: Nexus, Artifactory, Archiva … .v.v.

At MeU Solutions, we use Jenkins Docker, Cucumber for this article.

BDD in Continuous Integration

Seeing the value of BDD and CI / CD during the deployment and development of a software project, our company has combined the two models together to get a complete model in the operation of the company in particular as well as in the operation and development of software in general.

To create a successful application with unique ideas, convey ideas is important. Not only customers, software engineers or automated test engineers must concentrate highly to do their job well. Although there are external factors that can affect this, there are also some internal factors that need to be taken into account. In fact, the way we work can affect the quality of the results the most.

The state in which the participants complete 100% of the effort to complete a project can be called a flow.

The flow that can combine stakeholders according to the process that everyone in the group understands, the flow will become a successful workflow.

Behavior-driven development (BDD), continuous integration and continuous deployment (CI / CD) can help us complete them.

Conclusion

Xray plug-in for JIRA provides an ideal solution for test management on all projects where JIRA is used for tracking requirements and bugs. Besides, it supports management of test cases, test planning, test execution and requirements coverage with its built-in support of Cucumber and other test automation frameworks enabling support whether is needed manual or automated testing. In combining with Jenkins, we made a complete solution for test automation framework and continuous integration solution.

We hope this article was useful and gave you useful information for building your own automation tests pipeline, maybe from scratch. Now, you might want to learn more about Selenium, Jenkins and automation tests in general, there are plenty of tutorials you can find and dive into. Let us know your thoughts and share your own opinion on the chosen stack of technologies!

See our case study