LINE Engineering
Blog

Agile Testing Days 2017

Marc Lihan and Kent Han 2018.07.20

Marc and Kent are SQA engineers in LINE Taiwan. They are in charge of quality assurance and delivery management.​

Introduction to Agile Testing Days

Last winter in 2017, we—Marc and Kent, QA engineers from LINE Taiwan office—had an opportunity to attend one of the most well known software testing conferences in Europe called Agile Testing Days 2017 in Potsdam, Germany. Agile Testing Days or ATD has been hosting this conference every year for almost a decade and the community grows each year. It was a four full day conference with over 600 attendees, 160 sessions and 130 speakers. The conference week was jam-packed, that at times there were at most 8 parallel on-going sessions, which makes it difficult for us to decide on which track to attend because there were so many interesting topics.

Opening 1 Opening 2

There were a wide variety of sessions and tutorials which included topics about test automation, DevOps, best practices in testing and agile development. Before attending the conference, we had a little idea on how software testing is being done on the other side of the world. By attending the conference, we looked forward to learning the testing trends, major development process and popular testing tools in Europe. And lastly yet equally important, get to know people who has similar roles as ours. Here are a few sessions that gave us a lasting impression from the week-long conference.

Opening 3 Timetable

Conference Details

There were many talks and workshop in this conference. Let us go through some interesting topics.

  • Continuous Integration and Delivery
  • Altrun - Mobile Automation Framework Based on Appium
  • Regression Testing
  • Exploratory Testing
  • LEGO Product Exploration
  • Continuous Integration and Delivery

The first keynote was presented by Jez Humble, the co-author of the bestseller book, Continuous Delivery. In this keynote, he mentioned many excuses why companies and teams cannot do Continuous Integration and Continuous Delivery (CI/CD) well.

keynote

For the excuse—"We're not building websites—he gave us a positive example about the project 'FutureSmart' in HP, which was about upgrading firmware for laser printers. Before the change, developers wasted a lot of time working on porting code and manual testing. They could not invest time for innovation. Moreover, the release cycle was long due to manual regression testing. After the change, they tried to reduce the development effort by reducing hardware variation, creating a simulator for testing, and implementing test automation. The outcome was good. The development process was improved and developers had been given time for innovation.

Jez highlighted, no pain no gain. In total, 23% of resources was paid on implementing and managing test automation. We should focus on long-term success.

Altrun - Mobile Automation Framework Based on Appium

The presenter Ru Cindrea is a test consultant in Altom, a consulting company in Romania. Altom provides testing service to their clients, like functional testing, performance testing, and automation testing. Besides using the Appium framework to build up automated testing on mobile devices, they tried to build up a tool, Altrun in the middle layer, to handle the details of Appium.

Altrun

Altrun is developed in D language. It provides:

  • Appium server management
  • Multi-platform drivers
  • Reporting feature
  • Interfaces to collect screenshot, logs, and videos

Since Android and iOS have native differences, people experience difficulties handling devices on different platforms. With Altrun, you get a common CLI and API for both platforms. The best part of Altrun is that you can define the threshold in configuration files, which makes Altrun trigger testing on matching devices. For example, the battery should be above 40%. Ru also mentioned the importance of Page Object Model. With this model, test cases can be more maintainable.

After her presentation, I talked with her for a while. I think if they open Altrun as an open source project, it will be very popular. Altrun will help lower the entry point for beginners in Appium development.

By the way, one news estimates the global test automation market will rise at a solid 15.4% CAGR between 2017 and 2025, to be worth $109.69 billion (USD). It reminds us we should learn and practice more about test automation since it's a powerful weapon in our career.

Regression Testing

This session was very interesting, especially with so many questions asked by the audiences in Q&A. This session was shared by Carlo Van Driel and Marc Van Den Berg, starting with rock-n-roll music as the theme. Presenters boldly stated that "Regression test is dead. There is no more regression test.", which drew attention from the audience.

Regression Testing

The presenters shared a meaningful example; car maintenance. Have you ever got a bill with prices listed for the parts of your car checked by a mechanic? Some of us would be dismayed at the number of defects in our car and wouldn't want to accept the bill. Let's compare this to regression test in which the QA has to check all the parts of the service. Is the cost for testing ignorable? It seems regression test cannot be ignored. We need some kind of tests to ensure that bug fixes don't cause any side effects. It makes us assured with the service quality.

Regression Testing Example

The following is a list of tasks suggested by Carlo and Marc.

  • Unit test everything
  • Check the code change
  • Use variable test data
  • Use automation test
  • Do exploratory test
  • Think before you test

In the end of the presentation, the audience asked many questions. Seeing the active interaction, I thought to myself that most QA engineers have the same pain point in having to run an endless cycle of regression tests before every service release. It's boring and unnecessary. If developers and QA engineers work closer, QA engineers can be aware of more details about code changes, can design the scope for tests, and shorten the working hours on regression test.

Exploratory Testing

What is exploratory test? Is it related to ad-hoc test? Or monkey test? By Anne-Marie Charrett's hosting, I got the chance to experience it. She called it a sapient testing. It's iterative and like a discovery process. Each team was given a Big Trak - a programmable electric vehicle. The challenge is to find out what the function key x2 represents.

Exploratory Testing

The process is like the experiment we did in high school. You make some assumption, design the steps, and verify the result to approve your assumption. Team members design the steps together. After that, one member executes the experiment, and another one writes down the results. After 30 minutes with one-by-one experiments, each team knows exactly what the x2 represents. Anne gave us a bigger version of the experiment. The key x2 represents something different to our expectation. It's surprising that none of the teams found out what it meant. I think the reason for it is because we are stuck to routinely ideas and our repetitive discovery process didn't run fast enough to find out the real meaning of the key, x2.

For me, the process was like playing a toy with my brain. Every participant enjoyed the workshop a lot. It sure was more fun than handling routine procedures.

Before service releases, QA engineers are responsible to run many test cases. We all know that test scripts have limitations. Some issues will not be found out by these routine test cases. We need to allocate time and resources to run the exploratory test. It helps us to expand the coverage, find new bugs, and give feedback to test script.

LEGO Product Exploration

This workshop was hosted by Anton Zotin. All attendees were separated into groups. The purpose of the workshop was to practice "team innovation". He told us the basic steps of team innovation: build, explain, and merge. The attendees were to "build" up an artifact or ideas, "explain" to others, and team "merge" the final result. The first task was to build a duck with LEGO blocks. Each member was to build a duck by themselves, and explain the reason for building it the way they did, within the time allocated; everyone was given the same period of time. In the end, all the teams presented the "team duck" which was composed of members' ideas.

Lego

After the practice session, we learnt how to run the process. The second task was to design a persona and product features for an imaginary service: smart diaper with sensors. Luckily, I'm a newbie father, which was why I was very interested in the topic. Many ideas popped up in my head; poo-poo and pee-pee detection, alerts in app, baby movement detection for you to know when a baby wake up and so on.

The details of our persona was this. Michael, a software engineer works remotely. The problem is that he cannot focus on the work since he has to take care of his baby. He loves to buy fancy gadgets so he might be interested in the product we propose. After each team presented their persona and feature sets, Anton asked us whether we have similar workshops for our own projects or services? If you do, it really makes you know more about your target customers. Team innovation can help improve your project. Besides all feature requests are from planners, developers and QA engineers can contribute in feature planning and development. The steps, build, explain, and merge can be applied to any kind of team innovation, like Scrum's retrospective meeting. Teams should merge action items from many ideas after the retrospective meeting.

Lego

No more flaky tests

If you perform automated tests, then you are most likely to experience flakiness in your tests. Flaky test is basically a term given to tests that gives inconsistent results after several reruns without changing test code. For example, the first time you run a test, it passes, however after running the test more it fails from time to time. One of the toughest tasks of being a test automation engineer is not creating tests, but actually maintaining them. The reason being that a lot of tests are flaky and you would need to constantly update these tests on a regular basis.

There are several ways of identifying whether your test is flaky or not. Based on Emanuil Slavov's slide below, running the test for 100 times in a loop and if it fails even once, exit the loop and the test is considered flaky.

Flaky Test

The top three most common causes for flaky tests are as follows:

  • Architecture, testing directly on UI level instead of testing on API level
  • Shared resources
  • Handling of async operations

Things to do in order to avoid flaky tests are as follows:

  • No record and replay
  • Don't use Xpath locators
  • Smart wait for UI elements
  • Wait for all Ajax calls completion

Fortunately, I bumped into Mr. Slavov while eating lunch and was able to chat with him more. He has a lot of insights on doing automation the right way which are beneficial to project development. He also recommended to investigate flaky tests immediately and collaborate with developers more closely.

Mr. Slavov

Rethinking Agile Leadership

Based on the colors defined in the book called Reinventing organizations by Frederic Laloux, the speaker, Andrea Provaglio talked about the colors that represent Agile. The book describes how companies evolve, throughout human history, to different organizational paradigms and each paradigm is represented with colors ranging from Red to Teal. The speaker discussed briefly what each color represents and provided some examples, however I'd recommended you to read the book.

Agile color chart

In Andrea's session, Rethinking Agile Leadership, he applied organizational paradigm defined in the book into the characteristics of Agile. What is the color for Agile? A lot of companies have applied Agile already on their day-to-day work. In an Agile team, there is always a scrum master who facilitates and manages the process of how information is exchanged. Scrum master is actually a servant-leader whose focus is on the needs of the team members, with the goal of achieving results in line with organization. Imagine in 'Amber organization' such as the army, leaders state what everyone needs to do, orders need to be strictly followed. In 'Teal organization', self management and self awareness are the key. It is a decentralized structure consisting of small teams that take responsibility for their own organization and how they interact with other organizations. Members in the team are not instructed by orders from the hierarchy but instead they listen to the organization's purpose and decide actions on their own accord. The structure of Teal is characterized by rapid change and adaptation, as adjustments are continuously made in order to better implement the organization's purpose. Indeed, Agile sounds a lot more like Teal organization.

Agile Presentation

Continuous Deployment with Jenkins and Docker Swarm

The author of the book and the series "The DevOps Toolkit", Viktor Farcic hosted a tutorial on how to run a continuous deployment with Jenkins and Docker swarm in AWS. He is a funny guy with a lot of sarcastic comments. His slides and commands can be found here(http://vfarcic.github.io/jenkins-docker/workshop.html). His tutorial took over three hours and we didn't even finish his entire slides. The session started with over 20 people, in the end there were only three people left. The content was probably too technical for the audience or they didn't want to register their credit card in AWS. When creating an account in AWS, you need to register a credit card, however AWS will not charge you anything as long as you clean up the servers before the trial period ends.

Jenkins

The entire tutorial session was practically a hands-on session on how to setup a CI/CD with Jenkins and Docker. Although setup may sound simple, he wanted to let his audience learn more than just setup. He introduced several ways of implementing it in order to prove the following concepts.

  • High availability
  • Fault tolerance
  • Unexposed ports
  • Fully automated

More than that he also covered how to manage Docker images in Docker hub, setting up Jenkins Pipeline. Also running functional tests that is immutable and running inside a Docker registry in order to prove that he can run services under test anywhere and with any number of instances. Lastly, he talked about how to deploy into production as frequent as required, perform rolling updates without downtime and performing rollback. He wanted to discuss pipeline in a Version Control System (VCS), however, we ran out of time. More topics are discussed in detail in his book.

Conclusion and Feedbacks

People from Various Countries

In this conference, we met a lot of people from many countries, including European countries like German, Russia, France. However, we also met people from the Middle East like Israel. Even though we don't speak the same language, we instantly understood each other when discussing testing tools and Agile concepts. Of course we used English, our second language, to communicate with each other. It's a great feeling to know that we have a common understanding even though we live far away from each other.

Automation is Encouraged

We saw a booth that promoted institute for QA engineers. They provide the learning courses for Agile processes and automation skills. We can see Agile and automation becoming trends in Taiwan, too. However, it doesn't mean 100% of automation testing is really successful. One QA engineer from Trivago told me, his daily job is writing some automation, finding bugs manually, and managing the release. It's actually very similar to our life in Taiwan office.

For the Future

It was an excellent experience to know what QA engineers in other countries are doing. We learned a lot from each other through several sessions, brainstorming, and even by just having lunch with them. From this conference, we recognized what we are doing well, and where we can make some improvements.

QA Testing DevOps CI/CD Appium Agile

Marc Lihan and Kent Han 2018.07.20

Marc and Kent are SQA engineers in LINE Taiwan. They are in charge of quality assurance and delivery management.​

Add this entry to Hatena bookmark

Back to blog list