e-commerce cloud can’t perform well if you can’t measure it

Today's e-commerce systems don’t resemble those from the past. They not only provide multimedia content and allow you to place orders and payments online, but conform with user experience.

Designed as Single Page Application, in accordance with the principles of Progressive Web Application, they are resistant to the most important security threats (TOP 10 OWASP) and are highly responsive. The difficulty of developing such applications is growing - as are customers' expectations and time pressure for fast delivery of business functions. How to ensure the highest performance of such applications?

Performance is a feature

The failure of many complex projects stems from neglecting performance. It doesn’t exist directly in the sprint backlog, it isn’t perceived as a feature of the system, but the team and the client treat it as a obviousness. That is why it is easy to disregard or omit this aspect when the client requests for further system development, while adding more features will require additional work to keep up the expected level of performance.

When designing large systems, understanding the importance of performance is crucial. Taking care of it should be a part of the development sprints. The client must understand the importance of this issue and needs to treat performance as a feature. Apart from adopting the "performance as a feature" philosophy, we also need to unambiguously verify whether this task meets the DoD (Definition of Done) principle, and so there must be a reliable way to measure it.

Performance measurement methodology

Tom de Marco's famous words “You can’t control what you can’t measure” from "Controlling Software Project" book refer to the estimation of software size. If you want to control the cost of software development, you need a methodology for estimating the size of the software and be able to measure it. This is also true for performance: you need undoubtful metrics. End users most often express imprecisely: "Application works slowly" - it’s more perception than reliable metrics.

The simplest parameter to define is a server’s response time, which can be easily collected from the application logs and forwarded for statistical analyzing. To take into account the user's perception, it’s better to rely on the time of loading the application in the browser (Load Time, Finish Time). Please note that Finish Time can be significantly different from Load Time because it includes loading additional elements on the page (ajax requests, pictures, videos). This approach of methodology introduces additional elements over which we have no control, for example the results depend on the network traffic between the server and the client and the speed of the user's computer. Measuring only a response time from the server allows you to focus on measuring the application itself, which can be informative, but it will not verify why the site is loading slowly, eg due to the large number of images on the page.

Is my system performing well?

Serving e-commerce pages with an average of more than 4 seconds can have a significant impact on financial losses measured by lost sales. End users' impatience may encourage them to find another shop. The key aspect at this point is live monitoring the "health status" of the application. While de Marco suggests developing his own measurement methodology, in the case of performance verification it is not necessary to reinvent the wheel. We can use ready-made tools, such as Nagios, Grafana, Splunk, which provide many parameters - e.g., the current number of requests per second and the CPU usage of servers.

Application Performance Management standard in SAP Commerce Cloud

Measuring and monitoring is not enough when we experience regression in the application. To meet the expected high standards, Application Performance Management technology is a right solution. In a nutshell, APM is an advanced tool for software diagnostics, which introduces to the measured system so-called agents. They collect specific metrics, without disrupting the monitored environment. All data is being sent to a data logs storage, which allows you the analyze the information about the system. We have been using APM platform in B2B e-commerce solution for Inter Cars S.A. The application is hosted in SAP Commerce Cloud (former SAP Hybris Commerce Cloud), a cloud solution for applications based on the SAP Hybris framework. It uses the Dynatrace tool, a leading manufacturer in the APM industry.

After deploying the application in the cloud, the monitoring from the Dynatrace tool is ready to use. All data that would have to be painfully collected from various system resources are provided by the APM system live in the monitoring application (AppMon Client). It is also possible to work offline (data is being recorded for further analysis). There are many available parameters: from typical JVM data to very interesting information from the application development point of view, for example, which parts of the application code are the slowest one. Injecting a Javascript script into the html pages allows you to measure Customer Experience directly in the user's browser. Of course, you have to remember about the observer effect and avoid the excessive number of sensors not to disrupt the application. However, it is better to have full application monitoring, at the expense of a tiny speed loss. The application maintenance team will be able to quickly discover issues and react before end users report it.

Testing on production

The classic approach to test the performance (stress tests) assumes simulating production conditions in the test environment, generating high user traffic and then observing the behavior of the application, finding bottlenecks and fix them. But why not do it on production? This approach solves the problem of simulating production traffic to the application - customers will generate real traffic and the proper data volume. APM sensors will provide reliable information about system behavior.

Tuning Sessions - support from SAP team

Proper diagnosis of bottlenecks may require looking at the environment cross-sectionally - from the network and system layers to aspects of the Java Virtual Machine. In such situations, the expert knowledge of the SAP Commerce Cloud teams seems invaluable. In our project, technical meetings with the SAP maintenance team have shortened the repair time of defects, which is beneficial for the client. Technical cooperation of e-point, SAP and Inter Cars teams in tuning sessions ensured proper preparation of production infrastructure for the production launch.

Performance measurement between application deployments

Standardizing the testing process allows you to run the performance comparison process between successive versions of the application. The client can check the effect of adding new features against the responsiveness (the time that the user has to wait from, for example, clicking on the link until the response is rendered by the browser). In addition, the rapid development of the Hybris platform gives access to all performance improvements and new functionalities introduced when upgrading the Hybris version.

Customer satisfaction measurement

From the business perspective, unambiguous information about the proper operation of the e-commerce system is essential for sales. However, from the technical point of view, it is better to rely on the Apdex (Application Performance Index). This standard is intended to measure what is important, and therefore the satisfaction of customers. Simplifying: the parameter values are between 0 and 1 and values below 0.5 indicate poor results. Implementation of the Apdex metric is also available for SAP Commerce Cloud in the Dynatrace tool. It allows not only to track this parameter in real time, but also to tune it to the needs of your own application.

Using the above-described measures and tools, we are able to gain invaluable knowledge about how our system is currently operating, which translates directly into profits.