Comparison of the Results from Pattern Lab and Manual Backtest

How does the Pattern Lab’s backtester work? Why should I always validate my strategies by doing a manual backtest? You will find the answers to these questions in this article. For the comparison, we used the first version of Pattern Lab, released on 2021/12/01.

Whenever using the Pattern Lab’s backtester or any other backtester for that matter, make sure to check your results by doing a manual backtest as well. That is because of the limitations that any backtesting tool may have. The Pattern Lab notifies users whenever a candlestick pattern they chose to trade appeared on the price chart. Users receive an e-mail notification after the end of the trading day, and then they have enough time to place an order by the beginning of the next trading day.

Let’s have a look at the current limitations of the Pattern Lab’s backtester. The different results of the Pattern Lab and the manual backtest are caused either by the lack of intraday data in Pattern Lab or by the Pattern Lab’s rounding feature for detecting a specific candlestick pattern described below.

At this point, the daily data is the lowest time frame to which the Pattern Lab backtest has access. Because of that, it is not possible to open and close trades within one trading day. As a result, the Pattern Lab backtest has fewer trades executed than the manual backtest made with intraday data available. And for the same reason, the Pattern Lab’s backtester sometimes has opposite results for the same trades as the manual backtest. You can see examples of both situations in the pictures below.

In this first picture, you can see why Pattern Lab’s backtester executes fewer trades than manual backtest. While doing a manual backtest, we opened and closed a position during the same day on 2021/08/30 (green arrow). The Pattern Lab’s backtester opened a trade on this day but it closed it two days later on 2021/09/01 (lime green arrow). The outcome was that the Pattern Lab’s backtester executed one trade less than the manual backtest had. In the manual backtest, we were already opening (and closing) another trade on 2021/09/01.  The Pattern Lab’s backtaster is always waiting for all the opened positions to close before placing a new order. That is why it couldn’t open a new trade on 2021/09/01, because there still was an opened trade waiting to be closed at the end of this trading day.

In this picture, you can see why the Pattern Lab backtest does have different results for the same trades as the manual backtest. In the manual backtest, we opened and closed a trade during one single trading day on 2021/09/07 with profit. But as the Pattern Lab’s backtester cannot open and close trades in one trading day, it closed the trade on 2021/09/08 with a loss because the price hit the stop loss.

Those are the issues caused by the lack of intraday data and now let’s discuss how the rounding feature for detecting the specific candlestick pattern works. For each backtest, we used the momentum formula candlestick pattern. It is a pattern exclusive to the Pattern Lab that consists of three candlesticks, and it is defined as follows. The high of the day of the second candle must be higher than the highs of the first and third candle. That is a definition of the bullish momentum formula candlestick pattern. The opposite rules apply to the bearish momentum formula. In other words, the low of the second candlestick must be lower than the lows of the first and third candle.

When a bullish momentum formula pattern appears, the high of the second candlestick marks the entry price for a long position. For a bearish momentum formula pattern, the low of the second candle marks the entry price for a short position.

The Pattern Lab backtester rounds price data to 1 decimal place for this pattern. It does so to make this pattern more effective. With this logic in place, the total number of patterns detected by Pattern Lab and manual backtest will vary. It is something to keep in mind because fewer patterns detected may equal fewer trades executed.

What is the upgrade we prepare for the forthcoming versions of the Pattern Lab’s backtester?

We are going to make our backtester more accurate by implementing the intraday data price feed in the future. In the meantime, we will change the logic of the Pattern Lab’s backtester to reflect a real-world scenario better. This change will enable the backtester to open and close trades within a single day in some instances. If the stop-loss price is between the low and the high of the day on which we entered a trade, the backtester will automatically evaluate such trade as a loss. With the current data, we are not able to determine whether the stop loss was processed before or after we entered a trade. To be on the safe side, we chose to implement logic with a slight handicap. After this upgrade, all the backtested strategies will have lower returns. On the other hand, the very same strategies should surpass the backtest results when implemented.