How Do You Measure Block Performance?

By Dr Darren Toulson, Founder, LiquidMetrix

 

Background – Impact of MiFID 2 on Equity Trading

MiFID 2, implemented in January 2018, introduced significant changes to trading rules in Europe.

Focussing on equity markets, three of the most significant changes have been

–       the closure of Broker Crossing Networks (BCNs),

–       their partial replacement with pre-trade transparent Systematic Internalisers (SIs) and

–       the ‘Double Volume Caps’ (DVCs) applied to non-large in scale trading on MTF Dark Pools.

In the lead up to MiFID 2’s implementation, participants began to wonder where the volumes previously executed on BCN’s and MTF Dark pools might migrate to. Broadly, the possibilities appeared to be a shift to more SI trading, more on-book lit trading or more Large In Scale trading happening on specialised block venues. Whatever the case, it was anticipated that many existing methods of trading may be disrupted – either directly or indirectly – as certain existing trading options could become limited.

In 2018, Buy and Sell side firms need to both search for and follow shifting liquidity wherever it may go while also taking into account likely impacts in the function and performance of existing algorithms or methods of trading. For instance, how will dark liquidity seeking algorithms be affected by instruments being suspended from trading in small sizes on MTF Dark Pools due to the DVC breaches that triggered in March 2018? Furthermore, how might that affect the choices firms make when routing orders?

Amidst this uncertainty, MiFID 2 additionally introduced more stringent criteria for measuring and ensuring Best Execution for both Sell Side and Buy Side firms (RTS 28). Under RTS 28, firms are required to publish lists of the most common Brokers/Venues that they route their orders to and, perhaps more significantly, show evidence of regular ongoing procedures and analysis justifying the choice of Brokers/Venues used.

The challenge facing firms in 2018 is therefore not just where to source liquidity amid a potentially fluid trading landscape, but also how should they assess (and justify) the performance of these choices to ensure their Best Execution obligations to clients.

This article will focus on how to monitor, measure and quantify the benefits of trading on a block venueand ways in which the potential benefits and performance of trading blocks can be quantified versus other ‘algorithmic’ ways of trading. If MiFID 2 eventually leads to more block trading, as many participants expect, it will be important to update TCA toolkits to include accepted methods of benchmarking block trading.

We will illustrate the techniques we suggest by looking at 12 months of Buy Side block submissions / fills and cancellations submitted to the Turquoise Plato Block Discovery™ venue. In doing so we will attempt to quantify the benefit that members were able to obtain by trading blocks on the venue versus other modes of trading.

 

Data Set Used

This analysis is based on a 12-month data set of complete order audit histories supplied to us directly from the Turquoise Plato Block Discovery™ platform. The dataset is a subset of all Turquoise Plato Block Discovery™ trading activity, relating to a number of specific Brokers who each consented to provide a pool of anonymised data for the study.

The data received contains full details of order submissions, amendments, cancellations and fills, allowing us to not only analyse what happens when block orders fill (as measured in most typical TCA scenarios) but also to measure what happens when orders only partially fill or do not fill at all. Due to relatively lower fill rates achieved in Blocks – as opposed to fixed participation style algorithms – it’s important that non-execution consequences are considered when comparing block versus algorithm performance.

The graphic overleaf shows a summary of the orders analysed with total submitted monetary values (filled in orange, unfilled in blue) and trends over the 12-month period (October 2016 to October 2017). Both submitted and filled values have increased steadily over the 12 months analysed, with monthly submission/fill rates more than doubling over this period.

 

 

We also show a breakdown of orders split by primary market venue of the instruments traded:

 

 

From this we see that UK stocks are the most heavily traded having almost double the value of other countries.

For the rest of this analysis we will present results for orders submitted for these UK stocks only. This is to simplify analysis, as we needn’t worry about country to country differences skewing results. We will follow up on this article with a longer White Paper in which we analyse full pan European results.

 

Ordersizes

The orders in this data set represent a range of small to very large order sizes. The pie chart below shows the breakdown of orders into small (<1%), medium (1-10%) and large (> 10%) ADV.

 

 

As the purpose of this article is to examine metrics for large block sized trading we will exclude the small orders (<1% ADV)in all but fill level analysis.

A final point to note , the order audit histories are complicated by the fact that order ids are changed frequently in response to firm-up messages or fills. Where possible we have stitched together orders we believe to be part of the same history, however, we probably have overly granular number of orders meaning that some statistics such as fill rates are being underestimated here.

 

Block Performance Metrics

One approach to measuring the performance of block trading is to think of the typical aims of a participant using a block venue and how we can measure if those aims are being fulfilled by a particular block venue.

Below we will split these aims into 5 categories, essentially Fill Rates, IS Performance of Filled Orders, Opportunity Cost of Unfilled Resting Orders, Fill Level Gaming/Information Leakage and finally bringing this all together in a Comparison of Block Performance with Algorithmic Performance.

 

We will start with the most basic question:

 

1. Am I Getting Filled?

One key aim when using a block venue is to get large volumes filled, so a crucial metric is how often a block trading strategy leads to orders being filled, especially if orders are left resting on the venue for appreciable amounts of time. We will consider two sets of measures:

  • Percentage of Value Filled versus Value Submitted. This is an intuitive measure and corresponds to what a participant might view as a ‘fill rate’. However, it takes no account of how large orders submitted are, how long orders rest on a venue or how much trading volume happens on lit markets whilst a block is resting. As a result, its use as a comparative measure for different blocks submitted to different block venues is more limited and may give misleading results.

 

Effective Percentage of Trading Achieved. This measure normalises the rates of getting filled on a block venue by comparing the volume filled on the block venue versus the amount of market wide lit market volume that is executed while the block order was resting. It thus allows us to compare this ‘effective participation rate’ to the trading rates we might use in an electronic algorithm. As an example, if a total of €100 million was executed across 100 block orders on a block venue and while those 100 orders were resting, a total of €3 billion was executed on lit markets then the effective lit participation rate would have been 3.33%.

 

Results:

 

The chart and tables above show fill rates for the 12 months of study data. The headline averages are that:

  • 47% of orders had at least one fill associated with them before cancellation
  • 36% of the total submitted order value was filled
  • This represented an effective 3.02% lit participation rate when comparing market volumes that executed versus block volumes that executed whilst orders were resting.

 

 

It is interesting to note that both fill rates and effective market participation rates more than doubled over the 12 months of the study in the lead up to MiFID 2 implementation. By September 2017, around 2% of order volumes were filled leading to an effective lit market participation rate of around 4.7%.

 

2. Implementation Shortfall (IS) Cost for Filled Orders

Generally, the aim for any block order should be that the average execution price achieved is close to the market mid-price at the time the order was first submitted. This measure is in fact the commonly used IS (Arrival Mid) TCA benchmarkand expresses in Basis Points (BPS) the difference between arrival mid-price and the filled block price. Thus a negative IS number means the price achieved was worse than the price at arrival.

 

Results:

 

The results above show that for filled / partially filled orders the performance of the filled blocks versus arrival mid-price is extremely good with the prices missing benchmark by just 1 BPS. For context, the value weighted top of book spread of the instruments traded for these orders is 9.65 BPS, so the IS performance means that the average achieved block prices are well inside the lit market BBO spread at the time of submitting orders.

This performance is fairly consistent over the 12-month period with monthly average IS figures ranging from –3 BPS to + 0.5 BPS (i.e. always within market BBO) at time of order submission.

 

 

3. Opportunity Cost for Resting Orders

Measuring IS costs for fully / partially filled block orders often presents a rosy picture. It is usual that observed block IS costs are significantly lower than IS costs of similar orders traded algorithmically.

However, to give a more complete picture, we need to consider costs associated with block orders that do not fill. We need to consider to what extent, if any, the market might, on average, be moving disadvantageously in the direction of our unfilled orders, whilst we are waiting.

For instance, if we submit an order to buy 1 million shares in a stock when the market mid-price was 100.0 and three hours later the order is cancelled (unfilled) but the market price had risen to 102.0, then assuming we still need to fill this order, we have suffered a 200 BPS ‘opportunity cost’ whilst waiting.

 

Results:

The table above shows market movements (in BPS) between order submission and either fill or cancellation. A negative number means the movement is disadvantageous, i.e. that the market price has risen for a buy order or fallen for a sell order. We have split the results into orders that fill completely, fill partially or (most commonly) do not fill at all.

The striking point is that the market moves most strongly in an unfavourable direction for unfilled orders, whereas for partially filled orders the effect is smaller. For filled orders, the average movement is actually very close to zero.

The trend charts below show that these numbers are fairly stable month on month.

 

 

The key take-away from these numbers is that for unfilled orders, the market mid-price moved on average 9 BPS in an unfavourable direction whilst the orders were resting. So, the average ‘cost’ of resting on the dark pool unfilled (assuming, that the order still needs to be filled when it is cancelled) is 9 BP.

There is an important caveat to these numbers:

Because Turquoise Plato Block Discovery™ allows contingent orders (i.e. orders where volumes are ‘firmed up’ just before actual matching) it is possible for participants to trade part or even all their orders in parallel on lit or other dark venues whilst resting on Turquoise Plato Block Discovery™. In such cases the 9 BPS ‘penalty’ is not really relevant as resting on Turquoise Plato Block Discovery™ was not at the expense of pursuing other execution strategies. In fact, it may be that the 9 BPS movement we see for unfilled orders reflects the fact that some of the members were indeed executing volumes on lit venues in parallel to resting blocks (in the hope of a large fill). However, for participants that do not execute elsewhere in parallel to resting on Turquoise Plato Block Discovery™, this 9 BPS, if it applied to their orders, would be more of a concern.

 

 4. Am I Getting Fair Fill Prices and Are the Fills Leaking Information?

One of the attractions of using a dark pool is that there should be little information leakage whilst resting on the pool, even after partial fills occur. For this to be true we assume that our fills are being matched with relatively benign counterparties and are executed at exact market mid-price, so the public print of the fill will give no directional information to the wider market.

Also, assuming that the platform has anti-gaming measures in place, we would hope that any fills we do get are at ‘fair’ market mid-prices. To be considered ‘fair’ fills should not occur (at least not very often) at times when the primary market mid-price is skewed compared to wider EBBO market price. Also, market mid-prices should not show signs of being manipulated just before or after the fill, implying our counterparty was trying to game the price at which we were matched.

Some metrics that can help us monitor if any of these unfavourable outcomes are occurring at the block fill level are:

  • %EBBO spread capture– If matches are occurring at ‘neutral times’ then we would expect on average to receive prices that are at EBBO mid (i.e. 50% EBBO spread capture). If we have a number consistently below 50%, it implies our counterparties are outperforming us (either innocently or by design).

 

  • %EBBO outliers– This will count the number of times that our filled price lies outside of EBBO. If a significant number of our fills are happening outside of EBBO then we might worry that counterparties are timing orders to pick off advantageous prices at these anomalous times.

 

  • Market Mid-Price movementsin the milliseconds before and after a fill. From both a signalling and gaming perspective, we would prefer our fills to be happening at times that market mid-prices are stable.

 

Results:

 

 

The tables and charts overleaf summarise fill level performance for the 83,615 Fills in this study. The key points are:

  • The vast majority of fills happen at EBBO mid-price as evidenced by the EBBO spread capture histogram showing a clear spike at spread capture 50% and the overall average EBBO spread capture being 50.07%.

 

  • Only 0.19% of fills were outside of market EBBO. Also, the value ‘lost’ by those fills is minuscule at 0.001 BPS, indicating that even when fills do miss they are only marginally outside EBBO.

 

  • In terms of price movements following publicly reported fills, about 19.5% of fills were followed by a movement in market EBBO. This is low (for instance, the corresponding number on dark pools such as XUBS, CHID, BATD etc is more like 40-50%) but we do observe slightly lower numbers on pools such as LIQU (closer to 10%).

Based on this we can surmise that fill level execution on Turquoise Plato Block Discovery™ shows no real evidence of gaming primary mid-prices versus EBBO prices (low outlier rate, near 50% spread capture) and relatively low levels of market price movement following fills indicating little information leakage.

 

5. How Much Better (Worse) Off Was I Using a Block Strategy?

The metrics we have looked at so far, give us a good idea of how often we are getting filled, how close our filled block prices are to arrival prices, what kind of opportunity costs (adverse price moves), if any, we are suffering from whilst resting and whether the dark fills we get are at fair market mid prices and not leaking significant information to the market. These metrics taken together will be particularly useful when comparing execution quality across different block trading venues to find which venues give the highest fill rates and cleanest (best) execution.

However, these numbers alone make it difficult to compare the results of trading on a block venue versus using an algorithmic strategy. We can compare simple Implementation Shortfall (IS) of filled blocks with expected costs of executing similar orders using algorithms, but as we mentioned earlier, this is likely to flatter block trading performance as the instances in which blocks fully fill is the ‘best case scenario’ for block trading.

We suggest three different measures that will attempt to quantify the benefits of block trading versus sending the same order to a participation style algorithm for execution.

 

  • Block Benefit for Filled Orders– Here we measure theobservedIS outcome of filled blocks versus the estimatedIS cost of executing an equivalent order algorithmically. So, if a block order fills with an IS of -2 BPS and we estimate that an equivalent order sent to an electronic algorithm would have cost 15 BPS, we say the block provided a 13 BPS benefit for this order.

 

  • Block Benefit Averaged Over All Orderstakes the measure above but dilutes it based on how often blocks actually fill. This provides an indication of the average benefit of block trading across both filled and unfilled orders. So, if the average block benefit for a set of filled orders is 17 BPS but only 3% of orders get filled then we would say that the average benefit across all submitted orders would be 17 BPS * 3% = 0.51 BPS.

 

  • Opportunity Cost Adjusted Benefit is the toughest measure. In this measure we calculate the expected costs ‘with clean up’ of all submitted block orders. ‘Clean up’ means that if there is any volume left unfilled once a block is cancelled, we will assume that the remaining volume is submitted to an electronic algorithm whose cost will be estimated in the usual way based on the remaining volume to be filled. However, in addition to the clean-up cost there will also be an additional cost to reflect any negative (or positive) market price movements between the time the order was first submitted and the time the order was cancelled. These will be referred to as ‘Opportunity costs’. The fully cleaned up block order cost is then compared to the expected cost of executing the order purely algorithmically from the time of the order submission to give an overall benefit (penalty).

 

Results:

 

Block Benefit for Filled Orders

The 13,563 orders that were either filled or partially filled achieved a performance of -0.99 BPS versus arrival mid-price. If we simulate equivalent orders (based on filled volumes) submitted in the same stocks under similar market conditions, we estimate on average the algorithmic orders would have had a shortfall of 20.74 BPS versus arrival mid.

The benefit in terms of improved IS for FILLED block orders is therefore around 19.75 BPS. In terms of value this is a benefit of around €9M on the 4.5BN filled.

 

Block Benefit Averaged Over All Orders

Around 1.36% of order value submitted was filled. If we assume that the benefit of block trading is 19.75 BPS for orders that fill (from above) and is neutral (no gain or loss) for unfilled orders then the average benefit across all orders submitted (filled, partially filled or unfilled) is about 0.27 BPS.

This gives a more realistic idea of the net benefit of submitting block orders across both filled and unfilled orders.

 

Benefit Assuming Opportunity Cost

The table above summarises the total costs including clean-up costs of orders submitted to Turquoise Plato Block Discovery™. The three components to the cost are:

  • IS Contribution Based on Filled Blocks. This is based on 1.36% of order volumes filled with an IS of 0.99 BPS. So, this contributes 0.01 BPS to the overall cost.
  • The remaining unfilled order volume suffers a market movement (opportunity) cost that contributes 7.85 BPS to overall costs.
  • Finally, the cost of ‘cleaning’ up by trading algorithmically adds an additional 22.6 BPS to overall cost.

So, the overall cost of the Turquoise Plato Block Discovery™ orders including opportunity costs and clean-up costs for unfilled orders is 30.46 BPS.

This compares unfavourably with the estimated cost of 23.01 BPS that would be incurred if the orders had instead been sent directly to an algorithm for execution.

 

So, in summary

  • For Turquoise Plato Block Discovery™ orders that fill, the IS cost observed is much better than the cost of executing equivalent orders algorithmically. The benefit is 75 BPS

 

  • If we dilute this observed benefit for filled orders to also consider times that no fill (and hence no improvement) is achieved, we get an improvement of 27 BPSover all order value submitted.

 

  • If we assume that orders posted on Turquoise Plato Block Discovery™ are not traded in parallel on other venues / strategies and that upon cancellation of an order, any unfilled volume must be ‘cleaned up’ by sending it to an electronic algorithm, then the costs of the orders are actually higher than the cost of executing the original order algorithmically. This is principally because market mid prices appear to move unfavourably whilst unfilled orders rest on the dark pool.

 

The conclusion is therefore that block trading on Turquoise Plato Block Discovery™ offers the ability to significantly reduce IS costs of orders that fill on the venue. The caveat is that this benefit is cancelled (and reversed) if participants were to submit orders and not take advantage of the ability (through conditional orders) to trade the orders in parallel electronically.

So, one way of seeing this would be that a firm not currently using the Turquoise Plato Block Discovery™ platform and executing purely algorithmically, might expect an improvement of around 0.27 BPS across all orders if they were submitted to Turquoise Plato Block Discovery™ on a conditional basis alongsidetheir existing strategy.

 

Note on Estimating Algorithmic Execution Costs

The analysis above relies on the ability to estimate the cost of an order if it were submitted to an electronic algorithm. LiquidMetrix has developed a means of doing this by taking a broad set of attributes of an order and the market conditions in which it is to execute (%ADV, instrument spread, price volatility/momentum, market volumes, etc) and from this, identifying 1000 similar orders from our database of tens of millions of algorithmically executed orders. With these 1000 matches of usually very similar orders, we can infer the likely IS cost / distribution of the order we are analysing.

This is the method we have used to infer expected costs of algorithmically executing the orders and the costs of cleaning up unfilled Turquoise Plato Block Discovery™ orders with an algorithm.