Machine learning in quantitative finance

Challenges of ML in Quantitative Finance

Unique characteristics of financial data impose great challenges in the applications of machine learning.

• Low signal-to-noise ratio. Financial markets are highly uncertain, and the signal/information hidden behind financial data is weak compared with the amount of noise. A three-year-old child can easily recognise numbers in images, while investors with 20 years of experience still make unwise trading decisions to suffer from substantial financial loss. The challenges caused by uncertainty and randomness of financial markets make us appreciate Warren Buffett’s success in investment and led to the recognition of Fama and French’s empirical work in factor models by the Nobel prize in Economics. The low signal-to-noise property may lead to the severe overfitting issues of machine learning methods and inaccurate prediction of the future financial data.
• Data availability. Some financial data may be limited in terms of data availability, e.g. some financial instruments exist only for a short period, which may lead to insufficient data for sophisticated machine learning techniques. Last but not least, financial data may not be stationary, which means that they may be prone to regime changes and render older data less relevant for prediction.
• Interacting systems. Financial markets are highly competitive, and people’s decisions affect each other. Even for the most skilled investors, their profitable investment strategies will be exploited by other market participants sooner or later, and those strategies will stop making money. Hence, developing a good investment strategy to make a profit consistently for a long-term run is incredibly challenging, if not impossible. This difficulty is starkly different from machine learning tasks in other domains, for example, image recognition and natural language processing. Machine learning algorithms have made impressive progress on the applications where deterministic rules exist underneath data, albeit the rules might be complicated.
• Unstructured data. An increasing amount of unstructured data, e.g., the financial news, satellite images for earth observation data or investment forum chat, are potentially valuable to provide useful information to financial services. However, traditional statistical analysis cannot process and analyse these unstructured data, which is also called alternative data.

Having known these difficulties, they are not entirely unsolvable. To deal with the low signal-to-noise ratio problem, we can apply some data processing techniques or even unsupervised learning to extract information. These methods can reduce the amount of noise. Although the financial market is an interacting system that seems to make developing new trading strategies worthless, there are ways to keep the edge longer. Some strategies require lots of input to develop, be it computational resources or unique alternative data. Those who have access to these resources can gain an edge that does not fade in a short time. That’s why hedge funds are very generous in building infrastructure, obtaining new data and collaborating with top universities, especially in the era of machine learning.