Under the following headings this blog sets out an explanation of what our framework is and what it aims to achieve. It provides links to resources to access and utlise the framework, and it summarises and provides links to our NN framework research findings as we write them up.
- Introduction to the framework
- The framework - access to resources
- Investigations/Research output
Introduction to the framework
We have developed a ‘NN framework’ to introduce consistency and allow us to compare results and assess the success of different models.
A framework allows us to build up knowledge and compare aspects of the modelling process. For example, we can start to understand what aspects of a particular approach work better than others: Is it the type of Neural Network that has been used, or the way the data has been structured going into the model, or is it to do with the nature and characteristics of the data it has been applied to? We hope to build up our understanding of what models work best when, i.e. which approach to take under which circumstances.
We also hope it provides an easy way in for reserving actuaries to learn how to use Neural Networks. The framework provides an off-the-shelf package which is ready to go, saving individuals time by not having to write everything from scratch.
In addition, by providing all the code behind the models, it allows others to replicate our work and hopefully build on it. In fact, a key aim of the working party is to encourage reserving actuaries to try out these models on their own data.
NB: The framework specifically focuses on applying Neural Networks to individual claims data. The working party as a whole does undertake other research looking at a broader range of machine learning applications for reserving. These and other outputs can be found in our list of previous posts, or in our book which has collated most of our past output.
Why individual claims?
The real advantage to using machine learning for reserving is likely to be in the ability to incorporate a wider range of data into the reserving process. Using individual claims allows us to use additional information about the claim, the exposure/policyholder or the wider environment.
Aggregate triangles by definition limit us to a very small number of parameters. With individual claims, theoretically the model will be able to work out, for example, which claim types, or subsections of policyholders, have claims which develop in different ways; have different inflationary pressures, or different notification or settlement patterns.
A known issue with using individual claims is that pure IBNR or IBNYR (Incurred But Not Yet Reported) claims will not be included in the projections. Hence models applied to individual claims will only project IBNER (Incurred But Not Enough Reported) as they will only project values for the claims that are in the model - claims not yet reported cannot by definition be in the data. We recognise this limitation but still think it is worth investigating techniques that estimate IBNER at this stage. There are techniques which can separately estimate IBNYR.
Why Neural Networks?
Whilst we haven’t completely forgotten about GBMs and their like, the focus here is on Neural Networks. One reason is that because when we started investigating they were not being used as much by actuaries and we wanted to promote use of them - they can potentially provide greater flexibility than tree based methods. This article sets out this thinking Data science lab – GBMs or neural networks? – Machine Learning in Reserving Working Party
The Framework
How to access the model
Our practical introduction to NNs series of blogs in 2024 introduces a NN model. A practical introduction to using neural networks in reserving on individual claims data
We also introduced it in our 2024 IFoA Spring Conference Presentation A practical introduction to using neural networks in reserving on individual claims data
Our base NN model is a basic FeedForward Neural Network applied to simulated data. This blog steps you through what is happening in the code that runs the model How to run a basic neural network in Python: code explainer
Diagnostics
A significant part of the framework is diagnostics. When we started it seemed that the relative success of a particular ML model was based solely on the loss function; i.e. the best model was deemed to be the one with the lowest error term.
We looked into other measures that could be used to assess how well a particular model is working compared to another model, but also to gain insight and build up knowledge about what the model is doing. These findings can be found in our Diagnostics blog Neural Network Diagnostics
We are continuing to update the diagnostics and tools we use.
Hyperparameters
An important part of getting a NN to work optimally is in the tuning of the hyperparameters. This can seem daunting to the new user, and we investigated this in our hyperparameters blog on our base model Neural Network Hyperparameters. It gave us an understanding of which are likely to be the most significant parameters and what impact they are likely to have on this model. Hopefully the blog goes a long way to demystify these for the new user, as well as giving clear instructions as to how to practically apply them.
Data
We have started with simulated data - specifically SPLICE data which is parameterised to loosely resemble the experience of a specific (but anonymous) Auto Bodily Injury portfolio. It has been designed to give the user control of the mechanics of the evolution of an individual claim, hence the flexibility to create claims data of any level of complexity. For more details see here: SPLICE – Machine Learning in Reserving Working Party
We very much encourage reserving actuaries to download our code and see how these models work on their own data.
Investigations/Research output
We are building up our knowledge by testing out different scenarios, be it by seeing what happens with different data, models, or parameter selection. We have done much more than is published so far here - not everything is worthy of a writeup, and we continue to investigate in the background. For example, one area we are we are focussing on is to replicate models from existing papers in our framework. We need more manpower to achieve all we want to - please see the ‘Can you help?’ section below!
Data structures
Data Structures - Initial Findings
How you structure your data going into a model matters. This blog explains what we mean by this and summarises some of our initial findings. It shows how we have found one structure that we think is the best for now. However we think it does not pick up development patterns well and can be seen to not work well on outstanding claims (i.e. claims still open at the valuation date). We suspect using time series based NNs might work better (and we have since seen this with a GRU model).
Models
Our basic FeedForward Neural Network
Comparision of a NN to the Chain Ladder - coming soon
GBM vs NN – coming soon
An investigation into time series NNs - coming once we have finished writing the findings up!
Can you help?
We have a long ‘to do’ list but not enough manpower. All our outputs come with code you can download and apply to your own data.
To join the working party simply apply through the IFoA’s volunteering page - scroll down and look for us under ‘On-going Vacancies’.