Models & Scores
An introduction to the different types of validation samples Model validation is an essential step in evaluating and verifying a model’s performance during development before finalizing the design and proceeding with implementation. More specifically, during a predictive model’s development, the objective of a model validation is to measure the model’s accuracy in predicting the expected outcome. For a credit risk model, this may be predicting the likelihood of good or bad payment behavior, depending on the predefined outcome. Two general types of data samples can be used to complete a model validation. The first is known as the in-time, or holdout, validation sample and the second is known as the out-of-time validation sample. So, what’s the difference between an in-time and an out-of-time validation sample? An in-time validation sample sets aside part of the total sample made available for the model development. Random partitioning of the total sample is completed upfront, generally separating the data into a portion used for development and the remaining portion used for validation. For instance, the data may be randomly split, with 70 percent used for development and the other 30 percent used for validation. Other common data subset schemes include an 80/20, a 60/40 or even a 50/50 partitioning of the data, depending on the quantity of records available within each segment of your performance definition. Before selecting a data subset scheme to be used for model development, you should evaluate the number of records available in your target performance group, such as number of bad accounts. If you have too few records in your target performance group, a 50/50 split can leave you with insufficient performance data for use during model development. A separate blog post will present a few common options for creating alternative validation samples through a technique known as resampling. Once the data has been partitioned, the model is created using the development sample. The model is then applied to the holdout validation sample to determine the model’s predictive accuracy on data that wasn’t used to develop the model. The model’s predictive strength and accuracy can be measured in various ways by comparing the known and predefined performance outcome to the model’s predicted performance outcome. The out-of-time validation sample contains data from an entirely different time period or customer campaign than what was used for model development. Validating model performance on a different time period is beneficial to further evaluate the model’s robustness. Selecting a data sample from a more recent time period having a fully mature set of performance data allows the modeler to evaluate model performance on a data set that may more closely align with the current environment in which the model will be used. In this case, a more recent time period can be used to establish expectations and set baseline parameters for model performance, such as population stability indices and performance monitoring. Learn more about how Experian Decision Analytics can help you with your custom model development needs.
According to our recent research for the State of Alternative Credit Data, more lenders are using alternative credit data to determine if a consumer is a good or bad credit risk. In fact, when it comes to making decisions: More than 50% of lenders verify income, employment and assets as well as check public records before making a credit decision. 78% of lenders believe factoring in alternative data allows them to extend credit to consumers who otherwise would be declined. 70% of consumers are willing to provide additional financial information to a lender if it increases their chance for approval or improves their interest rate. The alternative financial services space continues to grow with products like payday loans, rent-to-own products, short-term loans and more. By including alternative financial data, all types of lenders can explore both universe expansion and risk mitigation. State of Alternative Credit Data
The traditional credit score has ruled the financial services space for decades, but it‘s clear the way in which consumers are managing their money and credit has evolved. Today’s consumers are utilizing different types of credit via various channels. Think fintech. Think short-term loans. Think cash-checking services and payday. So, how do lenders gain more visibility to a consumer’s credit worthiness in 2018? Alternative credit data has surfaced to provide a more holistic view of all consumers – those on the traditional file and those who are credit invisibles and emerging. In an all-new report, Experian dives into “The State of Alternative Credit Data,” providing in-depth coverage on how alternative credit data is defined, regulatory implications, consumer personas attached to the alternative financial services industry, and how this data complements traditional credit data files. “Alternative credit data can take the shape of alternative finance data, rental, utility and telecom payments, and various other data sources,” said Paul DeSaulniers, Experian’s senior director of Risk Scoring and Trended/Alternative Data and attributes. “What we’ve seen is that when this data becomes visible to a lender, suddenly a much more comprehensive consumer profile is formed. In some instances, this helps them offer consumers new credit opportunities, and in other cases it might illuminate risk.” In a national Experian survey, 53% of consumers said they believe some of these alternative sources like utility bill payment history, savings and checking account transactions, and mobile phone payments would have a positive effect on their credit score. Of the lenders surveyed, 80% said they rely on a credit report, plus additional information when making a lending decision. They cited assessing a consumer’s ability to pay, underwriting insights and being able to expand their lending universe as the top three benefits to using alternative credit data. The paper goes on to show how layering in alternative finance data could allow lenders to identify the consumers they would like to target, as well as suppress those that are higher risk. “Additional data fields prove to deliver a more complete view of today’s credit consumer,” said DeSaulniers. “For the credit invisible, the data can show lenders should take a chance on them. They may suddenly see a steady payment behavior that indicates they are worthy of expanded credit opportunities.” An “unscoreable” individual is not necessarily a high credit risk — rather they are an unknown credit risk. Many of these individuals pay rent on time and in full each month and could be great candidates for traditional credit. They just don’t have a credit history yet. The in-depth report also explores the future of alternative credit data. With more than 90 percent of the data in the world having been generated in just the past five years, there is no doubt more data sources will emerge in the coming years. Not all will make sense in assessing credit decisions, but there will definitely be new ways to capture consumer-permissioned data to benefit both consumer and lender. Read Full Report
In my first blog post on the topic of customer segmentation, I shared with readers that segmentation is the process of dividing customers or prospects into groupings based on similar behaviors. The more similar or homogeneous the customer grouping, the less variation across the customer segments are included in each segment’s custom model development. A thoughtful segmentation analysis contains two phases: generation of potential segments, and the evaluation of those segments. Although several potential segments may be identified, not all segments will necessarily require a separate scorecard. Separate scorecards should be built only if there is real benefit to be gained through the use of multiple scorecards applied to partitioned portions of the population. The meaningful evaluation of the potential segments is therefore an essential step. There are many ways to evaluate the performance of a multiple-scorecard scheme compared with a single-scorecard scheme. Regardless of the method used, separate scorecards are only justified if a segment-based scorecard significantly outperforms a scorecard based on a broader population. To do this, Experian® builds a scorecard for each potential segment and evaluates the performance improvement compared with the broader population scorecard. This step is then repeated for each potential segmentation scheme. Once potential customer segments have been evaluated and the segmentation scheme finalized, the next step is to begin the model development. Learn more about how Experian Decision Analytics can help you with your segmentation or custom model development needs.
Marketers are keenly aware of how important it is to “Know thy customer.” Yet customer knowledge isn’t restricted to the marketing-savvy. It’s also essential to credit risk managers and model developers. Identifying and separating customers into distinct groups based on various types of behavior is foundational to building effective custom models. This integral part of custom model development is known as segmentation analysis. Segmentation is the process of dividing customers or prospects into groupings based on similar behaviors such as length of time as a customer or payment patterns like credit card revolvers versus transactors. The more similar or homogeneous the customer grouping, the less variation across the customer segments are included in each segment’s custom model development. So how many scorecards are needed to aptly score and mitigate credit risk? There are several general principles we’ve learned over the course of developing hundreds of models that help determine whether multiple scorecards are warranted and, if so, how many. A robust segmentation analysis contains two components. The first is the generation of potential segments, and the second is the evaluation of such segments. Here I’ll discuss the generation of potential segments within a segmentation scheme. A second blog post will continue with a discussion on evaluation of such segments. When generating a customer segmentation scheme, several approaches are worth considering: heuristic, empirical and combined. A heuristic approach considers business learnings obtained through trial and error or experimental design. Portfolio managers will have insight on how segments of their portfolio behave differently that can and often should be included within a segmentation analysis. An empirical approach is data-driven and involves the use of quantitative techniques to evaluate potential customer segmentation splits. During this approach, statistical analysis is performed to identify forms of behavior across the customer population. Different interactive behavior for different segments of the overall population will correspond to different predictive patterns for these predictor variables, signifying that separate segment scorecards will be beneficial. Finally, a combination of heuristic and empirical approaches considers both the business needs and data-driven results. Once the set of potential customer segments has been identified, the next step in a segmentation analysis is the evaluation of those segments. Stay tuned as we look further into this topic. Learn more about how Experian Decision Analytics can help you with your segmentation or custom model development needs.
Alternative credit data sources make it possible for lenders to gain a more holistic view of existing and potential client bases, enabling them to better determine the risk of lending to someone with little or no credit history. These sources also can provide lenders with a competitive edge by preparing them to be better equipped to mitigate losses, expand their scope of potential applicants, give them another look at the creditworthiness of previously-denied applicants, and enable them to properly adjust credit terms and pricing to better speak to compatible consumers. “Experian is a big supporter of the widespread reporting of alternative data—including, rental payments, utility payments and cellular telephone payments to name a few,” said Alex Lintner, president of Experian’s Consumer Information Services. Here are four FCRA-compliant alternative credit data sources to consider when evaluating consumers for credit: Alternative Financial Services: Alternative financial credit information, including payday and short-term installment loans and inquiries, on the majority of the United States’ subprime population. Rental Data: RentBureau includes detailed, positive and negative rental payment history information on more than 18 million consumers in the United States. Extended View Score: A risk model designed to evaluate the creditworthiness of thin-file and no-file consumers who have little to no traditional credit history. Account Aggregation: Permissioned by the consumer, a real-time data collection of his/her financial accounts (bank, credit card, investment and business) in a single location to digitally verify income and assets. Tapping into these resources can give deeper and richer results—improving overall data analytics and, subsequently, possibilities for future lending.
Traditional credit attributes provide immense value for lenders when making decisions, but when used alone, they are limited to capturing credit behavior during a single moment of time. To add a deeper layer of insight, Experian® today unveiled new trended attributes, aimed at giving lenders a wider view into consumer credit behavior and patterns over time. Ultimately, this helps them expand into new risk segments and better tailor credit offers to meet consumer needs. An Experian analysis shows that custom models developed using Trended 3DTM attributes provide up to a 7 percent lift in predictive performance when compared with models developed using traditional attributes only. “While trended data has been shown to provide additional insight into a consumer’s credit behavior, lack of standardization across different providers has made it a challenge to gain those insights,” said Steve Platt, Experian’s Group President of Decision Analytics and Data Quality. “Trended 3D makes it easy for our clients to get value from trended data in a consistent manner, so they can make more informed decisions across the credit life cycle and, more importantly, give consumers better access to lending options.” Experian’s Trended 3D attributes help lenders unlock valuable insights hidden within credit reports. For example, two people may have similar balances, utilization and risk scores, but their paths to that point may be substantially different. The solution synthesizes a 24-month history of five key credit report fields — balance, credit limit or original loan amount, scheduled payment amount, actual payment amount and last payment date. Lenders can gain insight into: Changes in balances over time Migration patterns from one tradeline or multiple tradelines to another Variations in utilization and credit limits Changes in payment activity and collections Balance transfer and debt consolidation behavior Behavior patterns of revolving trades versus transactional trades Additionally, Trended 3D leverages machine learning techniques to evaluate behavioral data and recognize patterns that previously may have gone undetected. To learn more information about Experian’s Trended 3D attributes, click here.
Today’s consumer lending environment is more dynamic and competitive than ever, with renewed focus on personal loans, marketplace lending and the ever-challenging credit card market. One of the significant learnings from the economic crisis is how digging deeper into consumer credit data can help provide insights into trending behavior and not just point-in-time credit evaluation. For example, I’ve found consumer trending behavior to be very powerful when evaluating risks of credit card revolvers versus transactors. However, trended data can come with its own challenges when the data isn’t interpreted uniformly across multiple data sources. To address these challenges, Experian® has developed trended attributes, which can provide significant lift in the development of segmentation strategies and custom models. These Trended 3DTM attributes are used effectively across the life cycle to drive balance transfers, mitigate high-risk exposure and fine-tune strategies for customers near score cutoffs. One of the things I look for when exploring new trended data is the ability to further understand payment velocity. These characteristics go far beyond revolver and transactor flags, and into the details of consumer usage and trajectory. As illustrated in the chart, a consumer isn’t easily classified into one borrowing persona (revolver, transactor, etc.) or another — it’s a spectrum of use trends. Experian’s Trended 3D provides details needed to understand payment rates, slope of balance growth and even trends in delinquency. These trends provide strong lift across all decisioning strategies to improve your business performance. In recent engagements with lenders, new segmentation tools and data for the development of custom models is at the forefront of the conversation. Risk managers are looking for help leveraging new modeling techniques such as machine learning, but often have challenges moving from prior practices. In addition, attribute governance has been a key area of focus that is addressed with Trended 3D, as it was developed using machine learning techniques and is delivered with the necessary documentation for regulatory conformance. This provides an impressive foundation, allowing you to integrate the most advanced analytics into your credit decisioning. Alternative data isn’t the only source for new consumer insights. Looking at the traditional credit report can still provide so much insight; we simply need to take advantage of new techniques in analytics development. Trended attributes provide a high-definition lens that opens a world of opportunity.
You just finished redeveloping an existing scorecard, and now it’s time to replace the old with the new. If not properly planned, switching from one scorecard to another within a decisioning or scoring system can be disruptive. Once a scorecard has been redeveloped, it’s important to measure the impact of changes within the strategy as a result of replacing the old model with the new one. Evaluating such changes and modifying the strategy where needed will not only optimize strategy performance, but also maximize the full value of the newly redeveloped model. Such an impact assessment can be completed with a swap set analysis. The phrase swap set refers to “swapping out” a set of customer accounts — generally bad accounts — and replacing them with, or “swapping in,” a set of good customer accounts. Swap-ins are the customer population segment you didn’t previously approve under the old model but would approve with the new model. Swap-outs are the customer population segment you previously approved with the old model but wouldn’t approve with the new model. A worthy objective is to replace bad accounts with good accounts, thereby reducing the overall bad rate. However, different approaches can be used when evaluating swap sets to optimize your strategy and keep: The same overall bad rate while increasing the approval rate. The same approval rate while lowering the bad rate. The same approval and bad rate but increase the customer activation or customer response rates. It’s also important to assess the population that doesn’t change — the population that would be approved or declined using either the old or new model. The following chart highlights the three customer segments within a swap set analysis. With the incumbent model, the bad rate is 8.3%. With the new model, however, the bad rate is 4.9%. This is a reduction in the bad rate of 3.4 percentage points or a 41% improvement in the bad rate. This type of planning also is beneficial when replacing a generic model with another or a custom-developed model. Click here to learn more about how the Experian Decision Analytics team can help you manage the impacts of migrating from a legacy model to a newly developed model.
In 2017, 81 percent of U.S. Americans have a social media profile, representing a five percent growth compared to the previous year. Pick your poison. Facebook. Instagram. Twitter. Snapchat. LinkedIn. The list goes on, and it is clear social media is used by all. Grandma and grandpa are hooked, and tweens are begging for accounts. Factor in the amount of data being generated by our social media obsession – one report claims Americans are using 2,675,700 GB of Internet data per minute – and it makes some lenders wonder if social media insights can be used to assess credit risk. Can banks, credit unions and online lenders look at social media profiles when making a loan decision and garner intel to help them make a credit decision? After all, in some circles, people believe a person’s character is just as important as their income and assets when making a lending decision. Certainly, some businesses are seeing value in collecting social media insights for marketing purposes. An individual’s interests, likes and click-throughs reveal a lot about their lifestyle and potential brand linkages. But credit decisions are different. In fact, there are two key concerns when considering social media data as it pertains to financial decisions. There is that little rule called the Equal Credit Opportunity Act, which states credit must be extended to all creditworthy applicants regardless of race, religion, gender, marital status, age and other personal characteristics. A quick scan of any Facebook profile can reveal these things, and more. Credit applications do not ask for these specific details for this very reason. Social media data can also be manipulated. One can “like” financial articles, participate in educational quizzes and represent themselves as if they are financially responsible. Social media can be gamed. On the flip side, a consumer can’t manipulate their payment history. There is no question that data is essential for all aspects of the financial services industry, but when it comes to making credit decisions on a consumer, FCRA data trumps everything. In the consumer’s best interest, it is essential that credit data be both displayable and disputable. The right data must be used. For lenders, their primary goal is to assess a consumer’s stability, ability and willingness to pay. Today, social media can’t address those needs. It’s not to say that social media data can’t be used in the future, but financial institutions are still grappling with how it can be predictive of credit behavior over time. In the meantime, other sources of data are being evaluated. Everything from including on-time utility and rental payments, insights on smaller dollar loans and various credit attributes can help to provide a more holistic view of today’s credit consumer. There is no question social media data will continue to grow exponentially. But in the world of credit decisioning, the “like” button cannot be given quite yet.
We use our laptops and mobile phones every day to communicate with our friends, family, and co-workers. But how do software programs communicate with each other? APIs--Application Programming Interfaces--are the hidden backbone of our modern world, allowing software programs to communicate with one another. Behind the scenes of every app and website we use, is a mesh of systems “talking” to each other through a series of APIs. Today, the API economy is quickly changing how the world interacts. Everything from photo sharing, to online shopping, to hailing a cab is happening through APIs. Because of APIs, technical innovation is happening at a faster pace than ever. We caught up with Edgar Uaje, senior product manager at Experian, to find out more about APIs in the financial services space. What exactly are APIs and why are they so important? And how do they apply to B2B? APIs are the building blocks of many of our applications that exist today. They are an intermediary that allows application programs to communicate, interact, and share data with various operating systems or other control programs. In B2B, APIs allow our clients to consume our data, products, and services in a standard format. They can utilize the APIs for internal systems to feed their risk models or external applications for their customers. As Experian has new data and services available, our clients can quickly access the data and services. Are APIs secure? APIs are secure as long as the right security measures are put in place. There are many security measures that can be utilized such as authentication, authorization, channel encryption and payload encryption. Experian takes security seriously and ensures that the right security measures are put in place to protect our data. For example, one of the recent APIs that was built this year utilizes OAuth, channel encryption, and payload encryption. The central role of APIs is promoting innovation and rapid but stable evolution, but they seem to only have taken hold selectively in much of the business world. Is the world of financial services truly ready for APIs? APIs have been around for a long time, but they are getting much more traction recently. Financial tech and online market place lending companies are leading the charge of consuming data, products, and services through APIs because they are nimble and fast. With standard API interfaces, these companies can move as fast as their development teams can. The world of financial services is evolving, and the time is now for them to embrace APIs in day-to-day business. How can APIs benefit a bank or credit union, for example? APIs can benefit a bank or credit union by allowing them to consume Experian data, products, and services in a standard format. The value to them is faster speed to market for applications (internal/external), ease of integration, and control over the user’s experience. APIs allow a bank or credit union to quickly develop new and innovative applications quickly, with the support of their development teams. Can you tell us more about the API Developer Portal? Experian will publish the documentation of our available APIs on our Developer Portal over time as they become available. Our clients will have a one-stop shop to view available APIs, review API documentation, obtain credentials, and test APIs. This is simplifying data access by utilizing REST API, making it easier for our clients.
Looking to score more consumers, but worried about increased risk? A recent VantageScore LLC study found that consumers rendered “unscoreable” by commonly used credit scoring models are nearly identical in their financial and credit behavior to scoreable consumers. To get a more detailed financial portrait of the “expanded” population, credit files were supplemented with demographic and economic data. The study found: Consumers who scored above 620 using the VantageScore 3.0 model exhibited profiles of sufficient quality to justify mortgage loans on par with those of conventionally scoreable consumers. 3 to 2.5 million – a majority of the 3.4 million consumers categorized as potentially eligible for mortgages – demonstrated sufficient income to support a mortgage in their geographic areas. The findings demonstrate that VantageScore is a scalable solution to expanding mortgage credit without relaxing credit standards should the FHFA and GSEs accept VantageScore 3.0. Want to know more?
Lenders are looking for ways to accurately score more consumers and grow their applicant pool without increasing risk. And it looks like more and more are turning to VantageScore® to help achieve their goals. So, who’s using VantageScore? 9 of the top 10 financial institutions. 18 of the top 25 credit card issuers. 21 of the top 25 credit unions. VantageScore leverages the collective experience of the industry’s leading experts on credit data, modeling and analytics to provide a more consistent and highly predictive credit score. >>Want to know more?
As credit behavior and economic conditions continue to evolve, using a model that is validated regularly can give lenders greater confidence in the model’s performance. VantageScore Solutions, LLC validates all its models annually to promote transparency and support financial institutions with model governance. The results of the most recent validation demonstrate the consistent ability of VantageScore® 3.0 to accurately score more than 30 million to 35 million consumers considered unscoreable by other models — including 9.5 million Hispanic and African-American consumers. The findings reinforce the importance of using advanced credit scoring models to make more accurate decisions while providing consumers with access to fair and equitable credit. >> VantageScore Annual Validation Results 2016 VantageScore® is a registered trademark of VantageScore Solutions, LLC
A recent study by VantageScore Solutions, LLC analyzed if consumers would continue to be creditworthy over time. Highlights about consumer credit score migration over time include: Over 3 months, 49% of consumers experienced a credit score improvement averaging 19 points, and 30% experienced a score decrease averaging 24 points At 12 months, 51% of consumers had credit score increases averaging 27 points, while 38% had their scores decline by an average of 34 points 6.4% of the total population would have received the opposite credit decision because of score migration if they had been reviewed three months later Lenders looking to enhance credit strategies may benefit from using quarterly or monthly score trends for credit approval rather than a conventional, single-point-in-time credit score. >> Credit score migration study
Learn More Image
