Part II: Where are Models Most Needed Now in Mortgages? (Click here if you missed Part I of this post.) By: John Straka A first important question should always be are all of your models, model uses, and model testing strategies, and your non-model processes, sound and optimal for your business?  But in today’s environment, two areas in mortgage stand out where better models and decision systems are most needed now: mortgage servicing and loan-quality assurance.  I will discuss loan-quality assurance in a future installment. Mortgage servicing and loss mitigation are clearly one area where better models and new decision analytics continue to have a seemingly great potential today to add significant new value.  At the risk of oversimplifying, it is possible that a number of the difficulties and frustrations of mortgage servicers (and regulators) and borrowers in recent years may have been lessened through more efficient automated decision tools and optimization strategies.  And because these problems will continue to persist for quite some time, it is certainly not too late to envision and move now towards an improved future state of mortgage servicing, or to continue to advance your existing new strategic direction by adding to enhancements already underway. Much has been written about the difficulties faced by many mortgage servicers who have been overwhelmed by the demands of many more delinquent and defaulted borrowers and very extensive, evolving government involvements in new programs, performance incentives and standards.  A strategic question on the minds of many executives and others in the industry today seems to be, where is all of this going?  Is there a generally viable strategic direction for mortgage servicers that can help them to emerge from their current issues—perhaps similar to the improved data, standards, modeling, and technologies that allowed the mortgage industry in the 1990s to emerge overall quite successfully from the problems of the late 1980s and early 90s? To review briefly, mortgage industry problems of the early 1990s were less severe, of course—but really not dissimilar to the current environment.  There had been a major home-price correction in California, in New England, and in a number of large metro areas elsewhere.  A “low doc” mortgage era (and other issues) had left Citicorp nearly insolvent, for example, and caused other significant losses on top of the losses generated by the home prices.  A major source of most mortgage funding, the Savings & Loan industry, had largely collapsed, with losses having to be resolved by a special government agency. Statistical mortgage credit scoring and automated underwriting resulted from the improved data, standards, modeling, and technologies that allowed the mortgage industry to recover in the 1990s, allowing mortgages to catch up with the previously established use of this decision technology in cards, autos, etc., thus benefiting the mortgage industry with reduced costs and significant gains in efficiency and risk management.  An important question today is, is there a similar “renaissance,” so to speak, now in the offing or at hand for mortgage servicers?  Despite all of the still ongoing problems? Let me offer here a very simple analogy—with a disclaimer that this is only a basic starting viewpoint, an oversimplification, recognizing that mortgage servicing and loss mitigation is extraordinarily complex in its details, and often seems only to grow more complex by the day (with added constraints and uncertainties piling on). The simple analogy is this: consider your loan-level Net Present Value (NPV) or other key objective of loan-level decisions in servicing and loss mitigation to be analogous to the statistically based mortgage default “Score” of automated underwriting for originations in the 1990s.  Viewed in this way, a simple question stemming from the figure below is:  can you reduce costs and satisfy borrowers and performance standards better by automating and focusing your servicing representatives more, or primarily, on the “Refer” group of borrowers?  A corollary question is can more automated model-based decision engines confidently reduce the costs and achieve added insights and efficiencies in servicing the lowest and highest NPV delinquent borrowers and the Refer range?  Another corollary question is, are new government-driven performance standards helpful or hindering (or even preventing) particular moves toward this type of objective. Is this a generally viable strategic direction for the future (or even the present) of mortgage servicing?  Is it your direction today?  What is your vision for the future of your quality mortgage servicing?

By: Joel Pruis One might consider this topic redundant to the last submission around application requirements and that assessment would be partially true.  As such we are not going to go over the data that has already been collected in the application such as the demographic information of the applicant and guarantors or the business financial information or personal financial information.  That discussion like Elvis has “left the building”. Rather, we will discuss the use of additional data to support the underwriting/decisioning process – namely: Personal/Consumer credit data Business data Scorecards Fraud data Let’s get a given out in the open.  Personal credit data has a high correlation to the payment performance of a small business.  The smaller the business the higher the correlation. “Your honor, counsel requests the above be stipulated in the court records.” “So stipulated for the record.” “Thank you, your honor.” With that put to rest (remember you can always comment on the blog if you have any questions or want to comment on any of the content). The real debate in small business lending revolves around the use of business data. Depth and availability of business data There are some challenges with the gathering and dissemination of business data for use in decisioning – mainly around the history of the data for the individual entity.  More specifically, while a consumer is a single entity and for the vast majority of consumers, one does not bankrupt one entity and then start a new person to refresh their credit history.  No, that is actually bankruptcy and the bankruptcy stays with the individual. Businesses, however, can and in fact do close one entity and start up another.  Restaurants and general contractors come to mind as two examples of individuals who will start up a business, go bankrupt and then start another business under a new entity repeating the cycle multiple times.  While this scenario is a challenge, one cannot refute the need to know how both the individual consumer as well as the individual business is handling its obligations whether they are credit cards, auto loans or trade payables. I once worked for a bank president in a small community bank who challenged me with the following mantra, “It’s not what you know that you don’t know that can hurt you, it is what you think you know but really don’t that hurts you the most.”  I will admit that it took me a while to digest that statement when I first heard it.  Once fully digested the statement was quite insightful.  How many times do we think we know something when we really don’t?  How many times do we act on an assumed understanding but find that our understanding was flawed?  How sound was our decision when we had the flawed understanding?  The same holds true as it relates to the use (or lack thereof) of business information.  We assume that we don’t need business information because it will not tell us much as it relates to our underwriting.  How can the business data be relevant to our underwriting when we know that the business performance is highly correlated to the performance of the owner? Let’s look at a study done a couple of years ago by the Business Information group at Experian.  The data comes from a whitepaper titled “Predicting Risk: the relationship between business and consumer scores” and was published in 2008.  The purpose of the study was to determine which goes bad first, the business or the owner.  At a high level the data shows the following:                 If you're interested, you can download the full study here. So while a majority of time and without any additional segmentation, the business will show signs of stress before the owner. If we look at the data using length of time in business we see some additional insights.               Figure: Distribution of businesses by years in business Interesting distinction is that based upon the age of the business we will see the owner going bad before the business if the business age is 5 years or less.  Once we get beyond the 5 year point the “first bad” moves to the business. In either case, there is no clear case to be made to exclude one data source in favor of the other to predict risk in a small business origination process.  While we can look at see that there is an overall majority where the business goes bad first or that if we have a young small business the owner will more likely go bad first, in either case, there is still a significant population where the inverse is true. Bottom line, gathering both the business and the consumer data allows the financial institution to make a better and more informed decision.  In other words, it prevents us from the damage caused by “thinking we know something when we really don’t”. Coming up next month – Decisioning Strategies. 

Part I: Types and Complexity of Models, and Unobservable or Omitted Variables or Relationships By: John Straka Since the financial crisis, it’s not unusual to read articles here and there about the “failure of models.” For example, a recent piece in Scientific American critiqued financial model “calibration,” proclaiming in its title, Why Economic Models are Always Wrong. In the mortgage business, for example, it is important to understand where models have continued to work, as well as where they failed, and what this all means for the future of your servicing and origination business. I also see examples of loose understanding about best practices in relation to the shortcomings of models that do work, and also about the comparative strengths and weaknesses of alternative judgmental decision processes.  With their automation efficiencies, consistency, valuable added insights, and testability for reliability and robustness, statistical business models driven by extensive and growing data remain all around us today, and they are continuing to expand.  So regardless of your views on the values and uses of models, it is important to have a clear view and sound strategies in model usage. A Categorization: Ten Types of Models Business models used by financial institutions can be placed in more than ten categories, of course, but here are ten prominent general types of models: Statistical credit scoring models (typically for default) Consumer- or borrower-response models Consumer- or borrower-characteristic prediction models Loss given default (LGD) and Exposure at default (EAD) models Optimization tools (these are not models, per se, but mathematical algorithms that often use inputs from models) Loss forecasting and simulation models and Value-at-risk (VAR) models Valuation, option pricing, and risk-based pricing models Profitability forecasting and enterprise-cash-flow projection models Macroeconomic forecasting models Financial-risk models that model complex financial instruments and interactions Types 8, 9 and 10, for example, are often built up from multiple component models, and for this reason and others, these model categories are not mutually exclusive.  Types 1 through 3, for example, can also be built from individual-level data (typical) or group-level data.  No categorical type listing of models is perfect, and this listing is also not intended to be completely exhaustive. The Strain of Complexity (or Model Ambition) The principle of Occam’s razor in model building, roughly translated, parallels the business dictum to “keep it simple, stupid.”  Indeed, the general ordering of model types 1 through 10 above (you can quibble on the details) tends to correspond to growing complexity, or growing model ambition. Model types 1 and 2 typically forecast a rank-ordering, for example, rather than also forecasting a level.  Credit scores and credit scoring typically seek to rank-order consumers in their default, loss, or other likelihoods, without attempting to project the actual level of default rates, for example, across the score distribution.  Scoring models that add the dimension of level prediction increase this layer of complexity. In addition, model types 1 through 3 are generally unconditional predictors.  They make no attempt to add the dimension of predicting the time path of the dependent variable.  Predicting not just a consumer’s relative likelihood of an event over a future time period as a whole, for example, but also the event’s frequency level and time path of this level each year, quarter, or month, is a more complex and ambitious modeling endeavor.  (This problem is generally approached through continuous or discrete hazard models.) While generalizations can be hazardous (exceptions can typically be found), it is generally true that, in the events leading up to and surrounding the financial crisis, greater model complexity and ambition was correlated with greater model failure.  For example, at what is perhaps an extreme, Coval, Jurek, and Stafford (2009) have demonstrated how, for model type 10, even slight unexpected changes in default probabilities and correlations had a substantial impact on the expected payoffs and ratings of typical collateralized debt obligations (CDOs) with subprime residential mortgage-backed securities as their underlying assets.  Nonlinear relationships in complex systems can generate extreme unreliability of system predictions. To a lesser but still significant degree, the mortgage- or housing-related models included or embedded in types 6 through 10 were heavily dependent on home-price projections and risk simulation, which caused significant “expected”-model failures after 2006.  Home-price declines in 2007-2009 reached what had previously only been simulated as extreme and very unlikely stress paths.  Despite this clear problem, given the inescapable large impact of home prices on any mortgage model or decision system (of any kind), it is generally acceptable to separate the failure of the home-price projection from any failure of the relative default and other model relationships built around the possible home-price paths.  In other words, if a model of type 8, for example, predicted the actual profitability and enterprise cash flow quite well given the actual extreme path of home prices, then this model can be reasonably regarded as not having failed as a model per se, despite the clear, but inescapable reliance of the model’s level projections on the uncertain home-price outcomes. Models of type 1, statistical credit scoring models, generally continued to work well or reasonably well both in the years preceding and during the home-price meltdown and financial crisis.  This is very largely due to these models’ relatively modest objective of simply rank-ordering risks, in general.  To be sure, scoring models in mortgage, and more generally, were strongly impacted by the home price declines and unusual events of the bubble and subsequent recession, with deteriorated strength in risk separation.  This can be seen, for example, in the recent VantageScore® credit score stress-test study, VantageScore® Stress Testing, which shows the lowest risk separation ability in the states with the worst home-price and unemployment outcomes (CA, AZ, FL, NV, MI).  But these kinds of significant but comparatively modest magnitudes of deterioration were neither debilitating nor permanent for these models.   In short, even in mortgage, scoring models generally held up pretty well, even through the crisis—not perfectly, but comparatively better than the more complex level-, system-, and path-prediction models. (see footnote 1) Scoring models have also relied more exclusively on microeconomic behavioral stabilities, rather than including macroeconomic risk modeling.  Fortunately the microeconomic behavioral patterns have generally been much more stable.  Weak-credit borrowers, for example, have long tended to default at significantly higher rates than strong credit borrowers—they did so preceding, and right through, the financial crisis, even as overall default levels changed dramatically; and they continue to do so today, in both strong and weak housing markets. (see footnote 2) As a general rule overall, the more complex and ambitious the model, the more complex are the many questions that have to be asked concerning what could go wrong in model risks.  But relative complexity is certainly not the only type of model risk.  Sometimes relative simplicity, otherwise typically desirable, can go in a wrong direction. Unobservable or Omitted Variables or Relationships No model can be perfect, for many reasons.  Important determining variables may be unmeasured or unknown.  Similarly, important parameters and relationships may differ significantly across different types of populations, and different time periods.  How many models have been routinely “stress tested” on their robustness in handling different types of borrower populations (where unobserved variables tend to lurk) or different shifts in the mix of borrower sub-populations?  This issue is more or less relevant depending on the business and statistical problem at hand, but overall, modeling practice has tended more often than not to neglect robustness testing (i.e., tests of validity and model power beyond validation samples). Several related examples from the last decade appeared in models that were used to help evaluate subprime loans.  These models used generic credit scores together with LTV, and perhaps a few other variables (or not), to predict subprime mortgage default risks in the years preceding the market meltdown.  This was a hazardous extension of relatively simple model structures that worked better for prime mortgages (but had also previously been extended there).  Because, for example, the large majority of subprime borrowers had weak credit records, generic credit scores did not help nearly as much to separate risk.  Detailed credit attributes, for example, were needed to help better predict the default risks in subprime.  Many pre-crisis subprime models of this kind were thus simplified but overly so, as they began with important omitted variables. This was not the only omitted-variables problem in this case, and not the only problem.  Other observable mortgage risk factors were oddly absent in some models.  Unobserved credit risk factors also tend to be correlated with observed risk factors, creating greater volatility and unexplained levels of higher risk in observed higher-credit-risk populations.  Traditional subprime mortgages also focused mainly on poor-credit borrowers who needed cashout refinancing for debt consolidation or some other purpose.  Such borrowers, in shaky financial condition, were more vulnerable to economic shocks, but a debt consolidating cashout mortgage could put them in a better position, with lower total monthly debt payments that were tax deductible.  So far, so good—but an omitted capacity-risk variable was the number of previous cashout refinancings done (which loan brokers were incented to “churn”).  The housing bubble allowed weak-capacity borrowers to sustain themselves through more extracted home equity, until the music stopped.  Rate and fee structures of many subprime loans further heightened capacity risks.  A significant population shift also occurred when subprime mortgage lenders significantly raised their allowed LTVs and added many more shaky purchase-money borrowers last decade; previously targeted affordable-housing programs from the banks and conforming-loan space had instead generally required stronger credit histories and capacity.  Significant shifts like this in any modeled population require very extensive model robustness testing and scrutiny.  But instead, projected subprime-pool losses from the major purchasers of subprime loans, and the ratings agencies, went down in the years just prior to the home-price meltdown, not up (to levels well below those seen in widely available private-label subprime pool losses from 1990’s loans). Rules and Tradition in Lieu of Sound Modeling Interestingly, however, these errant subprime models were not models that came into use in lender underwriting and automated underwriting systems for subprime—the front-end suppliers of new loans for private-label subprime mortgage-backed securities.  Unlike the conforming-loan space, where automated underwriting using statistical mortgage credit scoring models grew dramatically in the 1990s, underwriting in subprime, including automated underwriting, remained largely based on traditional rules. These rules were not bad at rank-ordering the default risks, as traditional classifications of subprime A-, B, C and D loans showed.  However, the rules did not adapt well to changing borrower populations and growing home-price risks either.  Generic credit scores improved for most subprime borrowers last decade as they were buoyed by the general housing boom and economic growth.  As a result, subprime-lender-rated C and D loans largely disappeared and the A- risk classifications grew substantially. Moreover, in those few cases where statistical credit scoring models were estimated on subprime loans, they identified and separated the risks within subprime much better than the traditional underwriting rules.  (I authored an invited article early last decade, which included a graph, p. 222, that demonstrated this, Journal of Housing Research.)  But statistical credit scoring models were scarcely or never used in most subprime mortgage lending. In Part II, I’ll discuss where models are most needed now in mortgages. Footnotes: [1] While credit scoring models performed better than most others, modelers can certainly do more to improve and learn from the performance declines at the height of the home-price meltdown.  Various approaches have been undertaken to seek such improvements. [2] Even strategic mortgage defaults, while comprising a relatively larger share of strong-credit borrower defaults, have not significantly changed the traditional rank-ordering, as strategic defaults occur across the credit spectrum (weaker credit histories include borrowers with high income and assets).